diff --git a/.agents/skills/coding/SKILL.md b/.agents/skills/coding/SKILL.md deleted file mode 100644 index 30ddb1dc1..000000000 --- a/.agents/skills/coding/SKILL.md +++ /dev/null @@ -1,69 +0,0 @@ ---- -name: coding -description: Apply CROWDB conventions while changing production or test code; not for read-only questions or reviews. ---- - - - - -# Coding - -Inspect the touched module and its callers. Read `doc/doc_index.md` and the one -matched design section only when the change alters documented behavior, -crosses module boundaries, or leaves architectural intent unclear. - -## Logging - -- `critical!`: broken invariant; include `next step:`. -- `error!`: recoverable failure and its handling. -- `warn!`: actionable anomaly or transient failure. -- `info!`: lifecycle and topology changes only; never per request. -- `debug!`: requests, hot-path decisions, and routine detail. -- Do not leave production `trace!` calls. - -Use structured Paxos fields when available: `store_id`, `group_id`, -`replica_l_id`, `replica_r_id`, `slot`, `ballot`. Instrument public Paxos -object methods so fields propagate. Defaults remain file=`debug`, console=`info`. - -## Tests and status - -- Put integration tests in `/tests/*_test.rs`; helpers belong in - `tests/common/` and use `Test*` names. Do not add inline test modules, - `tests/testkit/`, or fixtures under `src/`. -- Keep Paxos tests under `lib/crowdb-kv/tests/paxos_test/`, entered by - `lib/crowdb-kv/tests/paxos_test.rs`. -- Use `CROWDB_TEST_LOG=1` and `tests/common/logging.rs` for test tracing. -- Expose useful new state through `StatusLevel` or the relevant `*Status` - type in `lib/crowdb-kv/src/cluster/status.rs`; shared wire types live in - `lib/crowdb-protocol/src/mgmt.rs`. -- Track new TODO/FIXME items in `doc/todo_code.md`; remove both together. - -## Layout - -- Keep roots for entry points, facades, ABI boundaries, configuration, and - genuinely shared primitives. Put implementation under its owning product - domain; avoid catch-all `handlers`, `types`, and `utils` folders. -- Use the non-`mod.rs` layout: `foo.rs` + `foo/`; `foo.rs` contains module docs, - declarations, and deliberate re-exports. Keep internal children private - unless callers need them. -- Mirror public C++ subsystem ownership under `include//` and `src/`; - keep private headers with their implementation. Avoid forwarding headers - unless compatibility requires them. -- Use established domain names consistently. Keep one nameable responsibility - per module and separate domain invariants from runtime wiring. -- Improve nearby layout only when cohesive with the requested change. -- Keep code files near 300 lines; split before adding to one over 1000. Keep - functions near 40 lines, at most 80 for orchestration; split over 150. -- Use the narrowest visibility. Test hooks require `test-util` and `_for_tests`. - -## C++ clang-tidy - -- Run `pixi run tree-lint` before committing; it must exit 0. -- No per-line `NOLINT`. Disable checks in `.clang-tidy` with a comment. -- Auto-fix only with `--extra-arg=-w` (blocks compiler fix-its leaking - in). Build and run `pixi run test-cpp` after each check group. -- Review fix-its that change behavior or perf: `static`/`const` method - conversions, `std::move` removals, value-param → `const &`. -- Suppressed checks and reasons are listed in `.clang-tidy`. - -For visible console UI or Playwright work, also apply `/console-ui-e2e`. diff --git a/.agents/skills/console-ui-e2e/SKILL.md b/.agents/skills/console-ui-e2e/SKILL.md deleted file mode 100644 index 37a1fc056..000000000 --- a/.agents/skills/console-ui-e2e/SKILL.md +++ /dev/null @@ -1,69 +0,0 @@ ---- -name: console-ui-e2e -description: Change visible CROWDB console UI or work on its Playwright E2E tests; not for backend-only changes or general questions. ---- - - - - -# Console UI E2E - -Apply to `app/crowdb-web/ui/src/**` and `e2e-test/clients/console-ui/**`, including -browser-test failures and runtime regressions. Run the affected spec for every -visible change; add a regression assertion for every UI bug fix. - -## Structure and cost - -- Specs: `e2e/flows/NN--.spec.ts`. Prefixes order the - single-worker suite: `0x` shell, `1x` physical, `2x` KV cluster, `3x` KV - data, `4x` inspector, `5x` capacity/DiskDB, `9x` cross-function. -- Extend the existing page-function spec; add one only for a new function. -- Register each mutating flow under the root runner with an owned release - fixture. Use inspected node IPs and fixed ports; host browser clients never - launch server binaries. Fresh fixtures provide empty-backend isolation. -- Preserve original distinct assertions when coalescing flow owners. Important - deterministic UI logic remains in component tests. -- Record `// Baseline: Xs (date)`; investigate runtime above 2x. -- Measure the current exact test before changing it; saved timing reports may - describe an earlier worktree. Compare Playwright's per-test time, not the - command wall time that includes UI build and web-server startup. -- Instrument setup, mutation response, lifecycle poll, DOM refresh, and - teardown separately with `stepTimer`. Optimize the first slow boundary. -- Before removing repeated setup from a cross-function flow, locate the - dedicated specs that retain each behavior and assertion. - -## Assertions - -- Trace handler -> JSON -> hook -> props -> DOM before asserting unfamiliar - state. Probe raw API data when its shape is uncertain. -- Prefer role, label, test-id, and scoped locators. Add `data-testid` for - ambiguity; never resolve it with page-level `.first()` or `.last()`. -- Assertions time out at 3 seconds; leader election at 10 seconds. - `expect.poll` uses `intervals: [100]`. -- Poll lifecycle state; never sleep. -- Do not assert on toast alerts. If one intercepts a click, click via `evaluate`. -- Never swallow errors, weaken assertions, add retries, or inflate waits. - Use `waitForResponse` only when the response is an assertion target, not as - a substitute for observing lifecycle or DOM state. - -## Diagnose - -For a failing, flaky, or slow E2E test, read -[references/diagnosis.md](references/diagnosis.md). Ordinary UI implementation -does not need that reference. - -## Run - -Follow [test and CI progression](../debug-test/SKILL.md#test-and-ci-progression) -when sequencing suites or CI jobs. Fix a failure with its affected spec/group, -then resume the remaining selection; use this skill for browser diagnosis. - -```sh -pixi run clean --runtime-only -pixi run test-container-e2e --scope console.operations --case --concurrency 1 --node-cpus 0.25 --cpus 8 --memory-mib 16384 -``` - -Use the root runner for the full Console container domain. `pixi run test-console-ui` -runs UI component tests and types without servers or Docker. Browser tests use -an installed system browser; do not install a private browser. If blocked, -keep the regression assertion and report why. diff --git a/.agents/skills/console-ui-e2e/references/diagnosis.md b/.agents/skills/console-ui-e2e/references/diagnosis.md deleted file mode 100644 index d2d7a1c55..000000000 --- a/.agents/skills/console-ui-e2e/references/diagnosis.md +++ /dev/null @@ -1,26 +0,0 @@ - - - -# Console E2E diagnosis - -- Reproduce the exact test, then its spec. If isolated execution passes, run - the original ordered selection and inspect shared state, IDs, ports, and - leaked processes; compare serial execution before changing code. -- Locate the first divergence with the trace, `error-context.md`, console - errors, raw APIs, and `stepTimer`. Separate backend mutation from DOM refresh. -- Scope strict locators to the named region. Never hide ambiguity with - page-level `.first()` or `.last()`. -- Reuse one `APIRequestContext` per poll phase. Poll resulting API state for - asynchronous services; use an explicit test-mode cadence only when the real - production interval dominates runtime. -- Verify lifecycle through both API state and the OS process list. - -Useful log signals: - -- `new standalone instance created`: transport was not shared. -- `no mgmt seeds configured`: invalid empty seed set. -- `no KV servers deployed`: cluster is not initialized. -- `no rpc endpoint resolved` or `no group-0 endpoint found`: no group-0 route. -- `group-0 query failed`: RPC failed and config fallback was used. -- `topology refresh failed` or `topology discovery failed`: verify whether the - seed is a store listen address or node RPC address. diff --git a/.agents/skills/debug-test/SKILL.md b/.agents/skills/debug-test/SKILL.md deleted file mode 100644 index 76d865870..000000000 --- a/.agents/skills/debug-test/SKILL.md +++ /dev/null @@ -1,70 +0,0 @@ ---- -name: debug-test -description: Run CROWDB test and CI jobs incrementally, and diagnose non-browser failures from first divergence to root cause. ---- - - - - -# Debug Test - -Do not increase timeouts, suppress errors, weaken assertions, add caller-side -retries, or otherwise make only the symptom pass. - -## Test and CI progression - -For expensive suites or multiple jobs, preserve an ordered list of commands, -completed results, logs, relevant source/image inputs, and the next unrun item. -Use existing runner records where available. - -- Run until an error, then fix it before advancing the affected sequence. Do - not first run every job against a known defect. Prioritize bugs that make - subsequent tests slow, hang, or fail through the same dependency. -- For container E2E, use `--fail-fast` to stop queued cases while letting - already owned fixtures finish and clean up. Read failure records throughout - the log, not only its tail or completed-case count. Do not interrupt both a - launcher and its child; forwarded duplicate signals can break cleanup. -- Reproduce and repair the failure with its test or smallest meaningful test - group. Run that group's regression and relevant quality gates after the fix. -- Resume the remaining tests/jobs from the saved position. Retain earlier - passing results; repeat them only when the fix changes their behavior or - inputs. Preserve original ordering and coverage when selecting a remainder. -- Independent work already in flight may finish when it is unaffected. Stop - dispatching work that depends on the defect or would repeat its slowdown. -- Complete shared build prerequisites before parallel test jobs. Never run - concurrent builders against the same output directory; use independent - output directories when parallel builds are necessary. Perform shared - environment cleanup before dispatch; do not repeat `clean --runtime-only` while other - local jobs still own running fixtures. Host browser fixtures that require - stable Docker networks also share that constraint across local CI jobs; - keep their browser phases separate from other jobs' bridge creation/removal. -- Before long suites, inspect actual fixture slots and declared CPU/memory - costs. An outer parallel job count does not prove cases run concurrently. - Distinguish build, fixture startup, client execution, and cleanup time. -- Reuse valid builds and images. Rebuild when their relevant inputs change; - avoid unrelated feature/profile changes and repeated rebuilds while debugging. -- After the requested jobs have each been repaired and locally verified, run - the requested or required overall verification once as a closing gate. If - that gate finds another defect, repair its group and resume the affected - remainder rather than restarting valid completed groups from the beginning. -- Report actual coverage, failures, and unrun/blocked work. A focused pass does - not by itself establish that a whole suite or CI job passed. - -## Diagnose a failure - -1. Check relevant environment variables, stale processes, logs, persisted - state, and build artifacts. -2. Reproduce the exact test. If it passes alone, compare the affected suite - under its default concurrency and serial execution; inspect shared IDs, - ports, temp directories, and leaked processes. -3. List setup, action, and assertion steps. Find the first divergence using - logs, APIs, persisted data, and timing. -4. Classify it as code/design, timing/order, or environment. -5. For an unexplained process exit, analyze its crash report before changing code. -6. Add focused temporary instrumentation only when evidence is insufficient; - remove it afterward. -7. Fix the earliest upstream cause. Rerun the test, affected suite, and quality - gate. Add a regression test for a code bug. - -For browser or console UI tests, use `/console-ui-e2e` instead. During active -requirement work, its retry and blocking rules remain authoritative. diff --git a/.agents/skills/doc-backlog/SKILL.md b/.agents/skills/doc-backlog/SKILL.md deleted file mode 100644 index f8d0ebd89..000000000 --- a/.agents/skills/doc-backlog/SKILL.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -name: doc-backlog -description: Write a CROWDB requirement document as a testable high-level design and contract. ---- - - - - -# Backlog Requirement - -Write `doc/backlog/R**--.md` as one requirement's high-level -design and testable contract. It is removed by `/implement-requirement`. - -Keep the contract concrete: describe CrowDB data, state transitions, APIs, and -verification. Do not invent concepts for completeness. Cite primary sources -when borrowing terminology or behavior from another system, and distinguish -that reference from CrowDB's own semantics. - -Use this exact order: - -1. `### R**: — ` after the license. -2. Optional `Status` when the item is explicitly deferred or blocked; state the - reason and what would unblock it. -3. `Problem`: current behavior, impact, root cause, root-design link, and - concrete operator/system scenarios. -4. `Solution`: architecture, invariants, component responsibilities, numbered - work items naming real modules, and edge-case outcomes. If unsettled, list - the human decisions and alternatives instead of inventing one. -5. `Dependencies`: incoming/outgoing requirements, named artifacts, and - fallback for unlanded work. -6. `Acceptance`: one test case per solution and edge-case claim. -7. `Open Questions`, only for human decisions; include alternatives and trade-offs. - -Each acceptance case states setup -> action -> assertion, names the invariant, -and ends with `Unit test`, `Integration test`, or `E2E test`. End with exact -pixi test, fmt, and clippy commands. - -The requirement is the implementation's high-level design. Keep signatures, -file lists, sequencing, and other execution detail in the working plan; do not -create a separate working design. Requirement numbers are allowed here. Use an -ASCII flow only for three or more components or a primary/fallback path. Do not -use tables. diff --git a/.agents/skills/doc-design/SKILL.md b/.agents/skills/doc-design/SKILL.md deleted file mode 100644 index 28d4eae27..000000000 --- a/.agents/skills/doc-design/SKILL.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -name: doc-design -description: Write or refine permanent CROWDB architecture when that document itself is in scope; not a mandatory requirement cleanup step. ---- - -<!-- Copyright 2026-present Gian <crow.db@outlook.com> --> -<!-- Licensed under the Apache License, Version 2.0. --> - -# Formal Design - -Root docs define an area's architecture, rationale, non-goals, and sub-design -map. Sub-designs hold one topic's detail and link upstream with `Depends on:` -and `Satisfies:`. - -Use a license, `# CROWDB - Design: <Title>`, short scope introduction, a table -of contents, and numbered `## N. <Title>` sections. Include only applicable -content: problem/why, concepts and named invariants, design, neighbor -interactions, tunables, correctness, risks, and open questions. Give invariants -stable IDs such as `I1`. - -Describe current state only. Omit requirement numbers, change history, -before/after prose, file paths, and line numbers. Refer to searchable symbols. -Keep architecture in root docs, detail in sub-designs, and operations in the -relevant design or component README; link rather than repeat. - -Keep the design concrete and implementation-oriented. Define only concepts -needed by CrowDB's data structures, state transitions, interfaces, and -correctness rules; do not introduce product concepts merely to make the prose -sound complete. When reusing terminology or behavior from another system, -cite its primary documentation and state which part is only a reference, not a -CrowDB semantic. - -When explicitly promoting a working draft, remove temporary scaffolding and -requirement references, rewrite as current state, update `doc/doc_index.md`, -and delete the draft. diff --git a/.agents/skills/doc-working-design/SKILL.md b/.agents/skills/doc-working-design/SKILL.md deleted file mode 100644 index 671c43b5e..000000000 --- a/.agents/skills/doc-working-design/SKILL.md +++ /dev/null @@ -1,39 +0,0 @@ ---- -name: doc-working-design -description: Write a temporary design exploration only when the user explicitly requests a separate draft; not for normal requirement implementation. ---- - -<!-- Copyright 2026-present Gian <crow.db@outlook.com> --> -<!-- Licensed under the Apache License, Version 2.0. --> - -# Working Design - -Write `doc/working/design-<topic>.md` only for an explicitly requested design -exploration. Normal `/implement-requirement` work uses the requirement as its -high-level design and records implementation detail in the working plan. - -Keep the exploration concrete and implementation-oriented. Introduce only -concepts needed to explain CrowDB data, state, interfaces, or verification. -When borrowing terminology or behavior, cite primary documentation and mark it -as external reference rather than CrowDB semantics. - -Link the backlog and root design; do not repeat their problem, dependencies, -architecture, or rationale. - -Include: - -- License, `# <Title> (R**)`, and a short introduction with upstream links and - landed dependencies. -- Numbered solution sections by component: why a new mechanism is needed, - signatures, behavior, failures, and fallback. -- `Scope`: every changed path and intended change. -- `Complexity`: High/Medium/Low and the actual implementation difficulty. -- `Test Design`: concrete unit and E2E cases for every acceptance item and - edge case; each states setup -> action -> assertion and invariant. -- `Module Structure`: annotated file tree. -- `Config Extensions` and `Server Wiring` when applicable. -- `Open Questions` only for unresolved human decisions, with trade-offs. - -Use real paths and symbols. Fence signatures and diagrams; use ordered prose -for behavior. Requirement references are allowed only while the draft is -temporary and are removed when it is folded into formal design. diff --git a/.agents/skills/doc-working-plan/SKILL.md b/.agents/skills/doc-working-plan/SKILL.md deleted file mode 100644 index 499d72282..000000000 --- a/.agents/skills/doc-working-plan/SKILL.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -name: doc-working-plan -description: Write and maintain a requirement execution plan. ---- - -<!-- Copyright 2026-present Gian <crow.db@outlook.com> --> -<!-- Licensed under the Apache License, Version 2.0. --> - -# Working Plan - -Write `doc/working/plan-<topic>.md` from the requirement's solution and -acceptance criteria. It holds implementation detail, is kept live, and is -deleted after completion. - -Keep tasks tied to concrete CrowDB data structures, state transitions, APIs, -and tests. Do not create new product concepts just to organize prose. Cite -primary sources for borrowed terminology or behavior and keep those references -separate from CrowDB's own contract. - -- Start with the license, `# <Title> Plan`, upstream links, and one-line goal. -- Group dependency-ordered tasks by phase or component. -- Use `- [ ] **Name**: action. Files: paths.`; `[~]` marks the only active - task and `[x]` a verified task. -- Split tasks until each is completable and maps to identifiable diff hunks. -- Record relevant symbols, signatures, ordering, edge cases, and migration - steps needed to execute the high-level design without duplicating it. -- Include a consolidated file list and tests grouped by unit, integration, E2E. -- Keep status truthful as work proceeds. -- Add `## Blocked` only under `/implement-requirement` conditions, recording - the decision/failure, alternatives or retries, and why work cannot continue. - -For a persistent plan, state the exception in its header and remove completed -tasks instead of deleting the file. diff --git a/.agents/skills/doc/SKILL.md b/.agents/skills/doc/SKILL.md deleted file mode 100644 index 1f2f0ace0..000000000 --- a/.agents/skills/doc/SKILL.md +++ /dev/null @@ -1,34 +0,0 @@ ---- -name: doc -description: Route an unclear CROWDB documentation task or maintain general docs; do not combine with a known doc-type skill. ---- - -<!-- Copyright 2026-present Gian <crow.db@outlook.com> --> -<!-- Licensed under the Apache License, Version 2.0. --> - -# Documentation - -Start at `doc/doc_index.md`; open only the row and section matching the task. - -## Locations - -- `doc/design/<area>/design-crowdb-<area>.md`: architecture and rationale. -- `doc/design/<area>/design-crowdb-<area>-<topic>.md`: permanent topic detail. -- `doc/design/kv/kv-*-flow-analysis.md`: permanent KV path analysis. -- `doc/design/`: permanent architecture and operational behavior. -- `doc/backlog/`: requirement index and analysis. -- `doc/working/`: implementation plans and explicitly requested design drafts. - -## Rules - -- Keep one source of truth. Link by section anchor instead of copying text. -- Fix conflicting upstream design before downstream docs or code. -- Index permanent document changes, not backlog or working files. -- Permanent design describes current state; backlog requirements hold - high-level proposed design, and working plans hold execution detail. -- Split independent topics; delete working files when their work completes. -- Prefer concrete, tight prose and raw-readable bullets. Remove filler, - repetition, rhetorical openings, adjective lists, and excessive em dashes. - -When the target is a design, backlog requirement, design draft, or working -plan, use only its matching `/doc-*` guide instead of this router. diff --git a/.agents/skills/implement-requirement/SKILL.md b/.agents/skills/implement-requirement/SKILL.md deleted file mode 100644 index c2dda3418..000000000 --- a/.agents/skills/implement-requirement/SKILL.md +++ /dev/null @@ -1,39 +0,0 @@ ---- -name: implement-requirement -description: Implement one CROWDB backlog requirement from its high-level design through focused verification and cleanup. ---- - -<!-- Copyright 2026-present Gian <crow.db@outlook.com> --> -<!-- Licensed under the Apache License, Version 2.0. --> - -# Implement Requirement - -Use for an item in `doc/backlog/backlog.md`. Its detail is the high-level -design and testable contract; do not create a separate design draft. - -1. Read the matched requirement and relevant code. Read the backlog index only - for selection, dependency ordering, or status; read design sections only to - resolve behavior or module interaction. -2. Update a missing high-level decision with `/doc-backlog`; block only on a - real human choice. -3. Create one `doc/working/plan-<topic>.md` with `/doc-working-plan`; keep all - file, symbol, sequencing, and test detail there. -4. Implement in plan order, keep it current, and commit coherent tasks. -5. Run affected acceptance tests and fmt/lint gates separately. -6. Update permanent design only for an in-scope architecture change. -7. Delete the completed requirement, backlog entry, and plan in a final cleanup - commit. - -Find tests with `pixi task list`; prefix server-spawning tests with -`pixi run clean --runtime-only &&`. Before push run fmt and `pixi run rs-lint`. - -Proceed autonomously except when a design has multiple valid choices with no -clear winner, or a test still fails after five root-cause-driven attempts. -Then record the alternatives or failed command, attempts, diagnosis, and exact -failure under plan `## Blocked`; commit that state and ask the user. - -`Open Questions` contains only decisions requiring user input. Keep unfinished -implementation work in the plan instead of presenting it as an open issue. - -This overrides ordinary commit cadence. Every commit passes its applicable -gate. diff --git a/.agents/skills/review/SKILL.md b/.agents/skills/review/SKILL.md deleted file mode 100644 index 8e908fd86..000000000 --- a/.agents/skills/review/SKILL.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -name: review -description: Review a CROWDB diff when code review or pre-push review is requested; not an automatic coding handoff step. ---- - -<!-- Copyright 2026-present Gian <crow.db@outlook.com> --> -<!-- Licensed under the Apache License, Version 2.0. --> - -# Review - -Review the diff, callers, and tests. Report concrete findings before style. -Apply repository code, test-placement, and module-ownership rules without -loading another skill solely for review. - -Treat `propose`, `accept`, `learn`, `kv_get`, `kv_put`, `kv_delete`, -and `kv_batch_write` as hot paths: - -- Borrow instead of copying; justify non-trivial clones. -- Pre-size collections or use stack storage. -- Do not add locks. `std::sync::Mutex` is limited to lifecycle work; - `tokio::sync::Mutex` requires state held across `.await` and user approval. -- Avoid nested `Arc`; retain it only when data outlives its parent. Never move - a borrow into a task that may outlive it. - -For crowdb-rpc flatbuffers, read fields from the frame. Do not create owned -intermediates or use allocating accessors on hot paths. Shared wrappers belong -in `crowdb-protocol`; copy payload only beyond frame lifetime. See -`doc/design/rpc/design-crowdb-rpc.md` section 6. - -Check correctness invariants, errors, shutdown, async lifetimes, status -exposure, dead code, dependencies, visibility, `Debug`, module cohesion, -integration-test placement, and performance regressions. Avoid non-test panic. -Prefer enum dispatch unless implementations are open across crates. Use -`get_or_try_init` for fallible initialization. - -For module cohesion, flag implementation left at roots, catch-all folders, -inconsistent Rust/C++ ownership, or unclear names. Recommend a concrete owner, -but keep relocation findings scoped to the changed area. - -`async_fn_in_trait` is the only documented clippy exception. Add any new -exception here before suppressing it. diff --git a/.clang-tidy b/.clang-tidy index cda17cdd9..e33df3a26 100644 --- a/.clang-tidy +++ b/.clang-tidy @@ -1,8 +1,10 @@ --- # Suppressed checks — fix-its are unsafe or the warning source is -# third-party macro noise. See coding SKILL.md § "C++ clang-tidy". +# third-party macro noise. # These checks can change ownership, ABI, overload resolution, or hot-path # behavior through their fix-its; keep them visible only when reviewed locally. +# Public member const-qualification changes C++ ABI and requires deliberate +# interface review rather than a lint-driven conversion. # Fixed async completion tokens transfer heap ownership through opaque context # pointers. NewDeleteLeaks does not model the backend invoking their deleters; # completion ownership is covered by ASan and async lifetime tests instead. @@ -24,6 +26,7 @@ Checks: >- -performance-unnecessary-value-param, -performance-move-const-arg, -readability-convert-member-functions-to-static, + -readability-make-member-function-const, -modernize-use-ranges, -modernize-avoid-c-arrays, -readability-non-const-parameter, diff --git a/AGENTS.md b/AGENTS.md deleted file mode 100644 index 79396c51e..000000000 --- a/AGENTS.md +++ /dev/null @@ -1,67 +0,0 @@ -<!-- Copyright 2026-present Gian <crow.db@outlook.com> --> -<!-- Licensed under the Apache License, Version 2.0. --> - -# CROWDB - -Distributed storage platform: Paxos consensus, per-key slots, WAL durability, -crowdb-tree engine, and a disk-block allocator. Rust workspace with C++ storage -and transport exposed through FFI. - -## Terminology - -- **Repo chunk** is the discussion term for any user-data chunk, including - S3, IcebergTable, and future Dataset chunks. It excludes system chunks - such as WAL and tree storage chunks. -- Implementations distinguish these user-data uses with specific chunk types; - do not retain or introduce a generic `Repo` chunk type. The discussion term - does not imply a shared implementation type. - -## Project rules - -- Run every build, test, lint, and executable through `pixi run`. -- `unsafe_code = deny` by default. Keep unsafe confined to existing scoped FFI - or low-level modules; raise any new exception before adding it. -- Keep hot paths lock-free. Before adding a lock, stop and ask the user with - the contention, ordering, and complexity trade-off. -- Add Rust tests under each crate's `tests/`; crates with `test-util` enable it - for their own tests through self dev-dependencies. -- Preserve existing work. Never hard-reset, revert, or restore files from an - older commit without explicit approval. Use stash or a temporary branch. -- Ordinary interactive work is committed only when asked, as one coherent - commit. Requirement work follows `/implement-requirement`. -- Commit messages are single-line subjects without bodies, trailers, doc - references, or requirement numbers. Code comments also omit those references. -- Before committing, run the relevant gate through pixi: - Rust fmt and clippy, changed C++ format/tree-lint, and affected tests. Fix - ordinary failures up to three times; report confirmed pre-existing failures. - Requirement work uses `/implement-requirement` retry and blocking rules. -- Playwright uses an installed system browser. Do not install one locally. -- Use 60-second shell timeouts by default. Start hang-prone commands in the - background and poll. Show complete output; do not hide errors with filters. -- On test or CI failure, print the failing case and test error to the console. - Enable `CROWDB_E2E_PRINT_SERVICE_ERRORS=1` in CI or when debugging to also print - container service file errors. This extra output defaults off locally; - artifacts retain complete logs in either mode. -- Markdown is primarily read raw. Prefer bullets; use tables for real - comparisons only. `doc/doc_index.md` always uses tables. When a table is - used, pad columns with spaces so the raw markdown aligns visually. -- Console UI changes follow `doc/design/console/design-crowdb-console-ui.md`. - Before changing visible behavior, update the UI spec and its ASCII layout - after agreement with the user; only then implement the UI change. Do not - unilaterally change the spec or introduce UI behavior that is absent from - the agreed spec. Each visible UI change needs focused test or E2E coverage. - -## Skill dispatch - -- Repository skills live only under `.agents/skills/`; do not create legacy - aliases or compatibility symlinks such as `.devin`. -- Code changes: `/coding`; add `/console-ui-e2e` only for visible UI or - Playwright work. -- Test execution and CI progression: `/debug-test`; use `/console-ui-e2e` for - browser/UI diagnosis. -- Docs: one matched `/doc-*` guide; `/doc` only for general or unclear targets. -- Requirements: `/implement-requirement`; open the detail directly and consult - the backlog index only for selection, ordering, or status. -- Pre-push or explicitly requested code review: `/review`. -- Design questions: one section selected through `doc/doc_index.md`. -- Operations/user behavior: the relevant `doc/design/` or component README. diff --git a/CHANGELOG.md b/CHANGELOG.md deleted file mode 100644 index 9a94a77dc..000000000 --- a/CHANGELOG.md +++ /dev/null @@ -1,67 +0,0 @@ -<!-- Copyright 2026-present Gian <crow.db@outlook.com> --> -<!-- Licensed under the Apache License, Version 2.0. --> - -# Changelog - -CROWDB is developing `0.3.0`. This is a development version, not a -production release or a compatibility promise. - -CROWDB does not yet maintain compatibility for persisted data, WAL, metadata, -or other on-disk formats. A newer checkout may be unable to read data created by -an older checkout. Use disposable data only. - -Changelog history begins with the first public Docker preview. Development work -before that baseline remains available in Git history and project requirements -but is intentionally not reconstructed as released change history. - -The changelog will follow [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) -from that baseline. Version tags alone do not imply Semantic Versioning -compatibility until the project explicitly adopts and documents a compatibility -policy. - -## [Unreleased] - -The following changes are being prepared for `0.3.0`. - -### Added - -- Dataset access support with namespace, manifest, cursor, retention, transport, - and publication primitives. -- A Python dataset client surface for iterating dataset records. - -### Changed - -- S3 and Iceberg use separate chunk types, storage policies, and write pools - behind one access-server process. -- Single-node and three-node profiles now make storage protection explicit, - including strip-level mirror and EC I/O with repairable degraded placement. - -### Internal - -- Local and CI tests are organized around component tasks rather than package- - level Pixi tasks. - -## [0.2.2] - 2026-10-06 - -### Fixed - -- CI builds service binaries before stream and console acceptance tests. -- Console workspace reset waits for stopped KV children before deleting runtime - state. - -## [0.1.0] - -- Single-node Linux amd64 container with native Iceberg REST catalog and FileIO, - backed by CROWDB metadata, chunk storage and disk services. -- Persistent bootstrap, generated client credentials, health checks, bounded - service recovery and restart validation. -- S3 object access through an optional published endpoint. -- Host builds and runtime-only container packaging, with a manual Docker Hub - publication workflow for version and commit tags, signatures, SBOM and provenance. - -The preview image is `crowdb/crowdb-iceberg:0.1.0`; no published digest is -recorded here. Multi-node deployment, -production hardening and data-format upgrades are outside this release. - -See the container deployment files for supported startup, persistence, -credentials and recovery behavior. diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index f68c57706..e0a5c0f31 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -5,14 +5,9 @@ ## Our pledge -We pledge to make participation in CROWDB a harassment-free experience for -everyone, regardless of age, body size, visible or invisible disability, -ethnicity, sex characteristics, gender identity and expression, level of -experience, education, socioeconomic status, nationality, personal appearance, -race, caste, color, religion, or sexual identity and orientation. +We pledge to make participation in CROWDB a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socioeconomic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation. -We will act and interact in ways that contribute to an open, welcoming, -diverse, inclusive, and healthy community. +We will act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. ## Expected behavior @@ -29,50 +24,33 @@ diverse, inclusive, and healthy community. - Sexualized language, imagery, attention, or advances. - Insults, derogatory comments, trolling, or personal and political attacks. - Publishing private information without explicit permission. -- Pressuring anyone to disclose identity, credentials, employer, or private - communications. -- Retaliation against a person who raises a concern or participates in an - investigation. -- Conduct that would reasonably be considered inappropriate in a professional - community. +- Pressuring anyone to disclose identity, credentials, employer, or private communications. +- Retaliation against a person who raises a concern or participates in an investigation. +- Conduct that would reasonably be considered inappropriate in a professional community. ## Scope -This code applies in project repositories, issue trackers, reviews, discussions, -chat, events, and other CROWDB community spaces. It also applies when someone is -officially representing the project in public. +This code applies in project repositories, issue trackers, reviews, discussions, chat, events, and other CROWDB community spaces. It also applies when someone is officially representing the project in public. ## Reporting -Report abusive, harassing, or otherwise unacceptable behavior privately to -**crow.db@outlook.com**. Do not include sensitive personal information in a -public issue. +Report abusive, harassing, or otherwise unacceptable behavior privately to **crow.db@outlook.com**. Do not include sensitive personal information in a public issue. -Reports will be reviewed promptly, impartially, and as confidentially as -possible. People handling a report must disclose conflicts of interest and -recuse themselves when necessary. The project will protect the privacy and -safety of reporters and affected community members to the extent possible. +Reports will be reviewed promptly, impartially, and as confidentially as possible. People handling a report must disclose conflicts of interest and recuse themselves when necessary. The project will protect the privacy and safety of reporters and affected community members to the extent possible. -Security vulnerabilities follow [SECURITY.md](SECURITY.md), not the conduct -reporting process. +Security vulnerabilities follow [SECURITY.md](SECURITY.md), not the conduct reporting process. ## Enforcement -Project maintainers may remove, edit, or reject comments, commits, code, issues, -and other contributions that violate this code. Responses will be proportionate -to the behavior, its impact, and any pattern of prior conduct. Actions may -include: +Project maintainers may remove, edit, or reject comments, commits, code, issues, and other contributions that violate this code. Responses will be proportionate to the behavior, its impact, and any pattern of prior conduct. Actions may include: 1. A private correction and explanation of the impact. 2. A formal warning with conditions for continued participation. 3. A temporary restriction from project interaction or representation. 4. A permanent ban from project spaces and representation. -Maintainers will not publicly identify a reporter or disclose private report -details without permission, except when required to protect people or comply -with law. +Maintainers will not publicly identify a reporter or disclose private report details without permission, except when required to protect people or comply with law. ## Attribution -This code is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/), -version 2.1. +This code is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/), version 2.1. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index de5c99a6a..11e0b1826 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -3,35 +3,25 @@ # Contributing to CROWDB -Thank you for contributing to CROWDB. - ## Development status -CROWDB is under active development at version `0.3.0`. It has not reached -alpha, is not recommended for production, and must be tested with disposable -data. Compatibility is not yet maintained for persisted data, WAL, metadata, or -other on-disk formats. A change may deliberately replace an unreleased format -without migration support when its requirement says so. +CROWDB is under active development. It has not reached alpha, is not recommended for production, and must be tested with disposable data. Compatibility is not yet maintained for persisted data, WAL, metadata, or other on-disk formats. A change may deliberately replace an unreleased format without migration support when its requirement says so. -The root [VERSION](VERSION) file is the project version source of truth. Cargo, -Pixi, the web package, and lockfiles must match it. Do not bump the version in an -ordinary contribution unless the pull request is explicitly release work. +The root [VERSION](VERSION) file defines the server version; `pixi run check-version` checks the corresponding manifests and lockfiles. The portable SDK has its own version. Change versions only as part of release work. ## Community and security -Follow [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md). Report vulnerabilities through -[SECURITY.md](SECURITY.md), not a public issue. +Follow [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md). Report vulnerabilities through [SECURITY.md](SECURITY.md), not a public issue. ## Development environment -CROWDB uses [Pixi](https://pixi.sh) to pin Rust, C++, Node.js, and native build -dependencies. Run builds, tests, linters, and project executables through Pixi. +CROWDB uses [Pixi](https://pixi.sh) to pin Rust, C++, Node.js, and native build dependencies. Run builds, tests, linters, and project executables through Pixi. ```bash # Build C++, the Rust workspace, and the web UI. pixi run build -# Run host component, UI and Iceberg SDK suites. +# Run host test suites. pixi run test-suite # Check version metadata. @@ -41,20 +31,14 @@ pixi run check-version pixi run rs-fmt-check pixi run rs-lint -# Format and lint changed C++ code. -pixi run tree-fmt -pixi run tree-lint - # Check TypeScript and browser behavior when the UI changes. pixi run ts-lint pixi run test-console-ui ``` -Playwright uses an installed system browser; do not install a repository-local -browser. See [pixi.toml](pixi.toml) for focused component tasks and -[tools/README.md](tools/README.md) for their scripts. Native Iceberg and official -SDK suites use the `iceberg-e2e` environment; see -[the test inventory](doc/working/test.md) for CI coverage and timing. +For changed C++ files, run `clang-format --dry-run --Werror` and `tree-lint` through Pixi, passing the affected file paths. `tree-fmt` rewrites all C++ trees listed in its task; it is not limited to changed files. + +Playwright uses an installed system browser; do not install one locally. See [pixi.toml](pixi.toml) for tasks and environments, and [tools/README.md](tools/README.md) for test selection, scripts, and CI coverage. Container acceptance is a separate Linux amd64 gate requiring Docker: @@ -65,39 +49,28 @@ pixi run test-single-node-container ## Before writing code - Search existing requirements, designs, tests, and neighboring components. -- For architectural or externally visible behavior, agree on the requirement or - design before implementation. -- Preserve the existing authority and recovery model; do not create a local - fallback that can diverge from Group 0 or another durable authority. -- Do not add a lock to a hot path without discussing contention, ordering, - progress, and complexity trade-offs. -- Never commit credentials, tokens, private keys, production data, or generated - runtime directories. -- Add dependencies through the owning package manager and avoid newly published - versions until they have had time for ecosystem review. - -Start documentation work at `doc/doc_index.md`. Permanent architecture and -operational behavior belong under `doc/design/` or the relevant component -README, future contracts in `doc/backlog/`, and temporary execution plans in -`doc/working/`. +- For architectural or externally visible behavior, agree on the requirement or design before implementation. +- Preserve the existing authority and recovery model; do not create a local fallback that can diverge from Group 0 or another durable authority. +- Do not add a lock to a hot path without discussing contention, ordering, progress, and complexity trade-offs. +- Never commit credentials, tokens, private keys, production data, or generated runtime directories. +- Add dependencies through the owning package manager and avoid newly published versions until they have had time for ecosystem review. + +Public [design articles](doc/design/README.md) explain architecture and advantages in English; update them only for an explicit documentation task. Keep usage instructions in the owning README and public documents self-contained. ## Code and tests ### Rust -- Workspace crates deny unsafe code by default. Keep necessary unsafe code - confined to the existing FFI and low-level boundaries. +- Workspace crates deny unsafe code by default. Keep necessary unsafe code confined to the existing FFI and low-level boundaries. - Put integration tests in each crate's `tests/` directory. -- Put shared integration-test helpers in `tests/common/` and name helper types - with a `Test` prefix. +- Put shared integration-test helpers in `tests/common/` and name helper types with a `Test` prefix. - Use structured `tracing` fields and established domain identifiers. - Follow the existing `foo.rs` plus `foo/` module layout; do not add `mod.rs`. ### C++ - Follow the repository `.clang-format` and `.clang-tidy` configuration. -- Keep public subsystem headers under the matching `include/<library>/` tree and - private headers with their implementation. +- Keep public subsystem headers under the matching `include/<library>/` tree and private headers with their implementation. - Add GoogleTest coverage under the owning component's `tests/` directory. ### Web UI @@ -106,37 +79,23 @@ README, future contracts in `doc/backlog/`, and temporary execution plans in - Add focused unit tests and update Playwright E2E coverage for visible behavior. - Verify the real backend path when UI behavior depends on service state. -A bug fix should normally add a failing regression test first, then fix the root -cause. Do not weaken assertions, add retries, disable durability, or bypass -security boundaries to make a test pass. +A bug fix should normally add a failing regression test first, then fix the root cause. Do not weaken assertions, add retries, disable durability, or bypass security boundaries to make a test pass. ## Pull requests 1. Create a focused branch from `main`. -2. Keep the change aligned with one requirement or one coherent maintenance - purpose. -3. Add or update tests and documentation with the implementation. +2. Keep the change aligned with one requirement or one coherent maintenance purpose. +3. Add or update tests and affected behavior specs or usage instructions with the implementation. Public architecture overviews are a separate documentation task. 4. Run the relevant Pixi gates; run the full suite for cross-component changes. 5. Review generated files and the complete diff for secrets and unrelated edits. -6. Open a pull request explaining the problem, design choice, verification, and - known limitations. - -Use concise, single-line commit subjects. Keep unrelated refactors in separate -commits or pull requests. Do not force-push shared branches or bypass hooks and -release gates. +6. Open a pull request explaining the problem, design choice, verification, and known limitations. -## Changelog and releases +Use concise, single-line commit subjects. Keep unrelated refactors in separate commits or pull requests. Do not force-push shared branches or bypass hooks and release gates. -CROWDB begins public change history with its first Docker preview. Before that -baseline, Git history and requirement documents are the development record; do -not fabricate historical releases. Release preparation creates the first -versioned changelog entry. After that baseline, user-visible changes belong -under `Unreleased` and move to a dated section during release. +## Releases -Only maintainers publish releases. A version or image tag is not a production or -compatibility promise unless the release notes explicitly make that promise. +Git history records development changes. Only maintainers publish releases; summarize notable changes and compatibility in the release notes. Version and image tags alone do not promise production readiness or compatibility. ## License -By contributing, you agree that your contribution is licensed under the Apache -License 2.0. +By contributing, you agree that your contribution is licensed under the Apache License 2.0. diff --git a/Cargo.lock b/Cargo.lock index 13bfcda59..b8510c249 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -306,8 +306,6 @@ checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" [[package]] name = "bytes" version = "1.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" [[package]] name = "cast" @@ -648,7 +646,7 @@ dependencies = [ "crowdb-chunk-client", "crowdb-chunk-kv-client", "crowdb-common", - "crowdb-kv-client", + "crowdb-paxos-kv-client", "crowdb-protocol", "data-encoding", "flatbuffers", @@ -694,7 +692,7 @@ dependencies = [ "crowdb-chunk-client", "crowdb-chunk-kv-client", "crowdb-common", - "crowdb-kv-client", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "flatbuffers", @@ -740,8 +738,8 @@ dependencies = [ "crowdb-console-shared", "crowdb-diskdb-client", "crowdb-diskio-client", - "crowdb-kv", - "crowdb-kv-client", + "crowdb-paxos-kv", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "crowdb-test-harness", @@ -782,7 +780,7 @@ dependencies = [ "crowdb-common", "crowdb-diskdb-client", "crowdb-diskio-client", - "crowdb-kv-client", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "crowdb-test-harness", @@ -822,7 +820,7 @@ dependencies = [ "async-trait", "bytes", "clap", - "crowdb-kv-client", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "futures", @@ -848,8 +846,8 @@ dependencies = [ "crowdb-chunk-kv-server", "crowdb-chunk-stream", "crowdb-common", - "crowdb-kv", - "crowdb-kv-client", + "crowdb-paxos-kv", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "crowdb-test-harness", @@ -876,8 +874,8 @@ dependencies = [ "crowdb-chunk-client", "crowdb-chunk-stream", "crowdb-common", - "crowdb-kv", - "crowdb-kv-client", + "crowdb-paxos-kv", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-test-harness", "futures", @@ -900,6 +898,9 @@ dependencies = [ "clap", "crossbeam-skiplist", "crowdb-chunk-client", + "crowdb-chunk-kv", + "crowdb-chunk-kv-server", + "crowdb-chunk-stream", "crowdb-chunkdb", "crowdb-chunkdb-client", "crowdb-common", @@ -907,11 +908,12 @@ dependencies = [ "crowdb-diskdb", "crowdb-diskdb-client", "crowdb-diskio-client", - "crowdb-kv", - "crowdb-kv-client", + "crowdb-paxos-kv", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "crowdb-test-harness", + "crowdb-tree-ffi", "dashmap", "flatbuffers", "futures", @@ -934,7 +936,7 @@ version = "0.3.0" dependencies = [ "arc-swap", "bytes", - "crowdb-kv-client", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "flatbuffers", @@ -956,7 +958,7 @@ dependencies = [ "crowdb-common", "crowdb-console-shared", "crowdb-diskdb-client", - "crowdb-kv-client", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "crowdb-test-harness", @@ -1006,9 +1008,9 @@ dependencies = [ "clap", "crowdb-chunkdb-client", "crowdb-console-shared", - "crowdb-kv", - "crowdb-kv-client", - "crowdb-kv-server", + "crowdb-paxos-kv", + "crowdb-paxos-kv-client", + "crowdb-paxos-kv-server", "crowdb-protocol", "crowdb-rpc-ffi", "crowdb-test-harness", @@ -1047,11 +1049,12 @@ dependencies = [ "crowdb-diskdb", "crowdb-diskdb-client", "crowdb-diskio-client", - "crowdb-kv", - "crowdb-kv-client", + "crowdb-paxos-kv", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "crowdb-test-harness", + "crowdb-tree-ffi", "dashmap", "flatbuffers", "futures", @@ -1073,7 +1076,7 @@ dependencies = [ "arc-swap", "bytes", "crowdb-diskdb-client", - "crowdb-kv-client", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "crowdb-test-harness", @@ -1093,8 +1096,8 @@ dependencies = [ "bytes", "crowdb-common", "crowdb-diskio-client", - "crowdb-kv", - "crowdb-kv-client", + "crowdb-paxos-kv", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "flatbuffers", @@ -1105,7 +1108,40 @@ dependencies = [ ] [[package]] -name = "crowdb-kv" +name = "crowdb-monitor" +version = "0.3.0" +dependencies = [ + "axum", + "clap", + "crowdb-chunk-kv-client", + "crowdb-console-shared", + "crowdb-diskio-client", + "crowdb-monitor", + "crowdb-paxos-kv", + "crowdb-paxos-kv-client", + "crowdb-protocol", + "crowdb-rpc-ffi", + "crowdb-test-harness", + "flatbuffers", + "futures", + "if-addrs 0.13.4", + "mdns-sd", + "rand 0.8.6", + "reqwest", + "rustix", + "serde", + "serde_json", + "sha2", + "ssh-key", + "thiserror 2.0.18", + "tokio", + "toml", + "tracing", + "uuid", +] + +[[package]] +name = "crowdb-paxos-kv" version = "0.3.0" dependencies = [ "arc-swap", @@ -1113,8 +1149,8 @@ dependencies = [ "criterion", "crossbeam-skiplist", "crowdb-common", - "crowdb-kv", - "crowdb-kv-client", + "crowdb-paxos-kv", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "crowdb-test-harness", @@ -1138,7 +1174,7 @@ dependencies = [ ] [[package]] -name = "crowdb-kv-client" +name = "crowdb-paxos-kv-client" version = "0.3.0" dependencies = [ "arc-swap", @@ -1146,8 +1182,8 @@ dependencies = [ "bincode", "bytes", "crowdb-common", - "crowdb-kv", - "crowdb-kv-client", + "crowdb-paxos-kv", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "dashmap", @@ -1163,7 +1199,7 @@ dependencies = [ ] [[package]] -name = "crowdb-kv-server" +name = "crowdb-paxos-kv-server" version = "0.3.0" dependencies = [ "arc-swap", @@ -1171,8 +1207,8 @@ dependencies = [ "bytes", "clap", "crowdb-common", - "crowdb-kv", - "crowdb-kv-client", + "crowdb-paxos-kv", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "crowdb-test-harness", @@ -1188,39 +1224,6 @@ dependencies = [ "uuid", ] -[[package]] -name = "crowdb-monitor" -version = "0.3.0" -dependencies = [ - "axum", - "clap", - "crowdb-chunk-kv-client", - "crowdb-console-shared", - "crowdb-diskio-client", - "crowdb-kv", - "crowdb-kv-client", - "crowdb-monitor", - "crowdb-protocol", - "crowdb-rpc-ffi", - "crowdb-test-harness", - "flatbuffers", - "futures", - "if-addrs 0.13.4", - "mdns-sd", - "rand 0.8.6", - "reqwest", - "rustix", - "serde", - "serde_json", - "sha2", - "ssh-key", - "thiserror 2.0.18", - "tokio", - "toml", - "tracing", - "uuid", -] - [[package]] name = "crowdb-protocol" version = "0.3.0" @@ -1291,7 +1294,7 @@ dependencies = [ "crowdb-chunk-client", "crowdb-chunk-kv-client", "crowdb-diskio-client", - "crowdb-kv-client", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "crowdb-sdk-native", @@ -1336,6 +1339,7 @@ dependencies = [ "crowdb-rpc-ffi", "crowdb-test-harness", "crowdb-tree-ffi", + "serde", "tokio", ] @@ -1355,9 +1359,9 @@ dependencies = [ "crowdb-common", "crowdb-console-shared", "crowdb-diskdb-client", - "crowdb-kv", - "crowdb-kv-client", "crowdb-monitor", + "crowdb-paxos-kv", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "crowdb-test-harness", @@ -1572,9 +1576,9 @@ dependencies = [ "crowdb-console-shared", "crowdb-diskdb-client", "crowdb-diskio-client", - "crowdb-kv", - "crowdb-kv-client", "crowdb-monitor", + "crowdb-paxos-kv", + "crowdb-paxos-kv-client", "crowdb-protocol", "crowdb-rpc-ffi", "crowdb-sdk-native", diff --git a/Cargo.toml b/Cargo.toml index 737475838..ec5caf1cd 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -2,10 +2,10 @@ resolver = "2" members = [ "lib/crowdb-common/rust", - "lib/crowdb-kv", - "lib/crowdb-kv-client", + "lib/crowdb-paxos-kv", + "lib/crowdb-paxos-kv-client", "lib/crowdb-protocol", - "app/crowdb-kv-server", + "app/crowdb-paxos-kv-server", "app/crowdb-diskdb", "lib/crowdb-diskdb-client", "app/crowdb-chunkdb", @@ -33,9 +33,9 @@ members = [ "lib/crowdb-sdk-native", "lib/crowdb-sdk-native-python", ] -exclude = ["third-party/hyper", "sdk"] +exclude = ["third-party/hyper", "third-party/bytes", "sdk"] # : crowdb-tree/ffi moved from `exclude` into `members` now that -# crowdb-kv depends on it (CrowdbTreeEngine) -- building crowdb-kv already triggers +# crowdb-paxos-kv depends on it (CrowdbTreeEngine) -- building crowdb-paxos-kv already triggers # the C++ build step via crowdb-tree-ffi's build.rs regardless of workspace # membership, so the original "keep the C build out of the main workspace" # rationale for excluding it no longer holds; membership instead unifies @@ -66,6 +66,7 @@ pin-project-lite = "0.2" hyper = { git = "https://github.com/buzzcrow/hyper.git", branch = "feature-crowdb", version = "1.11.1", default-features = false } [patch.crates-io] +bytes = { path = "third-party/bytes" } hyper = { git = "https://github.com/buzzcrow/hyper.git", branch = "feature-crowdb" } [workspace.lints.rust] diff --git a/README.md b/README.md index 0170ff4c6..36695e6da 100644 --- a/README.md +++ b/README.md @@ -3,81 +3,55 @@ # CROWDB -[![CI](https://github.com/buzzcrow/crowdb/actions/workflows/ci.yml/badge.svg)](https://github.com/buzzcrow/crowdb/actions/workflows/ci.yml) -[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE) +[![CI](https://github.com/buzzcrow/crowdb/actions/workflows/ci.yml/badge.svg)](https://github.com/buzzcrow/crowdb/actions/workflows/ci.yml) [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE) -**A distributed storage platform where ownership, data movement, and recovery -share one durable control model.** +**Distributed storage for Iceberg tables, AI datasets, and S3 objects.** -CROWDB provides a shared storage core for **Iceberg tables, S3 objects, and AI -datasets**. Each access model keeps its own semantics while sharing one storage -core for distributed state, placement, protection, streaming, and recovery. +CROWDB brings native table and dataset semantics to a shared storage core. It owns the path from metadata publication to physical I/O, so catalog operations, data placement, buffer movement, and recovery can be designed together. -- **Iceberg:** native catalog and FileIO, implemented. -- **S3:** core HTTP object operations, implemented. -- **Dataset:** native access and direct GPU delivery, in design. +Access Server and OCI container deployment are implemented, with deployments tested on physical Linux hosts. A shared Web console and CLI manage the cluster, services, and storage. -## Why CROWDB? +[Quick start](https://crowdb.dev/docs/quickstart/) · [Design](doc/design/README.md) · [Deployment](container/oci-image/README.md) · [Website](https://crowdb.dev/) -The hard part of distributed storage is keeping authority correct while data -moves and workers fail. CROWDB treats ownership and these transitions as -durable state, so physical preparation does not implicitly grant serving -authority. +[![CrowDB Console showing cluster topology, service health, and capacity](doc/assets/crowdb-cluster.png)](doc/assets/crowdb-cluster.png) -Storage bottlenecks move—from disks to CPUs, networks, and data movement—but -system boundaries tend to stay. A change that crosses a metadata service, an -object gateway, and a separate storage engine can become a negotiation between -systems rather than an improvement to one data path. +## What you can build on it -CROWDB owns enough of that path to change it when workloads and hardware change. -S3 objects, Iceberg tables, and planned AI datasets are native access models -built over shared infrastructure, not conventions layered on top of one -another. The goal is not to claim novelty for Paxos, WALs, trees, or erasure -coding; it is to make their contracts agree on durability, placement, bounded -buffers, and recovery. +- **An Iceberg lakehouse.** Use a native REST catalog and file storage integration with compatible analytical clients. Tables, snapshots, manifests, and files belong to the same storage system, connecting table publication with file cleanup. +- **Dataset readers and training pipelines.** Organize datasets into immutable Snapshots, use Splits to select samples, and read the fields each workload needs. Rust and Python clients support portable HTTP access or trusted native access with retained host buffers. +- **S3 applications.** Use familiar bucket, object, and multipart operations. Large transfers stream through bounded buffers; small writes share physical storage work. -Read the full motivation in -[Why we’re building CROWDB](https://buzzcrow.github.io/blog/why-we-are-building-crowdb/). +## Why the architecture matters -## Development preview +CROWDB separates replicated control state from range metadata on shared storage. Paxos-KV makes durable decisions; Chunk-KV lets metadata serving move independently of its stored pages and journal. The Tree engine understands distributed versions, and the Chunk layer supplies placement, protection, and streaming for all access models. -Linux amd64 containers are available for public evaluation: +That control extends to the data path. The native SDK and TCP/RDMA transport preserve buffer ownership across asynchronous work, creating opportunities to avoid staging copies and bound memory across the stack. Mirror and erasure-coded layouts provide different choices for write cost, recovery, and capacity. -- [Iceberg container](https://hub.docker.com/r/crowdb/crowdb-iceberg) provides - an Apache Iceberg REST catalog and native CROWDB storage. -- [S3 container](https://hub.docker.com/r/crowdb/crowdb-s3) provides - S3-compatible object storage with native CROWDB storage. +![Iceberg, Dataset, and S3 share CROWDB's metadata, Chunk storage, physical I/O, and management.](doc/assets/crowdb-overview.svg) -Both images use the same CROWDB runtime and can serve both interfaces from one -container. Use disposable data. Production use and on-disk upgrade -compatibility are not supported, and Dataset and direct GPU delivery are not -available yet. The container's PyIceberg write and pandas example is in the -[quick start](https://crowdb.dev/docs/quickstart/). See the -[S3 container example](container/single-node-container/README.md#s3-container-usage) -for port mapping and boto3 setup. +Read the [design overview](doc/design/design-crowdb.md) for CROWDB's strengths and the [architecture](doc/design/design-crowdb-architecture.md) for the mechanisms behind them. -- [Project homepage](https://crowdb.dev/) -- [Quick start](https://crowdb.dev/docs/quickstart/) -- [Documentation](https://crowdb.dev/docs/) -- [Architecture](https://crowdb.dev/docs/architecture/) -- [Demos](https://crowdb.dev/demo/) +## Run CROWDB -## Repository +- **Try Iceberg or S3:** follow the [quick start](https://crowdb.dev/docs/quickstart/) or the [local node and client guide](container/single-node-container/README.md). +- **Build and deploy containers:** the [OCI guide](container/oci-image/README.md) covers shared images, local clusters, persistent storage, and lifecycle. +- **Deploy on physical Linux hosts:** use the [host deployment guide](container/oci-image/README.md#linux-host-deployment). +- **Use Dataset from Python or Rust:** start with the [HTTP SDK](sdk/README.md), or use the [native SDK](lib/crowdb-sdk-native/README.md) inside a trusted deployment. -The repository contains the storage implementation, tests, and source design -documents. Start with: +The [access acceptance guide](app/crowdb-access-server/tests/README.md) and [E2E guide](e2e-test/README.md) describe tested clients, deployment scenarios, and restart checks. Compatibility depends on the interface and operation; the native SDK exposes host buffers, not direct GPU delivery. -- [Contributing](CONTRIBUTING.md) for the development environment and workflow. -- [Documentation index](doc/doc_index.md) for subsystem designs. -- [Backlog](doc/backlog/backlog.md) for current delivery scope. -- [Single-node container](container/single-node-container/README.md) for image - development and packaging. +CROWDB is a development release. Use disposable data for evaluation; production use and on-disk upgrade compatibility are not supported. + +## Explore and contribute + +- [Product design](doc/design/design-crowdb.md): workloads, scope, and advantages. +- [Design library](doc/design/README.md): architecture and component articles. +- [Contributing](CONTRIBUTING.md): environment, validation, and development workflow. +- [Demos](https://crowdb.dev/demo/): see CROWDB in use. ## AI-assisted development -The code in this project was written with AI assistance. The architecture, -naming, module boundaries, and trade-offs remain human choices. AI is the -compiler. The intent is mine. +The code in this project was written with AI assistance. The architecture, naming, module boundaries, and trade-offs remain human choices. AI is the compiler. The intent is mine. ## License diff --git a/SECURITY.md b/SECURITY.md index 90e08ba65..ec6678396 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -8,18 +8,13 @@ If you discover a security vulnerability in CROWDB, please report it responsibly: 1. **Do not** open a public GitHub issue. -2. Email **crow.db@outlook.com** with the affected version or commit, a description - of the vulnerability, its impact, and reproduction steps. +2. Email **crow.db@outlook.com** with the affected version or commit, a description of the vulnerability, its impact, and reproduction steps. 3. Omit live credentials and private user data from the report. ## Scope -CROWDB `0.3.0` is a development version for evaluation with disposable data. -There is no production support commitment, supported stable release series, or -guaranteed response time. Security reports are reviewed by the maintainers. +CROWDB is a development release for evaluation with disposable data. There is no production support commitment, supported stable release series, or guaranteed response time. Security reports are reviewed by the maintainers. ## Disclosure -Please coordinate public disclosure with the maintainers while a report is -investigated and a fix is prepared. Reporter credit should follow the reporter's -preference, including anonymity. +Please coordinate public disclosure with the maintainers while a report is investigated and a fix is prepared. Reporter credit should follow the reporter's preference, including anonymity. diff --git a/app/crowdb-access-server/Cargo.toml b/app/crowdb-access-server/Cargo.toml index b36a54e3c..c8a3c9124 100644 --- a/app/crowdb-access-server/Cargo.toml +++ b/app/crowdb-access-server/Cargo.toml @@ -30,7 +30,7 @@ crowdb-access-s3 = { path = "../../lib/crowdb-access-s3" } crowdb-chunk-client = { path = "../../lib/crowdb-chunk-client" } crowdb-chunk-kv-client = { path = "../../lib/crowdb-chunk-kv-client" } crowdb-common = { path = "../../lib/crowdb-common/rust" } -crowdb-kv-client = { path = "../../lib/crowdb-kv-client" } +crowdb-paxos-kv-client = { path = "../../lib/crowdb-paxos-kv-client" } crowdb-protocol = { path = "../../lib/crowdb-protocol" } crowdb-rpc-ffi = { path = "../../lib/crowdb-rpc/ffi" } futures = "0.3" @@ -56,7 +56,7 @@ sync_wrapper = { version = "1", features = ["futures"] } [dev-dependencies] crowdb-access-dataset = { path = "../../lib/crowdb-access-dataset", features = ["test-util"] } -crowdb-kv = { path = "../../lib/crowdb-kv" } +crowdb-paxos-kv = { path = "../../lib/crowdb-paxos-kv" } bytes = "1" tempfile = "3" bincode = "1" diff --git a/app/crowdb-access-server/conf/crowdb_access_server_config.toml b/app/crowdb-access-server/conf/crowdb_access_server_config.toml index 9a89cdd34..dcdfd81ad 100644 --- a/app/crowdb-access-server/conf/crowdb_access_server_config.toml +++ b/app/crowdb-access-server/conf/crowdb_access_server_config.toml @@ -1,3 +1,4 @@ +# Generated by tools/ports/generate.py from ports.rs; edit the .in template. # Canonical configuration for the S3 and Iceberg access processes. # Credentials and bearer tokens are supplied through environment variables. @@ -6,7 +7,7 @@ mode = "production" max_node_failures = 1 [common] -management_seeds = ["http://127.0.0.1:10000"] +management_seeds = ["http://127.0.0.1:10135"] diskio_connections_per_endpoint = 2 diskio_rpc_workers = 2 @@ -31,7 +32,7 @@ chunk_capacity_bytes = 268435456 small_strip_prefetch_count = 32 [s3] -listen = "127.0.0.1:9091" +listen = "127.0.0.1:7135" tenant = "preview" region = "us-east-1" trusted_network = false @@ -46,7 +47,7 @@ ec_code = 4 max_chunk_size = 1073741824 [iceberg] -listen = "127.0.0.1:9092" +listen = "127.0.0.1:8135" native_budget_bytes = 268435456 max_chunk_size = 1073741824 diff --git a/app/crowdb-access-server/conf/crowdb_access_server_config.toml.ports.in b/app/crowdb-access-server/conf/crowdb_access_server_config.toml.ports.in new file mode 100644 index 000000000..b6d7b48e9 --- /dev/null +++ b/app/crowdb-access-server/conf/crowdb_access_server_config.toml.ports.in @@ -0,0 +1,69 @@ +# Canonical configuration for the S3 and Iceberg access processes. +# Credentials and bearer tokens are supplied through environment variables. + +[deployment] +mode = "production" +max_node_failures = 1 + +[common] +management_seeds = ["http://127.0.0.1:@PAXOS_KV_SERVER_MGMT_BASE@"] +diskio_connections_per_endpoint = 2 +diskio_rpc_workers = 2 + +[read] +stream_window_bytes = 1048576 +stream_slots = 3 +global_stream_bytes = 268435456 +recovery_memory_bytes = 268435456 + +[small_write] +threshold_ratio = 0.9 +disk_block_bytes = 1048576 +conversion_enabled = true +ec_data = 8 +ec_code = 4 +memory_budget_bytes = 1342177280 +queue_capacity = 1024 +min_pipelines = 1 +max_pipelines = 32 +max_batch_bytes = 1048576 +chunk_capacity_bytes = 268435456 +small_strip_prefetch_count = 32 + +[s3] +listen = "127.0.0.1:@ACCESS_SERVER_HTTP_BASE@" +tenant = "preview" +region = "us-east-1" +trusted_network = false +small_object_limit = 8388608 +list_scan_items = 1001 +list_scan_bytes = 4194304 +continuation_ttl_seconds = 900 +native_budget_bytes = 268435456 +cleanup_backlog_limit = 10000 +ec_data = 8 +ec_code = 4 +max_chunk_size = 1073741824 + +[iceberg] +listen = "127.0.0.1:@ACCESS_SERVER_ICEBERG_HTTP_BASE@" +native_budget_bytes = 268435456 +max_chunk_size = 1073741824 + +[iceberg.gc] +enabled = false +interval_ms = 1800000 +step_bytes = 8388608 +step_ms = 1000 +page_items = 64 +page_bytes = 49152 +concurrency = 1 +kv_bytes = 67108864 +kv_requests = 128 +chunk_bytes = 8388608 +chunk_requests = 128 +retry_base_ms = 1000 +retry_max_ms = 60000 +corruption_attempts = 3 +minimum_retention_ms = 604800000 +catalogs = [] diff --git a/app/crowdb-access-server/src/credentials.rs b/app/crowdb-access-server/src/credentials.rs index 4e42c014a..14e41e875 100644 --- a/app/crowdb-access-server/src/credentials.rs +++ b/app/crowdb-access-server/src/credentials.rs @@ -8,7 +8,7 @@ use std::sync::Arc; use crowdb_access_s3::auth::{ Credential, CredentialCipher, DurableCredentialRecord, IssuedUserToken, SecretError, }; -use crowdb_kv_client::{CrowdbKvClient, Error as KvError, ReadMode}; +use crowdb_paxos_kv_client::{Error as KvError, PaxosKvClient, ReadMode}; mod ensure; @@ -16,7 +16,7 @@ const CREDENTIAL_PREFIX: &[u8] = b"crowdb/s3/credential/"; const CREATE_ATTEMPTS: usize = 8; pub struct CredentialAuthority { - control: Arc<CrowdbKvClient>, + control: Arc<PaxosKvClient>, cipher: Arc<CredentialCipher>, } @@ -40,7 +40,7 @@ pub enum CredentialAuthorityError { impl CredentialAuthority { #[must_use] - pub fn new(control: Arc<CrowdbKvClient>, cipher: Arc<CredentialCipher>) -> Self { + pub fn new(control: Arc<PaxosKvClient>, cipher: Arc<CredentialCipher>) -> Self { Self { control, cipher } } diff --git a/app/crowdb-access-server/src/credentials/ensure.rs b/app/crowdb-access-server/src/credentials/ensure.rs index 6a0693e7d..40763279e 100644 --- a/app/crowdb-access-server/src/credentials/ensure.rs +++ b/app/crowdb-access-server/src/credentials/ensure.rs @@ -5,7 +5,7 @@ use super::{credential_key, CredentialAuthority, CredentialAuthorityError}; use crowdb_access_s3::auth::{DurableCredentialRecord, IssuedUserToken}; -use crowdb_kv_client::{Error as KvError, GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{Error as KvError, GetOutcome, ReadMode}; use sha2::{Digest, Sha256}; impl CredentialAuthority { diff --git a/app/crowdb-access-server/src/dataset.rs b/app/crowdb-access-server/src/dataset.rs index 1e11e221a..c70f40cf4 100644 --- a/app/crowdb-access-server/src/dataset.rs +++ b/app/crowdb-access-server/src/dataset.rs @@ -18,7 +18,7 @@ use crowdb_access_dataset::{ use crowdb_chunk_kv_client::{ ChunkKvClient, ChunkKvRpcTransport, ClientConfig as ChunkKvConfig, Group0ChunkKvRangeCatalogSource, }; -use crowdb_kv_client::{ClientConfig as KvConfig, CrowdbKvClient}; +use crowdb_paxos_kv_client::{ClientConfig as KvConfig, PaxosKvClient}; use http_body_util::{BodyExt, Full}; use hyper::body::{Bytes, Incoming}; use hyper::server::conn::http1; @@ -151,7 +151,7 @@ pub async fn run_with_shutdown( .listen .clone() .or_else(|| std::env::var("CROWDB_DATASET_LISTEN").ok()) - .unwrap_or_else(|| "127.0.0.1:9093".into()); + .unwrap_or_else(|| format!("127.0.0.1:{}", crowdb_protocol::ACCESS_SERVER_DATASET_HTTP_BASE)); let _: std::net::SocketAddr = address.parse()?; let seeds = if access.common.management_seeds.is_empty() { std::env::var("CROWDB_MANAGEMENT_SEEDS")? @@ -166,7 +166,7 @@ pub async fn run_with_shutdown( if seeds.is_empty() { return Err("one or more management seeds are required".into()); } - let control = Arc::new(CrowdbKvClient::new(KvConfig::new(seeds.clone()))); + let control = Arc::new(PaxosKvClient::new(KvConfig::new(seeds.clone()))); let source = Arc::new(Group0ChunkKvRangeCatalogSource::from_shared(Arc::clone(&control))); let chunk_config = ChunkKvConfig::default(); let transport = Arc::new(ChunkKvRpcTransport::with_config( @@ -240,7 +240,7 @@ pub fn start_dataset_cleanup_for_tests( async fn connect_payloads( seeds: Vec<String>, - control: Arc<CrowdbKvClient>, + control: Arc<PaxosKvClient>, capacity: usize, mut small_write: crowdb_chunk_client::SmallWritePolicy, ) -> Result<crowdb_chunk_client::ChunkIoClient, BoxError> { diff --git a/app/crowdb-access-server/src/iceberg/runtime.rs b/app/crowdb-access-server/src/iceberg/runtime.rs index 65acdceaa..7410a76cf 100644 --- a/app/crowdb-access-server/src/iceberg/runtime.rs +++ b/app/crowdb-access-server/src/iceberg/runtime.rs @@ -66,7 +66,7 @@ impl IcebergRuntimeConfig { .listen .clone() .or_else(|| std::env::var("CROWDB_ICEBERG_LISTEN").ok()) - .unwrap_or_else(|| "127.0.0.1:9092".into()); + .unwrap_or_else(|| format!("127.0.0.1:{}", crowdb_protocol::ACCESS_SERVER_ICEBERG_HTTP_BASE)); let _: std::net::SocketAddr = listen.parse()?; Ok(Self { listen, diff --git a/app/crowdb-access-server/src/main.rs b/app/crowdb-access-server/src/main.rs index 82504ab04..39667c7af 100644 --- a/app/crowdb-access-server/src/main.rs +++ b/app/crowdb-access-server/src/main.rs @@ -32,7 +32,7 @@ use crowdb_access_server::storage::S3StorageClients; #[cfg(feature = "s3")] use crowdb_chunk_client::LargeWritePolicy; #[cfg(feature = "s3")] -use crowdb_kv_client::{ClientConfig as KvConfig, CrowdbKvClient}; +use crowdb_paxos_kv_client::{ClientConfig as KvConfig, PaxosKvClient}; #[cfg(feature = "s3")] use tokio::net::TcpListener; #[cfg(feature = "s3")] @@ -271,7 +271,7 @@ async fn bind_health_listener(access_config: &AccessConfig) -> std::io::Result<T .listen .clone() .or_else(|| std::env::var("CROWDB_ACCESS_HEALTH_LISTEN").ok()) - .unwrap_or_else(|| "127.0.0.1:9094".into()); + .unwrap_or_else(|| format!("127.0.0.1:{}", crowdb_protocol::ACCESS_SERVER_HEALTH_HTTP_BASE)); TcpListener::bind(address).await } @@ -435,7 +435,7 @@ async fn issue_user(args: &[String], access: &AccessConfig) -> Result<(), Box<dy } else { access.common.management_seeds.clone() }; - let control = Arc::new(CrowdbKvClient::new(KvConfig::new(seeds))); + let control = Arc::new(PaxosKvClient::new(KvConfig::new(seeds))); let authority = CredentialAuthority::new(control, cipher); let token = match command.as_str() { "ensure-user" => authority.ensure_user(user.as_bytes()).await?, diff --git a/app/crowdb-access-server/tests/README.md b/app/crowdb-access-server/tests/README.md index c240a481f..af21e3147 100644 --- a/app/crowdb-access-server/tests/README.md +++ b/app/crowdb-access-server/tests/README.md @@ -3,169 +3,56 @@ # Access server acceptance -Protocol authority: [native Iceberg design](../../../doc/design/access-server/iceberge/design-crowdb-iceberg.md). -Client recipes: [container development](../../../container/single-node-container/README.md). - -- REST/official-SDK conformance, native durability and ecosystem compatibility - are distinct evidence. Passing SDK tests cannot certify compute engines. -- The [manual ecosystem fixture](../../../e2e-test/clients/ecosystem/README.md) - uses a separate locked environment and workflow dispatch. Its matrix records - only actual pinned client operations against the container and a volume restart. -- [S3 language SDKs](../../../e2e-test/clients/S3-SDK.md) and the Rust Iceberg SDK retain - independent manual workflows. They are not default test dependencies. -- Selected ORC and optional caches remain independent backlog ownership under - [ORC validation](../../../doc/backlog/R186-access-iceberg-orc-validation.md) - and [cache optimization](../../../doc/backlog/R185-access-iceberg-cache-invalidation.md). - The functional fixtures select Parquet and establish no cache speed guarantee. -- Physical GC remains an independent opt-in runtime contract. Logical drop, - clear, expiry and CAS loss do not authorize physical deletion by TTL alone. - Provisioned allocation capacity bounds storage; cumulative orphan storage - is not bounded by active request limits. +Architecture: [native Iceberg storage](../../../doc/design/design-crowdb-access-iceberg.md). Protocol implementation: [Iceberg wire types](../../../lib/crowdb-access-iceberg/src/wire.rs) and [request handlers](../../../lib/crowdb-access-iceberg/src). Client recipes: [container development](../../../container/single-node-container/README.md). + +- REST/official-SDK conformance, native durability and ecosystem compatibility are distinct evidence. Passing SDK tests cannot certify compute engines. +- The [manual ecosystem fixture](../../../e2e-test/clients/ecosystem/README.md) uses a separate locked environment and workflow dispatch. Its matrix records only actual pinned client operations against the container and a volume restart. +- [S3 language SDKs](../../../e2e-test/clients/S3-SDK.md) and the Rust Iceberg SDK retain independent manual workflows. They are not default test dependencies. +- Selected ORC validation and optional cache optimization remain deferred. The functional fixtures select Parquet and establish no cache speed guarantee. +- Physical GC remains an independent opt-in runtime contract. Logical drop, clear, expiry and CAS loss do not authorize physical deletion by TTL alone. Provisioned allocation capacity bounds storage; cumulative orphan storage is not bounded by active request limits. ## Main integration acceptance -The 2026-10-26 integration rebases all 252 local commits onto main -`94135bfeca2a04e59b5b3c7339d6300dc17f2042`. The range comparison retains -210 identical patches, 41 adapted patches and the corresponding standalone RPC -benchmark move. Integration commit `c550f4c5` preserves ordinary V2 frames, -owner-backed payload views, the reader layout cache, single-request synchronized -DiskIO writes and ordered Stream chunk records. Serving metadata maintenance -keeps progressing after one-record deletions and revisits unchanged generations. -Nonzero WAL trimming still requires durable expired-request floors and fails -closed while those floors are unavailable. +The 2026-10-26 integration rebases all 252 local commits onto main `94135bfeca2a04e59b5b3c7339d6300dc17f2042`. The range comparison retains 210 identical patches, 41 adapted patches and the corresponding standalone RPC benchmark move. Integration commit `c550f4c5` preserves ordinary V2 frames, owner-backed payload views, the reader layout cache, single-request synchronized DiskIO writes and ordered Stream chunk records. Serving metadata maintenance keeps progressing after one-record deletions and revisits unchanged generations. Nonzero WAL trimming still requires durable expired-request floors and fails closed while those floors are unavailable. The frozen service image is `crowdb-node:access-cleanup-v11-20261026`: - Build revision: `c550f4c5bd977c1287778fa17add9fb6fd79328f`. - Source tree: `3483a0c16d1ad44e27626be1e2262ee4ddf1659b`. -- Imported Docker ID: - `sha256:10c1042b734b9af43223db3d3a71817839d0327e644b6588abb2bfed1599b255`. -- Runtime SHA-256: - `1f3f5444b801d5099dc6b1d6aeb9880c76db8803e09a8606edc61d23c1439443`. -- OCI receipt/config/layer digests, the imported Docker runtime configuration - and uncompressed layer identities, staged runtime hash and source archive - are independently verified. Docker conversion changes the local image ID; - it is recorded separately from the OCI manifest and config digests. - -The completed component inventories contain 129 storage/transport targets -(716 passing tests, one configured ignore) and 411 Access/KV targets -(2,438 passing tests, 13 configured ignores). Monitor preview adds four passes; -the independent HTTP SDK has 43 passing tests and all-target clippy success. -Workspace all-feature/all-target clippy, normal `rs-lint` with version/DashMap -checks, formatting and C++ tree-lint pass. C++ Tree, RPC, DiskIO and common -inventories complete, retaining their configured hardware skips. Python fixture -tests pass 252 cases and the benchmark runner passes 17. - -The first five-node KV group run missed its initial three-second leader wait. -The unchanged exact case passes alone; the complete group subsequently passes -131 cases under both default concurrency and serial execution. The failed log -and both full regressions are retained. High filesystem I/O pressure was present -on the first run; its causal role is unproven. Election timers and caller retry -behavior were unchanged. Other integration test repairs replace V1 geometry -literals with V2 protocol sizes and retain exact buffer-address, feed-count, -ownership and error assertions. - -Current-source real-service checks pass all five Dataset cases, including -installed portable SDKs and restart publication semantics. The Split case -performs five storage/Access restarts in 219.99 seconds within its unchanged -600-second bound. The extended two-cleaner GC matrix passes in 122.12 seconds -with three simultaneous cleaner restart rounds, nine dead files and successful -foreground work during deletion. It records the exact accepted large range -`[0,16786211)`, sealed boundary `16786432`, unchanged layout/epoch and -115,288 ms of final continuous drain. The external/tree/live-gap case passes -in 161.14 seconds. These establish notification acceptance and retained live -ranges; they make no physical-space-release or production latency claim. - -Strict local-HCA checks pass HTTP/ChunkIO to the separate O_DIRECT DiskIO -process, its interrupted-download case, and Chunk-KV large logical controls. -The normal DiskIO RDMA payload-copy, fallback and CQ error counters remain zero. -Chunk-KV uses the production RPC/service/native tree/WAL with injected catalog, -authority and memory ChunkStream backing. The separately activated native -Dataset accepted-read failure test also passes: exactly two accepted reads and -one reply, no replay or metadata re-selection, a valid held first buffer through -shutdown and returned credits after its final release. This is same-host HCA -functional evidence rather than cross-host or throughput measurement. - -The installed native Python SDK also passes the separate semantics-only workload: -mixed-size publication, continuation replay, cancellation and buffer lifetime, -65 seconds idle, and storage/Access restart. Both complete native S3 and Dataset -HCA corpora pass write and post-storage-restart verification. S3 covers host and -non-root container clients; Dataset uses the installed native Python client. -Strict RDMA remains required, and Dataset receive credits return after the last -held buffer is released. - -All ten ordered jobs, including image creation, finish successfully. Independent -verification preserves 1,432 raw evidence files with byte counts and SHA-256 -digests under -`.crowdb-runtime/persistent/task-checkpoints/access-simplification-20261026/evidence-current/`. -Its `verified-summary.json` maps cases to their original run identities; OCI, -source and provenance records are retained alongside them. The additional SDK -HCA terminal-read test is recorded separately in -`current-dataset-hca-terminal.json` and its complete log. Installed SDK, S3 HCA -and Dataset HCA run identities are respectively -`dataset-workload/65c5124702d840bcb825d8d42a6a42f2`, -`native-s3-hca/56ee513daa424136bcda3c580e8ab4fe` and -`native-dataset-hca/9465ca2f4c754fd096b519b0691cb214`. - -The preceding v10 throughput trials retain their original source boundary; -they are not measurements of this rebased service image. +- Imported Docker ID: `sha256:10c1042b734b9af43223db3d3a71817839d0327e644b6588abb2bfed1599b255`. +- Runtime SHA-256: `1f3f5444b801d5099dc6b1d6aeb9880c76db8803e09a8606edc61d23c1439443`. +- OCI receipt/config/layer digests, the imported Docker runtime configuration and uncompressed layer identities, staged runtime hash and source archive are independently verified. Docker conversion changes the local image ID; it is recorded separately from the OCI manifest and config digests. + +The completed component inventories contain 129 storage/transport targets (716 passing tests, one configured ignore) and 411 Access/KV targets (2,438 passing tests, 13 configured ignores). Monitor preview adds four passes; the independent HTTP SDK has 43 passing tests and all-target clippy success. Workspace all-feature/all-target clippy, normal `rs-lint` with version/DashMap checks, formatting and C++ tree-lint pass. C++ Tree, RPC, DiskIO and common inventories complete, retaining their configured hardware skips. Python fixture tests pass 252 cases and the benchmark runner passes 17. + +The first five-node KV group run missed its initial three-second leader wait. The unchanged exact case passes alone; the complete group subsequently passes 131 cases under both default concurrency and serial execution. The failed log and both full regressions are retained. High filesystem I/O pressure was present on the first run; its causal role is unproven. Election timers and caller retry behavior were unchanged. Other integration test repairs replace V1 geometry literals with V2 protocol sizes and retain exact buffer-address, feed-count, ownership and error assertions. + +Current-source real-service checks pass all five Dataset cases, including installed portable SDKs and restart publication semantics. The Split case performs five storage/Access restarts in 219.99 seconds within its unchanged 600-second bound. The extended two-cleaner GC matrix passes in 122.12 seconds with three simultaneous cleaner restart rounds, nine dead files and successful foreground work during deletion. It records the exact accepted large range `[0,16786211)`, sealed boundary `16786432`, unchanged layout/epoch and 115,288 ms of final continuous drain. The external/tree/live-gap case passes in 161.14 seconds. These establish notification acceptance and retained live ranges; they make no physical-space-release or production latency claim. + +Strict local-HCA checks pass HTTP/ChunkIO to the separate O_DIRECT DiskIO process, its interrupted-download case, and Chunk-KV large logical controls. The normal DiskIO RDMA payload-copy, fallback and CQ error counters remain zero. Chunk-KV uses the production RPC/service/native tree/WAL with injected catalog, authority and memory ChunkStream backing. The separately activated native Dataset accepted-read failure test also passes: exactly two accepted reads and one reply, no replay or metadata re-selection, a valid held first buffer through shutdown and returned credits after its final release. This is same-host HCA functional evidence rather than cross-host or throughput measurement. + +The installed native Python SDK also passes the separate semantics-only workload: mixed-size publication, continuation replay, cancellation and buffer lifetime, 65 seconds idle, and storage/Access restart. Both complete native S3 and Dataset HCA corpora pass write and post-storage-restart verification. S3 covers host and non-root container clients; Dataset uses the installed native Python client. Strict RDMA remains required, and Dataset receive credits return after the last held buffer is released. + +All ten ordered jobs, including image creation, finish successfully. Independent verification preserves 1,432 raw evidence files with byte counts and SHA-256 digests under `.crowdb-runtime/persistent/task-checkpoints/access-simplification-20261026/evidence-current/`. Its `verified-summary.json` maps cases to their original run identities; OCI, source and provenance records are retained alongside them. The additional SDK HCA terminal-read test is recorded separately in `current-dataset-hca-terminal.json` and its complete log. Installed SDK, S3 HCA and Dataset HCA run identities are respectively `dataset-workload/65c5124702d840bcb825d8d42a6a42f2`, `native-s3-hca/56ee513daa424136bcda3c580e8ab4fe` and `native-dataset-hca/9465ca2f4c754fd096b519b0691cb214`. + +The preceding v10 throughput trials retain their original source boundary; they are not measurements of this rebased service image. ## Expanded cleanup and read observations -The 2026-10-25 cleanup batch passes 1,842 tests across 407 targets in six -affected crates, with zero failures and 13 separately configured cases ignored. -Workspace all-target/all-feature clippy, normal lint, formatting, C++ tree lint, -184 fixture unit checks and the independent HTTP SDK gates pass. These gates -and the measurements below precede the subsequent rebase onto newer `main`. +The 2026-10-25 cleanup batch passes 1,842 tests across 407 targets in six affected crates, with zero failures and 13 separately configured cases ignored. Workspace all-target/all-feature clippy, normal lint, formatting, C++ tree lint, 184 fixture unit checks and the independent HTTP SDK gates pass. These gates and the measurements below precede the subsequent rebase onto newer `main`. -The measured image is `crowdb-node:access-cleanup-v10-20261025`, Docker ID -`sha256:c54fca81a58c4255ee38668a95f0dfa615111c35eda574cc0c9895fc043446ce`. -Its build revision is `8611270da82b1b49cf4619092f7961c3ff1c7ea5`, captured tree -`b733912cba7ea0c88d2c4d6a8ceb23043694f45f`, and runtime SHA256 -`00189c332c7eafc461390b8f09266f441c9af7e3de4eeb7912fc5c1ccd3dcccc`. -The later host-test correction changes no production source. Every selected -service run verifies the matching image and server-source identity. +The measured image is `crowdb-node:access-cleanup-v10-20261025`, Docker ID `sha256:c54fca81a58c4255ee38668a95f0dfa615111c35eda574cc0c9895fc043446ce`. Its build revision is `8611270da82b1b49cf4619092f7961c3ff1c7ea5`, captured tree `b733912cba7ea0c88d2c4d6a8ceb23043694f45f`, and runtime SHA256 `00189c332c7eafc461390b8f09266f441c9af7e3de4eeb7912fc5c1ccd3dcccc`. The later host-test correction changes no production source. Every selected service run verifies the matching image and server-source identity. Seven real-service cases pass with their original budgets and deadlines: -- The new two-cleaner matrix passes in - `container-e2e/run-0c6216e0dd5b4d3297eb08926ad1986a`. Both Access processes - are killed and restarted three times. Nine dead canonical files disappear, - their candidates and purge task complete, and 16,777,304 logical bytes are - accounted. A future deleted file remains retained at its original deadline - while a higher-ID expired file completes. Each cleaner records ten errors - scanning an independent corrupt catalog; its original bytes remain intact - while the healthy catalog converges. -- A real 16-MiB dedicated object is written and verified before cleanup. The - accepted retirement is exactly `[0,16785954)` within the unchanged sealed - extent of 16,786,432 bytes, with writer epoch zero. This verifies a range - larger than the unchanged 8-MiB payload budget: a deletion control notification - consumes one chunk-operation request and no referenced payload bytes. -- The matrix takes 119.03 seconds; its final uninterrupted drain takes - 105,874 ms. Forty-eight official-SDK table lifecycles and 64 native small - writes complete, with successful SDK requests inside three observed deletion - intervals. Table create/commit/load/read-metadata p99 values are - 305.612/44.717/4.532/14.729 ms. Table actions have 48 samples each; namespace - create/drop have four. These are finite same-host observations, without an - isolated baseline, a physical-capacity claim or an arbitrary-restart SLO. -- The unchanged external/tree/live-gap case passes in 167.38 seconds. All five - Dataset HTTP, SDK and restart cases also pass on this image. - -The first matrix execution fails its overlap assertion: sparse multipart -cleanup starts pressure at 51.59 seconds, but purge deletion starts at -106.92 seconds after the SDK load has ended. The host test now starts and -observes pressure using the selected purge's deletion count. The original -overlap assertion, deadlines, workload and production implementation remain. -The failed run `run-5de26741afd24a4ba63215ddf20a9d04` is retained separately. - -Dataset workload `dataset-workload/fb782f743944466ea6193a148dc46e69` passes all -18 trials, 4,608 complete requests and 18 completed observations, with zero -errors. Installed Python byte/continuation/Split checks and storage restart -also pass. The original eight sizes, concurrency one/three, two warmup rounds, -three repeats, sixteen rounds, integrity checks and 600-second child bound -remain. Client CPU is pinned to 2, services to 4–15, and metadata replicas retain -their 2-GiB cap. This is the TCP profile; earlier HCA evidence retains its own -image identity in [the Dataset/HCA record](#dataset-httpnative-comparison). +- The new two-cleaner matrix passes in `container-e2e/run-0c6216e0dd5b4d3297eb08926ad1986a`. Both Access processes are killed and restarted three times. Nine dead canonical files disappear, their candidates and purge task complete, and 16,777,304 logical bytes are accounted. A future deleted file remains retained at its original deadline while a higher-ID expired file completes. Each cleaner records ten errors scanning an independent corrupt catalog; its original bytes remain intact while the healthy catalog converges. +- A real 16-MiB dedicated object is written and verified before cleanup. The accepted retirement is exactly `[0,16785954)` within the unchanged sealed extent of 16,786,432 bytes, with writer epoch zero. This verifies a range larger than the unchanged 8-MiB payload budget: a deletion control notification consumes one chunk-operation request and no referenced payload bytes. +- The matrix takes 119.03 seconds; its final uninterrupted drain takes 105,874 ms. Forty-eight official-SDK table lifecycles and 64 native small writes complete, with successful SDK requests inside three observed deletion intervals. Table create/commit/load/read-metadata p99 values are 305.612/44.717/4.532/14.729 ms. Table actions have 48 samples each; namespace create/drop have four. These are finite same-host observations, without an isolated baseline, a physical-capacity claim or an arbitrary-restart SLO. +- The unchanged external/tree/live-gap case passes in 167.38 seconds. All five Dataset HTTP, SDK and restart cases also pass on this image. + +The first matrix execution fails its overlap assertion: sparse multipart cleanup starts pressure at 51.59 seconds, but purge deletion starts at 106.92 seconds after the SDK load has ended. The host test now starts and observes pressure using the selected purge's deletion count. The original overlap assertion, deadlines, workload and production implementation remain. The failed run `run-5de26741afd24a4ba63215ddf20a9d04` is retained separately. + +Dataset workload `dataset-workload/fb782f743944466ea6193a148dc46e69` passes all 18 trials, 4,608 complete requests and 18 completed observations, with zero errors. Installed Python byte/continuation/Split checks and storage restart also pass. The original eight sizes, concurrency one/three, two warmup rounds, three repeats, sixteen rounds, integrity checks and 600-second child bound remain. Client CPU is pinned to 2, services to 4–15, and metadata replicas retain their 2-GiB cap. This is the TCP profile; earlier HCA evidence retains its own image identity in [the Dataset/HCA record](#dataset-httpnative-comparison). | Path | Concurrency | MiB/s | Inner client CPU (s) | Lifetime client RSS (MiB) | | ------------- | ----------- | ------ | -------------------- | ------------------------ | @@ -176,253 +63,74 @@ image identity in [the Dataset/HCA record](#dataset-httpnative-comparison). | Native Rust | 3 | 271.95 | 2.73 | 31.54 | | Native Python | 3 | 269.04 | 2.82 | 63.96 | -Throughput divides summed bytes by summed inner trial time; client CPU sums -inner ticks, while RSS is the maximum lifetime high-water observation. These -independent runs do not isolate the speedup of any individual patch. - -Every trial now records cumulative client-visible batch-header, first nonempty -delivery and verified-EOF offsets alongside the unchanged completion latency. -They are not wire-first-byte timings or additive phase durations. Python's -explicit `to_bytes` copies are counted separately. All twelve native trials -validate the sixteen existing shared-reader counter deltas, including positive -layout-query counts, against the raw boundary snapshots. Native metadata, -aggregate and data sent groups remain separate from those shared-reader metrics. -The selected counters are not all RPCs or all copies. Service CPU windows include -background work and sequential capture; they cannot be added to inner client CPU -as exact total-system cost. Process/cgroup peaks and held high-water observations -are cumulative, not per-trial peaks. TCP receive totals, all-copy bytes, precise -GC attribution and per-request server CPU remain unmeasured. - -The former `/tmp` summaries are absent when work resumes on 2026-10-26. The -original repository artifacts and OCI receipt remain available. A read-only -reconstruction verifies all eighteen raw phase/count/copy reports, resource and -transport deltas, and the seven passing service cases without rerunning them. -Raw records, reconstructed summaries, verification source, image/source archives -and a verified Git checkpoint are preserved locally under -`.crowdb-runtime/persistent/task-checkpoints/access-simplification-20261026`. -The immutable image results above do not certify later production changes from -rebasing onto `main`. +Throughput divides summed bytes by summed inner trial time; client CPU sums inner ticks, while RSS is the maximum lifetime high-water observation. These independent runs do not isolate the speedup of any individual patch. + +Every trial now records cumulative client-visible batch-header, first nonempty delivery and verified-EOF offsets alongside the unchanged completion latency. They are not wire-first-byte timings or additive phase durations. Python's explicit `to_bytes` copies are counted separately. All twelve native trials validate the sixteen existing shared-reader counter deltas, including positive layout-query counts, against the raw boundary snapshots. Native metadata, aggregate and data sent groups remain separate from those shared-reader metrics. The selected counters are not all RPCs or all copies. Service CPU windows include background work and sequential capture; they cannot be added to inner client CPU as exact total-system cost. Process/cgroup peaks and held high-water observations are cumulative, not per-trial peaks. TCP receive totals, all-copy bytes, precise GC attribution and per-request server CPU remain unmeasured. + +The former `/tmp` summaries are absent when work resumes on 2026-10-26. The original repository artifacts and OCI receipt remain available. A read-only reconstruction verifies all eighteen raw phase/count/copy reports, resource and transport deltas, and the seven passing service cases without rerunning them. Raw records, reconstructed summaries, verification source, image/source archives and a verified Git checkpoint are preserved locally under `.crowdb-runtime/persistent/task-checkpoints/access-simplification-20261026`. The immutable image results above do not certify later production changes from rebasing onto `main`. ## Historical evidence retention after shared-directory removal -During an earlier Dataset run, the repository's `.crowdb-runtime` and -`target` directories disappeared. The deletion source is unconfirmed. Artifact -IDs below identify their completed executions; the original runtime directories -are no longer present. Source commits and Git bundle backups remain intact. - -The new image's 12-case acceptance stdout, component/lint logs, recovered GC -observations and all 18 Dataset trial JSON records are retained in -`/tmp/crowdb-access-continuation-evidence-20261025`. The Dataset trials complete -4,608 reads with zero errors, but the semantic process exits unsuccessfully and -its deleted stderr cannot be recovered. The runner then fails on missing log and -manifest paths. This execution does not pass installed-binding or restart -acceptance, and its service cgroup manifest is unavailable. Rerun only with a -stable artifact directory; do not substitute the completed trials for the missing -semantic/restart gate. The imported Docker image remains available independently -of the removed OCI archive. +During an earlier Dataset run, the repository's `.crowdb-runtime` and `target` directories disappeared. The deletion source is unconfirmed. Artifact IDs below identify their completed executions; the original runtime directories are no longer present. Source commits and Git bundle backups remain intact. + +The new image's 12-case acceptance stdout, component/lint logs, recovered GC observations and all 18 Dataset trial JSON records are retained in `/tmp/crowdb-access-continuation-evidence-20261025`. The Dataset trials complete 4,608 reads with zero errors, but the semantic process exits unsuccessfully and its deleted stderr cannot be recovered. The runner then fails on missing log and manifest paths. This execution does not pass installed-binding or restart acceptance, and its service cgroup manifest is unavailable. Rerun only with a stable artifact directory; do not substitute the completed trials for the missing semantic/restart gate. The imported Docker image remains available independently of the removed OCI archive. ## Performance work to consolidate later -- A native fault-matrix diagnostic run returned `Store(Client(Deadline))` from - the independent verification client's first file-record load, after HTTP replay - succeeded. No request timeout or caller retry was changed; two subsequent complete - 44-case runs passed. The cause of that one five-second client deadline remains - unconfirmed. Capture fresh client routing/transport and backend timing if it - recurs; do not describe it as fixed by FileIO scheduling changes. -- Native multipart diagnostics exposed unequal competing copy windows, tiny - checkpoint-only leaves, repeated directory reads and duplicate JSON digest - passes. Assembly now recomputes from the complete selected-part input without - persisting byte progress. Broader batching, shared decoded caches, - sustained throughput and recovery-page scaling remain measurement work, not - implied guarantees from the original-bound functional fixture passing. - -- SDK diagnostic: the first expanded in-memory Java lifecycle run returned 503 - at purge on 2026-09-24. One instrumented rerun and two fixed diagnostic batches - (five and ten runs) passed without changing timeouts, adding retries or suppressing - assertions. No server diagnostic was captured for the original failure; its root - cause remains unconfirmed. Keep this as a follow-up observation, not a fixed bug - or a reason to claim a stronger latency guarantee. Preserve the unchanged SDK - command and capture request-admission/deadline diagnostics if it recurs. - -- Historical namespace diagnostics measured roughly 45–75 ms per durable phase - and intermittent failure under a 500-ms total bound. Refresh measurements before - attributing current cost to any component; these are not current p95/p99 values. -- Full namespace CRUD uses the bounded 300,000-ms functional profile with - delegation disabled; raw HTTP client timeouts remain five seconds. The separate - maintenance fixture retains 500 ms. Their passing results are not evidence that - every namespace mutation meets 500 ms. Earlier redundant immutable-payload and - terminal-cleanup writes were fixed without altering publication CAS. -- Maintenance fixture repair errors for synthetic reserved name mappings without - journals are expected from `verify_name_index`; retain the diagnostics rather - than interpreting them as production corruption or suppressing them. -- Profile journal/retry-ledger round trips and durable payload/checkpoint writes - on identical storage, concurrency and data. Prior redundant writes already - received no-op/read-before-put fixes; do not reimplement them blindly. -- Consider batching or pipeline changes only with measured evidence and preserved - publication/clear/replay invariants. Record before/after latency, KV round trips, - storage I/O, CPU and memory alongside failure-injection regression results. -- Create one consolidated optimization backlog later. No new performance backlog - or latency guarantee is introduced by the functional-test split itself. +- A native fault-matrix diagnostic run returned `Store(Client(Deadline))` from the independent verification client's first file-record load, after HTTP replay succeeded. No request timeout or caller retry was changed; two subsequent complete 44-case runs passed. The cause of that one five-second client deadline remains unconfirmed. Capture fresh client routing/transport and backend timing if it recurs; do not describe it as fixed by FileIO scheduling changes. +- Native multipart diagnostics exposed unequal competing copy windows, tiny checkpoint-only leaves, repeated directory reads and duplicate JSON digest passes. Assembly now recomputes from the complete selected-part input without persisting byte progress. Broader batching, shared decoded caches, sustained throughput and recovery-page scaling remain measurement work, not implied guarantees from the original-bound functional fixture passing. + +- SDK diagnostic: the first expanded in-memory Java lifecycle run returned 503 at purge on 2026-09-24. One instrumented rerun and two fixed diagnostic batches (five and ten runs) passed without changing timeouts, adding retries or suppressing assertions. No server diagnostic was captured for the original failure; its root cause remains unconfirmed. Keep this as a follow-up observation, not a fixed bug or a reason to claim a stronger latency guarantee. Preserve the unchanged SDK command and capture request-admission/deadline diagnostics if it recurs. + +- Historical namespace diagnostics measured roughly 45–75 ms per durable phase and intermittent failure under a 500-ms total bound. Refresh measurements before attributing current cost to any component; these are not current p95/p99 values. +- Full namespace CRUD uses the bounded 300,000-ms functional profile with delegation disabled; raw HTTP client timeouts remain five seconds. The separate maintenance fixture retains 500 ms. Their passing results are not evidence that every namespace mutation meets 500 ms. Earlier redundant immutable-payload and terminal-cleanup writes were fixed without altering publication CAS. +- Maintenance fixture repair errors for synthetic reserved name mappings without journals are expected from `verify_name_index`; retain the diagnostics rather than interpreting them as production corruption or suppressing them. +- Profile journal/retry-ledger round trips and durable payload/checkpoint writes on identical storage, concurrency and data. Prior redundant writes already received no-op/read-before-put fixes; do not reimplement them blindly. +- Consider batching or pipeline changes only with measured evidence and preserved publication/clear/replay invariants. Record before/after latency, KV round trips, storage I/O, CPU and memory alongside failure-injection regression results. +- Create one consolidated optimization backlog later. No new performance backlog or latency guarantee is introduced by the functional-test split itself. ## Previous cleanup regression gates -The preceding v9 six-crate all-target/all-feature gate passes 1,828 tests across 403 -targets, with zero failures and 13 separately configured cases ignored. -Workspace all-feature clippy, normal lint, both workspace formatting checks, -the container client build and 179 fixture unit tests pass. The matching local -image is `crowdb-node:access-cleanup-v9-20261025`, imported Docker ID -`sha256:67a1ef4f24901cee4be69b6951461b00059fcf58bed9d5d6960c0dcfd94f5de6`. -Its revision label is `0225659a7d632140e7d24fb86aa7fa989f51ffb8`; captured source -tree `cc7421928d395ee3a2e6a64f4eb35aca836d6d25` also includes the budgeted -Locations notification fix. Runtime SHA256 is -`e994c5dc0d7f89c4ec03519ec69f1ec3ea8046e53a53a6a2cbed497ec6812683`. -All eleven selected real-service cases pass with the original case deadlines -and GC budgets: - -- External Locations/tree notification convergence: - `container-e2e/run-48bc19a9be0a48dbaf013cf41efefe8a`. -- Catalog recovery through real Chunk-KV restart: - `container-e2e/run-c521bc97e14c42059584f25a8d3cf494`. -- Official namespace CRUD through storage/listener restart: - `container-e2e/run-42f6dcb9f2b2472c80bd774e2c8ed7d5`. -- Official SDK foreground work during GC backlog: - `container-e2e/run-8e25a3f3d11c428bb6beef0f15319fb4`. -- Seven S3 cases cover replacement parts, UploadPartCopy, boto3 checksums, - user metadata, lost replies, an open/replaced-part upload through metadata - restart, and HTTP/native parity. Their exact case/run mappings, every client - exit code and matching image identities are retained in - `/tmp/crowdb-access-cleanup-v9-evidence-20261025/acceptance-status.json`. - -Depth-one empty namespace deletion uses six GETs, two CAS operations, one exact -mapping delete and two scans, compared with the measured journal path's 51 -GETs, 13 CAS operations, one delete and two scans. Depth three uses ten GETs. -The two complete child proofs, generation fences and four catalog checks remain. -A fresh retry may delete a recreated empty name; one selected invocation cannot -delete its replacement, and a nonempty namespace still fails. Deterministic -component races cover late creation, rename, property updates, unknown writes -and retained legacy recovery against permanent retired identities. - -An ordinary independent MPU part's metadata path uses three GETs and two -conditional writes, compared with two GETs and four conditional writes. Distinct -parts perform no shared upload CAS. An initially matching pointer retains two -GETs and no metadata writes. Frozen generation A remains authoritative when an -already admitted replacement B completes later; the deterministic component -barrier proves that interleaving. The service cases do not claim to force it. -Strict older resources and tighter quotas retain their reservation protocol. -These are logical metadata-action counts, not routed wire totals or isolated -end-to-end TPS measurements. - -The external GC case completes its convergence window in 161,787 ms at task -revision 56 after one actual Access restart; the full client case takes 162.47 s. -Both original file keys are absent, their candidates and task are Complete, -and the fixed task identity/deadline survive restart. Two dead files contain -33,280 logical payload bytes. Three tree blocks and one external frame produce -the exact durable retired range union `[0,16964)` and `[17510,34212)`, totaling -33,666 chunk bytes including frame/directory overhead. The interleaved live -range `[16964,17510)` remains readable, layout identity is unchanged, and a -stale marker cannot retire the newer live head. This establishes notification -acceptance, without a physical free-space or GC latency SLO claim. - -The preceding v8 external case failed at its unchanged 480-second bound because -`BudgetedGcBlocks` inherited the default deferred `reclaim_location` instead of -dispatching to its configured provider. The wrapper now charges its existing -request/byte budget and forwards the operation and result unchanged. The failed -v8 evidence remains preserved independently. Final v9 source/image archives, -raw logs, range validation and gate outputs are retained outside the repository -in `/tmp/crowdb-access-cleanup-v9-evidence-20261025`. Dataset read and transport -paths did not change in this batch; their completed v8 workload/HCA evidence -below retains its own image and source identity. - -Source `b566fdcd` adds compact upload-resource storage, one-step ordinary inline -purge completion and request-local Dataset delivery reductions. The upload/GC -batch passes 1,709 tests across five affected crates, with 13 independently -configured cases ignored. The subsequent Dataset/Access/native batch passes -669 tests, with the same 13 configured cases ignored. These suites overlap; -their counts are not additive. Workspace all-target/all-feature clippy, normal -lint, formatting and focused final regressions pass. The matching v7 image from -`a78e981f` passes six S3, five Dataset and five GC cases, plus the complete -TCP/installed-Python/restart workload. The v6 results below cover `c22bc40e`. - -Source `d400ae21` also discovers expired multipart uploads by immutable bucket -identity after a bucket name is deleted and recreated. A production-actor -regression fails on the preceding scanner and passes with the fix. The affected -S3/Access/native suite passes 495 tests, with 12 separately configured cases -ignored; the final multipart target passes all 64 cases. Only expired Open -uploads are marked Aborted. Index, part and generation rows remain, and this -does not establish physical space recovery. Workspace all-feature clippy, -normal lint and formatting pass. Matching v7 service acceptance passes. - -The upload codec keeps all owner identities and the public JSON contract while -writing a fixed 52-byte header plus 16 bytes per identity. Its maximum record is -69,684 bytes. Legacy JSON keeps its original validation, and CAS conditions use -the actual stored bytes. JSON-only Access servers and direct native clients -require a coordinated upgrade. No resource key or foreground RPC is added. -The fixed legacy authority-read budget is retained. - -An exclusive CPU-2 release measurement uses seven samples of 128 iterations, -with the same logical 4,095 retired and two live owners. Canned retirement -includes GET cloning and exact CAS comparisons, without KV, WAL or network. -Compact input takes a median 25.644 us after sorted insertion, compared with -29.395 us using append-and-sort; validated compact round trips take 7.589 us. -Legacy JSON migration takes 654.204 us in the later run and 629.274 us in the -preceding run, so it is not reported as an insertion speedup. Raw samples, -timer-boundary measurements, source/binary hashes and scope are retained in -`/tmp/crowdb-c12-owner-cpu-20261025` and -`/tmp/crowdb-c12-owner-cpu-insert-20261025`. These are CPU observations, not TPS -or total-system efficiency claims. - -Source `c22bc40e` additionally includes retained point-operation buffers across -routed retries, observed-authority multipart expiry and reused validated inline -digests. The six affected crates pass 1,109 tests, with 13 separately configured -cases ignored. Workspace clippy, normal lint and formatting pass. Its matching -`access-cleanup-v6-20261025` image passes six S3 protocol cases and all five -Dataset cases (`run-8376c20aa6ea425583e2d8d6be33fcd9`), plus the complete -[TCP/restart workload](#dataset-httpnative-comparison). The five-case GC scope also passes in -`run-d56438cb607349f3a4d819402ee8d484`, including exact full inline-backlog -convergence: 16 real-service cases in total. - -Boundary regressions preserve the caller's point payload allocations through -first dispatch and routed retry. Expired multipart refresh uses one GET and one -CAS instead of two GETs and one CAS; independent public abort still reads its -own authority. Both native Dataset entry points verify inline corruption before -reusing the immutable result. These are copy/scan and logical-operation results, -not an isolated end-to-end throughput measurement of the changes. Raw v6 -artifacts and immutable image identity are independently retained in -`/tmp/crowdb-access-cleanup-v6-evidence-20261025`. - -The preceding source `ae4e1b8a` includes the bounded upload-retirement, Split-membership and -Iceberg receiver optimizations. All-target/all-feature Dataset/Access -Server/native coverage passes: 645 tests, 13 separately configured cases -ignored. Workspace clippy, `rs-lint` and formatting pass. -The matching `access-cleanup-v5-20261025` image passes 21 real-service cases: +The preceding v9 six-crate all-target/all-feature gate passes 1,828 tests across 403 targets, with zero failures and 13 separately configured cases ignored. Workspace all-feature clippy, normal lint, both workspace formatting checks, the container client build and 179 fixture unit tests pass. The matching local image is `crowdb-node:access-cleanup-v9-20261025`, imported Docker ID `sha256:67a1ef4f24901cee4be69b6951461b00059fcf58bed9d5d6960c0dcfd94f5de6`. Its revision label is `0225659a7d632140e7d24fb86aa7fa989f51ffb8`; captured source tree `cc7421928d395ee3a2e6a64f4eb35aca836d6d25` also includes the budgeted Locations notification fix. Runtime SHA256 is `e994c5dc0d7f89c4ec03519ec69f1ec3ea8046e53a53a6a2cbed497ec6812683`. All eleven selected real-service cases pass with the original case deadlines and GC budgets: + +- External Locations/tree notification convergence: `container-e2e/run-48bc19a9be0a48dbaf013cf41efefe8a`. +- Catalog recovery through real Chunk-KV restart: `container-e2e/run-c521bc97e14c42059584f25a8d3cf494`. +- Official namespace CRUD through storage/listener restart: `container-e2e/run-42f6dcb9f2b2472c80bd774e2c8ed7d5`. +- Official SDK foreground work during GC backlog: `container-e2e/run-8e25a3f3d11c428bb6beef0f15319fb4`. +- Seven S3 cases cover replacement parts, UploadPartCopy, boto3 checksums, user metadata, lost replies, an open/replaced-part upload through metadata restart, and HTTP/native parity. Their exact case/run mappings, every client exit code and matching image identities are retained in `/tmp/crowdb-access-cleanup-v9-evidence-20261025/acceptance-status.json`. + +Depth-one empty namespace deletion uses six GETs, two CAS operations, one exact mapping delete and two scans, compared with the measured journal path's 51 GETs, 13 CAS operations, one delete and two scans. Depth three uses ten GETs. The two complete child proofs, generation fences and four catalog checks remain. A fresh retry may delete a recreated empty name; one selected invocation cannot delete its replacement, and a nonempty namespace still fails. Deterministic component races cover late creation, rename, property updates, unknown writes and retained legacy recovery against permanent retired identities. + +An ordinary independent MPU part's metadata path uses three GETs and two conditional writes, compared with two GETs and four conditional writes. Distinct parts perform no shared upload CAS. An initially matching pointer retains two GETs and no metadata writes. Frozen generation A remains authoritative when an already admitted replacement B completes later; the deterministic component barrier proves that interleaving. The service cases do not claim to force it. Strict older resources and tighter quotas retain their reservation protocol. These are logical metadata-action counts, not routed wire totals or isolated end-to-end TPS measurements. + +The external GC case completes its convergence window in 161,787 ms at task revision 56 after one actual Access restart; the full client case takes 162.47 s. Both original file keys are absent, their candidates and task are Complete, and the fixed task identity/deadline survive restart. Two dead files contain 33,280 logical payload bytes. Three tree blocks and one external frame produce the exact durable retired range union `[0,16964)` and `[17510,34212)`, totaling 33,666 chunk bytes including frame/directory overhead. The interleaved live range `[16964,17510)` remains readable, layout identity is unchanged, and a stale marker cannot retire the newer live head. This establishes notification acceptance, without a physical free-space or GC latency SLO claim. + +The preceding v8 external case failed at its unchanged 480-second bound because `BudgetedGcBlocks` inherited the default deferred `reclaim_location` instead of dispatching to its configured provider. The wrapper now charges its existing request/byte budget and forwards the operation and result unchanged. The failed v8 evidence remains preserved independently. Final v9 source/image archives, raw logs, range validation and gate outputs are retained outside the repository in `/tmp/crowdb-access-cleanup-v9-evidence-20261025`. Dataset read and transport paths did not change in this batch; their completed v8 workload/HCA evidence below retains its own image and source identity. + +Source `b566fdcd` adds compact upload-resource storage, one-step ordinary inline purge completion and request-local Dataset delivery reductions. The upload/GC batch passes 1,709 tests across five affected crates, with 13 independently configured cases ignored. The subsequent Dataset/Access/native batch passes 669 tests, with the same 13 configured cases ignored. These suites overlap; their counts are not additive. Workspace all-target/all-feature clippy, normal lint, formatting and focused final regressions pass. The matching v7 image from `a78e981f` passes six S3, five Dataset and five GC cases, plus the complete TCP/installed-Python/restart workload. The v6 results below cover `c22bc40e`. + +Source `d400ae21` also discovers expired multipart uploads by immutable bucket identity after a bucket name is deleted and recreated. A production-actor regression fails on the preceding scanner and passes with the fix. The affected S3/Access/native suite passes 495 tests, with 12 separately configured cases ignored; the final multipart target passes all 64 cases. Only expired Open uploads are marked Aborted. Index, part and generation rows remain, and this does not establish physical space recovery. Workspace all-feature clippy, normal lint and formatting pass. Matching v7 service acceptance passes. + +The upload codec keeps all owner identities and the public JSON contract while writing a fixed 52-byte header plus 16 bytes per identity. Its maximum record is 69,684 bytes. Legacy JSON keeps its original validation, and CAS conditions use the actual stored bytes. JSON-only Access servers and direct native clients require a coordinated upgrade. No resource key or foreground RPC is added. The fixed legacy authority-read budget is retained. + +An exclusive CPU-2 release measurement uses seven samples of 128 iterations, with the same logical 4,095 retired and two live owners. Canned retirement includes GET cloning and exact CAS comparisons, without KV, WAL or network. Compact input takes a median 25.644 us after sorted insertion, compared with 29.395 us using append-and-sort; validated compact round trips take 7.589 us. Legacy JSON migration takes 654.204 us in the later run and 629.274 us in the preceding run, so it is not reported as an insertion speedup. Raw samples, timer-boundary measurements, source/binary hashes and scope are retained in `/tmp/crowdb-c12-owner-cpu-20261025` and `/tmp/crowdb-c12-owner-cpu-insert-20261025`. These are CPU observations, not TPS or total-system efficiency claims. + +Source `c22bc40e` additionally includes retained point-operation buffers across routed retries, observed-authority multipart expiry and reused validated inline digests. The six affected crates pass 1,109 tests, with 13 separately configured cases ignored. Workspace clippy, normal lint and formatting pass. Its matching `access-cleanup-v6-20261025` image passes six S3 protocol cases and all five Dataset cases (`run-8376c20aa6ea425583e2d8d6be33fcd9`), plus the complete [TCP/restart workload](#dataset-httpnative-comparison). The five-case GC scope also passes in `run-d56438cb607349f3a4d819402ee8d484`, including exact full inline-backlog convergence: 16 real-service cases in total. + +Boundary regressions preserve the caller's point payload allocations through first dispatch and routed retry. Expired multipart refresh uses one GET and one CAS instead of two GETs and one CAS; independent public abort still reads its own authority. Both native Dataset entry points verify inline corruption before reusing the immutable result. These are copy/scan and logical-operation results, not an isolated end-to-end throughput measurement of the changes. Raw v6 artifacts and immutable image identity are independently retained in `/tmp/crowdb-access-cleanup-v6-evidence-20261025`. + +The preceding source `ae4e1b8a` includes the bounded upload-retirement, Split-membership and Iceberg receiver optimizations. All-target/all-feature Dataset/Access Server/native coverage passes: 645 tests, 13 separately configured cases ignored. Workspace clippy, `rs-lint` and formatting pass. The matching `access-cleanup-v5-20261025` image passes 21 real-service cases: - All 16 Iceberg file cases: `run-f2d2bfc503214a77bae216d1cc8942f6`. - All five Dataset cases: `run-23ba53d4761b494781c86b799899ad17`. -The file scope includes the size matrix, 32 concurrent PUTs, 100-MiB PUT/MPU, -slow uploads, capacity failure/recovery, explicit cleanup and storage restart. -These functional checks do not measure receiver fast-path hit rate or copy CPU. -Executable cost regressions show ten shared upload references need one upload -record GET plus one CAS within a cleanup window, and twenty candidates in one -Split membership page need one bounded page read. Previously those paths used -ten and twenty reads respectively. These count logical metadata actions; -routed frame counts and end-to-end TPS savings are not inferred from them. - -Logs, image identity, summaries and raw evidence are independently backed up in -`/tmp/crowdb-access-cleanup-v5-evidence-20261025`. The -[current Dataset comparison](#dataset-httpnative-comparison) records the same -image's complete TCP workload and installed-binding restart acceptance. - -The preceding source `15efaaec` passes the broader affected -Dataset/Iceberg/S3/Access Server/native coverage across full runs and corrected -HTTP targets: -1,644 passed, 13 cases requiring a separate environment or measurement run -ignored. Workspace clippy, `rs-lint`, formatting, SDK dependency boundaries -and C++ tree lint pass. -The installed portable Python package passes 21 contract tests. - -That matching `access-cleanup-v4-20261025` image passes 18 real-service cases. -Artifact identifiers are relative to `.crowdb-runtime/artifacts/container-e2e/`: +The file scope includes the size matrix, 32 concurrent PUTs, 100-MiB PUT/MPU, slow uploads, capacity failure/recovery, explicit cleanup and storage restart. These functional checks do not measure receiver fast-path hit rate or copy CPU. Executable cost regressions show ten shared upload references need one upload record GET plus one CAS within a cleanup window, and twenty candidates in one Split membership page need one bounded page read. Previously those paths used ten and twenty reads respectively. These count logical metadata actions; routed frame counts and end-to-end TPS savings are not inferred from them. + +Logs, image identity, summaries and raw evidence are independently backed up in `/tmp/crowdb-access-cleanup-v5-evidence-20261025`. The [current Dataset comparison](#dataset-httpnative-comparison) records the same image's complete TCP workload and installed-binding restart acceptance. + +The preceding source `15efaaec` passes the broader affected Dataset/Iceberg/S3/Access Server/native coverage across full runs and corrected HTTP targets: 1,644 passed, 13 cases requiring a separate environment or measurement run ignored. Workspace clippy, `rs-lint`, formatting, SDK dependency boundaries and C++ tree lint pass. The installed portable Python package passes 21 contract tests. + +That matching `access-cleanup-v4-20261025` image passes 18 real-service cases. Artifact identifiers are relative to `.crowdb-runtime/artifacts/container-e2e/`: - Catalog creation/recovery: `run-3cbb504df0c6455e9613caf31be0ae1f`. - Namespace CRUD/restart: `run-c1b02d1e879e49a881fbccbfec31cb8d`. @@ -436,23 +144,13 @@ Artifact identifiers are relative to `.crowdb-runtime/artifacts/container-e2e/`: - Selected multipart publication: `run-1cd9d713bdea4fac8b6104bbd79a7f99`. - S3 metadata/copy/multipart: `run-51db5b3ff45340b18187286ca0a6cb65`. -Logs, manifests, image identity and trial summaries are also retained outside -the runtime/build directories in `/tmp/crowdb-access-cleanup-v4-evidence-20261025`. -Its complete TCP workload also passed in -`dataset-workload/50925d2ddaf04a5da8fb1687132c4228`; that earlier image's -measurements remain evidence for its own source. +Logs, manifests, image identity and trial summaries are also retained outside the runtime/build directories in `/tmp/crowdb-access-cleanup-v4-evidence-20261025`. Its complete TCP workload also passed in `dataset-workload/50925d2ddaf04a5da8fb1687132c4228`; that earlier image's measurements remain evidence for its own source. ## Earlier cleanup regression gates -At source `be5eca5c`, the affected Iceberg crate passes all 823 tests; Access -Server passes 201 tests with 11 existing ignored cases. Workspace all-target / -all-feature clippy, normal `rs-lint`, formatting, the 163 fixture unit tests and -container-client compilation pass. +At source `be5eca5c`, the affected Iceberg crate passes all 823 tests; Access Server passes 201 tests with 11 existing ignored cases. Workspace all-target / all-feature clippy, normal `rs-lint`, formatting, the 163 fixture unit tests and container-client compilation pass. -The matching `access-cleanup-v3-20261025` image passes 12 affected real-service -cases. GC evidence is below. Namespace CRUD and native-storage/listener restart -pass in `container-e2e/run-79033ed427994b0ca117463063119374`. -Seven independent `s3.protocol` cases pass with the original checks and timeouts: +The matching `access-cleanup-v3-20261025` image passes 12 affected real-service cases. GC evidence is below. Namespace CRUD and native-storage/listener restart pass in `container-e2e/run-79033ed427994b0ca117463063119374`. Seven independent `s3.protocol` cases pass with the original checks and timeouts: - AWS chunked trailers: `run-b59ea5a1911247dc802bbe593b2357df`. - Default boto3 checksums and multipart: `run-704e090a405f47b9835b83e98e96b75d`. @@ -462,57 +160,25 @@ Seven independent `s3.protocol` cases pass with the original checks and timeouts - Truncated signed upload and credit release: `run-b757efd59d334ed58237fd9ae846556c`. - Signed raw HTTP wire contract: `run-5d9fbbfe7b894eb685dd6c68ac037ed3`. -These are functional and request-cost regressions. They do not measure the hit -rate or end-to-end throughput benefit of forwarding complete S3 body owners. +These are functional and request-cost regressions. They do not measure the hit rate or end-to-end throughput benefit of forwarding complete S3 body owners. ## GC scheduling and foreground observations -The v7 five-case scope passes in -`container-e2e/run-35fb5dec04fa4fe69dc9200665a10519`. The unchanged 128-file, -1,554-logical-byte case completes in 2,396,811 ms, with final revision 792. -All original canonical keys are absent and all candidates/task are Complete; -no deferred range, retry, stall or fence remains. First deletion occurs at -2,002,959 ms, revision 663. The observer records 2,393 task GETs and 256 final -verification GETs, preserving all 792 observed revisions. - -The separate v6 run below completes at revision 920 in 2,785,403 ms. The current -inline path removes 128 checkpoint-only worker steps, consistent with the -128-revision difference. These independent same-host runs are not an isolated -timing comparison or a general GC capacity/SLO guarantee. - -Four SDK workers complete 48 table lifecycles while 64 native writes succeed; -two observed deletion intervals overlap the finite load. SDK create/commit/drop -p50/p99 are 89.42/338.77, 47.15/56.36 and 41.39/46.37 ms. Native write p50/p99 -are 0.72/84.88 ms. Full per-operation samples and exact image identity are in -`/tmp/crowdb-access-cleanup-v7-evidence-20261025`. This proves inline metadata -convergence and bounded overlap, without external range/tree notification or -physical capacity recovery evidence. - -The preceding v6 five-case scope passes in -`container-e2e/run-d56438cb607349f3a4d819402ee8d484`. Its separate expired-inline -case starts with 128 canonical files (1,554 logical bytes), a genuinely elapsed -fixed deletion deadline, one-item pages and the production 30-minute idle -interval. Other window, concurrency and I/O budgets remain unchanged. +The v7 five-case scope passes in `container-e2e/run-35fb5dec04fa4fe69dc9200665a10519`. The unchanged 128-file, 1,554-logical-byte case completes in 2,396,811 ms, with final revision 792. All original canonical keys are absent and all candidates/task are Complete; no deferred range, retry, stall or fence remains. First deletion occurs at 2,002,959 ms, revision 663. The observer records 2,393 task GETs and 256 final verification GETs, preserving all 792 observed revisions. + +The separate v6 run below completes at revision 920 in 2,785,403 ms. The current inline path removes 128 checkpoint-only worker steps, consistent with the 128-revision difference. These independent same-host runs are not an isolated timing comparison or a general GC capacity/SLO guarantee. + +Four SDK workers complete 48 table lifecycles while 64 native writes succeed; two observed deletion intervals overlap the finite load. SDK create/commit/drop p50/p99 are 89.42/338.77, 47.15/56.36 and 41.39/46.37 ms. Native write p50/p99 are 0.72/84.88 ms. Full per-operation samples and exact image identity are in `/tmp/crowdb-access-cleanup-v7-evidence-20261025`. This proves inline metadata convergence and bounded overlap, without external range/tree notification or physical capacity recovery evidence. + +The preceding v6 five-case scope passes in `container-e2e/run-d56438cb607349f3a4d819402ee8d484`. Its separate expired-inline case starts with 128 canonical files (1,554 logical bytes), a genuinely elapsed fixed deletion deadline, one-item pages and the production 30-minute idle interval. Other window, concurrency and I/O budgets remain unchanged. - First actual deletion: 2,005,848 ms, task revision 664. -- Complete verification: 2,785,403 ms, final task revision 920, deleted 128 and - logical bytes 1,554. All 128 original file keys are absent and their exact - candidates are Complete; no deferred ranges, retry, stalled state or fence - remains. This is logical inline metadata retirement, not chunk notification - or physical capacity release. -- The observer performs 2,781 task GETs and 256 separate final verification - GETs. All 919 observed revisions and phase transitions remain in the raw log. -- After first deletion, four official SDK workers each process twelve tables - while the native writer completes 64 writes. Both loads remain active across - one observed interval with increasing deletion count, and a successful SDK - operation finishes inside it. This is finite overlap, not sustained traffic - throughout the 46-minute cleanup or an isolated GC performance baseline. -- One supplemental admin metrics request is recorded separately. Those counters - belong to the foreground store; GC's independent store is not included. - -The new case retains raw latencies and nearest-rank quantiles. SDK table actions -have 48 samples; namespace actions have four. Intentional 500-ms inter-table -and 250-ms inter-write delays are excluded from the operation timings. +- Complete verification: 2,785,403 ms, final task revision 920, deleted 128 and logical bytes 1,554. All 128 original file keys are absent and their exact candidates are Complete; no deferred ranges, retry, stalled state or fence remains. This is logical inline metadata retirement, not chunk notification or physical capacity release. +- The observer performs 2,781 task GETs and 256 separate final verification GETs. All 919 observed revisions and phase transitions remain in the raw log. +- After first deletion, four official SDK workers each process twelve tables while the native writer completes 64 writes. Both loads remain active across one observed interval with increasing deletion count, and a successful SDK operation finishes inside it. This is finite overlap, not sustained traffic throughout the 46-minute cleanup or an isolated GC performance baseline. +- One supplemental admin metrics request is recorded separately. Those counters belong to the foreground store; GC's independent store is not included. + +The new case retains raw latencies and nearest-rank quantiles. SDK table actions have 48 samples; namespace actions have four. Intentional 500-ms inter-table and 250-ms inter-write delays are excluded from the operation timings. | Operation | Samples | p50 (ms) | p99 (ms) | | ------------------ | ------- | -------- | -------- | @@ -526,23 +192,11 @@ and 250-ms inter-write delays are excluded from the operation timings. | Drop namespace | 4 | 245.05 | 253.12 | | Native chunk write | 64 | 0.71 | 84.79 | -Raw output, exact candidate checks, summaries and image identity are retained -in `/tmp/crowdb-access-cleanup-v6-evidence-20261025`. The original foreground -case remains independently unchanged. Its earlier v3 observations follow. +Raw output, exact candidate checks, summaries and image identity are retained in `/tmp/crowdb-access-cleanup-v6-evidence-20261025`. The original foreground case remains independently unchanged. Its earlier v3 observations follow. -The four GC container cases pass on `crowdb-node:access-cleanup-v3-20261025` -in `container-e2e/run-15dee04958f34b6f9ad7207fcff1d679`. The imported Docker ID is -`sha256:6636ec6d437cc3bbe32d4c0d355cd1b0a778fbc156797fb39f00ee8eed3cbe0d`, -source revision `be5eca5c52e8f64bdbc4c5c2e96fc6091af6729e`, source tree -`c4e3b70c54ba6c4b4aef2d54b0af21cf98b3f473`. +The four GC container cases pass on `crowdb-node:access-cleanup-v3-20261025` in `container-e2e/run-15dee04958f34b6f9ad7207fcff1d679`. The imported Docker ID is `sha256:6636ec6d437cc3bbe32d4c0d355cd1b0a778fbc156797fb39f00ee8eed3cbe0d`, source revision `be5eca5c52e8f64bdbc4c5c2e96fc6091af6729e`, source tree `c4e3b70c54ba6c4b4aef2d54b0af21cf98b3f473`. -The foreground case uses the production default 30-minute idle interval, -one-item discovery pages, 128 seeded files and the original per-window budgets. -Four PyIceberg workers each create, commit, read and drop three tables while a -native chunk writer completes 64 writes of 128 bytes. All operations succeed. -The task is observed at revisions 1 and 2 at 0 and 1,018 ms; the continuation -check observes revision 3 at 4,077 ms. It remains in `Discover`: this proves -continued advancement, not complete backlog retirement or a sustained cleanup rate. +The foreground case uses the production default 30-minute idle interval, one-item discovery pages, 128 seeded files and the original per-window budgets. Four PyIceberg workers each create, commit, read and drop three tables while a native chunk writer completes 64 writes of 128 bytes. All operations succeed. The task is observed at revisions 1 and 2 at 0 and 1,018 ms; the continuation check observes revision 3 at 4,077 ms. It remains in `Discover`: this proves continued advancement, not complete backlog retirement or a sustained cleanup rate. The test prints every successful operation's latency and nearest-rank quantiles: @@ -558,13 +212,7 @@ The test prints every successful operation's latency and nearest-rank quantiles: | Drop namespace | 4 | 243.95 | 244.53 | | Native chunk write | 64 | 0.54 | 85.17 | -Chunk timing includes writer preparation, payload delivery and finish; it excludes -the intentional 50-ms sleeps and initial client connection. It does not measure -an HTTP object PUT or object metadata publication. Its first sample is 85.17 ms -and remains included. The observer adds one task GET per 100 ms during overlap -and continuation checks. These same-host, unpinned, small-sample observations have -no isolated baseline; their p99 values are near maxima, not production latency -guarantees or evidence that GC adds no foreground cost. +Chunk timing includes writer preparation, payload delivery and finish; it excludes the intentional 50-ms sleeps and initial client connection. It does not measure an HTTP object PUT or object metadata publication. Its first sample is 85.17 ms and remains included. The observer adds one task GET per 100 ms during overlap and continuation checks. These same-host, unpinned, small-sample observations have no isolated baseline; their p99 values are near maxima, not production latency guarantees or evidence that GC adds no foreground cost. ```sh pixi run clean-env && pixi run python e2e-test/run.py --layer iceberg --scope iceberg.gc --image crowdb-node:access-cleanup-v3-20261025 --concurrency 1 --cpus 4 --memory-mib 12288 @@ -572,39 +220,11 @@ pixi run clean-env && pixi run python e2e-test/run.py --layer iceberg --scope ic ## Dataset HTTP/native comparison -The workload runner now records finite observations around each original trial. -The child captures local cached status, announces its exact run/path/concurrency/ -repeat/Snapshot identity, and waits while the parent reads owned service cgroups -and process counters. A second handshake captures the end before the independent -digest kernel or next trial. The original mixed sizes, concurrency, warmups, -three repeats, 4,608 requests, verification and 600-second child bound remain. -Direct child execution without the observation environment variable stays -standalone. The tool passes all 179 fixture unit checks, including 16 -protocol/resource cases, and all 18 matching-image trial observations complete. - -Each path writes its raw log and `*-observations.json`. Service CPU deltas include -background work and the explicitly timed sequential capture interval. Actual -member-process RSS, cumulative process HWM, charged cgroup memory and cumulative -cgroup peaks are retained separately; none is a per-trial peak. Local native -metadata, aggregate and data transport groups retain selected sent-frame/byte -deltas without adding repeated process RUSAGE counters. Missing, reset or changed -identities remain unavailable. In this initial observation revision, receive -traffic, all-copy bytes, layout queries and per-request header/first-byte/EOF -timing remain unmeasured; the [expanded observations](#expanded-cleanup-and-read-observations) -later add layout counters and client-visible delivery phases. The enclosing -service and inner client windows are not additive total-system CPU measurements. - -The v8 complete TCP/installed-Python/storage-restart acceptance passes in -`dataset-workload/c660f7ea934540a3bcb527482ecc0da1`: 18 trials, 4,608 requests, -zero errors and 18 completed observations. The five Dataset service cases pass -in `container-e2e/run-7d55f126f179499fa00b288de78aae5f`. Image -`crowdb-node:access-cleanup-v8-20261025` has source revision -`0174f6b98b8cb5051cdeaff1c7436d9683806e28`, tree -`394adb644056e73cf480a19005807796198a92fa` and Docker ID -`sha256:daaf9de782e91b232ebb288ccaf479c4646c2c7529c9110d115ba90bbfca61e1`. -The original eight sizes, CPU placement, concurrency, repeats, budgets and -verification remain unchanged. Raw artifacts, validated summaries and hashes -are retained in `/tmp/crowdb-access-cleanup-v8-evidence-20261025`. +The workload runner now records finite observations around each original trial. The child captures local cached status, announces its exact run/path/concurrency/repeat/Snapshot identity, and waits while the parent reads owned service cgroups and process counters. A second handshake captures the end before the independent digest kernel or next trial. The original mixed sizes, concurrency, warmups, three repeats, 4,608 requests, verification and 600-second child bound remain. Direct child execution without the observation environment variable stays standalone. The tool passes all 179 fixture unit checks, including 16 protocol/resource cases, and all 18 matching-image trial observations complete. + +Each path writes its raw log and `*-observations.json`. Service CPU deltas include background work and the explicitly timed sequential capture interval. Actual member-process RSS, cumulative process HWM, charged cgroup memory and cumulative cgroup peaks are retained separately; none is a per-trial peak. Local native metadata, aggregate and data transport groups retain selected sent-frame/byte deltas without adding repeated process RUSAGE counters. Missing, reset or changed identities remain unavailable. In this initial observation revision, receive traffic, all-copy bytes, layout queries and per-request header/first-byte/EOF timing remain unmeasured; the [expanded observations](#expanded-cleanup-and-read-observations) later add layout counters and client-visible delivery phases. The enclosing service and inner client windows are not additive total-system CPU measurements. + +The v8 complete TCP/installed-Python/storage-restart acceptance passes in `dataset-workload/c660f7ea934540a3bcb527482ecc0da1`: 18 trials, 4,608 requests, zero errors and 18 completed observations. The five Dataset service cases pass in `container-e2e/run-7d55f126f179499fa00b288de78aae5f`. Image `crowdb-node:access-cleanup-v8-20261025` has source revision `0174f6b98b8cb5051cdeaff1c7436d9683806e28`, tree `394adb644056e73cf480a19005807796198a92fa` and Docker ID `sha256:daaf9de782e91b232ebb288ccaf479c4646c2c7529c9110d115ba90bbfca61e1`. The original eight sizes, CPU placement, concurrency, repeats, budgets and verification remain unchanged. Raw artifacts, validated summaries and hashes are retained in `/tmp/crowdb-access-cleanup-v8-evidence-20261025`. | Path | Concurrency | MiB/s | Inner client CPU (s) | Lifetime client RSS (MiB) | | ------------- | ----------- | ------ | -------------------- | ------------------------- | @@ -615,33 +235,11 @@ are retained in `/tmp/crowdb-access-cleanup-v8-evidence-20261025`. | Native Rust | 3 | 270.53 | 2.75 | 30.66 | | Native Python | 3 | 264.55 | 2.88 | 61.70 | -Native Rust throughput is about 21%/7% above HTTP in this run, with higher -client CPU and RSS. For 128 bytes, HTTP/native Rust median-trial p50/p99 are -563/1,071 versus 623/710 microseconds at concurrency one, and 818/15,919 versus -1,080/7,105 at three. At 1 MiB and concurrency one, they are 6,561/9,286 versus -3,503/3,870 microseconds. Full per-size quantiles remain in the summary and raw -trials; each trial has only 16/48 observations per size. The separate resident -64-MiB MD5 kernel takes about 86 ms in Rust and 63–67 ms in Python. Normal reads -retain all integrity work and Python byte copies. - -All twelve native trials confirm one fewer bounded metadata request per read: -aggregate sent frames are 896 per 128-request trial, versus v7's 1,024, and -2,688 at concurrency three. Sent aggregate bytes are 273,664/820,992. -Metadata and data groups remain 192/288 frames and 39,040/21,952 sent bytes -at concurrency one, with exact threefold counts at three. The strict scan -preserves a validated terminal owner page, removing its empty confirmation -request. These selected sent counters prove the request reduction; independent -v7/v8 throughput or CPU differences do not isolate its timing benefit. The -observation-window, cumulative-memory and unmeasured-counter scopes above apply. - -The preceding v7 complete TCP/installed-Python/storage-restart acceptance passes in -`dataset-workload/419095e911824fdc9aefe50bbc5bacc9`: 18 trials, 4,608 requests, -zero errors. Source revision is `a78e981fdb8f3f8251aa3bc99e7beb88aa078d80`, tree -`32ac07470397bb295e58d58abf79d72f3dbb15a1`, imported Docker ID -`sha256:979f2646dc189f3c0cb6e8a936faf2e9380716b404372c680b87d40863f060a2`. -The same eight sizes, CPU placement, concurrency, warmups and budgets as the -preceding v6 run below are retained. Raw records, validated summaries and hashes -are backed up in `/tmp/crowdb-access-cleanup-v7-evidence-20261025`. +Native Rust throughput is about 21%/7% above HTTP in this run, with higher client CPU and RSS. For 128 bytes, HTTP/native Rust median-trial p50/p99 are 563/1,071 versus 623/710 microseconds at concurrency one, and 818/15,919 versus 1,080/7,105 at three. At 1 MiB and concurrency one, they are 6,561/9,286 versus 3,503/3,870 microseconds. Full per-size quantiles remain in the summary and raw trials; each trial has only 16/48 observations per size. The separate resident 64-MiB MD5 kernel takes about 86 ms in Rust and 63–67 ms in Python. Normal reads retain all integrity work and Python byte copies. + +All twelve native trials confirm one fewer bounded metadata request per read: aggregate sent frames are 896 per 128-request trial, versus v7's 1,024, and 2,688 at concurrency three. Sent aggregate bytes are 273,664/820,992. Metadata and data groups remain 192/288 frames and 39,040/21,952 sent bytes at concurrency one, with exact threefold counts at three. The strict scan preserves a validated terminal owner page, removing its empty confirmation request. These selected sent counters prove the request reduction; independent v7/v8 throughput or CPU differences do not isolate its timing benefit. The observation-window, cumulative-memory and unmeasured-counter scopes above apply. + +The preceding v7 complete TCP/installed-Python/storage-restart acceptance passes in `dataset-workload/419095e911824fdc9aefe50bbc5bacc9`: 18 trials, 4,608 requests, zero errors. Source revision is `a78e981fdb8f3f8251aa3bc99e7beb88aa078d80`, tree `32ac07470397bb295e58d58abf79d72f3dbb15a1`, imported Docker ID `sha256:979f2646dc189f3c0cb6e8a936faf2e9380716b404372c680b87d40863f060a2`. The same eight sizes, CPU placement, concurrency, warmups and budgets as the preceding v6 run below are retained. Raw records, validated summaries and hashes are backed up in `/tmp/crowdb-access-cleanup-v7-evidence-20261025`. | Path | Concurrency | MiB/s | Inner client CPU (s) | Lifetime client RSS (MiB) | | ------------- | ----------- | ------ | -------------------- | ------------------------- | @@ -652,42 +250,16 @@ are backed up in `/tmp/crowdb-access-cleanup-v7-evidence-20261025`. | Native Rust | 3 | 280.41 | 2.68 | 30.56 | | Native Python | 3 | 275.40 | 2.76 | 60.90 | -Service capture spans are 5.80–10.18 ms. For the three concurrency-one trials, -HTTP/native-Rust enclosing Access CPU is 0.809512/0.002916 s; metadata/control -CPU is 0.286016/0.336947 s. These are separate role windows including background -work and capture slack, not exact total-system CPU. The native metadata, -aggregate and data groups send 192/1,024/288 frames per 128-request trial. -Aggregate here names the bounded metadata transport, not business-message -aggregation. These groups omit control/layout RPCs and expose outgoing request -bytes rather than payload receive bytes. The observed compatibility fallback -copy counter stays zero; Python byte copies remain included in client work. -Differences from v6 are independent observations, not isolated patch effects. - -The preceding v6 complete same-host TCP workload and installed Python/storage-restart -acceptance pass in `dataset-workload/d1fa8f646f124426b6c9e24688d92786` under -`.crowdb-runtime/artifacts/`. The manifest records image labels, CPU placement -and per-phase service counters; client logs retain all per-size p50/p99 and -digest timings. Raw trials and the aggregate summary are independently retained -in `/tmp/crowdb-access-cleanup-v6-evidence-20261025`. - -- Image: `crowdb-node:access-cleanup-v6-20261025`, imported Docker ID - `sha256:cc1f879276f039a13a0ed0b7232833d176a1659fc10bf508700a43ef980b4e2f`. -- Source revision: `c22bc40ebb6663b4b3cc4f0a89075a41f1c9f5a6`; source tree: - `e2d0f660762290085649081fd2b437b53cd343da`. -- Client CPU 2; service CPUs 4–15; each metadata replica capped at 2 GiB. - Metadata cgroup peaks were 204.61/204.60/204.68 MiB, without a stopped replica. -- Sizes: 0, 128, 513, 4,096, 65,535, 262,144, 1,048,576 and 4,194,321 bytes. - Two warmup rounds precede three trials at each concurrency. Each worker - performs 128 requests per trial: 4,608 measured requests, zero errors. - Every path verifies exact bytes and integrity; Python byte-copy costs remain - included. No workload budget, deadline or retry bound was increased. - -Throughput is total bytes divided by total elapsed time across three trials. -Client CPU is the three-trial sum; RSS is the largest observed client value. -HTTP runs authority and payload processing in Access Server, while native -runs that work in the client. Client CPU alone does not measure total system -efficiency. HTTP v1 uses Manifest projection and native uses inspection -metadata; the product comparison includes this implementation difference. +Service capture spans are 5.80–10.18 ms. For the three concurrency-one trials, HTTP/native-Rust enclosing Access CPU is 0.809512/0.002916 s; metadata/control CPU is 0.286016/0.336947 s. These are separate role windows including background work and capture slack, not exact total-system CPU. The native metadata, aggregate and data groups send 192/1,024/288 frames per 128-request trial. Aggregate here names the bounded metadata transport, not business-message aggregation. These groups omit control/layout RPCs and expose outgoing request bytes rather than payload receive bytes. The observed compatibility fallback copy counter stays zero; Python byte copies remain included in client work. Differences from v6 are independent observations, not isolated patch effects. + +The preceding v6 complete same-host TCP workload and installed Python/storage-restart acceptance pass in `dataset-workload/d1fa8f646f124426b6c9e24688d92786` under `.crowdb-runtime/artifacts/`. The manifest records image labels, CPU placement and per-phase service counters; client logs retain all per-size p50/p99 and digest timings. Raw trials and the aggregate summary are independently retained in `/tmp/crowdb-access-cleanup-v6-evidence-20261025`. + +- Image: `crowdb-node:access-cleanup-v6-20261025`, imported Docker ID `sha256:cc1f879276f039a13a0ed0b7232833d176a1659fc10bf508700a43ef980b4e2f`. +- Source revision: `c22bc40ebb6663b4b3cc4f0a89075a41f1c9f5a6`; source tree: `e2d0f660762290085649081fd2b437b53cd343da`. +- Client CPU 2; service CPUs 4–15; each metadata replica capped at 2 GiB. Metadata cgroup peaks were 204.61/204.60/204.68 MiB, without a stopped replica. +- Sizes: 0, 128, 513, 4,096, 65,535, 262,144, 1,048,576 and 4,194,321 bytes. Two warmup rounds precede three trials at each concurrency. Each worker performs 128 requests per trial: 4,608 measured requests, zero errors. Every path verifies exact bytes and integrity; Python byte-copy costs remain included. No workload budget, deadline or retry bound was increased. + +Throughput is total bytes divided by total elapsed time across three trials. Client CPU is the three-trial sum; RSS is the largest observed client value. HTTP runs authority and payload processing in Access Server, while native runs that work in the client. Client CPU alone does not measure total system efficiency. HTTP v1 uses Manifest projection and native uses inspection metadata; the product comparison includes this implementation difference. | Path | Concurrency | MiB/s | Client CPU (s) | Peak RSS (MiB) | | ------------- | ----------- | ------ | -------------- | -------------- | @@ -698,28 +270,11 @@ metadata; the product comparison includes this implementation difference. | Native Rust | 3 | 268.17 | 2.77 | 31.21 | | Native Python | 3 | 266.02 | 2.86 | 61.30 | -Native Rust throughput is about 19% higher at concurrency one and 5% higher at -concurrency three in this execution. Client CPU and memory increase. Individual -sizes also differ: at concurrency one, the median trial p50 for 128 bytes is -557 µs over HTTP and 700 µs in native Rust; for 1 MiB it is 6,788 and 3,681 µs -respectively. At concurrency three, the 128-byte p50 is 798 versus 3,854 µs, -while its p99 is 9,548 versus 7,268 µs. These are medians of three trial -quantiles, not pooled percentiles. Only 16/48 observations per size enter each trial, so these tails -do not establish a production latency guarantee. -The separately timed resident-buffer 64-MiB MD5 kernel takes approximately -85 ms in Rust and 63 ms in Python; normal reads still perform integrity checks. - -The shared fixture's service `memory.peak` is cumulative, and HTTP service CPU -also includes initial publication. Those counters are not isolated per-path -server peaks or a fair read-only server-CPU comparison. This is a pinned -same-host functional workload, not cross-host RDMA performance evidence. - -The semantic phase compares HTTP/native ReadSpec and Split batch headers, -sample IDs and bytes; replays the original cursor on a second runtime and after -storage restart; and checks publication conflicts, multipart ownership, a -65-second idle, cancellation, caller-held buffers and an explicit complete-Split -deletion whose fixed deadline survives retry and storage restart. The native -runtime never uses HTTP as an implicit fallback. +Native Rust throughput is about 19% higher at concurrency one and 5% higher at concurrency three in this execution. Client CPU and memory increase. Individual sizes also differ: at concurrency one, the median trial p50 for 128 bytes is 557 µs over HTTP and 700 µs in native Rust; for 1 MiB it is 6,788 and 3,681 µs respectively. At concurrency three, the 128-byte p50 is 798 versus 3,854 µs, while its p99 is 9,548 versus 7,268 µs. These are medians of three trial quantiles, not pooled percentiles. Only 16/48 observations per size enter each trial, so these tails do not establish a production latency guarantee. The separately timed resident-buffer 64-MiB MD5 kernel takes approximately 85 ms in Rust and 63 ms in Python; normal reads still perform integrity checks. + +The shared fixture's service `memory.peak` is cumulative, and HTTP service CPU also includes initial publication. Those counters are not isolated per-path server peaks or a fair read-only server-CPU comparison. This is a pinned same-host functional workload, not cross-host RDMA performance evidence. + +The semantic phase compares HTTP/native ReadSpec and Split batch headers, sample IDs and bytes; replays the original cursor on a second runtime and after storage restart; and checks publication conflicts, multipart ownership, a 65-second idle, cancellation, caller-held buffers and an explicit complete-Split deletion whose fixed deadline survives retry and storage restart. The native runtime never uses HTTP as an implicit fallback. Reproduce after building/importing the matching service image: @@ -727,39 +282,9 @@ Reproduce after building/importing the matching service image: pixi run clean-env && pixi run python tools/sdk/dataset_workload.py --image crowdb-node:access-cleanup-v6-20261025 ``` -The strict same-host HCA profile passes on image -`crowdb-node:access-cleanup-v8-20261025` in -`native-dataset-hca/b8c70fd1b93443c8918e26122b168eb9`. Its immutable source is -`0174f6b98b8cb5051cdeaff1c7436d9683806e28`, tree -`394adb644056e73cf480a19005807796198a92fa`; imported Docker ID is -`sha256:daaf9de782e91b232ebb288ccaf479c4646c2c7529c9110d115ba90bbfca61e1`. -The installed native Python consumer publishes through the shared runtime, -then verifies after all storage services restart. Each phase establishes twelve -data connections with TCP fallback disabled. Write records 48 sent / 60 received -RPC frames; verify records 18 / 30. Fallback, protocol, CQ, send and receive-post -error counters remain zero. Registered memory is 436,404,224 bytes within the -512-MiB budget. A registered receive lease remains valid through runtime shutdown, -and its final release returns held credits. Raw observations and image/source -identity are retained in `/tmp/crowdb-access-cleanup-v8-evidence-20261025`. -This is local-HCA functional and ownership evidence, not a throughput or -cross-host measurement; its counters have a separate scope from the TCP workload. -The separately activated host test -`hardware_accepted_dataset_read_failure_is_sticky_without_replay` also passes -with `CROWDB_RDMA_E2E=1`, the same address and device. It verifies an accepted -second read interrupted after a registered first buffer remains terminal, -without reconnect, retry, metadata re-selection or loss of held ownership. -Its complete output is retained alongside the HCA evidence. - -The preceding v5 run was -`dataset-workload/def69a711d4e468880c5cb9fae82e32e`; its native Rust throughput -was about 36%/5% above HTTP at concurrency one/three. The preceding v4 run was -`dataset-workload/50925d2ddaf04a5da8fb1687132c4228`; its image and raw evidence -remain preserved separately. Earlier accepted runs were -`dataset-workload/9dbf56c386e142e995b8eb400305b467` -and `dataset-workload/abea0a0568824233a32ba85a063e522a`; their original runtime -directories were removed as described above. -Differences between independent runs -are not an isolated before/after measurement of any one code change. +The strict same-host HCA profile passes on image `crowdb-node:access-cleanup-v8-20261025` in `native-dataset-hca/b8c70fd1b93443c8918e26122b168eb9`. Its immutable source is `0174f6b98b8cb5051cdeaff1c7436d9683806e28`, tree `394adb644056e73cf480a19005807796198a92fa`; imported Docker ID is `sha256:daaf9de782e91b232ebb288ccaf479c4646c2c7529c9110d115ba90bbfca61e1`. The installed native Python consumer publishes through the shared runtime, then verifies after all storage services restart. Each phase establishes twelve data connections with TCP fallback disabled. Write records 48 sent / 60 received RPC frames; verify records 18 / 30. Fallback, protocol, CQ, send and receive-post error counters remain zero. Registered memory is 436,404,224 bytes within the 512-MiB budget. A registered receive lease remains valid through runtime shutdown, and its final release returns held credits. Raw observations and image/source identity are retained in `/tmp/crowdb-access-cleanup-v8-evidence-20261025`. This is local-HCA functional and ownership evidence, not a throughput or cross-host measurement; its counters have a separate scope from the TCP workload. The separately activated host test `hardware_accepted_dataset_read_failure_is_sticky_without_replay` also passes with `CROWDB_RDMA_E2E=1`, the same address and device. It verifies an accepted second read interrupted after a registered first buffer remains terminal, without reconnect, retry, metadata re-selection or loss of held ownership. Its complete output is retained alongside the HCA evidence. + +The preceding v5 run was `dataset-workload/def69a711d4e468880c5cb9fae82e32e`; its native Rust throughput was about 36%/5% above HTTP at concurrency one/three. The preceding v4 run was `dataset-workload/50925d2ddaf04a5da8fb1687132c4228`; its image and raw evidence remain preserved separately. Earlier accepted runs were `dataset-workload/9dbf56c386e142e995b8eb400305b467` and `dataset-workload/abea0a0568824233a32ba85a063e522a`; their original runtime directories were removed as described above. Differences between independent runs are not an isolated before/after measurement of any one code change. ## Bounded immutable write comparison @@ -769,32 +294,11 @@ Build/import the matching service image, then run: pixi run clean-env && pixi run python e2e-test/write_batch_comparison.py --image crowdb-node:access-cleanup-20261025 ``` -The owned persistent Chunk-KV fixture compares release point writes and routed -batch writes on client CPU 2 and service CPUs 4–15. Both paths admit at most -16 rows / 2 MiB of logical key/value input, plus one prepared row. Three warmups -precede three paired, alternating trials for small schema/membership/field rows -and 64-KiB inline payloads with their actual JSON encoding. The test records typed -results, routes, owner RPC attempts, completed TCP frames and protocol bytes, -client CPU ticks and RSS. Artifacts include the image/source identity, client -binary/source hashes, original output and `comparison.json`. -Cold warmups account for one capability-handshake frame per new pooled TCP -connection. Measured trials require no new connections and exact equality -between owner RPC attempts and completed protocol frames. - -This comparison does not publish a Dataset head. Logical admission bytes exclude -transport copies; TCP protocol bytes exclude IP/retransmissions. Allocation and -copied-byte counts are unmeasured. Process memory peaks are cumulative, and -short trials can report zero CPU ticks without having zero CPU cost. Report -the measured routing scope rather than assuming multiple partitions or owners. - -The comparison passes in -`write-batch-comparison/acd38136ec0943aeac3aaa9d849c0a5c` with -`crowdb-node:access-cleanup-20261025` from source `7b2c61aa`, tree `aecf740c`. -Its manifest retains the exact imported image and client binary/source hashes. -All warmup and measured creates return typed Applied results. The actual route -is one partition, one owner and catalog generation 1; it is not a multi-partition -measurement. Each measured trial keeps both connections open and has no setup -frames or failed owner attempts. +The owned persistent Chunk-KV fixture compares release point writes and routed batch writes on client CPU 2 and service CPUs 4–15. Both paths admit at most 16 rows / 2 MiB of logical key/value input, plus one prepared row. Three warmups precede three paired, alternating trials for small schema/membership/field rows and 64-KiB inline payloads with their actual JSON encoding. The test records typed results, routes, owner RPC attempts, completed TCP frames and protocol bytes, client CPU ticks and RSS. Artifacts include the image/source identity, client binary/source hashes, original output and `comparison.json`. Cold warmups account for one capability-handshake frame per new pooled TCP connection. Measured trials require no new connections and exact equality between owner RPC attempts and completed protocol frames. + +This comparison does not publish a Dataset head. Logical admission bytes exclude transport copies; TCP protocol bytes exclude IP/retransmissions. Allocation and copied-byte counts are unmeasured. Process memory peaks are cumulative, and short trials can report zero CPU ticks without having zero CPU cost. Report the measured routing scope rather than assuming multiple partitions or owners. + +The comparison passes in `write-batch-comparison/acd38136ec0943aeac3aaa9d849c0a5c` with `crowdb-node:access-cleanup-20261025` from source `7b2c61aa`, tree `aecf740c`. Its manifest retains the exact imported image and client binary/source hashes. All warmup and measured creates return typed Applied results. The actual route is one partition, one owner and catalog generation 1; it is not a multi-partition measurement. Each measured trial keeps both connections open and has no setup frames or failed owner attempts. | Rows | Path | Min / median / max (ms) | Owner RPCs | Sent protocol bytes | | ---------------------- | -------------- | ------------------------ | ---------- | ------------------- | @@ -803,44 +307,17 @@ frames or failed owner attempts. | 16 inline 64-KiB rows | Sliding points | 286.7 / 318.0 / 320.5 | 16 | 4,199,136 | | 16 inline 64-KiB rows | Routed batch | 545.6 / 550.9 / 561.5 | 3 | 12,593,170 | -Keep the existing bounded sliding point writes for inspection preparation: -the current batch reduces RPC count but is slower in both profiles. Inline -values already encode bytes as JSON numbers in the Dataset row; the group -envelope JSON-encodes that row again, increasing protocol bytes to about three -times the point path. Both large-value paths actually admit seven rows at a -time because the logical byte bound wins before the row-count bound. - -Across the three large-value trials, client CPU is 4 ticks for points and -21 ticks for batch at 100 Hz. Small-row CPU totals are only 3 and 2 ticks and -cannot establish a useful CPU advantage. The lifetime HWM reaches 60,116 KiB -during warmup and is shared by subsequent paths; it is not a per-path memory -comparison. Automatic catalog source refresh occurs twice in measured small -batches and once in measured large points; those calls remain included in -elapsed time and separately reported from owner RPCs. - -The service processes each batch's items sequentially, while the point window -can overlap row completion. This and encoding/copying are concrete differences; -their individual contributions were not profiled. Alternating trials do not -reset accumulated persistent state, and three rounds give no stable latency -SLO. Reconsider grouped writes after improving the wire/processing path and -repeating this comparison, without relaxing ordering or publication barriers. +Keep the existing bounded sliding point writes for inspection preparation: the current batch reduces RPC count but is slower in both profiles. Inline values already encode bytes as JSON numbers in the Dataset row; the group envelope JSON-encodes that row again, increasing protocol bytes to about three times the point path. Both large-value paths actually admit seven rows at a time because the logical byte bound wins before the row-count bound. + +Across the three large-value trials, client CPU is 4 ticks for points and 21 ticks for batch at 100 Hz. Small-row CPU totals are only 3 and 2 ticks and cannot establish a useful CPU advantage. The lifetime HWM reaches 60,116 KiB during warmup and is shared by subsequent paths; it is not a per-path memory comparison. Automatic catalog source refresh occurs twice in measured small batches and once in measured large points; those calls remain included in elapsed time and separately reported from owner RPCs. + +The service processes each batch's items sequentially, while the point window can overlap row completion. This and encoding/copying are concrete differences; their individual contributions were not profiled. Alternating trials do not reset accumulated persistent state, and three rounds give no stable latency SLO. Reconsider grouped writes after improving the wire/processing path and repeating this comparison, without relaxing ordering or publication barriers. ## Shared upload owner CPU -The release CPU decomposition passes on client CPU 2 in -`upload-owner-cpu/10866bc17bf14e53877a2d3be19c390e`. Its manifest retains binary -and source hashes; `measurement.log` and `measurements.json` retain every sample. -The fixture has two live owners and 1/64/256/4,095 retired owners. A retirement -removes one live owner and adds its permanent negative fence. The last case -reaches the existing 4,096-entry bound; it does not change the saturated-resource -format or late-claim rules. - -Each mode has eight warmups and seven fixed 128-iteration samples. Fixture -setup is separate. Values below are medians of calling-thread CPU per operation, -with Linux scheduler accounting refreshed at both boundaries by a safe yield. -The empty boundary costs approximately 6.6 µs per sample and is reported without -subtraction. Wall-clock samples are retained separately. The initial unrefreshed -counter was tick-quantized and is not used for these results. +The release CPU decomposition passes on client CPU 2 in `upload-owner-cpu/10866bc17bf14e53877a2d3be19c390e`. Its manifest retains binary and source hashes; `measurement.log` and `measurements.json` retain every sample. The fixture has two live owners and 1/64/256/4,095 retired owners. A retirement removes one live owner and adds its permanent negative fence. The last case reaches the existing 4,096-entry bound; it does not change the saturated-resource format or late-claim rules. + +Each mode has eight warmups and seven fixed 128-iteration samples. Fixture setup is separate. Values below are medians of calling-thread CPU per operation, with Linux scheduler accounting refreshed at both boundaries by a safe yield. The empty boundary costs approximately 6.6 µs per sample and is reported without subtraction. Wall-clock samples are retained separately. The initial unrefreshed counter was tick-quantized and is not used for these results. | Retired owners | Input record (bytes) | Serde roundtrip (µs) | Canned retirement (µs) | | -------------- | -------------------- | -------------------- | ---------------------- | @@ -849,20 +326,9 @@ counter was tick-quantized and is not used for these results. | 256 | 15,535 | 55.9 | 57.5 | | 4,095 | 245,760 | 832.1 | 849.2 | -The canned path calls production `retire_owner`, including its record checks, -one GET clone and one exact expected/result comparison. Its CAS returns a fixed -Applied outcome and does not persist. Standalone serde modes exclude session -validation. They are independently timed; subtracting or adding their medians -cannot apportion production CPU. At the largest input, decode and encode alone -measure approximately 574 and 236 µs. A separate test-store CAS/reset pair costs -about 106 µs and includes two mutations and fixture map/Vec cloning; it is never -subtracted from the authority measurement. - -This establishes substantial whole-record codec cost with large shared-owner -histories. It is not routed RPC latency, WAL cost, allocation counting or an -end-to-end throughput limit. Keep the CAS and retired-owner safety contract; -review a smaller resource representation or different layout before changing -publication/reclaim ordering. +The canned path calls production `retire_owner`, including its record checks, one GET clone and one exact expected/result comparison. Its CAS returns a fixed Applied outcome and does not persist. Standalone serde modes exclude session validation. They are independently timed; subtracting or adding their medians cannot apportion production CPU. At the largest input, decode and encode alone measure approximately 574 and 236 µs. A separate test-store CAS/reset pair costs about 106 µs and includes two mutations and fixture map/Vec cloning; it is never subtracted from the authority measurement. + +This establishes substantial whole-record codec cost with large shared-owner histories. It is not routed RPC latency, WAL cost, allocation counting or an end-to-end throughput limit. Keep the CAS and retired-owner safety contract; review a smaller resource representation or different layout before changing publication/reclaim ordering. Reproduce in an exclusive window, with no service workload running: @@ -874,28 +340,16 @@ pixi run taskset -c 2 cargo test --release --locked -p crowdb-access-dataset --t ## Verification and execution notes - Library: `pixi run -- cargo test -p crowdb-access-iceberg --all-targets`. -- HTTP: `pixi run clean --runtime-only && pixi run -- cargo test -p crowdb-access-server --all-targets`. - Default server tests alone skip the Iceberg suites. +- HTTP: `pixi run clean --runtime-only && pixi run -- cargo test -p crowdb-access-server --all-targets`. Default server tests alone skip the Iceberg suites. - SDK: `pixi run -- cargo test -p crowdb-access-server --features iceberg-e2e --test iceberg_table_sdk_test -- --ignored --nocapture --test-threads=1`. -- Namespace SDK: `pixi run -- cargo test -p crowdb-access-server --features iceberg-e2e --test iceberg_namespace_sdk_test -- --ignored --nocapture`. - Set `CROWDB_ICEBERG_E2E_PYTHON=$PWD/.pixi/envs/iceberg-e2e/bin/python` and the - Java environment below. The pinned PyIceberg method requests complete lists; - Java RESTCatalog implements token continuation. Neither client is patched. +- Namespace SDK: `pixi run -- cargo test -p crowdb-access-server --features iceberg-e2e --test iceberg_namespace_sdk_test -- --ignored --nocapture`. Set `CROWDB_ICEBERG_E2E_PYTHON=$PWD/.pixi/envs/iceberg-e2e/bin/python` and the Java environment below. The pinned PyIceberg method requests complete lists; Java RESTCatalog implements token continuation. Neither client is patched. - Native storage namespace: `pixi run test-container-e2e --scope iceberg.namespace --cpus 16 --memory-mib 16384`. - Native SDK: `pixi run test-container-e2e --scope iceberg.sdk.java --cpus 16 --memory-mib 16384`. - Native FileIO faults/lifecycle: `pixi run test-container-e2e --scope iceberg.files --cpus 16 --memory-mib 16384`. - Keep component crash-cut matrices: `pixi run cargo test -p crowdb-access-server --test iceberg_commit_recovery_test --test iceberg_file_recovery_test`. -- Manual concurrent SF1 loaders use the [locked host TPC client](../../../e2e-test/clients/tpc-loader/README.md) - and explicit `iceberg.manual.tpc` scope. They are excluded from ordinary CI. -- For Java tests, use default Pixi for Cargo; set - `JAVA_HOME=$PWD/.pixi/envs/iceberg-e2e/lib/jvm` and - `CROWDB_ICEBERG_E2E_MVN=$PWD/.pixi/envs/iceberg-e2e/bin/mvn`. -- Prefix native tests with `clean --runtime-only` using the same isolated - `CROWDB_RUNTIME_ROOT=$PWD/.crowdb-runtime/ephemeral/iceberg-catalog-e2e`; - preserve unrelated persistent port claims. Do not clean while another test runs. -- Gates: `pixi run -- cargo fmt --all -- --check`, `pixi run rs-lint`, and - `pixi run -- cargo clippy -p crowdb-access-server --features iceberg-e2e --all-targets -- -D warnings`. -- Long native/full-suite commands run in the background and are polled, rather - than being mistaken for failures at the default sixty-second shell cutoff. -- Maven SDK shutdown-thread/logging warnings are nonfatal in the passing native - fixture. Pinned SDK dependency order must precede Hadoop's older transitives. +- Manual concurrent SF1 loaders use the [locked host TPC client](../../../e2e-test/clients/tpc-loader/README.md) and explicit `iceberg.manual.tpc` scope. They are excluded from ordinary CI. +- For Java tests, use default Pixi for Cargo; set `JAVA_HOME=$PWD/.pixi/envs/iceberg-e2e/lib/jvm` and `CROWDB_ICEBERG_E2E_MVN=$PWD/.pixi/envs/iceberg-e2e/bin/mvn`. +- Prefix native tests with `clean --runtime-only` using the same isolated `CROWDB_RUNTIME_ROOT=$PWD/.crowdb-runtime/ephemeral/iceberg-catalog-e2e`; preserve unrelated persistent port claims. Do not clean while another test runs. +- Gates: `pixi run -- cargo fmt --all -- --check`, `pixi run rs-lint`, and `pixi run -- cargo clippy -p crowdb-access-server --features iceberg-e2e --all-targets -- -D warnings`. +- Long native/full-suite commands run in the background and are polled, rather than being mistaken for failures at the default sixty-second shell cutoff. +- Maven SDK shutdown-thread/logging warnings are nonfatal in the passing native fixture. Pinned SDK dependency order must precede Hadoop's older transitives. diff --git a/app/crowdb-access-server/tests/common/access_probe_budget.rs b/app/crowdb-access-server/tests/common/access_probe_budget.rs index 5bacf305b..1c232acfd 100644 --- a/app/crowdb-access-server/tests/common/access_probe_budget.rs +++ b/app/crowdb-access-server/tests/common/access_probe_budget.rs @@ -13,7 +13,7 @@ use crowdb_access_server::s3::{ S3OperationsFuture, }; use crowdb_chunk_client::{metrics::ReadinessMetrics, ChunkIoClient, DiskWriter, Result}; -use crowdb_kv_client::DiskRecord; +use crowdb_paxos_kv_client::DiskRecord; use crowdb_protocol::{ common::{DiskId, HwStatus}, diskdb::rpc::{DiskValue, Segment}, diff --git a/app/crowdb-access-server/tests/common/credentials.rs b/app/crowdb-access-server/tests/common/credentials.rs index 74b0fbe78..2655bdc00 100644 --- a/app/crowdb-access-server/tests/common/credentials.rs +++ b/app/crowdb-access-server/tests/common/credentials.rs @@ -3,15 +3,15 @@ use crowdb_access_s3::auth::{CredentialCipher, MasterKey}; use crowdb_access_server::credentials::CredentialAuthority; -use crowdb_kv::cluster::{ - group::PxGroup, kv_server::KvServer, PxKvStore, PxLocalReplica, PxLocalReplicaRole, +use crowdb_paxos_kv::cluster::{ + group::PxGroup, paxos_kv_server::PaxosKvServer, PxKvStore, PxLocalReplica, PxLocalReplicaRole, }; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient}; use std::sync::Arc; pub struct TestCredentials { store: Arc<PxKvStore>, - pub control: Arc<CrowdbKvClient>, + pub control: Arc<PaxosKvClient>, cipher: Arc<CredentialCipher>, } @@ -23,7 +23,7 @@ impl TestCredentials { PxLocalReplica::new(1, PxLocalReplicaRole::Leader), )); store.start().await.unwrap(); - let client = CrowdbKvClient::new(ClientConfig::new(vec![])); + let client = PaxosKvClient::new(ClientConfig::new(vec![])); client.seed_leader(0, 0, store.listen_addr().unwrap().to_string()); let key = MasterKey::from_hex(&"11".repeat(32)).unwrap(); Self { diff --git a/app/crowdb-access-server/tests/credential_authority_test.rs b/app/crowdb-access-server/tests/credential_authority_test.rs index 192a15aaa..fd4e1e275 100644 --- a/app/crowdb-access-server/tests/credential_authority_test.rs +++ b/app/crowdb-access-server/tests/credential_authority_test.rs @@ -5,7 +5,7 @@ mod credentials; use credentials::TestCredentials; use crowdb_access_server::credentials::CredentialAuthorityError; -use crowdb_kv_client::{GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{GetOutcome, ReadMode}; const USER: &[u8] = b"console-root"; const PREPARED: &[u8] = b"crowdb/s3/user-bootstrap/"; diff --git a/app/crowdb-chunk-kv-server/Cargo.toml b/app/crowdb-chunk-kv-server/Cargo.toml index 5d23d95c9..f442813af 100644 --- a/app/crowdb-chunk-kv-server/Cargo.toml +++ b/app/crowdb-chunk-kv-server/Cargo.toml @@ -22,7 +22,7 @@ crowdb-chunk-kv = { path = "../../lib/crowdb-chunk-kv" } crowdb-chunk-client = { path = "../../lib/crowdb-chunk-client" } crowdb-chunk-stream = { path = "../../lib/crowdb-chunk-stream" } crowdb-common = { workspace = true } -crowdb-kv-client = { path = "../../lib/crowdb-kv-client" } +crowdb-paxos-kv-client = { path = "../../lib/crowdb-paxos-kv-client" } crowdb-protocol = { path = "../../lib/crowdb-protocol" } crowdb-rpc-ffi = { path = "../../lib/crowdb-rpc/ffi" } crowdb-tree-ffi = { path = "../../lib/crowdb-tree/ffi", features = ["chunk-rpc"] } @@ -35,10 +35,11 @@ toml = "0.8" tracing = { workspace = true } [dev-dependencies] +crowdb-tree-ffi = { path = "../../lib/crowdb-tree/ffi", features = ["test-util"] } crowdb-test-harness = { path = "../../lib/crowdb-test-harness" } crowdb-chunk-kv-client = { path = "../../lib/crowdb-chunk-kv-client" } sha2 = "0.10" -crowdb-kv = { path = "../../lib/crowdb-kv", features = ["test-util"] } +crowdb-paxos-kv = { path = "../../lib/crowdb-paxos-kv", features = ["test-util"] } crowdb-chunk-kv-server = { path = ".", features = ["test-util"] } crowdb-chunk-kv = { path = "../../lib/crowdb-chunk-kv", features = ["test-util"] } crowdb-chunk-stream = { path = "../../lib/crowdb-chunk-stream", features = ["test-util"] } diff --git a/app/crowdb-chunk-kv-server/conf/crowdb_chunk_kv_server_config.toml b/app/crowdb-chunk-kv-server/conf/crowdb_chunk_kv_server_config.toml new file mode 100644 index 000000000..6df492061 --- /dev/null +++ b/app/crowdb-chunk-kv-server/conf/crowdb_chunk_kv_server_config.toml @@ -0,0 +1,292 @@ +# Copyright 2026-present Gian <crow.db@outlook.com> +# Licensed under the Apache License, Version 2.0. +# +# Chunk KV Server +# crowdb-chunk-kv-server --config <this-file> +# Read at startup. Sizes are integer bytes; 1 MiB = 1,048,576 bytes. +# Find maximum sustainable flush throughput with ample resources first, then +# reduce resources using measurements. Queues retain work; they do not add throughput. + +# Unique process identity. Set node_id when registering through node management. +instance_id = 1 +# node_id = 1 +# Bind addresses may be unspecified; advertised addresses must be routable. +rpc_listen_addr = "0.0.0.0:21135" +rpc_advertise_addr = "127.0.0.1:21135" +http_listen_addr = "0.0.0.0:20135" +# Omit to leave the optional management endpoint out of instance observations. +# http_advertise_addr = "127.0.0.1:20135" +# Paxos KV Group-0 management endpoints used to discover storage and assignments. +group0_mgmt_seeds = ["http://127.0.0.1:10135"] +# RPC I/O workers are separate from Tree CPU/backend workers. +rpc_workers = 2 +# Partitions share one process-wide Tree runtime. Per-partition caches/budgets +# still contribute to total memory use; this cap is not a process RSS limit. +max_hosted_partitions = 256 +# Fallback catalog/grant reconciliation interval; watch events trigger earlier refreshes. +catalog_refresh_interval_ms = 5_000 +# Remaining WAL records allowed before split preparation enters its final catch-up. +max_split_catchup_lag_records = 1_024 +# Deadline for draining admitted work and checkpointing during shutdown. +shutdown_drain_timeout_ms = 30_000 + +[storage] +# Paxos KV store containing the range catalog and durable root metadata. +metadata_store_id = 1 +# Stream ownership lease in milliseconds. +stream_writer_lease_ms = 30_000 +# Replica count for both WAL streams and Tree chunks (1..5). +stream_mirror_copies = 2 +# Tree and WAL allocation capacities, each in 1..256 MiB. +tree_chunk_capacity_bytes = 268_435_456 +stream_chunk_capacity_bytes = 268_435_456 +# 0 selects the route default; otherwise use a whole number of MiB in 1..256 MiB. +tree_strip_capacity_bytes = 0 +# Host-wide live pin and pending final-release slots for all Tree writers/readers. +tree_pin_release_slots = 1_048_576 +# DiskIO connections per endpoint and I/O workers, independent of the Tree pools. +diskio_connections_per_endpoint = 1 +diskio_rpc_workers = 2 + +[partition] +# Per-partition ingress bounds, including admitted requests still being processed. +# Larger queues absorb bursts and retain memory without increasing service rate. +queue_requests = 1_024 +queue_bytes = 67_108_864 +# Maximum journal batch count and target bytes. One admitted request may exceed +# batch_bytes and is then handled alone; apply remains ordered within each partition. +batch_requests = 64 +batch_bytes = 4_194_304 +# API key/value limits; native Tree key and frame limits also apply. +max_key_bytes = 65_536 +max_value_bytes = 16_777_216 +# Recent deduplication results kept for retry handling; older requests can expire. +retained_results = 65_536 +# Bound metadata reclamation work per maintenance pass. +metadata_reclaim_records_per_pass = 128 + +[monitor] +# Cluster-wide monitor identity and protocol versions. All owners in this domain +# must register a compatible descriptor; coordinate policy changes across owners. +domain = "chunk-kv" +service_registry_name = "chunk-kv" +driver_version = 1 +capability_version = 1 +# Heartbeat <= suspect <= dead. A dead owner must be fenced before lease expiry. +heartbeat_interval_ms = 2_000 +suspect_after_ms = 6_000 +dead_after_ms = 10_000 +lease_duration_ms = 12_000 +# Lease > clock skew + self-fence margin, and dead + margin <= lease. +max_clock_skew_ms = 1_000 +self_fence_margin_ms = 1_000 +# AutomaticSharedStorage permits automatic recovery; OperatorOnly requires intervention. +failure_policy = "AutomaticSharedStorage" +balance_policy = "count-first-v1" +# chunk_kv_range_balance is derived from [balance] on load; do not duplicate it here. + +[balance] +# Disable automatic owner moves; local partition splitting remains available. +enabled = true +# Reserved: must remain 0 until reliable per-range live-byte measurements are available. +byte_weight_percent = 0 +# Owner-count imbalance tolerated before proposing a move (0..100). +imbalance_tolerance_percent = 20 +# Desired minimum partition count per live owner and local split size threshold. +target_partitions_per_owner = 4 +target_partition_bytes = 1_073_741_824 +# Required weighted improvement for moves (0..100) and minimum time between moves. +minimum_weighted_improvement_percent = 25 +cooldown_ms = 60_000 +# 0 disables the request-rate ceiling for candidate target owners. +max_owner_request_rate = 0 + +# Optional first-partition bootstrap. Omit for ordinary catalog-driven startup. +# Use persistent, nonzero identities for an intentional empty-cluster bootstrap. +# [bootstrap_partition] +# partition_id = { high = 1, low = 1 } +# tree_id = 1 +# stream_name = { high = 1, low = 2 } +# owner_epoch = 1 +# metadata_group_id = 1 + +# Optional explicit RPC RDMA configuration. Omit to retain transport environment +# defaults. This complete example selects the 256 KiB control/data class. +# Uncomment the table and its fields together, then set device/source for this host. +# [rpc_rdma] +# enabled = true +# allow_tcp_fallback = true +# listen_enabled = true +# port = 1 +# device = "" +# source_address = "" +# gid = "" +# CQ/QP depths and posted receives; target <= rq_depth and SQ + target <= cq_depth. +# cq_count = 1 +# cq_depth = 8_192 +# sq_depth = 1_024 +# rq_depth = 1_024 +# posted_receive_target = 1_024 +# wire_layout = "control_data" +# wire_slot_capacity = 263_168 +# Send/receive batching and outstanding sends. A zero send window uses SQ depth; +# zero send_window_bytes disables the additional byte ceiling. +# send_batch_size = 16 +# receive_batch_size = 1 +# send_window_size = 0 +# send_window_bytes = 0 +# Completion polling limits and the power-of-two pending send queue capacity. +# poll_batch = 256 +# verbs_poll_batch = 32 +# poll_empty_budget = 256 +# send_queue_capacity = 4_096 +# Optional small-message aggregation (bounded by the transport frame capacity). +# aggregation_enabled = false +# aggregation_max_bytes = 65_536 +# aggregation_max_frames = 256 +# Registered send/receive arenas; size them together with the SQ/RQ windows. +# A zero registration budget derives the budget from the transport geometry. +# send_arena_bytes = 268_435_456 +# receive_arena_bytes = 269_484_032 +# registered_memory_budget_bytes = 0 +# Large control-message assembly bounds, timeout, and aggregate byte budgets. +# max_control_message_bytes = 33_554_432 +# control_assembly_timeout_ms = 30_000 +# control_send_budget_bytes = 67_108_864 +# control_receive_budget_bytes = 67_108_864 + +# Keep Tree field additions, removals, renames and explanations in sync with: +# app/crowdb-paxos-kv-server/conf/crowdb_paxos_kv_server_config.toml +# Each server owns its values independently; no shared include or override is loaded. + +# Process-wide CPU/backend pools. Check CPU quotas and utilization when scaling. +# Resource diagnostics sample every 100 ms and flush at the server metrics-log +# interval. Each host/Tree resource reports its limit, current usage, average +# held sampled usage, and sampled maximum for that log window. The maximum is +# not an exact transient peak; sample count, age, gaps and errors show coverage. +# Averages retain fractional counts: worker avg=0.750 means 75% of one worker. +# Reserved capacity and allocated storage are reported separately. Budget +# denial counters distinguish Tree limits from shared host limits. Queue-slot +# usage includes prepaid progress registrations even when no job is runnable. +# Tree cache rows cover decoded bytes, page slots and request slots; they +# overlap the storage charged to budget partitions and must not be added to it. +[tree_runtime.host] +workers = 4 +# Workers for synchronous PageStore calls; async concurrency also depends on the backend. +backend_workers = 2 +# Larger queues retain more work without increasing service rate. Recheck +# lifecycle/Routing reservations when increasing queue capacity. +cpu_queue_limit = 256 +backend_queue_limit = 256 +# Maximum pending waiters per resource partition. +max_waiters_per_class = 256 +# Retained queue/control reservation charged to host Routing, not an extra partition. +lifecycle_bytes = 4_194_304 + +# Shared limits for all Trees in this process. Resource classes cannot borrow +# from each other. These limits do not bound total process RSS. +[tree_runtime.host.limits] +# Source capture, run/range construction and complete-batch handoff. +handoff_bytes = 268_435_456 +# Routing, publication, cache/load and controls, including lifecycle reservations. +routing_bytes = 536_870_912 +# Consolidation, split/merge and checkpoint/writeback work. +reclaim_bytes = 134_217_728 +# Ordinary writes, payload owners and retained COW allocations. +writer_bytes = 536_870_912 +# Independent progress capacity for chosen/replay/gap apply work. +gap_repair_bytes = 268_435_456 +# In-flight I/O tickets. More tickets alone do not parallelize a serial write pipeline. +io_operations = 128 + +# Per-Tree limits. All Trees still compete for the shared host resources above. +[tree_runtime.tree] +# The producer window must fit this CPU queue. Local queues must fit host queues. +# This also bounds ordinary apply/backend operation owners; replay has two spare slots. +cpu_queue_limit = 64 +backend_queue_limit = 64 +# Fixed queue/core/cache descriptors, charged to this Tree's Routing partition. +# Increase this together with cpu_queue_limit and tuning.cache_max_pages; a +# larger Routing limit alone does not enlarge this retained control reserve. +lifecycle_bytes = 1_048_576 + +[tree_runtime.tree.limits] +# Same categories as the host limits. Check both levels when changing a partition. +handoff_bytes = 67_108_864 +routing_bytes = 268_435_456 +# 128 MiB is an adjustable default, not a fixed checkpoint memory target. +reclaim_bytes = 134_217_728 +writer_bytes = 268_435_456 +gap_repair_bytes = 268_435_456 +io_operations = 64 + +[tree] +# Durable frame size; the current public format supports at most 64 KiB. +# A logical leaf may exceed one frame. +# Must be at least 512 bytes and satisfy backend IU alignment. Logical working +# set size is independent of durable frame size. +frame_bytes = 65_536 +# Decoded-cache byte watermark; tuning.cache_max_pages also limits page count. +buffer_pool_bytes = 67_108_864 +# 0 = frame_bytes/4. Values larger than this spill to overflow; exactly 16 KiB +# remains inline with the default frame size. +max_inline_value = 0 +# LZ4 trades CPU for storage bandwidth/space. Keep compression fixed in comparisons. +compression_lz4 = false +# Local block allocation/reclamation unit; ignored by file/mem-block and injected Chunk storage. +block_size = 67_108_864 +# 0 = backend default: block 4 KiB, mem-block 1 byte, Chunk 64 KiB. A nonzero value +# must be a power of two that divides frame_bytes. +iu_size = 0 + +# Engine work granularity. Select these values from measurements. +[tree.tuning] +# Soft per-source rotation thresholds. Crossing either requests rotation; +# 0 requests rotation after every nonempty write. +# The source_max_* hard admission capacities below also apply. +memtable_flush_bytes = 16_777_216 +memtable_flush_entries = 100_000 +# Includes the writable source; the current read/handoff window requires 2..64. +max_memtable_count = 10 +# Per-source admitted version/key capacities, including writes not yet applied. +# Raising soft thresholds alone may not enlarge batches if these limits bind first. +source_max_entries = 4_096 +source_max_key_bytes = 1_048_576 +# Additional accepted/replay allowance. Size it for accepted apply bursts. +# Only the next contiguous slot may use gap source capacity; higher holes use ordinary capacity or wait. +source_max_gap_entries = 1_024 +source_max_gap_key_bytes = 1_048_576 +# 0 = min(host workers, Tree io_operations). Explicit values must fit both limits. +flush_parallelism = 0 +# Per-handoff-run entry/key/work-byte limits. Value owners are retained independently. +run_max_entries = 128 +# Effective key capacity is max(run_max_key_bytes, max_key_size). +run_max_key_bytes = 262_144 +run_max_bytes = 1_048_576 +# Key-arena allocation granularity; must not exceed run_max_key_bytes. +run_key_block_bytes = 1_024 +# Initial run-install window; later windows double. Balances early visible +# progress against prefix-array copying cost. +install_run_window = 64 +# Visits per step/pass; the step size must not exceed the pass limit. +maintenance_visits_per_step = 64 +maintenance_max_visits = 4_096 +# Candidates/steps per pass. Larger limits increase work and reservations; +# they do not repair starvation caused by an incorrect scan cursor. +maintenance_max_candidates = 256 +maintenance_max_steps = 1_024 +# Local scratch per maintenance operation; independent of the total Reclaim budget. +maintenance_metadata_bytes = 1_048_576 +# Decoded-cache page limit. Check this together with buffer_pool_bytes. +cache_max_pages = 256 +# Split/merge thresholds for the physical primary record region, not logical leaf limits. +# Split is capped by frame/format capacity. Merge=0 disables merging; merged +# output must also fit the hysteresis target of about 75% of the split threshold. +leaf_split_bytes = 65_502 +leaf_merge_bytes = 16_384 +# Inner separator limit; encoded output must also fit the frame byte capacity. +inner_max_keys = 256 +# 0 = max(1, inner_max_keys/4). +inner_merge_keys = 0 +# Maximum key size; 0 = frame_bytes/2. Oversized keys are rejected before admission. +max_key_size = 0 diff --git a/app/crowdb-chunk-kv-server/src/config.rs b/app/crowdb-chunk-kv-server/src/config.rs index ad44e2e38..b1839f4e7 100644 --- a/app/crowdb-chunk-kv-server/src/config.rs +++ b/app/crowdb-chunk-kv-server/src/config.rs @@ -8,7 +8,7 @@ use crowdb_protocol::chunk_kv::{ ChunkKvRangeBalancePolicy, DomainFailurePolicy, DomainMonitorDescriptor, Id128, }; use crowdb_protocol::chunk_stream::StreamName; -use crowdb_protocol::{CHUNK_KV_HTTP_BASE, CHUNK_KV_RPC_BASE, KV_SERVER_MGMT_BASE}; +use crowdb_protocol::{CHUNK_KV_HTTP_BASE, CHUNK_KV_RPC_BASE, PAXOS_KV_SERVER_MGMT_BASE}; use serde::{Deserialize, Serialize}; use thiserror::Error; @@ -41,6 +41,9 @@ pub struct ChunkKvServerConfig { pub rpc_workers: u32, pub rpc_rdma: Option<crowdb_rpc_ffi::RdmaConfig>, pub storage: StorageConfig, + pub partition: crowdb_chunk_kv::PartitionConfig, + pub tree_runtime: crowdb_tree_ffi::TreeRuntimeConfig, + pub tree: crowdb_tree_ffi::TreeEngineConfig, pub bootstrap_partition: Option<BootstrapPartitionConfig>, pub monitor: DomainMonitorDescriptor, pub balance: BalanceConfig, @@ -55,7 +58,7 @@ impl Default for ChunkKvServerConfig { rpc_advertise_addr: format!("127.0.0.1:{CHUNK_KV_RPC_BASE}"), http_listen_addr: format!("0.0.0.0:{CHUNK_KV_HTTP_BASE}"), http_advertise_addr: None, - group0_mgmt_seeds: vec![format!("http://127.0.0.1:{KV_SERVER_MGMT_BASE}")], + group0_mgmt_seeds: vec![format!("http://127.0.0.1:{PAXOS_KV_SERVER_MGMT_BASE}")], max_hosted_partitions: 256, catalog_refresh_interval_ms: 5_000, max_split_catchup_lag_records: 1_024, @@ -63,6 +66,9 @@ impl Default for ChunkKvServerConfig { rpc_workers: 2, rpc_rdma: None, storage: StorageConfig::default(), + partition: crowdb_chunk_kv::PartitionConfig::default(), + tree_runtime: crowdb_tree_ffi::TreeRuntimeConfig::default(), + tree: crowdb_tree_ffi::TreeEngineConfig::default(), bootstrap_partition: None, monitor: default_monitor(), balance: BalanceConfig::default(), @@ -71,6 +77,17 @@ impl Default for ChunkKvServerConfig { } impl ChunkKvServerConfig { + /// Build native engine and per-partition limits; the process runtime is attached by storage. + #[must_use] + pub fn native_tree_options(&self) -> crowdb_tree_ffi::Config { + let mut options = crowdb_tree_ffi::Config { + runtime_binding: self.tree_runtime.binding_options(), + ..Default::default() + }; + self.tree.apply_to(&mut options); + options + } + /// Loads a TOML configuration file and applies field defaults. /// /// # Errors @@ -131,7 +148,11 @@ impl ChunkKvServerConfig { "partition capacity and lifecycle intervals must be nonzero".into(), )); } + self.validate_tree_config()?; self.storage.validate()?; + self.partition + .validate() + .map_err(|error| ConfigError::Invalid(error.to_string()))?; if self .rpc_rdma .as_ref() @@ -164,6 +185,25 @@ impl ChunkKvServerConfig { } Ok(()) } + + fn validate_tree_config(&self) -> Result<(), ConfigError> { + self.tree_runtime.validate().map_err(ConfigError::Invalid)?; + self.tree + .validate(&self.tree_runtime) + .map_err(ConfigError::Invalid)?; + let iu = if self.tree.iu_size == 0 { + 64 * 1024 + } else { + self.tree.iu_size + }; + if self.tree.frame_bytes % iu != 0 { + return Err(ConfigError::Invalid( + "tree.frame_bytes must align to the Chunk backend IU (64 KiB when tree.iu_size is zero)" + .to_owned(), + )); + } + Ok(()) + } } #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] @@ -199,6 +239,7 @@ pub struct StorageConfig { pub stream_mirror_copies: u32, pub tree_chunk_capacity_bytes: u64, pub tree_strip_capacity_bytes: u32, + pub tree_pin_release_slots: usize, pub stream_chunk_capacity_bytes: u64, pub diskio_connections_per_endpoint: usize, pub diskio_rpc_workers: u32, @@ -212,6 +253,7 @@ impl Default for StorageConfig { stream_mirror_copies: 2, tree_chunk_capacity_bytes: 256 * 1024 * 1024, tree_strip_capacity_bytes: 0, + tree_pin_release_slots: 1_048_576, stream_chunk_capacity_bytes: 256 * 1024 * 1024, diskio_connections_per_endpoint: 1, diskio_rpc_workers: 2, @@ -221,6 +263,11 @@ impl Default for StorageConfig { impl StorageConfig { fn validate(&self) -> Result<(), ConfigError> { + if self.tree_pin_release_slots == 0 || self.tree_pin_release_slots > usize::MAX / 2 { + return Err(ConfigError::Invalid( + "storage.tree_pin_release_slots must fit a nonzero native queue capacity".into(), + )); + } if self.stream_writer_lease_ms == 0 || self.stream_mirror_copies == 0 || self.stream_mirror_copies > 5 diff --git a/app/crowdb-chunk-kv-server/src/control_store.rs b/app/crowdb-chunk-kv-server/src/control_store.rs index 0bb88108d..c06c2f337 100644 --- a/app/crowdb-chunk-kv-server/src/control_store.rs +++ b/app/crowdb-chunk-kv-server/src/control_store.rs @@ -6,7 +6,7 @@ use std::sync::Arc; use async_trait::async_trait; -use crowdb_kv_client::{CrowdbKvClient, Error as KvClientError, GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{Error as KvClientError, GetOutcome, PaxosKvClient, ReadMode}; use crowdb_protocol::chunk_kv::{ ChunkKvRangeCatalogHead, ChunkKvRangeCatalogPage, DomainMonitorDescriptor, EnsureDomainMonitorOutcome, ServingGrant, SplitTransition, TransferTransition, @@ -49,7 +49,7 @@ pub trait Group0Kv: Send + Sync { } #[async_trait] -impl Group0Kv for CrowdbKvClient { +impl Group0Kv for PaxosKvClient { async fn get(&self, key: &[u8]) -> Result<Option<VersionedValue>, Group0KvError> { match self .get(GROUP0_STORE, GROUP0_GROUP, key, ReadMode::Linearizable, None) @@ -125,7 +125,7 @@ impl Group0Kv for CrowdbKvClient { async fn put_cas(&self, key: &[u8], value: &[u8], expected_revision: u64) -> Result<(), Group0KvError> { const MAX_BUSY_ATTEMPTS: usize = 20; for attempt in 0..MAX_BUSY_ATTEMPTS { - match CrowdbKvClient::put_cas(self, GROUP0_STORE, GROUP0_GROUP, key, value, expected_revision) + match PaxosKvClient::put_cas(self, GROUP0_STORE, GROUP0_GROUP, key, value, expected_revision) .await { Ok(_) => return Ok(()), @@ -158,7 +158,7 @@ impl Group0ControlStore { } #[must_use] - pub fn from_client(kv: Arc<CrowdbKvClient>) -> Self { + pub fn from_client(kv: Arc<PaxosKvClient>) -> Self { Self { kv } } diff --git a/app/crowdb-chunk-kv-server/src/host_lifecycle.rs b/app/crowdb-chunk-kv-server/src/host_lifecycle.rs new file mode 100644 index 000000000..b204e1e8d --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/host_lifecycle.rs @@ -0,0 +1,96 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! One explicit host exit path also covers partially initialized services. + +use crowdb_chunk_kv_server::serving::process::ProcessLeaseSession; +use crowdb_chunk_kv_server::{ChunkKvRpcService, ChunkKvService, ChunkKvStorage}; +use crowdb_rpc_ffi::RpcServer; +use std::sync::{ + atomic::{AtomicBool, Ordering}, + Arc, +}; +use tokio::sync::Notify; +use tokio::task::JoinHandle; +use tracing::{error, warn}; + +pub(super) struct HostLifecycle { + pub storage: Arc<ChunkKvStorage>, + pub process: Option<Arc<ProcessLeaseSession>>, + pub service: Option<Arc<ChunkKvService>>, + pub rpc: Option<Arc<ChunkKvRpcService>>, + pub server: Option<Arc<RpcServer>>, + pub tasks: [Option<JoinHandle<()>>; 4], + pub stop: Arc<HostStop>, +} + +impl HostLifecycle { + pub fn new(storage: Arc<ChunkKvStorage>) -> Self { + Self { + storage, + process: None, + service: None, + rpc: None, + server: None, + tasks: std::array::from_fn(|_| None), + stop: Arc::default(), + } + } + + pub async fn close(&mut self) { + if let Some(service) = &self.service { + service.begin_drain(); + } + if let Some(server) = &self.server { + server.stop(); + } + self.stop.close(); + for task in &mut self.tasks { + if let Some(task) = task.take() { + let _ = task.await; + } + } + if let Some(service) = &self.service { + if let Err(error) = service.drain_writers().await { + warn!(%error, "WAL writer drain incomplete; restart must retain lease exclusion"); + } + } + if let Some(rpc) = &self.rpc { + rpc.close_and_drain().await; + } + if let Some(service) = &self.service { + if let Err(error) = service.release_hosted_trees().await { + error!(%error, "native Tree close failed during host shutdown"); + } + } + self.storage.drain_pin_releases().await; + if let Some(process) = &self.process { + if let Err(error) = process.withdraw_observation().await { + warn!(%error, "failed to unregister chunk KV instance"); + } + } + } +} + +#[derive(Default)] +pub(super) struct HostStop { + closed: AtomicBool, + wake: Notify, +} +impl HostStop { + fn close(&self) { + self.closed.store(true, Ordering::Release); + self.wake.notify_waiters(); + } + pub async fn wait(&self) { + loop { + let wake = self.wake.notified(); + tokio::pin!(wake); + wake.as_mut().enable(); + if self.closed.load(Ordering::Acquire) { + return; + } + wake.await; + } + } +} diff --git a/app/crowdb-chunk-kv-server/src/main.rs b/app/crowdb-chunk-kv-server/src/main.rs index 8320ab374..3211df7dd 100644 --- a/app/crowdb-chunk-kv-server/src/main.rs +++ b/app/crowdb-chunk-kv-server/src/main.rs @@ -14,7 +14,7 @@ use crowdb_chunk_kv_server::{ ChunkKvStorage, DomainMonitorRegistry, Group0ControlStore, ManagementState, StorageRuntimeError, TransitionExecutor, TransitionProcessor, }; -use crowdb_kv_client::{WatchNotifyClient, WatchSubscription}; +use crowdb_paxos_kv_client::{WatchNotifyClient, WatchSubscription}; use crowdb_protocol::chunk_kv::{ ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogHead, ChunkKvRangeCatalogPage, ChunkKvRangeCatalogPageRef, ChunkKvRangeCatalogPartitionState, EnsureDomainMonitorOutcome, EnsureDomainMonitorRequest, KeyRange, @@ -25,6 +25,8 @@ use crowdb_protocol::key::{ }; use tracing::{error, info, warn}; +mod host_lifecycle; + #[derive(Debug, Parser)] #[command( name = "crowdb-chunk-kv-server", @@ -146,7 +148,7 @@ async fn main() -> std::process::ExitCode { "crowdb-chunk-kv-server starting" ); - let mut metrics_runner = crowdb_chunk_kv_server::metrics::start_stream_perf_log( + let mut metrics_runner = crowdb_chunk_kv_server::metrics::create_perf_log( &log_dir, args.metrics_interval, args.log_max_file_mb, @@ -161,6 +163,33 @@ async fn main() -> std::process::ExitCode { return std::process::ExitCode::FAILURE; } }; + let mut host = host_lifecycle::HostLifecycle::new(storage); + let result = serve_host( + &mut host, + &config, + http_addr, + rpc_addr, + rpc_advertise_addr, + metrics_runner.as_mut(), + ) + .await; + host.close().await; + if let Some(runner) = &mut metrics_runner { + runner.stop().await; + } + result +} + +#[allow(clippy::too_many_lines)] +async fn serve_host( + host: &mut host_lifecycle::HostLifecycle, + config: &ChunkKvServerConfig, + http_addr: SocketAddr, + rpc_addr: SocketAddr, + rpc_advertise_addr: SocketAddr, + metrics_runner: Option<&mut crowdb_common::metrics::MetricsRunner>, +) -> std::process::ExitCode { + let storage = host.storage.clone(); let control_store = Arc::new(Group0ControlStore::from_client(Arc::clone(storage.kv()))); let monitor_registry = DomainMonitorRegistry::new( control_store.clone(), @@ -198,12 +227,9 @@ async fn main() -> std::process::ExitCode { return std::process::ExitCode::FAILURE; } }; - if let Err(error) = storage - .streams() - .registry() - .bind_owner_process(process_lease.authority()) - { - error!(%error, "failed to bind WAL stream process authority"); + host.process = Some(process_lease.clone()); + if let Err(error) = storage.bind_owner_process(process_lease.authority()) { + error!(%error, "failed to bind tree and WAL process authority"); return std::process::ExitCode::FAILURE; } let service = match ChunkKvService::new(process_lease.authority(), config.max_hosted_partitions) { @@ -214,6 +240,14 @@ async fn main() -> std::process::ExitCode { } }; + host.service = Some(service.clone()); + if let Some(runner) = metrics_runner { + runner.set_sampler( + crowdb_chunk_kv_server::metrics::TreeResources::new(storage.tree_runtime().clone(), &service) + .with_storage(&storage), + ); + runner.start(); + } let catalog = Arc::new(ChunkKvRangeCatalogPublisher::new(control_store.clone())); let recovery_storage = &storage; let recovery_service = &service; @@ -308,11 +342,13 @@ async fn main() -> std::process::ExitCode { } }; let refresh_interval = std::time::Duration::from_millis(config.catalog_refresh_interval_ms); + let stop = host.stop.clone(); let refresh_task = tokio::spawn(async move { let mut interval = tokio::time::interval(refresh_interval); interval.tick().await; loop { let grant_only = tokio::select! { + () = stop.wait() => break, _ = interval.tick() => false, alive = receive_transition_notify(&mut catalog_watch) => { if !alive { @@ -375,6 +411,7 @@ async fn main() -> std::process::ExitCode { } } }); + host.tasks[0] = Some(refresh_task); let transition_executor = match TransitionExecutor::new( config.instance_id, Arc::clone(&service), @@ -384,7 +421,6 @@ async fn main() -> std::process::ExitCode { Ok(executor) => Arc::new(executor), Err(error) => { error!(%error, "failed to initialize transition worker"); - refresh_task.abort(); return std::process::ExitCode::FAILURE; } }; @@ -409,10 +445,12 @@ async fn main() -> std::process::ExitCode { } }; let transition_interval = std::time::Duration::from_millis(config.catalog_refresh_interval_ms); + let stop = host.stop.clone(); let transition_task = tokio::spawn(async move { let mut interval = tokio::time::interval(transition_interval); loop { tokio::select! { + () = stop.wait() => break, _ = interval.tick() => {} alive = receive_transition_notify(&mut transfer_watch) => { if !alive { @@ -431,6 +469,7 @@ async fn main() -> std::process::ExitCode { } }); + host.tasks[1] = Some(transition_task); let capacity_bytes = u64::try_from(config.max_hosted_partitions) .unwrap_or(u64::MAX) .saturating_mul(config.balance.target_partition_bytes); @@ -445,13 +484,13 @@ async fn main() -> std::process::ExitCode { .await { error!(%error, "failed to register chunk KV instance"); - transition_task.abort(); - refresh_task.abort(); return std::process::ExitCode::FAILURE; } - install_latest_grant(&control_store, &service, &config).await; + install_latest_grant(&control_store, &service, config).await; let materialization_service = Arc::clone(&service); + let materialization_storage = Arc::clone(&storage); let materialization_interval = std::time::Duration::from_millis(config.catalog_refresh_interval_ms); + let stop = host.stop.clone(); let materialization_task = tokio::spawn(async move { let mut interval = tokio::time::interval(materialization_interval); interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); @@ -460,6 +499,7 @@ async fn main() -> std::process::ExitCode { loop { tokio::select! { biased; + () = stop.wait() => break, _ = interval.tick() => { if let Err(error) = materialization_service.materialize_split_overlays().await { warn!(%error, "chunk KV split-overlay materialization pass failed"); @@ -474,13 +514,22 @@ async fn main() -> std::process::ExitCode { false } }; + match materialization_service + .resume_legacy_retirement(&materialization_storage) + .await + { + Ok(backlog) => pending |= backlog, + Err(error) => warn!(%error, "legacy Chunk retirement deferred; retaining durable claims"), + } } }); + host.tasks[2] = Some(materialization_task); let heartbeat_service = Arc::clone(&service); let heartbeat_lease = Arc::clone(&process_lease); let heartbeat_store = Arc::clone(&control_store); let heartbeat_config = config.clone(); let heartbeat_endpoint = rpc_advertise_addr.to_string(); + let stop = host.stop.clone(); let heartbeat_task = tokio::spawn(async move { let mut interval = tokio::time::interval(std::time::Duration::from_millis( heartbeat_config.monitor.heartbeat_interval_ms, @@ -489,7 +538,10 @@ async fn main() -> std::process::ExitCode { let mut previous_requests = heartbeat_service.metrics().snapshot().requests; let mut previous_ms = heartbeat_service.monotonic_ms(); loop { - interval.tick().await; + tokio::select! { + () = stop.wait() => break, + _ = interval.tick() => {}, + } let now_ms = heartbeat_service.monotonic_ms(); let requests = heartbeat_service.metrics().snapshot().requests; let elapsed_ms = now_ms.saturating_sub(previous_ms).max(1); @@ -521,6 +573,7 @@ async fn main() -> std::process::ExitCode { } }); + host.tasks[3] = Some(heartbeat_task); let rpc_server = match ChunkKvRpcService::create_server(config.rpc_workers, config.rpc_rdma.as_ref()) { Ok(server) => server, Err(error) => { @@ -528,6 +581,7 @@ async fn main() -> std::process::ExitCode { return std::process::ExitCode::FAILURE; } }; + host.server = Some(rpc_server.clone()); if let Err(error) = rpc_server.listen(&rpc_addr.ip().to_string(), i32::from(rpc_addr.port())) { error!(%rpc_addr, %error, "data RPC bind failed"); return std::process::ExitCode::FAILURE; @@ -536,6 +590,7 @@ async fn main() -> std::process::ExitCode { Arc::clone(&service), tokio::runtime::Handle::current(), )); + host.rpc = Some(rpc_service.clone()); rpc_service.register_handlers(&rpc_server); rpc_server.start(); info!(%rpc_addr, "data RPC server listening"); @@ -574,35 +629,6 @@ async fn main() -> std::process::ExitCode { if let Err(error) = &serving { error!(%error, "HTTP management server failed"); } - heartbeat_task.abort(); - materialization_task.abort(); - transition_task.abort(); - refresh_task.abort(); - // These workers only observe the service and have already been fenced by - // begin_drain. Drop their handles after cancellation: a worker may be in - // a synchronous storage observation, and waiting for that observation here - // would hold the management restart request open until the supervisor kills - // the process. The service drain below is the durable shutdown barrier. - drop(( - heartbeat_task, - materialization_task, - transition_task, - refresh_task, - )); - info!("chunk KV background workers stopped"); - match service.drain_writers().await { - Ok(()) => info!("chunk KV WAL writer drain complete"), - Err(error) => { - warn!(%error, "WAL writer drain incomplete; restart must retain lease exclusion"); - } - } - if let Err(error) = process_lease.withdraw_observation().await { - warn!(%error, "failed to unregister chunk KV instance"); - } - info!("chunk KV observation withdrawal complete"); - if let Some(runner) = &mut metrics_runner { - runner.stop().await; - } if serving.is_ok() { std::process::ExitCode::SUCCESS } else { diff --git a/app/crowdb-chunk-kv-server/src/management.rs b/app/crowdb-chunk-kv-server/src/management.rs index 977285fae..33a2e30ca 100644 --- a/app/crowdb-chunk-kv-server/src/management.rs +++ b/app/crowdb-chunk-kv-server/src/management.rs @@ -28,7 +28,7 @@ impl ManagementState { #[derive(Debug, Serialize)] struct HealthResponse { instance_id: u64, - process_lease: crowdb_kv_client::StreamProcessLeaseStatus, + process_lease: crowdb_paxos_kv_client::StreamProcessLeaseStatus, lifecycle: &'static str, catalog_generation: u64, hosted_partitions: usize, diff --git a/app/crowdb-chunk-kv-server/src/metrics.rs b/app/crowdb-chunk-kv-server/src/metrics.rs index 0d92e219e..a9feb4f76 100644 --- a/app/crowdb-chunk-kv-server/src/metrics.rs +++ b/app/crowdb-chunk-kv-server/src/metrics.rs @@ -6,6 +6,9 @@ use std::sync::atomic::{AtomicU64, Ordering}; use crowdb_protocol::chunk_kv::{ChunkKvResponse, ChunkKvRpcErrorCode}; use serde::Serialize; +mod resources; +pub use resources::TreeResources; + #[derive(Debug, Default)] pub struct ServerMetrics { requests: AtomicU64, @@ -151,7 +154,8 @@ fn increment(counter: &AtomicU64) { counter.fetch_add(1, Ordering::Relaxed); } -/// Starts periodic stream phase output alongside the service logs. +/// Creates periodic stream phase output alongside the service logs. +/// Install the Tree sampler before calling `start()`. /// Zero interval disables output. Call `stop().await` to flush the last window. /// /// # Errors @@ -159,7 +163,7 @@ fn increment(counter: &AtomicU64) { /// /// # Panics /// Panics if the metrics registry is poisoned or a histogram name conflicts. -pub fn start_stream_perf_log( +pub fn create_perf_log( log_dir: &str, interval_secs: u64, max_file_mb: usize, @@ -171,10 +175,9 @@ pub fn start_stream_perf_log( let file = crowdb_common::logging::open_metrics_log(log_dir, "crowdb-chunk-kv-server", max_file_mb, max_files) .map_err(std::io::Error::other)?; - let mut runner = crowdb_common::metrics::MetricsRunner::new(file, interval_secs); + let runner = crowdb_common::metrics::MetricsRunner::new(file, interval_secs); crowdb_chunk_stream::register_stream_perf_metrics( &mut runner.registry().lock().expect("metrics registry poisoned"), ); - runner.start(); Ok(Some(runner)) } diff --git a/app/crowdb-chunk-kv-server/src/metrics/resources.rs b/app/crowdb-chunk-kv-server/src/metrics/resources.rs new file mode 100644 index 000000000..7ee540290 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/metrics/resources.rs @@ -0,0 +1,66 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use std::sync::{Arc, Weak}; +use std::time::Instant; + +use crowdb_common::metrics::MetricsSampler; +use crowdb_tree_ffi::runtime::{ResourceUsageLog, ResourceUsageSnapshot}; + +use crate::{ChunkKvService, ChunkKvStorage}; + +pub struct TreeResources { + runtime: Arc<crowdb_tree_ffi::TreeRuntime>, + service: Weak<ChunkKvService>, + storage: Weak<ChunkKvStorage>, + windows: ResourceUsageLog, +} + +impl TreeResources { + #[must_use] + pub fn new(runtime: Arc<crowdb_tree_ffi::TreeRuntime>, service: &Arc<ChunkKvService>) -> Self { + Self { + runtime, + service: Arc::downgrade(service), + storage: Weak::new(), + windows: ResourceUsageLog::default(), + } + } + + #[must_use] + pub fn with_storage(mut self, storage: &Arc<ChunkKvStorage>) -> Self { + self.storage = Arc::downgrade(storage); + self + } + + fn host_usage(&self) -> Result<ResourceUsageSnapshot, crowdb_tree_ffi::CtError> { + let mut snapshot = self.runtime.resource_usage()?; + if let Some(storage) = self.storage.upgrade() { + snapshot.resources[20] = storage.pin_release_resource_usage()?; + snapshot.resource_mask |= 1 << 20; + } + Ok(snapshot) + } +} + +impl MetricsSampler for TreeResources { + fn sample(&mut self) { + self.windows.begin_sample(); + match self.host_usage() { + Ok(snapshot) => self.windows.observe("host.tree", snapshot, Instant::now()), + Err(_) => self.windows.sample_failed("host.tree"), + } + if let Some(service) = self.service.upgrade() { + service.sample_tree_resources(|name, result| match result { + Ok(Some(snapshot)) => self.windows.observe(name, snapshot, Instant::now()), + Ok(None) => {} + Err(_) => self.windows.sample_failed(name), + }); + } + self.windows.end_sample(Instant::now()); + } + + fn flush(&mut self, writer: &mut dyn std::io::Write, timestamp: &str) -> std::io::Result<()> { + self.windows.flush(writer, timestamp, Instant::now()) + } +} diff --git a/app/crowdb-chunk-kv-server/src/rpc.rs b/app/crowdb-chunk-kv-server/src/rpc.rs index da3f7ef17..b777bd707 100644 --- a/app/crowdb-chunk-kv-server/src/rpc.rs +++ b/app/crowdb-chunk-kv-server/src/rpc.rs @@ -26,6 +26,7 @@ use tokio::runtime::Handle; use crate::ChunkKvService; mod response; +mod tasks; mod transport; use response::{submit_group_response, submit_response}; @@ -33,12 +34,42 @@ use response::{submit_group_response, submit_response}; pub struct ChunkKvRpcService { service: Arc<ChunkKvService>, runtime: Handle, + tasks: Arc<tasks::RpcTasks>, } impl ChunkKvRpcService { #[must_use] pub fn new(service: Arc<ChunkKvService>, runtime: Handle) -> Self { - Self { service, runtime } + Self { + service, + runtime, + tasks: Arc::new(tasks::RpcTasks::default()), + } + } + + /// Stop asynchronous handler admission and wait until their request, + /// partition and response owners have all been dropped. + pub async fn close_and_drain(&self) { + self.tasks.close_and_drain().await; + } + + fn spawn_owned<F>(&self, permit: tasks::RpcTask, future: F) + where + F: std::future::Future<Output = ()> + Send + 'static, + { + self.runtime.spawn(async move { + { + let mut future = std::pin::pin!(future); + future.as_mut().await; + } + drop(permit); + }); + } + + #[cfg(feature = "test-util")] + #[must_use] + pub fn hold_request_for_tests(&self) -> Option<impl Send + 'static> { + self.tasks.enter() } pub fn register_handlers(self: &Arc<Self>, server: &Arc<RpcServer>) { @@ -75,9 +106,13 @@ impl ChunkKvRpcService { } fn handle_point(&self, request: ServerRequest, server: &Arc<RpcServer>) { + let Some(permit) = self.tasks.enter() else { + request.close_connection(); + return; + }; let service = Arc::clone(&self.service); let server = Arc::clone(server); - self.runtime.spawn(async move { + self.spawn_owned(permit, async move { let rpc_request_id = request.request_id; let response = match decode_point_request(request.control()) { Ok(envelope) if envelope.rpc_request_id == rpc_request_id => { @@ -93,9 +128,13 @@ impl ChunkKvRpcService { } fn handle_seek(&self, request: ServerRequest, server: &Arc<RpcServer>) { + let Some(permit) = self.tasks.enter() else { + request.close_connection(); + return; + }; let service = Arc::clone(&self.service); let server = Arc::clone(server); - self.runtime.spawn(async move { + self.spawn_owned(permit, async move { let rpc_request_id = request.request_id; let response = match decode_seek_request(request.control()) { Ok(envelope) if envelope.rpc_request_id == rpc_request_id => { @@ -111,9 +150,13 @@ impl ChunkKvRpcService { } fn handle_scan(&self, request: ServerRequest, server: &Arc<RpcServer>) { + let Some(permit) = self.tasks.enter() else { + request.close_connection(); + return; + }; let service = Arc::clone(&self.service); let server = Arc::clone(server); - self.runtime.spawn(async move { + self.spawn_owned(permit, async move { let rpc_request_id = request.request_id; let response = match decode_scan_request(request.control()) { Ok(envelope) if envelope.rpc_request_id == rpc_request_id => { @@ -129,9 +172,13 @@ impl ChunkKvRpcService { } fn handle_bounded_scan(&self, request: ServerRequest, server: &Arc<RpcServer>) { + let Some(permit) = self.tasks.enter() else { + request.close_connection(); + return; + }; let service = Arc::clone(&self.service); let server = Arc::clone(server); - self.runtime.spawn(async move { + self.spawn_owned(permit, async move { let id = request.request_id; let response = match decode_bounded_scan_request(request.control()) { Ok((envelope, budget)) if envelope.rpc_request_id == id => { @@ -147,9 +194,13 @@ impl ChunkKvRpcService { } fn handle_multi_get(&self, request: ServerRequest, server: &Arc<RpcServer>) { + let Some(permit) = self.tasks.enter() else { + request.close_connection(); + return; + }; let service = Arc::clone(&self.service); let server = Arc::clone(server); - self.runtime.spawn(async move { + self.spawn_owned(permit, async move { let id = request.request_id; let create_nano = request.rpc_create_nano; let response = match decode_multi_get_request(request.control()) { @@ -177,9 +228,13 @@ impl ChunkKvRpcService { } fn handle_batch_mutation(&self, request: ServerRequest, server: &Arc<RpcServer>) { + let Some(permit) = self.tasks.enter() else { + request.close_connection(); + return; + }; let service = Arc::clone(&self.service); let server = Arc::clone(server); - self.runtime.spawn(async move { + self.spawn_owned(permit, async move { let id = request.request_id; let create_nano = request.rpc_create_nano; let response = match decode_batch_mutation_request(request.control()) { diff --git a/app/crowdb-chunk-kv-server/src/rpc/tasks.rs b/app/crowdb-chunk-kv-server/src/rpc/tasks.rs new file mode 100644 index 000000000..b2eef3202 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/rpc/tasks.rs @@ -0,0 +1,55 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +//! One atomic gate covers handler admission through the last owned request tail. +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Arc, +}; +use tokio::sync::Notify; +const CLOSED: usize = 1 << (usize::BITS - 1); +#[derive(Default)] +pub(super) struct RpcTasks { + state: AtomicUsize, + changed: Notify, +} +pub(super) struct RpcTask { + owner: Arc<RpcTasks>, +} +impl RpcTasks { + pub fn enter(self: &Arc<Self>) -> Option<RpcTask> { + let mut observed = self.state.load(Ordering::Acquire); + loop { + if observed & CLOSED != 0 || observed == CLOSED - 1 { + return None; + } + match self.state.compare_exchange_weak( + observed, + observed + 1, + Ordering::AcqRel, + Ordering::Acquire, + ) { + Ok(_) => return Some(RpcTask { owner: self.clone() }), + Err(actual) => observed = actual, + } + } + } + pub async fn close_and_drain(&self) { + self.state.fetch_or(CLOSED, Ordering::AcqRel); + loop { + let change = self.changed.notified(); + tokio::pin!(change); + change.as_mut().enable(); + if self.state.load(Ordering::Acquire) == CLOSED { + return; + } + change.await; + } + } +} +impl Drop for RpcTask { + fn drop(&mut self) { + if self.owner.state.fetch_sub(1, Ordering::AcqRel) == CLOSED + 1 { + self.owner.changed.notify_waiters(); + } + } +} diff --git a/app/crowdb-chunk-kv-server/src/server.rs b/app/crowdb-chunk-kv-server/src/server.rs index 60480a628..454e63dff 100644 --- a/app/crowdb-chunk-kv-server/src/server.rs +++ b/app/crowdb-chunk-kv-server/src/server.rs @@ -11,7 +11,7 @@ use crowdb_chunk_kv::{ ChunkKvError, CompareCondition, JournalPosition, MutationOperation, MutationResult, Partition, PreparedSplitWriterArtifact, RequestId, SplitArtifact, ValueRevision, }; -use crowdb_kv_client::StreamProcessAuthority; +use crowdb_paxos_kv_client::StreamProcessAuthority; use crowdb_protocol::chunk_kv::{ BatchMutationRequest, BatchMutationResponse, BatchMutationResult, ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogHead, ChunkKvRangeCatalogPage, ChunkKvRangeCatalogPartitionState, ChunkKvResponse, @@ -34,6 +34,7 @@ mod authority; mod drain; mod group; use assignments::{local_partition_for_entry, partition_matches_entry}; +mod legacy_retirement; mod load_sampling; mod maintenance; mod observation; @@ -42,6 +43,7 @@ mod readiness; mod recovery; pub use readiness::RangeReadiness; mod reconcile; +mod resources; mod routing; mod scan; pub(crate) use observation::PageQuery; @@ -378,9 +380,14 @@ impl ChunkKvService { #[must_use] pub fn registry_observation(&self, capacity_bytes: u64, request_rate: u64) -> ChunkKvExtra { let partitions = self.observed_partition_snapshot(); + let mut scopes = std::collections::BTreeSet::new(); let durable_bytes = partitions .values() - .filter_map(|partition| partition.estimated_bytes().ok()) + .filter_map(|partition| match partition.backing_storage_stats() { + Ok(Some(stats)) => scopes.insert(stats.scope_id).then_some(stats.allocated_bytes), + Ok(None) => partition.estimated_bytes().ok(), + Err(_) => None, + }) .fold(0_u64, u64::saturating_add); let mut hosted: Vec<_> = partitions .values() diff --git a/app/crowdb-chunk-kv-server/src/server/drain.rs b/app/crowdb-chunk-kv-server/src/server/drain.rs index ea1b7b427..d87b4ac38 100644 --- a/app/crowdb-chunk-kv-server/src/server/drain.rs +++ b/app/crowdb-chunk-kv-server/src/server/drain.rs @@ -22,8 +22,14 @@ impl ChunkKvService { self.begin_drain(); self.load_sampling.drain().await; let partitions = self.partitions.load_full(); + let prepared = self.prepared_transfer_targets.load_full(); + let splits = self.local_split_sessions.load_full(); let mut first_error = None; - for partition in partitions.values() { + for partition in partitions + .values() + .chain(prepared.values()) + .chain(splits.values().map(|entry| &entry.dispatcher)) + { if let Err(error) = partition .suspend_for_shutdown(partition.snapshot().ownership_epoch) .await @@ -33,4 +39,18 @@ impl ChunkKvService { } first_error.map_or(Ok(()), Err) } + + /// Release hosted Tree owners after background tasks and RPC handlers end. + /// Independent snapshots keep their exact backend owners until their own drop. + /// # Errors + /// Reports a failure of the host blocking task that closes native Trees. + pub async fn release_hosted_trees(&self) -> Result<(), ChunkKvError> { + self.begin_drain(); + let partitions = self.partitions.swap(std::sync::Arc::default()); + let prepared = self.prepared_transfer_targets.swap(std::sync::Arc::default()); + let splits = self.local_split_sessions.swap(std::sync::Arc::default()); + tokio::task::spawn_blocking(move || drop((partitions, prepared, splits))) + .await + .map_err(|error| ChunkKvError::MaintenanceDegraded(error.to_string())) + } } diff --git a/app/crowdb-chunk-kv-server/src/server/legacy_retirement.rs b/app/crowdb-chunk-kv-server/src/server/legacy_retirement.rs new file mode 100644 index 000000000..81184c062 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/server/legacy_retirement.rs @@ -0,0 +1,63 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Resume explicitly started namespace retirement through the existing host pump. +use super::{local_partition_for_entry, ChunkKvRangeCatalogPartitionState, ChunkKvService}; +use crate::storage::ChunkKvStorage; +use crowdb_chunk_kv::ChunkKvError; +use crowdb_chunk_stream::StreamRegistry; +use std::collections::HashSet; + +impl ChunkKvService { + /// Resume only durable, explicitly started legacy retirements for live local + /// metadata namespaces. Missing proof waits for the next scheduled pass. + /// # Errors + /// Returns metadata, ownership, or physical cleanup failures without dropping claims. + pub async fn resume_legacy_retirement(&self, storage: &ChunkKvStorage) -> Result<bool, ChunkKvError> { + let catalog = self.catalog.load_full(); + let partitions = self.partitions.load_full(); + let mut groups = HashSet::new(); + let mut pending = false; + for entry in &catalog.entries { + if entry.owner.instance_id != self.instance_id + || entry.state != ChunkKvRangeCatalogPartitionState::Serving + || entry.transition_id.is_some() + || self + .authority + .authorize( + catalog.generation, + entry.partition_id, + entry.owner_epoch, + self.monotonic_ms(), + ) + .is_err() + || partitions + .get(&entry.partition_id) + .and_then(|partition| local_partition_for_entry(partition, entry)) + .is_none() + { + continue; + } + let record = storage + .streams() + .registry() + .load(entry.artifact.stream_name) + .await + .map_err(|error| ChunkKvError::MaintenanceDegraded(error.to_string()))? + .ok_or_else(|| { + ChunkKvError::MaintenanceDegraded("legacy namespace stream is absent".into()) + })?; + if !groups.insert(record.metadata_group_id) { + continue; + } + if let Some(progress) = storage + .resume_legacy_storage(record.metadata_group_id, 64) + .await + .map_err(|error| ChunkKvError::MaintenanceDegraded(error.to_string()))? + { + pending |= !progress.complete && progress.pending.is_none(); + } + } + Ok(pending) + } +} diff --git a/app/crowdb-chunk-kv-server/src/server/resources.rs b/app/crowdb-chunk-kv-server/src/server/resources.rs new file mode 100644 index 000000000..ec6ee4475 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/server/resources.rs @@ -0,0 +1,21 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use super::ChunkKvService; + +impl ChunkKvService { + pub(crate) fn sample_tree_resources( + &self, + mut observe: impl FnMut( + &str, + crowdb_chunk_kv::Result<Option<crowdb_tree_ffi::runtime::ResourceUsageSnapshot>>, + ), + ) { + // Include prepared transfer targets, whose Trees already consume host + // resources before activation into the serving catalog. + for partition in self.observed_partition_snapshot().values() { + let name = format!("chunk.tree.{}", partition.tree_id()); + observe(&name, partition.tree_resource_usage()); + } + } +} diff --git a/app/crowdb-chunk-kv-server/src/serving/lease.rs b/app/crowdb-chunk-kv-server/src/serving/lease.rs index a626aba49..5224c6707 100644 --- a/app/crowdb-chunk-kv-server/src/serving/lease.rs +++ b/app/crowdb-chunk-kv-server/src/serving/lease.rs @@ -5,7 +5,7 @@ use std::sync::Arc; use std::time::Instant; use arc_swap::ArcSwap; -use crowdb_kv_client::StreamProcessAuthority; +use crowdb_paxos_kv_client::StreamProcessAuthority; use crowdb_protocol::chunk_kv::{DomainMonitorDescriptor, Id128, InstanceHealth, ServingGrant}; use thiserror::Error; diff --git a/app/crowdb-chunk-kv-server/src/serving/lease/context.rs b/app/crowdb-chunk-kv-server/src/serving/lease/context.rs index 8f5f57cad..df2ab5dcd 100644 --- a/app/crowdb-chunk-kv-server/src/serving/lease/context.rs +++ b/app/crowdb-chunk-kv-server/src/serving/lease/context.rs @@ -4,7 +4,7 @@ //! Immutable admission context and shared grant-snapshot validation. use super::{AuthorityError, AuthoritySnapshot, AuthorityState, ServingAuthority}; -use crowdb_kv_client::StreamProcessContext; +use crowdb_paxos_kv_client::StreamProcessContext; use crowdb_protocol::chunk_kv::{Id128, ServingAssignment}; use std::time::Instant; diff --git a/app/crowdb-chunk-kv-server/src/serving/process.rs b/app/crowdb-chunk-kv-server/src/serving/process.rs index 4c6d3406d..136bd99f5 100644 --- a/app/crowdb-chunk-kv-server/src/serving/process.rs +++ b/app/crowdb-chunk-kv-server/src/serving/process.rs @@ -8,8 +8,8 @@ use std::{ time::{Duration, Instant, SystemTime, UNIX_EPOCH}, }; -use crowdb_kv_client::{ - CrowdbKvClient, StreamProcessAuthority, StreamProcessContext, StreamProcessLeaseClient, +use crowdb_paxos_kv_client::{ + PaxosKvClient, StreamProcessAuthority, StreamProcessContext, StreamProcessLeaseClient, StreamProcessLeaseError, StreamProcessLeaseTiming, }; use crowdb_protocol::{ @@ -54,7 +54,7 @@ impl ProcessLeaseSession { /// # Errors /// Returns invalid policy, a changed predecessor or an unavailable lease. pub async fn start( - kv: Arc<CrowdbKvClient>, + kv: Arc<PaxosKvClient>, instance_id: u64, policy: &DomainMonitorDescriptor, ) -> Result<Self, StreamProcessLeaseError> { diff --git a/app/crowdb-chunk-kv-server/src/serving/worker.rs b/app/crowdb-chunk-kv-server/src/serving/worker.rs index 7d181f7e6..1c45e4dd3 100644 --- a/app/crowdb-chunk-kv-server/src/serving/worker.rs +++ b/app/crowdb-chunk-kv-server/src/serving/worker.rs @@ -4,7 +4,7 @@ use std::sync::Arc; use async_trait::async_trait; -use crowdb_chunk_kv::{Partition, PartitionConfig, SplitArtifact}; +use crowdb_chunk_kv::{Partition, SplitArtifact}; use crowdb_protocol::chunk_kv::{ AuthorityReleaseProof, ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogPartitionState, PartitionArtifact, SplitPhase, SplitReadinessProof, SplitTransition, TailOverlayArtifact, TargetReadinessProof, @@ -475,7 +475,7 @@ impl TransitionStorage for ChunkKvStorage { let artifact = prepared.artifact.clone(); let child = prepared .child - .open_warmed(PartitionConfig::default()) + .open_warmed(self.partition_config().clone()) .map_err(|error| plan_error(&error.to_string()))?; let retained_parent = parent .split_ingress() diff --git a/app/crowdb-chunk-kv-server/src/storage.rs b/app/crowdb-chunk-kv-server/src/storage.rs index ea7606605..b6815f253 100644 --- a/app/crowdb-chunk-kv-server/src/storage.rs +++ b/app/crowdb-chunk-kv-server/src/storage.rs @@ -7,12 +7,12 @@ use std::sync::Arc; use crowdb_chunk_client::{ChunkIoClient, ChunkIoClientConfig, ChunkReadPolicy, SmallWritePolicy}; use crowdb_chunk_kv::{ - Partition, PartitionConfig, PartitionId, PartitionJournal, PartitionRange, PartitionTree, PreparedSplit, + Partition, PartitionConfig, PartitionId, PartitionJournal, PartitionRange, PreparedSplit, PreparedSplitWriterArtifact, SplitArtifact, SplitChild, SplitPlan, SplitWriterTarget, StreamPartitionJournal, TransitionId, }; use crowdb_chunk_stream::{ChunkStream, ProductionStreamRuntime, StreamConfig, StreamName, StreamRegistry}; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient}; use crowdb_protocol::chunk_kv::{ ChunkKvRangeCatalogEntry, PartitionArtifact, SplitChildAssignment, SplitTransition, TailOverlayArtifact, TransferTransition, @@ -26,8 +26,14 @@ use thiserror::Error; use crate::{BootstrapPartitionConfig, ChunkKvServerConfig}; mod bootstrap; +pub mod legacy_retirement; mod root_catalog; -use bootstrap::{initialize_bootstrap_root, recover_bootstrap_partition}; +#[cfg(feature = "test-util")] +mod root_catalog_test; +mod tree_owner; +use bootstrap::{initialize_bootstrap_root, open_bootstrap_partition, recover_bootstrap_partition}; +#[cfg(feature = "test-util")] +pub use root_catalog_test::{open_root_catalog_with_process_for_tests, TestOwnedRootCatalog}; mod split_handoff; mod split_recovery; mod streams; @@ -47,19 +53,42 @@ pub enum StorageRuntimeError { /// Process-wide production clients shared by every hosted partition stream. pub struct ChunkKvStorage { - kv: Arc<CrowdbKvClient>, + kv: Arc<PaxosKvClient>, chunk_io: ChunkIoClient, streams: Arc<ProductionStreamRuntime>, tree_transport: Arc<ChunkTransport>, + tree_runtime: Arc<crowdb_tree_ffi::TreeRuntime>, + tree_options: crowdb_tree_ffi::Config, + partition_config: PartitionConfig, + pin_releases: crowdb_tree_ffi::chunk::pin_service::ChunkPinReleaseService, tree_mirror_copies: u32, tree_chunk_capacity_bytes: u64, metadata_store_id: u64, + owner_process: arc_swap::ArcSwapOption<crowdb_paxos_kv_client::StreamProcessAuthority>, } -struct TreeStorageGeometry { +struct TreeStorageOptions { mirror_copies: u32, chunk_capacity_bytes: u64, strip_capacity_bytes: u32, + pin_release_slots: usize, + runtime: crowdb_tree_ffi::TreeRuntimeOptions, + native: crowdb_tree_ffi::Config, + partition: PartitionConfig, +} + +impl Default for TreeStorageOptions { + fn default() -> Self { + Self { + mirror_copies: 2, + chunk_capacity_bytes: 256 * 1024 * 1024, + strip_capacity_bytes: 0, + pin_release_slots: 1_048_576, + runtime: crowdb_tree_ffi::TreeRuntimeOptions::default(), + native: crowdb_tree_ffi::Config::default(), + partition: PartitionConfig::default(), + } + } } impl ChunkKvStorage { @@ -70,7 +99,7 @@ impl ChunkKvStorage { /// /// Returns an endpoint discovery or storage configuration error. pub async fn connect(config: &ChunkKvServerConfig) -> Result<Self, StorageRuntimeError> { - let kv = Arc::new(CrowdbKvClient::new(ClientConfig::new( + let kv = Arc::new(PaxosKvClient::new(ClientConfig::new( config.group0_mgmt_seeds.clone(), ))); let chunk_io = ChunkIoClient::connect_with_kv( @@ -84,15 +113,19 @@ impl ChunkKvStorage { ) .await .map_err(|error| StorageRuntimeError::ChunkIo(error.to_string()))?; - Self::from_parts_with_geometry( + Self::from_parts_with_options( kv, chunk_io, config.storage.metadata_store_id, config.storage.stream_writer_lease_ms, - TreeStorageGeometry { + TreeStorageOptions { mirror_copies: config.storage.stream_mirror_copies, chunk_capacity_bytes: config.storage.tree_chunk_capacity_bytes, strip_capacity_bytes: config.storage.tree_strip_capacity_bytes, + pin_release_slots: config.storage.tree_pin_release_slots, + runtime: config.tree_runtime.runtime_options(), + native: config.native_tree_options(), + partition: config.partition.clone(), }, StreamConfig { chunk_capacity_bytes: config.storage.stream_chunk_capacity_bytes, @@ -106,12 +139,12 @@ impl ChunkKvStorage { .await } - async fn from_parts_with_geometry( - kv: Arc<CrowdbKvClient>, + async fn from_parts_with_options( + kv: Arc<PaxosKvClient>, chunk_io: ChunkIoClient, metadata_store_id: u64, writer_lease_ms: u64, - tree: TreeStorageGeometry, + tree: TreeStorageOptions, stream_config: StreamConfig, read_policy: ChunkReadPolicy, ) -> Result<Self, StorageRuntimeError> { @@ -135,21 +168,22 @@ impl ChunkKvStorage { /// /// Returns an error for an invalid stream lease or read policy. pub async fn from_parts( - kv: Arc<CrowdbKvClient>, + kv: Arc<PaxosKvClient>, chunk_io: ChunkIoClient, metadata_store_id: u64, writer_lease_ms: u64, stream_mirror_copies: u32, ) -> Result<Self, StorageRuntimeError> { - Self::from_parts_with_geometry( + Self::from_parts_with_options( kv, chunk_io, metadata_store_id, writer_lease_ms, - TreeStorageGeometry { + TreeStorageOptions { mirror_copies: stream_mirror_copies, chunk_capacity_bytes: 256 * 1024 * 1024, strip_capacity_bytes: 0, + ..TreeStorageOptions::default() }, StreamConfig::default(), ChunkReadPolicy::default(), @@ -158,12 +192,12 @@ impl ChunkKvStorage { } async fn assemble( - kv: Arc<CrowdbKvClient>, + kv: Arc<PaxosKvClient>, chunk_io: ChunkIoClient, streams: Arc<ProductionStreamRuntime>, metadata_store_id: u64, writer_lease_ms: u64, - tree: TreeStorageGeometry, + tree: TreeStorageOptions, ) -> Result<Self, StorageRuntimeError> { let (chunkdb, disks) = chunk_io .native_storage_routes(writer_lease_ms) @@ -189,19 +223,53 @@ impl ChunkKvStorage { }) .map_err(|error| StorageRuntimeError::Tree(error.to_string()))?, ); + let tree_runtime = Arc::new( + crowdb_tree_ffi::TreeRuntime::new(tree.runtime) + .map_err(|error| StorageRuntimeError::Tree(error.to_string()))?, + ); + let pin_releases = crowdb_tree_ffi::chunk::pin_service::ChunkPinReleaseService::new( + &tree_runtime, + tree.pin_release_slots, + ) + .map_err(|error| StorageRuntimeError::Tree(error.to_string()))?; Ok(Self { kv, chunk_io, streams, tree_transport, + tree_runtime, + tree_options: tree.native, + partition_config: tree.partition, + pin_releases, tree_mirror_copies: tree.mirror_copies, tree_chunk_capacity_bytes: tree.chunk_capacity_bytes, metadata_store_id, + owner_process: arc_swap::ArcSwapOption::empty(), }) } #[must_use] - pub fn kv(&self) -> &Arc<CrowdbKvClient> { + pub fn tree_runtime(&self) -> &Arc<crowdb_tree_ffi::TreeRuntime> { + &self.tree_runtime + } + + pub(crate) fn pin_release_resource_usage( + &self, + ) -> Result<crowdb_tree_ffi::runtime::ResourceUsage, crowdb_tree_ffi::CtError> { + self.pin_releases.resource_usage() + } + + pub(crate) fn partition_config(&self) -> &PartitionConfig { + &self.partition_config + } + + /// Await real final-pin metadata work before the host transport/runtime stop. + pub async fn drain_pin_releases(&self) { + self.pin_releases.shutdown().await; + } + + #[must_use] + pub fn kv(&self) -> &Arc<PaxosKvClient> { &self.kv } @@ -229,7 +297,34 @@ impl ChunkKvStorage { ) -> Result<Arc<PageStore>, StorageRuntimeError> { options.mirror_copies = self.tree_mirror_copies; options.max_chunk_bytes = self.tree_chunk_capacity_bytes; - PageStore::open_chunk(options, catalog, Some(&self.tree_transport)) + if self.owner_process.load().is_none() { + return Err(StorageRuntimeError::Tree( + "tree process authority is not bound".into(), + )); + } + self.pin_releases + .bind(&catalog) + .map_err(|error| StorageRuntimeError::Tree(error.to_string()))?; + let opened = PageStore::open_chunk_runtime( + &self.tree_runtime, + crowdb_tree_ffi::ChunkRuntimeOptions { + tree_id: options.tree_id, + owner_epoch: options.owner_epoch, + open_generation: options.open_generation, + iu_size: if options.iu_size != 0 { + options.iu_size + } else if self.tree_options.iu_size != 0 { + self.tree_options.iu_size + } else { + 64 * 1024 + }, + mirror_copies: options.mirror_copies, + max_chunk_bytes: options.max_chunk_bytes, + }, + catalog, + &self.tree_transport, + ); + opened .map(Arc::new) .map_err(|error| StorageRuntimeError::Tree(error.to_string())) } @@ -245,21 +340,9 @@ impl ChunkKvStorage { options: ChunkPageStoreOptions, metadata_group_id: u64, ) -> Result<Arc<PageStore>, StorageRuntimeError> { - let catalog = KvRootCatalogStore::claim( - Arc::clone(&self.kv), - self.metadata_store_id, - metadata_group_id, - options.tree_id, - options.owner_epoch, - ) - .await?; - self.open_tree_page_store( - options, - Arc::new( - ChunkRootCatalog::open_callback(Arc::new(catalog)) - .map_err(|error| StorageRuntimeError::Tree(error.to_string()))?, - ), - ) + self.open_durable_tree(options, metadata_group_id) + .await + .map(|opened| opened.0) } /// Initializes the stream selected by an Active catalog record. @@ -331,36 +414,7 @@ impl ChunkKvStorage { entry.owner_epoch )) })?; - let open_generation = entry - .artifact - .tail_overlay - .as_ref() - .map_or(0, |overlay| overlay.base_root_manifest_generation); - let options = ChunkPageStoreOptions { - tree_id: entry.artifact.tree_id, - owner_epoch: entry.owner_epoch, - open_generation, - pack_bytes: 0, - iu_size: 0, - max_concurrent_packs: 0, - materialization_bytes_per_pass: 0, - mirror_copies: 0, - max_chunk_bytes: 0, - }; - let catalog = KvRootCatalogStore::for_assignment( - self.kv.clone(), - self.metadata_store_id, - record.metadata_group_id, - entry, - ) - .await?; - let page_store = self.open_tree_page_store( - options, - Arc::new( - ChunkRootCatalog::open_callback(Arc::new(catalog)) - .map_err(|error| StorageRuntimeError::Tree(error.to_string()))?, - ), - )?; + let (page_store, catalog) = self.open_assignment_tree(entry, record.metadata_group_id).await?; if let Some(overlay) = &entry.artifact.tail_overlay { let parent_stream = self .streams @@ -373,18 +427,20 @@ impl ChunkKvStorage { )) })?; let artifact = prepared_overlay_artifact(entry, overlay); - return Partition::recover_native_prepared_overlay( + let partition = Partition::recover_native_prepared_overlay( artifact, - PartitionConfig::default(), - crowdb_tree_ffi::Config::default(), - page_store, + self.partition_config.clone(), + self.tree_config(), + page_store.clone(), stream, parent_stream, ) .await - .map_err(|error| overlay_recovery_error(entry, overlay, &error)); + .map_err(|error| overlay_recovery_error(entry, overlay, &error))?; + Self::complete_tree_recovery(&partition, &page_store, &catalog).await?; + return Ok(partition); } - Partition::recover_native_latest_prepared_assignment( + let partition = Partition::recover_native_latest_prepared_assignment( PartitionId { high: entry.partition_id.high, low: entry.partition_id.low, @@ -395,13 +451,15 @@ impl ChunkKvStorage { }, entry.owner_epoch, entry.artifact.tree_id, - PartitionConfig::default(), - crowdb_tree_ffi::Config::default(), - page_store, + self.partition_config.clone(), + self.tree_config(), + page_store.clone(), stream, ) .await - .map_err(|error| assignment_recovery_error(entry, &error)) + .map_err(|error| assignment_recovery_error(entry, &error))?; + Self::complete_tree_recovery(&partition, &page_store, &catalog).await?; + Ok(partition) } /// Advances one already-open transfer target from its preparation cursor @@ -507,8 +565,8 @@ impl ChunkKvStorage { ) .await .map_err(|error| StorageRuntimeError::Stream(error.to_string()))?; - let page_store = self - .open_durable_tree_page_store( + let (page_store, catalog) = self + .open_durable_tree( ChunkPageStoreOptions { tree_id: config.tree_id, owner_epoch: config.owner_epoch, @@ -523,35 +581,28 @@ impl ChunkKvStorage { config.metadata_group_id, ) .await?; - let partition_id = PartitionId { - high: config.partition_id.high, - low: config.partition_id.low, - }; - let range = PartitionRange { - start: Some(Vec::new()), - end: None, - }; if !fresh { - return recover_bootstrap_partition(config, page_store, stream).await; - } - let owner_epoch = config.owner_epoch; - let tree_id = config.tree_id; - let partition = tokio::task::spawn_blocking(move || { - Partition::open_native( - partition_id, - range, - owner_epoch, - PartitionConfig::default(), - tree_id, - crowdb_tree_ffi::Config::default(), - page_store, + let partition = recover_bootstrap_partition( + config, + self.partition_config.clone(), + self.tree_config(), + page_store.clone(), stream, ) - }) - .await - .map_err(|error| StorageRuntimeError::Partition(format!("bootstrap tree open: {error}")))? - .map_err(|error| StorageRuntimeError::Partition(format!("bootstrap tree open: {error}")))?; + .await?; + Self::complete_tree_recovery(&partition, &page_store, &catalog).await?; + return Ok(partition); + } + let partition = open_bootstrap_partition( + config, + self.partition_config.clone(), + self.tree_config(), + page_store.clone(), + stream, + ) + .await?; initialize_bootstrap_root(&partition, config).await?; + Self::complete_tree_recovery(&partition, &page_store, &catalog).await?; Ok(partition) } @@ -605,16 +656,14 @@ impl ChunkKvStorage { "nonempty child WAL has no committed handoff proof", )); } - let tree_id = child.tree_id; - let tree_config = child.tree_config.clone(); let transition_id = split_plan(transition).transition_id; - tokio::task::spawn_blocking(move || { - crowdb_chunk_kv::CrowdbPartitionTree::open(tree_id, &tree_config)? - .unpin_generation(transition_id) - }) - .await - .map_err(|error| storage_plan_error(&error.to_string()))? - .map_err(|error| storage_plan_error(&error.to_string()))?; + child + .tree_config + .page_store + .as_ref() + .ok_or_else(|| storage_plan_error("split child page store is absent"))? + .unpin_chunk_generation(child.tree_id, transition_id.high, transition_id.low) + .map_err(|error| storage_plan_error(&error.to_string()))?; child }; let handoff = Arc::new(split_handoff::SplitHandoffCommit { @@ -735,7 +784,7 @@ impl ChunkKvStorage { tree_id: artifact.tree_id, tree_config: crowdb_tree_ffi::Config { page_store: Some(page_store), - ..crowdb_tree_ffi::Config::default() + ..self.tree_config() }, journal: Arc::new( StreamPartitionJournal::new(stream, artifact.stream_name) @@ -886,7 +935,7 @@ fn assignment_recovery_error( /// Returns a root identity, KV availability or stale authority error. #[cfg(feature = "test-util")] pub async fn open_root_catalog_for_tests( - kv: Arc<CrowdbKvClient>, + kv: Arc<PaxosKvClient>, store_id: u64, group_id: u64, entry: &ChunkKvRangeCatalogEntry, diff --git a/app/crowdb-chunk-kv-server/src/storage/bootstrap.rs b/app/crowdb-chunk-kv-server/src/storage/bootstrap.rs index e7f818b78..b7c299616 100644 --- a/app/crowdb-chunk-kv-server/src/storage/bootstrap.rs +++ b/app/crowdb-chunk-kv-server/src/storage/bootstrap.rs @@ -7,16 +7,32 @@ use super::StorageRuntimeError; #[cfg(feature = "test-util")] impl super::ChunkKvStorage { + /// Open a private bootstrap with the ordinary production engine settings. + /// + /// # Errors + /// Returns invalid native configuration or worker initialization errors. + pub async fn open_bootstrap_partition_for_tests( + config: &BootstrapPartitionConfig, + partition_config: PartitionConfig, + tree_config: crowdb_tree_ffi::Config, + page_store: Arc<PageStore>, + stream: ChunkStream, + ) -> Result<Partition, StorageRuntimeError> { + open_bootstrap_partition(config, partition_config, tree_config, page_store, stream).await + } + /// Resume a private bootstrap before its first catalog publication. /// /// # Errors /// Returns durable identity, WAL replay, and checkpoint errors. pub async fn recover_bootstrap_partition_for_tests( config: &BootstrapPartitionConfig, + partition_config: PartitionConfig, + tree_config: crowdb_tree_ffi::Config, page_store: Arc<PageStore>, stream: ChunkStream, ) -> Result<Partition, StorageRuntimeError> { - recover_bootstrap_partition(config, page_store, stream).await + recover_bootstrap_partition(config, partition_config, tree_config, page_store, stream).await } } use crate::BootstrapPartitionConfig; @@ -27,16 +43,46 @@ use crowdb_chunk_stream::ChunkStream; use crowdb_tree_ffi::{CtError, PageStore}; use std::sync::Arc; -pub(super) async fn recover_bootstrap_partition( +pub(super) async fn open_bootstrap_partition( config: &BootstrapPartitionConfig, + partition_config: PartitionConfig, + tree_config: crowdb_tree_ffi::Config, page_store: Arc<PageStore>, stream: ChunkStream, ) -> Result<Partition, StorageRuntimeError> { - let pending = match page_store.chunk_manifest_generation() { - Ok(generation) => generation == 0, - Err(CtError::NotFound) => true, - Err(error) => return Err(StorageRuntimeError::Tree(error.to_string())), + let partition_id = PartitionId { + high: config.partition_id.high, + low: config.partition_id.low, }; + let owner_epoch = config.owner_epoch; + let tree_id = config.tree_id; + tokio::task::spawn_blocking(move || { + Partition::open_native( + partition_id, + PartitionRange { + start: Some(Vec::new()), + end: None, + }, + owner_epoch, + partition_config, + tree_id, + tree_config, + page_store, + stream, + ) + }) + .await + .map_err(|error| StorageRuntimeError::Partition(error.to_string()))? + .map_err(|error| StorageRuntimeError::Partition(format!("bootstrap tree open: {error}"))) +} + +pub(super) async fn recover_bootstrap_partition( + config: &BootstrapPartitionConfig, + partition_config: PartitionConfig, + tree_config: crowdb_tree_ffi::Config, + page_store: Arc<PageStore>, + stream: ChunkStream, +) -> Result<Partition, StorageRuntimeError> { let partition = Partition::recover_native_latest_prepared_assignment( PartitionId { high: config.partition_id.high, @@ -48,13 +94,18 @@ pub(super) async fn recover_bootstrap_partition( }, config.owner_epoch, config.tree_id, - PartitionConfig::default(), - crowdb_tree_ffi::Config::default(), - page_store, + partition_config, + tree_config, + page_store.clone(), stream, ) .await .map_err(|error| StorageRuntimeError::Partition(error.to_string()))?; + let pending = match page_store.chunk_manifest_generation() { + Ok(generation) => generation == 0, + Err(CtError::NotFound) => true, + Err(error) => return Err(StorageRuntimeError::Tree(error.to_string())), + }; if pending { partition .activate_recovered(config.owner_epoch) diff --git a/app/crowdb-chunk-kv-server/src/storage/legacy_retirement.rs b/app/crowdb-chunk-kv-server/src/storage/legacy_retirement.rs new file mode 100644 index 000000000..bad23cc8c --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/legacy_retirement.rs @@ -0,0 +1,250 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Explicit stopped-legacy-format retirement. Unknown historical owners retain their receipts. +use super::ChunkKvStorage; +use crowdb_chunk_client::{ChunkAllocator, DiskWriter}; +use crowdb_paxos_kv_client::{GetOutcome, PaxosKvClient, ReadMode, StreamProcessAuthority}; +use crowdb_tree_ffi::CtError; +use serde::{Deserialize, Serialize}; +use std::{sync::Arc, time::Instant}; +mod allocation; +mod proof; +pub(crate) mod record; +use record::{Namespace, Phase, NAMESPACE_KEY, ROOT_PREFIX}; + +/// A durable reason why a known legacy receipt is still physically retained. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub enum LegacyRetirementPending { + LegacyRoot { + tree: u64, + }, + MissingOwnerProof { + tree: u64, + generation: u64, + }, + LiveProcess { + tree: u64, + generation: u64, + instance: u64, + epoch: u64, + }, + GenerationPin { + tree: u64, + generation: u64, + }, + MissingAllocation { + high: u64, + low: u64, + }, + AllocationChanged { + high: u64, + low: u64, + }, +} +/// Work in this invocation; `complete` covers persisted, known manifest receipts only. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct LegacyRetirementProgress { + pub examined: u32, + pub reclaimed_chunks: u32, + pub reclaimed_bytes: u64, + pub complete: bool, + pub pending: Option<LegacyRetirementPending>, +} + +impl ChunkKvStorage { + /// Permanently stop legacy-format publication in this metadata namespace and + /// retire known receipts only after all registered legacy readers are excluded. + /// No writer-quiesced flag substitutes for authoritative process exclusion. + /// Historical allocations lacking a retained manifest are outside this census. + /// + /// # Errors + /// Returns fencing, metadata, or physical-IO errors. The durable cursor and + /// allocation claim survive unknown replies; the next call resumes them. + pub async fn collect_legacy_storage( + &self, + metadata_group_id: u64, + limit: u32, + ) -> Result<LegacyRetirementProgress, CtError> { + self.legacy_collector(metadata_group_id)?.step(limit).await + } + + pub(crate) async fn resume_legacy_storage( + &self, + metadata_group_id: u64, + limit: u32, + ) -> Result<Option<LegacyRetirementProgress>, CtError> { + self.legacy_collector(metadata_group_id)?.resume(limit).await + } + + fn legacy_collector(&self, metadata_group_id: u64) -> Result<Collector, CtError> { + let process = self.owner_process.load_full().ok_or(CtError::NotSupported)?; + let (allocator, disks) = self.chunk_io.storage_parts(); + Ok(Collector { + kv: self.kv.clone(), + store: self.metadata_store_id, + group: metadata_group_id, + process, + allocator, + disks, + }) + } +} + +pub(super) struct Collector { + pub kv: Arc<PaxosKvClient>, + pub store: u64, + pub group: u64, + pub process: Arc<StreamProcessAuthority>, + pub allocator: Arc<dyn ChunkAllocator>, + pub disks: Arc<dyn DiskWriter>, +} +impl Collector { + async fn resume(&self, limit: u32) -> Result<Option<LegacyRetirementProgress>, CtError> { + if self.get(NAMESPACE_KEY).await?.is_none() { + return Ok(None); + } + self.step(limit).await.map(Some) + } + + async fn get(&self, key: &[u8]) -> Result<Option<(bytes::Bytes, u64)>, CtError> { + match self + .kv + .get(self.store, self.group, key, ReadMode::Linearizable, None) + .await + .map_err(|_| CtError::Unavailable)? + { + GetOutcome::NotFound => Ok(None), + GetOutcome::Found { value, revision } => Ok(Some((value, revision))), + } + } + async fn put(&self, key: &[u8], bytes: &[u8], revision: u64) -> Result<(), CtError> { + self.kv + .put_cas(self.store, self.group, key, bytes, revision) + .await + .map(|_| ()) + .map_err(|_| CtError::Unavailable) + } + pub(super) async fn step(&self, limit: u32) -> Result<LegacyRetirementProgress, CtError> { + if self.group == 0 || limit == 0 || limit > 64 { + return Err(CtError::InvalidArgument); + } + self.process + .capture(Instant::now()) + .map_err(|_| CtError::Unavailable)?; + let (mut state, revision) = if let Some((bytes, revision)) = self.get(NAMESPACE_KEY).await? { + (record::decode::<Namespace>(&bytes)?, revision) + } else { + self.put(NAMESPACE_KEY, &record::encode(&Namespace::default())?, 0) + .await?; + let (bytes, revision) = self.get(NAMESPACE_KEY).await?.ok_or(CtError::Unavailable)?; + (record::decode::<Namespace>(&bytes)?, revision) + }; + if state.version != 1 { + return Err(CtError::Corruption); + } + state.pending = None; + let mut progress = LegacyRetirementProgress::default(); + while progress.examined < limit && state.phase != Phase::Complete { + let page = self + .kv + .scan( + self.store, + self.group, + ROOT_PREFIX, + &state.after, + &[], + 1, + ReadMode::Linearizable, + None, + true, + None, + ) + .await + .map_err(|_| CtError::Unavailable)?; + if page.timed_out { + return Err(CtError::Unavailable); + } + let Some((key, _)) = page.items.into_iter().next() else { + state.phase = if state.phase == Phase::Proving { + Phase::Retiring + } else { + Phase::Complete + }; + state.after.clear(); + state.pack_index = 0; + continue; + }; + progress.examined += 1; + let finished = if state.phase == Phase::Proving { + self.prove_key(&key, &mut state).await? + } else { + self.retire_key(&key, &mut state, &mut progress).await? + }; + if state.pending.is_some() { + break; + } + if finished { + state.after = key.to_vec(); + state.pack_index = 0; + state.proof_after.clear(); + state.pins_done = false; + state.owners_done = false; + } + } + self.process + .capture(Instant::now()) + .map_err(|_| CtError::Unavailable)?; + self.put(NAMESPACE_KEY, &record::encode(&state)?, revision) + .await?; + progress.pending = state.pending; + progress.complete = state.phase == Phase::Complete; + Ok(progress) + } +} + +/// Run the production collector against protocol fixture seams. +/// # Errors +/// Returns the same proof, metadata and physical IO failures as the host entry. +#[cfg(feature = "test-util")] +pub async fn collect_legacy_storage_for_tests( + kv: Arc<PaxosKvClient>, + process: Arc<StreamProcessAuthority>, + allocator: Arc<dyn ChunkAllocator>, + disks: Arc<dyn DiskWriter>, + limit: u32, +) -> Result<LegacyRetirementProgress, CtError> { + Collector { + kv, + store: 0, + group: 1, + process, + allocator, + disks, + } + .step(limit) + .await +} + +/// Run the host's resume-only path without constructing native RPC routes. +/// # Errors +/// Returns the production resume result. +#[cfg(feature = "test-util")] +pub async fn resume_legacy_storage_for_tests( + kv: Arc<PaxosKvClient>, + process: Arc<StreamProcessAuthority>, + allocator: Arc<dyn ChunkAllocator>, + disks: Arc<dyn DiskWriter>, + limit: u32, +) -> Result<Option<LegacyRetirementProgress>, CtError> { + Collector { + kv, + store: 0, + group: 1, + process, + allocator, + disks, + } + .resume(limit) + .await +} diff --git a/app/crowdb-chunk-kv-server/src/storage/legacy_retirement/allocation.rs b/app/crowdb-chunk-kv-server/src/storage/legacy_retirement/allocation.rs new file mode 100644 index 000000000..6fb0f1621 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/legacy_retirement/allocation.rs @@ -0,0 +1,194 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +use super::{record, Collector, LegacyRetirementPending, LegacyRetirementProgress}; +use crowdb_protocol::{ + chunkdb::rpc::{Chunk, ChunkState, DeleteChunkRequest, QueryChunkRequest, Strip, StripType}, + common::ChunkId, +}; +use crowdb_tree_ffi::{chunk::legacy::LegacyManifest, CtError}; +use std::time::{Duration, Instant}; + +impl Collector { + pub(super) async fn retire_key( + &self, + key: &[u8], + state: &mut record::Namespace, + progress: &mut LegacyRetirementProgress, + ) -> Result<bool, CtError> { + let (tree, kind, generation) = record::parse_key(key)?; + if kind != b"legacy-migration" { + return Ok(true); + } + let (bytes, _) = self.get(key).await?.ok_or(CtError::Corruption)?; + let manifest = LegacyManifest::decode(&bytes)?; + if manifest.tree_id() != tree || manifest.generation() != generation { + return Err(CtError::Corruption); + } + let Some(pack) = manifest.packs().nth(state.pack_index) else { + return Ok(true); + }; + let id = ChunkId { + high: pack.chunk_high, + low: pack.chunk_low, + }; + let end = pack + .offset + .checked_add(u64::from(pack.length)) + .ok_or(CtError::Corruption)?; + self.retire_allocation(id, end, state, progress).await?; + if state.pending.is_none() { + state.pack_index += 1; + } + let finished = state.pack_index == manifest.packs().len(); + Ok(finished) + } + async fn query(&self, id: ChunkId) -> Result<Option<Chunk>, CtError> { + match self + .allocator + .query_chunk(QueryChunkRequest { chunk_id: Some(id) }) + .await + { + Ok(response) => response.chunk.map(Some).ok_or(CtError::Corruption), + Err(crowdb_chunk_client::IoError::ChunkNotFound(_)) => Ok(None), + Err(_) => Err(CtError::Unavailable), + } + } + async fn retire_allocation( + &self, + id: ChunkId, + end: u64, + state: &mut record::Namespace, + progress: &mut LegacyRetirementProgress, + ) -> Result<(), CtError> { + let key = record::allocation_key(id.high, id.low); + let (mut receipt, revision) = if let Some((bytes, revision)) = self.get(&key).await? { + (record::decode::<record::AllocationReceipt>(&bytes)?, revision) + } else { + let Some(chunk) = self.query(id).await? else { + state.pending = Some(LegacyRetirementPending::MissingAllocation { + high: id.high, + low: id.low, + }); + return Ok(()); + }; + if !matches_allocation(&chunk, id, end) { + state.pending = Some(LegacyRetirementPending::AllocationChanged { + high: id.high, + low: id.low, + }); + return Ok(()); + } + let receipt = record::AllocationReceipt { + version: 1, + physical_bytes: physical_bytes(&chunk)?, + allocation: chunk, + complete: false, + }; + self.put(&key, &record::encode(&receipt)?, 0).await?; + let (bytes, revision) = self.get(&key).await?.ok_or(CtError::Unavailable)?; + (record::decode::<record::AllocationReceipt>(&bytes)?, revision) + }; + if receipt.version != 1 || receipt.allocation.id != Some(id) { + return Err(CtError::Corruption); + } + if receipt.complete { + return Ok(()); + } + let Some(current) = self.query(id).await? else { + state.pending = Some(LegacyRetirementPending::MissingAllocation { + high: id.high, + low: id.low, + }); + return Ok(()); + }; + if current.id != Some(id) + || current.create_ts_ms != receipt.allocation.create_ts_ms + || current.chunk_type != receipt.allocation.chunk_type + { + state.pending = Some(LegacyRetirementPending::AllocationChanged { + high: id.high, + low: id.low, + }); + return Ok(()); + } + if !physically_deleted(¤t) { + if current.strips != receipt.allocation.strips || !current.cleanup_intents.is_empty() { + state.pending = Some(LegacyRetirementPending::AllocationChanged { + high: id.high, + low: id.low, + }); + return Ok(()); + } + self.drain(&receipt.allocation).await?; + let deleted = self + .allocator + .delete_chunk(DeleteChunkRequest { chunk_id: Some(id) }) + .await + .map_err(|_| CtError::Unavailable)? + .chunk + .ok_or(CtError::Corruption)?; + if deleted.id != Some(id) + || deleted.create_ts_ms != receipt.allocation.create_ts_ms + || !physically_deleted(&deleted) + { + return Err(CtError::Unavailable); + } + } + receipt.complete = true; + self.put(&key, &record::encode(&receipt)?, revision).await?; + progress.reclaimed_chunks += 1; + progress.reclaimed_bytes = progress + .reclaimed_bytes + .checked_add(receipt.physical_bytes) + .ok_or(CtError::Corruption)?; + Ok(()) + } + async fn drain(&self, chunk: &Chunk) -> Result<(), CtError> { + let deadline = Instant::now() + Duration::from_secs(10); + for strip in &chunk.strips { + let Some(Strip::MirrorStrip(mirror)) = &strip.strip else { + return Err(CtError::Corruption); + }; + for segment in &mirror.segments { + self.disks + .drain_segment(segment, u64::from(strip.unit_kb) * 1024, deadline) + .await + .map_err(|_| CtError::Unavailable)?; + } + } + Ok(()) + } +} +fn physically_deleted(chunk: &Chunk) -> bool { + chunk.state == ChunkState::Deleted as i32 + && chunk.strips.is_empty() + && chunk.capacity == 0 + && chunk.cleanup_intents.is_empty() +} +fn matches_allocation(chunk: &Chunk, id: ChunkId, end: u64) -> bool { + chunk.id == Some(id) + && matches!(chunk.chunk_type, 2 | 3) + && u64::try_from(chunk.chunk_type).ok() == Some(id.high >> 56) + && chunk.create_ts_ms != 0 + && chunk.cleanup_intents.is_empty() + && (physically_deleted(chunk) || u64::from(chunk.capacity) >= end) +} +fn physical_bytes(chunk: &Chunk) -> Result<u64, CtError> { + let mut bytes = 0_u64; + for strip in &chunk.strips { + let Some(Strip::MirrorStrip(mirror)) = &strip.strip else { + return Err(CtError::Corruption); + }; + if strip.strip_type != StripType::Mirror as i32 || strip.unit_kb == 0 || mirror.segments.is_empty() { + return Err(CtError::Corruption); + } + for segment in &mirror.segments { + let segment_bytes = u64::from(segment.unit_count) + .checked_mul(u64::from(strip.unit_kb)) + .and_then(|units| units.checked_mul(1024)) + .ok_or(CtError::Corruption)?; + bytes = bytes.checked_add(segment_bytes).ok_or(CtError::Corruption)?; + } + } + Ok(bytes) +} diff --git a/app/crowdb-chunk-kv-server/src/storage/legacy_retirement/proof.rs b/app/crowdb-chunk-kv-server/src/storage/legacy_retirement/proof.rs new file mode 100644 index 000000000..1d915a6f1 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/legacy_retirement/proof.rs @@ -0,0 +1,235 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +use super::{record, Collector, LegacyRetirementPending}; +use bytes::Bytes; +use crowdb_paxos_kv_client::{BatchOp, ReadMode, StreamProcessLeaseClient}; +use crowdb_protocol::chunk_stream::StreamProcessIdentity; +use crowdb_tree_ffi::{chunk::legacy::LegacyManifest, CtError}; +use std::time::{SystemTime, UNIX_EPOCH}; + +impl Collector { + pub(super) async fn prove_key(&self, key: &[u8], state: &mut record::Namespace) -> Result<bool, CtError> { + let (tree, kind, generation) = record::parse_key(key)?; + if kind == b"authority" { + self.freeze_tree(tree, state).await?; + } else if kind == b"manifest" || kind == b"legacy-migration" { + if let Some((bytes, _)) = self.get(key).await? { + if record::is_legacy(&bytes) { + let manifest = LegacyManifest::decode(&bytes)?; + if manifest.tree_id() != tree || manifest.generation() != generation { + return Err(CtError::Corruption); + } + self.save_receipt(tree, generation, bytes.clone()).await?; + if !state.owners_done && !self.prove_pack_owners(manifest, state).await? { + return Ok(false); + } + return self.prove_generation(tree, generation, state).await; + } + } + } + Ok(state.pending.is_none()) + } + + async fn prove_pack_owners( + &self, + manifest: LegacyManifest<'_>, + state: &mut record::Namespace, + ) -> Result<bool, CtError> { + let own_key = record::key(manifest.tree_id(), b"legacy-inventory", 0); + if self + .get(&own_key) + .await? + .map_or(true, |(value, _)| value.as_ref() != b"1") + { + // Still retain the original directory when old publication provenance is absent. + state.pending = Some(LegacyRetirementPending::MissingOwnerProof { + tree: manifest.tree_id(), + generation: manifest.generation(), + }); + return Ok(false); + } + for pack in manifest.packs().skip(state.pack_index).take(64) { + let key = record::key(pack.owner_tree_id, b"legacy-inventory", 0); + if self + .get(&key) + .await? + .map_or(true, |(value, _)| value.as_ref() != b"1") + { + state.pending = Some(LegacyRetirementPending::MissingOwnerProof { + tree: pack.owner_tree_id, + generation: manifest.generation(), + }); + return Ok(false); + } + state.pack_index += 1; + } + state.owners_done = state.pack_index == manifest.packs().len(); + Ok(state.owners_done) + } + + async fn freeze_tree(&self, tree: u64, state: &mut record::Namespace) -> Result<(), CtError> { + let authority_key = record::key(tree, b"authority", 0); + let (authority, revision) = self.get(&authority_key).await?.ok_or(CtError::Corruption)?; + if authority.len() != 16 { + return Err(CtError::Corruption); + } + if let Some((current, _)) = self.get(&record::key(tree, b"current", 0)).await? { + if record::is_legacy(¤t) { + LegacyManifest::decode(¤t)?; + state.pending = Some(LegacyRetirementPending::LegacyRoot { tree }); + return Ok(()); + } + } else if authority[8..] != 0_u64.to_be_bytes() { + return Err(CtError::Corruption); + } + let key = record::key(tree, b"legacy-frozen", 0); + if self.get(&key).await?.is_some() { + return Ok(()); + } + let ops = [ + BatchOp::Put { + key: Bytes::from(authority_key.clone()), + value: authority, + }, + BatchOp::Put { + key: Bytes::from(key), + value: Bytes::from_static(b"1"), + }, + ]; + self.kv + .batch_write_cas(self.store, self.group, &ops, &authority_key, revision) + .await + .map(|_| ()) + .map_err(|_| CtError::Unavailable) + } + + async fn save_receipt(&self, tree: u64, generation: u64, manifest: Bytes) -> Result<(), CtError> { + let key = record::key(tree, b"legacy-migration", generation); + if let Some((old, _)) = self.get(&key).await? { + return if old == manifest { + Ok(()) + } else { + Err(CtError::Corruption) + }; + } + self.put(&key, &manifest, 0).await + } + + async fn prove_generation( + &self, + tree: u64, + generation: u64, + state: &mut record::Namespace, + ) -> Result<bool, CtError> { + if self + .get(&record::key(tree, b"legacy-tracked", generation)) + .await? + .map_or(true, |(bytes, _)| bytes.as_ref() != b"1") + || self + .get(&record::key(tree, b"legacy-untracked", generation)) + .await? + .is_some() + { + state.pending = Some(LegacyRetirementPending::MissingOwnerProof { tree, generation }); + return Ok(false); + } + let Some((current, _)) = self.get(&record::key(tree, b"current", 0)).await? else { + return Err(CtError::Corruption); + }; + record::typed_root(¤t, tree, generation)?; + if !state.pins_done && !self.check_pins(tree, generation, state).await? { + return Ok(false); + } + let mut prefix = record::key(tree, b"legacy-reader", generation); + prefix.push(b'/'); + let owners = self + .kv + .scan( + self.store, + self.group, + &prefix, + &state.proof_after, + &[], + 64, + ReadMode::Linearizable, + None, + false, + None, + ) + .await + .map_err(|_| CtError::Unavailable)?; + if owners.timed_out { + return Err(CtError::Unavailable); + } + if owners.items.is_empty() && state.proof_after.is_empty() { + state.pending = Some(LegacyRetirementPending::MissingOwnerProof { tree, generation }); + return Ok(false); + } + let now = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|_| CtError::Unavailable)?; + let now = u64::try_from(now.as_millis()).map_err(|_| CtError::Unavailable)?; + let leases = StreamProcessLeaseClient::new(self.kv.clone()); + for (key, bytes) in owners.items { + let owner: StreamProcessIdentity = record::decode(&bytes)?; + if leases + .confirm_exclusion(owner, now) + .await + .map_err(|_| CtError::Unavailable)? + .is_none() + { + state.pending = Some(LegacyRetirementPending::LiveProcess { + tree, + generation, + instance: owner.instance_id, + epoch: owner.process_epoch, + }); + return Ok(false); + } + state.proof_after = key.to_vec(); + } + Ok(!owners.truncated) + } + async fn check_pins( + &self, + tree: u64, + generation: u64, + state: &mut record::Namespace, + ) -> Result<bool, CtError> { + let mut prefix = record::key(tree, b"pin", 0); + prefix.truncate(prefix.len() - 8); + let pins = self + .kv + .scan( + self.store, + self.group, + &prefix, + &state.proof_after, + &[], + 64, + ReadMode::Linearizable, + None, + false, + None, + ) + .await + .map_err(|_| CtError::Unavailable)?; + if pins.timed_out { + return Err(CtError::Unavailable); + } + for (key, value) in pins.items { + let pinned = u64::from_be_bytes(value.as_ref().try_into().map_err(|_| CtError::Corruption)?); + if pinned == generation { + state.pending = Some(LegacyRetirementPending::GenerationPin { tree, generation }); + return Ok(false); + } + state.proof_after = key.to_vec(); + } + if pins.truncated { + return Ok(false); + } + state.proof_after.clear(); + state.pins_done = true; + Ok(true) + } +} diff --git a/app/crowdb-chunk-kv-server/src/storage/legacy_retirement/record.rs b/app/crowdb-chunk-kv-server/src/storage/legacy_retirement/record.rs new file mode 100644 index 000000000..8481f07fc --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/legacy_retirement/record.rs @@ -0,0 +1,131 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use super::LegacyRetirementPending; +use crowdb_protocol::chunkdb::rpc::Chunk; +use crowdb_tree_ffi::CtError; +use serde::{Deserialize, Serialize}; + +pub(crate) const NAMESPACE_KEY: &[u8] = b"\0crowdb/chunk-kv/legacy-retirement/v1/state"; +pub(super) const ROOT_PREFIX: &[u8] = b"\0crowdb/chunk-kv/root/v1/"; + +#[derive(Clone, Debug, Serialize, Deserialize)] +pub(super) struct Namespace { + pub version: u32, + pub phase: Phase, + pub after: Vec<u8>, + pub pack_index: usize, + pub proof_after: Vec<u8>, + pub pins_done: bool, + pub owners_done: bool, + pub pending: Option<LegacyRetirementPending>, +} +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub(super) enum Phase { + Proving, + Retiring, + Complete, +} +impl Default for Namespace { + fn default() -> Self { + Self { + version: 1, + phase: Phase::Proving, + after: Vec::new(), + pack_index: 0, + proof_after: Vec::new(), + pins_done: false, + owners_done: false, + pending: None, + } + } +} +#[derive(Clone, Debug, Serialize, Deserialize)] +pub(super) struct AllocationReceipt { + pub version: u32, + pub allocation: Chunk, + pub physical_bytes: u64, + pub complete: bool, +} + +pub(crate) fn is_legacy(bytes: &[u8]) -> bool { + bytes.get(..4) == Some(&7_u32.to_le_bytes()) +} +pub(super) fn key(tree: u64, kind: &[u8], generation: u64) -> Vec<u8> { + let mut key = ROOT_PREFIX.to_vec(); + key.extend_from_slice(&tree.to_be_bytes()); + key.push(b'/'); + key.extend_from_slice(kind); + key.push(b'/'); + key.extend_from_slice(&generation.to_be_bytes()); + key +} +pub(super) fn parse_key(bytes: &[u8]) -> Result<(u64, &[u8], u64), CtError> { + let tail = bytes.strip_prefix(ROOT_PREFIX).ok_or(CtError::Corruption)?; + if tail.len() < 18 || tail[8] != b'/' { + return Err(CtError::Corruption); + } + let end = tail[9..] + .iter() + .position(|byte| *byte == b'/') + .ok_or(CtError::Corruption)? + + 9; + if tail.len() < end + 9 { + return Err(CtError::Corruption); + } + Ok(( + u64::from_be_bytes(tail[..8].try_into().map_err(|_| CtError::Corruption)?), + &tail[9..end], + u64::from_be_bytes( + tail[end + 1..end + 9] + .try_into() + .map_err(|_| CtError::Corruption)?, + ), + )) +} +pub(super) fn allocation_key(high: u64, low: u64) -> Vec<u8> { + let mut key = b"\0crowdb/chunk-kv/legacy-retirement/v1/allocation/".to_vec(); + key.extend_from_slice(&high.to_be_bytes()); + key.extend_from_slice(&low.to_be_bytes()); + key +} +pub(super) fn encode<T: Serialize>(value: &T) -> Result<Vec<u8>, CtError> { + serde_json::to_vec(value).map_err(|_| CtError::Internal) +} +pub(super) fn decode<T: serde::de::DeserializeOwned>(bytes: &[u8]) -> Result<T, CtError> { + serde_json::from_slice(bytes).map_err(|_| CtError::Corruption) +} + +// This checks the immutable format envelope. The production recovery decoder +// additionally validates every directory entry before opening a typed root. +pub(super) fn typed_root(bytes: &[u8], tree: u64, generation: u64) -> Result<(), CtError> { + const HEADER: usize = 128; + const ENTRY: usize = 96; + if bytes.len() < HEADER + 4 { + return Err(CtError::Corruption); + } + let word = |index: usize| { + u64::from_le_bytes( + bytes[index * 8..index * 8 + 8] + .try_into() + .expect("header checked"), + ) + }; + let crc = u32::from_le_bytes( + bytes[bytes.len() - 4..] + .try_into() + .map_err(|_| CtError::Corruption)?, + ); + let entries = usize::try_from(word(13)).map_err(|_| CtError::Corruption)?; + if word(0) != 0x3354_4F4F_524B_4843 + || word(1) != 1 + || word(2) != tree + || word(3) <= generation + || entries != (bytes.len() - HEADER - 4) / ENTRY + || (bytes.len() - HEADER - 4) % ENTRY != 0 + || crowdb_tree_ffi::crc32c(&bytes[..bytes.len() - 4]) != crc + { + return Err(CtError::Corruption); + } + Ok(()) +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog.rs index be867266b..2f6775a38 100644 --- a/app/crowdb-chunk-kv-server/src/storage/root_catalog.rs +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog.rs @@ -7,22 +7,35 @@ use std::collections::HashMap; use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::Arc; -use arc_swap::ArcSwap; +use arc_swap::{ArcSwap, ArcSwapOption}; use bytes::Bytes; -use crowdb_kv_client::{BatchOp, CrowdbKvClient, GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{BatchOp, GetOutcome, PaxosKvClient, ReadMode}; +use crowdb_tree_ffi::chunk::stage::{StageId, StageRecord}; use crowdb_tree_ffi::{RootCatalogObject, RootCatalogStore}; +mod legacy_migration; +mod legacy_tracking; +mod prepared; +mod protection; +mod retention; +mod staging; +mod transaction; + use super::StorageRuntimeError; use crate::{ChunkKvRangeCatalogPublisher, Group0ControlStore}; use crowdb_protocol::chunk_kv::{ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogPartitionState}; pub(super) struct KvRootCatalogStore { - kv: Arc<CrowdbKvClient>, + kv: Arc<PaxosKvClient>, runtime: tokio::runtime::Handle, store_id: u64, group_id: u64, tree_id: u64, owner_epoch: u64, + process: Option<Arc<crowdb_paxos_kv_client::StreamProcessAuthority>>, + writer_incarnation: u64, + writer_quiesced: AtomicBool, + recovered: ArcSwapOption<protection::RecoveryReceipt>, publication_authorized: AtomicBool, prepared_assignment: Option<ChunkKvRangeCatalogEntry>, published_objects: ArcSwap<HashMap<Vec<u8>, Vec<u8>>>, @@ -30,7 +43,7 @@ pub(super) struct KvRootCatalogStore { impl KvRootCatalogStore { pub(super) async fn claim( - kv: Arc<CrowdbKvClient>, + kv: Arc<PaxosKvClient>, store_id: u64, group_id: u64, tree_id: u64, @@ -74,7 +87,9 @@ impl KvRootCatalogStore { .await { Ok(_) => break, - Err(crowdb_kv_client::Error::CasFailed { .. } | crowdb_kv_client::Error::CasBusy) => {} + Err( + crowdb_paxos_kv_client::Error::CasFailed { .. } | crowdb_paxos_kv_client::Error::CasBusy, + ) => {} Err(error) => return Err(StorageRuntimeError::Tree(error.to_string())), } } @@ -85,6 +100,10 @@ impl KvRootCatalogStore { group_id, tree_id, owner_epoch, + process: None, + writer_incarnation: 0, + writer_quiesced: AtomicBool::new(false), + recovered: ArcSwapOption::empty(), publication_authorized: AtomicBool::new(true), prepared_assignment: None, published_objects: ArcSwap::from_pointee(HashMap::new()), @@ -92,7 +111,7 @@ impl KvRootCatalogStore { } pub(super) async fn for_assignment( - kv: Arc<CrowdbKvClient>, + kv: Arc<PaxosKvClient>, store_id: u64, group_id: u64, entry: &ChunkKvRangeCatalogEntry, @@ -112,7 +131,7 @@ impl KvRootCatalogStore { } pub(super) fn prepare( - kv: Arc<CrowdbKvClient>, + kv: Arc<PaxosKvClient>, store_id: u64, group_id: u64, entry: &ChunkKvRangeCatalogEntry, @@ -127,6 +146,10 @@ impl KvRootCatalogStore { group_id, tree_id: entry.artifact.tree_id, owner_epoch: entry.owner_epoch, + process: None, + writer_incarnation: 0, + writer_quiesced: AtomicBool::new(false), + recovered: ArcSwapOption::empty(), publication_authorized: AtomicBool::new(false), prepared_assignment: Some(entry.clone()), published_objects: ArcSwap::from_pointee(HashMap::new()), @@ -136,7 +159,9 @@ impl KvRootCatalogStore { async fn authorize_publication(&self) -> Result<(), crowdb_tree_ffi::CtError> { self.claim_published(false) .await - .map_err(|_| crowdb_tree_ffi::CtError::Unavailable) + .map_err(|_| crowdb_tree_ffi::CtError::Unavailable)?; + self.register_writer().await?; + self.persist_completed_recovery().await } pub(super) async fn claim_published(&self, allow_catching_up: bool) -> Result<(), StorageRuntimeError> { @@ -194,10 +219,14 @@ impl KvRootCatalogStore { } } - async fn get(&self, key: &[u8]) -> Result<Option<(Bytes, u64)>, crowdb_kv_client::Error> { + async fn get(&self, key: &[u8]) -> Result<Option<(Bytes, u64)>, crowdb_paxos_kv_client::Error> { self.kv .get(self.store_id, self.group_id, key, ReadMode::Linearizable, None) .await + .map_err(|error| { + tracing::warn!(tree_id = self.tree_id, error = %error, "tree root catalog read failed"); + error + }) .map(|outcome| match outcome { GetOutcome::Found { value, revision } => Some((value, revision)), GetOutcome::NotFound => None, @@ -214,17 +243,62 @@ impl KvRootCatalogStore { } impl RootCatalogStore for KvRootCatalogStore { + fn prepare_legacy_migration( + &self, + tree: u64, + generation: u64, + manifest: &[u8], + ) -> Result<(), crowdb_tree_ffi::CtError> { + self.retain_legacy_migration(tree, generation, manifest) + } + fn load_legacy_migration( + &self, + tree: u64, + generation: u64, + ) -> Result<Option<Vec<u8>>, crowdb_tree_ffi::CtError> { + self.load_migration_receipt(tree, generation) + } + fn stage_protection(&self) -> Option<&dyn crowdb_tree_ffi::chunk::protection::StageProtectionCatalog> { + Some(self) + } + fn supports_staging(&self) -> bool { + true + } + fn load_stage(&self, tree_id: u64, id: StageId) -> Result<Option<StageRecord>, crowdb_tree_ffi::CtError> { + self.stage_load(tree_id, id) + } + fn compare_stage( + &self, + tree_id: u64, + owner_epoch: u64, + expected: Option<&StageRecord>, + next: Option<&StageRecord>, + ) -> Result<(), crowdb_tree_ffi::CtError> { + self.stage_compare(tree_id, owner_epoch, expected, next) + } + fn list_stages( + &self, + tree_id: u64, + after: Option<StageId>, + limit: u32, + ) -> Result<Vec<StageRecord>, crowdb_tree_ffi::CtError> { + self.stage_list(tree_id, after, limit) + } fn load( &self, tree_id: u64, object: RootCatalogObject, ) -> Result<Option<Vec<u8>>, crowdb_tree_ffi::CtError> { - if tree_id != self.tree_id { + if tree_id != self.tree_id && !matches!(object, RootCatalogObject::ReferenceSegment(_)) { return Err(crowdb_tree_ffi::CtError::InvalidArgument); } let key = object_key(tree_id, object); - if let Some(value) = self.published_objects.load().get(&key) { - return Ok(Some(value.clone())); + // Exact generations can be reclaimed by another catalog instance. + // A cached descriptor must not resurrect an absent durable root. + if matches!(object, RootCatalogObject::ReferenceSegment(_)) { + if let Some(value) = self.published_objects.load().get(&key) { + return Ok(Some(value.clone())); + } } let value = self .wait(self.get(&key)) @@ -242,6 +316,11 @@ impl RootCatalogStore for KvRootCatalogStore { } } } + if let Some((bytes, _)) = &value { + if !matches!(object, RootCatalogObject::ReferenceSegment(_)) { + self.wait(self.track_legacy_reader(bytes))?; + } + } Ok(value.map(|(bytes, _)| bytes.to_vec())) } @@ -282,10 +361,11 @@ impl RootCatalogStore for KvRootCatalogStore { else { return Err(crowdb_tree_ffi::CtError::Unavailable); }; + self.wait(self.check_registered_writer())?; if decode_authority(&authority) != Some((owner_epoch, expected_generation)) { return Err(crowdb_tree_ffi::CtError::Unavailable); } - let ops = [ + let mut ops = vec![ BatchOp::Put { key: Bytes::from(authority_key.clone()), value: Bytes::copy_from_slice(&encode_authority(owner_epoch, generation)), @@ -299,6 +379,7 @@ impl RootCatalogStore for KvRootCatalogStore { value: Bytes::copy_from_slice(manifest), }, ]; + ops.extend(self.wait(self.legacy_publication_ops(generation, manifest))?); self.wait( self.kv .batch_write_cas(self.store_id, self.group_id, &ops, &authority_key, revision), @@ -337,7 +418,9 @@ impl RootCatalogStore for KvRootCatalogStore { revision, )) { Ok(_) => return Ok(current), - Err(crowdb_kv_client::Error::CasFailed { .. } | crowdb_kv_client::Error::CasBusy) => {} + Err( + crowdb_paxos_kv_client::Error::CasFailed { .. } | crowdb_paxos_kv_client::Error::CasBusy, + ) => {} Err(_) => return Err(crowdb_tree_ffi::CtError::Unavailable), } } @@ -358,67 +441,8 @@ impl RootCatalogStore for KvRootCatalogStore { } fn reclaim_before(&self, tree_id: u64, generation: u64) -> u64 { - if tree_id != self.tree_id || generation <= 2 { - return 0; - } - let pin_prefix = catalog_pin_prefix(tree_id); - let pinned_generation = match self.wait(self.kv.scan( - self.store_id, - self.group_id, - &pin_prefix, - &[], - &[], - 0, - ReadMode::Linearizable, - None, - false, - None, - )) { - Ok(outcome) => { - let mut oldest = None; - for (_, value) in outcome.items { - let Some(pin) = decode_u64(&value) else { - return 0; - }; - oldest = Some(oldest.map_or(pin, |current: u64| current.min(pin))); - } - oldest - } - Err(_) => return 0, - }; - let generation = pinned_generation.map_or(generation, |pin| generation.min(pin)); - if generation <= 2 { - return 0; - } - let floor_key = catalog_key(tree_id, b"reclaim-floor", 0); - let (floor, revision) = match self.wait(self.get(&floor_key)) { - Ok(Some((value, revision))) => match decode_u64(&value) { - Some(floor) => (floor, revision), - None => return 0, - }, - Ok(None) => (1, 0), - Err(_) => return 0, - }; - let end = generation.saturating_sub(1).min(floor.saturating_add(128)); - if floor >= end { - return 0; - } - let mut ops = (floor..end) - .map(|old_generation| BatchOp::Delete { - key: Bytes::from(catalog_key(tree_id, b"manifest", old_generation)), - }) - .collect::<Vec<_>>(); - ops.push(BatchOp::Put { - key: Bytes::from(floor_key.clone()), - value: Bytes::copy_from_slice(&end.to_be_bytes()), - }); - self.wait( - self.kv - .batch_write_cas(self.store_id, self.group_id, &ops, &floor_key, revision), - ) - .map_or(0, |_| end - floor) + self.reclaim_generations(tree_id, generation) } - fn pin_generation( &self, tree_id: u64, @@ -426,53 +450,15 @@ impl RootCatalogStore for KvRootCatalogStore { transition_low: u64, generation: u64, ) -> Result<(), crowdb_tree_ffi::CtError> { - if tree_id != self.tree_id || (transition_high == 0 && transition_low == 0) || generation == 0 { - return Err(crowdb_tree_ffi::CtError::InvalidArgument); - } - if self - .wait(self.get(&catalog_key(tree_id, b"manifest", generation))) - .map_err(|_| crowdb_tree_ffi::CtError::Unavailable)? - .is_none() - { - return Err(crowdb_tree_ffi::CtError::NotFound); - } - let key = catalog_pin_key(tree_id, transition_high, transition_low); - for _ in 0..3 { - if let Some((value, _)) = self - .wait(self.get(&key)) - .map_err(|_| crowdb_tree_ffi::CtError::Unavailable)? - { - return if decode_u64(&value) == Some(generation) { - Ok(()) - } else { - Err(crowdb_tree_ffi::CtError::InvalidArgument) - }; - } - match self.wait( - self.kv - .put_cas(self.store_id, self.group_id, &key, &generation.to_be_bytes(), 0), - ) { - Ok(_) => return Ok(()), - Err(crowdb_kv_client::Error::CasFailed { .. } | crowdb_kv_client::Error::CasBusy) => {} - Err(_) => return Err(crowdb_tree_ffi::CtError::Unavailable), - } - } - Err(crowdb_tree_ffi::CtError::Unavailable) + self.retain_generation(tree_id, transition_high, transition_low, generation) } - fn unpin_generation( &self, tree_id: u64, transition_high: u64, transition_low: u64, ) -> Result<(), crowdb_tree_ffi::CtError> { - if tree_id != self.tree_id || (transition_high == 0 && transition_low == 0) { - return Err(crowdb_tree_ffi::CtError::InvalidArgument); - } - let key = catalog_pin_key(tree_id, transition_high, transition_low); - self.wait(self.kv.delete(self.store_id, self.group_id, &key, None)) - .map(|_| ()) - .map_err(|_| crowdb_tree_ffi::CtError::Unavailable) + self.release_generation(tree_id, transition_high, transition_low) } } diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/legacy_migration.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/legacy_migration.rs new file mode 100644 index 000000000..7ce29d486 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/legacy_migration.rs @@ -0,0 +1,89 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Immutable pre-CAS migration receipts also retain the legacy pack directory. +use super::{catalog_key, decode_authority, KvRootCatalogStore}; +use bytes::Bytes; +use crowdb_paxos_kv_client::BatchOp; +use crowdb_tree_ffi::{chunk::legacy::LegacyManifest, CtError}; + +impl KvRootCatalogStore { + pub(super) fn retain_legacy_migration( + &self, + tree: u64, + generation: u64, + manifest: &[u8], + ) -> Result<(), CtError> { + let original = LegacyManifest::decode(manifest)?; + if tree != self.tree_id + || original.tree_id() != tree + || original.generation() != generation + || original.owner_epoch() > self.owner_epoch + { + return Err(CtError::InvalidArgument); + } + self.wait(self.authority_mutation(|| async { + let authority_key = catalog_key(tree, b"authority", 0); + let (authority, revision) = self + .get(&authority_key) + .await + .map_err(|_| CtError::Unavailable)? + .ok_or(CtError::Unavailable)?; + self.check_registered_writer().await?; + let (epoch, current_generation) = decode_authority(&authority).ok_or(CtError::Corruption)?; + if epoch != self.owner_epoch || current_generation < generation { + return Err(CtError::Unavailable.into()); + } + let key = catalog_key(tree, b"legacy-migration", generation); + if let Some((existing, _)) = self.get(&key).await.map_err(|_| CtError::Unavailable)? { + return if existing.as_ref() == manifest { + Ok(()) + } else { + Err(CtError::Corruption.into()) + }; + } + let current = self + .get(&catalog_key(tree, b"current", 0)) + .await + .map_err(|_| CtError::Unavailable)? + .ok_or(CtError::Unavailable)? + .0; + if current_generation != generation || current.as_ref() != manifest { + return Err(CtError::Unavailable.into()); + } + let operations = [ + BatchOp::Put { + key: Bytes::from(authority_key.clone()), + value: authority, + }, + BatchOp::Put { + key: Bytes::from(key), + value: Bytes::copy_from_slice(manifest), + }, + ]; + self.kv + .batch_write_cas( + self.store_id, + self.group_id, + &operations, + &authority_key, + revision, + ) + .await + .map(|_| ()) + .map_err(|error| super::transaction::MutationError::from_cas(&error)) + })) + } + pub(super) fn load_migration_receipt( + &self, + tree: u64, + generation: u64, + ) -> Result<Option<Vec<u8>>, CtError> { + if tree != self.tree_id || generation == 0 { + return Err(CtError::InvalidArgument); + } + self.wait(self.get(&catalog_key(tree, b"legacy-migration", generation))) + .map(|value| value.map(|(bytes, _)| bytes.to_vec())) + .map_err(|_| CtError::Unavailable) + } +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/legacy_tracking.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/legacy_tracking.rs new file mode 100644 index 000000000..4c47f1b10 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/legacy_tracking.rs @@ -0,0 +1,134 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Generation provenance is committed with publication; late readers share its authority. +use super::{catalog_key, KvRootCatalogStore}; +use crate::storage::legacy_retirement::record::{is_legacy, NAMESPACE_KEY}; +use bytes::Bytes; +use crowdb_paxos_kv_client::BatchOp; +use crowdb_tree_ffi::{chunk::legacy::LegacyManifest, CtError}; + +impl KvRootCatalogStore { + fn legacy_owner_op(&self, generation: u64) -> Result<BatchOp, CtError> { + let Some(owner) = self.process_identity()? else { + return Ok(BatchOp::Put { + key: Bytes::from(catalog_key(self.tree_id, b"legacy-untracked", generation)), + value: Bytes::from_static(b"1"), + }); + }; + self.check_process_owner()?; + let mut key = catalog_key(self.tree_id, b"legacy-reader", generation); + key.push(b'/'); + key.extend_from_slice(&owner.instance_id.to_be_bytes()); + key.extend_from_slice(&owner.process_epoch.to_be_bytes()); + key.extend_from_slice(&owner.boot_nonce); + Ok(BatchOp::Put { + key: Bytes::from(key), + value: Bytes::from(serde_json::to_vec(&owner).map_err(|_| CtError::Internal)?), + }) + } + + pub(super) async fn legacy_publication_ops( + &self, + generation: u64, + bytes: &[u8], + ) -> Result<Vec<BatchOp>, CtError> { + if !is_legacy(bytes) { + return Ok(Vec::new()); + } + let manifest = LegacyManifest::decode(bytes)?; + if manifest.tree_id() != self.tree_id || manifest.generation() != generation { + return Err(CtError::InvalidArgument); + } + if self + .get(NAMESPACE_KEY) + .await + .map_err(|_| CtError::Unavailable)? + .is_some() + { + return Err(CtError::NotSupported); + } + let mut ops = vec![ + BatchOp::Put { + key: Bytes::from(catalog_key(self.tree_id, b"legacy-tracked", generation)), + value: Bytes::from_static(b"1"), + }, + self.legacy_owner_op(generation)?, + ]; + // Only the first publication can prove this lineage has no untracked history. + if generation == 1 && self.process_identity()?.is_some() { + ops.push(BatchOp::Put { + key: Bytes::from(catalog_key(self.tree_id, b"legacy-inventory", 0)), + value: Bytes::from_static(b"1"), + }); + } + Ok(ops) + } + + pub(super) async fn track_legacy_reader(&self, bytes: &[u8]) -> Result<(), CtError> { + if !is_legacy(bytes) { + return Ok(()); + } + let generation = LegacyManifest::decode(bytes)?.generation(); + self.authority_mutation(|| async { + let authority = self.protection_authority(false).await?; + if self + .get(&catalog_key(self.tree_id, b"legacy-frozen", 0)) + .await + .map_err(|_| CtError::Unavailable)? + .is_some() + { + return Err(CtError::Unavailable.into()); + } + let op = self.legacy_owner_op(generation)?; + self.protection_write(authority, vec![op]).await + }) + .await + } + + pub(super) async fn check_legacy_pin_admission(&self, generation: u64) -> Result<(), CtError> { + let Some((manifest, _)) = self + .get(&catalog_key(self.tree_id, b"manifest", generation)) + .await + .map_err(|_| CtError::Unavailable)? + else { + return Ok(()); + }; + if is_legacy(&manifest) + && self + .get(&catalog_key(self.tree_id, b"legacy-frozen", 0)) + .await + .map_err(|_| CtError::Unavailable)? + .is_some() + { + return Err(CtError::Unavailable); + } + Ok(()) + } + + pub(super) async fn legacy_history_ops(&self, generation: u64) -> Result<Vec<BatchOp>, CtError> { + let Some((manifest, _)) = self + .get(&catalog_key(self.tree_id, b"manifest", generation)) + .await + .map_err(|_| CtError::Unavailable)? + else { + return Ok(Vec::new()); + }; + if !is_legacy(&manifest) { + return Ok(Vec::new()); + } + LegacyManifest::decode(&manifest)?; + let key = catalog_key(self.tree_id, b"legacy-migration", generation); + if let Some((old, _)) = self.get(&key).await.map_err(|_| CtError::Unavailable)? { + return if old == manifest { + Ok(Vec::new()) + } else { + Err(CtError::Corruption) + }; + } + Ok(vec![BatchOp::Put { + key: Bytes::from(key), + value: manifest, + }]) + } +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/prepared.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/prepared.rs new file mode 100644 index 000000000..cd39eec2e --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/prepared.rs @@ -0,0 +1,84 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Preparation can stage private packs while the published source still writes. + +use super::{same_recovery_base, KvRootCatalogStore}; +use crate::Group0ControlStore; +use crowdb_protocol::chunk_kv::{ChunkKvRangeCatalogEntry, TransferPhase}; +use crowdb_tree_ffi::CtError; +use std::sync::atomic::Ordering; + +impl KvRootCatalogStore { + pub(super) async fn check_prepared_staging(&self, authority_epoch: u64) -> Result<(), CtError> { + self.check_process_owner()?; + let prepared = self.prepared_assignment.as_ref().ok_or(CtError::Unavailable)?; + let identity = self + .process + .as_ref() + .ok_or(CtError::Unavailable)? + .status(std::time::Instant::now()) + .lease + .identity; + let id = prepared.transition_id.ok_or(CtError::Unavailable)?; + let control = Group0ControlStore::from_client(self.kv.clone()); + let (transition, _) = control + .load_transfer_transition(id) + .await + .map_err(|_| CtError::Unavailable)? + .ok_or(CtError::Unavailable)?; + let allowed = matches!( + transition.phase, + TransferPhase::TargetPreparing + | TransferPhase::TargetPrepared + | TransferPhase::AwaitingFence + | TransferPhase::TargetCatchingUp + | TransferPhase::CatchupPublished + | TransferPhase::TargetReady + ); + let current = ChunkKvRangeCatalogEntry { + partition_id: transition.partition_id, + range: transition.range, + owner: transition.target, + owner_epoch: transition.target_epoch, + state: prepared.state, + artifact: transition.target_artifact, + transition_id: Some(transition.transition_id), + }; + if !allowed + || identity.instance_id != current.owner.instance_id + // A retained split keeps its Tree writer as partition epochs advance. + || (authority_epoch != self.owner_epoch + && (authority_epoch == 0 || authority_epoch > transition.source_epoch)) + || current.partition_id != prepared.partition_id + || current.range != prepared.range + || current.owner != prepared.owner + || current.owner_epoch != self.owner_epoch + || current.artifact.tree_id != self.tree_id + || current.artifact.stream_name != prepared.artifact.stream_name + || !same_recovery_base(¤t, prepared) + { + tracing::warn!(tree_id = self.tree_id, phase = ?transition.phase, authority_epoch, + owner_epoch = self.owner_epoch, process_owner = identity.instance_id, + target_owner = current.owner.instance_id, same_base = same_recovery_base(¤t, prepared), + "prepared tree staging no longer matches its transfer intent"); + return Err(CtError::Unavailable); + } + Ok(()) + } + + pub(super) async fn authorize_staging(&self) -> Result<(), CtError> { + if self.publication_authorized.load(Ordering::Acquire) { + return self.authorize_publication().await; + } + // Only the existing exact published-owner proof may advance root epoch. + // Otherwise a current durable preparation intent permits private packs. + if self.claim_published(false).await.is_ok() { + self.register_writer().await?; + return self.persist_completed_recovery().await; + } + let authority = self.protection_authority(false).await?; + self.check_prepared_staging(authority.epoch).await?; + self.register_writer().await + } +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection.rs new file mode 100644 index 000000000..4fa0071bd --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection.rs @@ -0,0 +1,186 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Reference intents, live pins and GC use one root authority revision. + +use bytes::Bytes; +use crowdb_paxos_kv_client::BatchOp; +use crowdb_tree_ffi::chunk::protection::{ + validate_operation, CandidateStatus, ManifestCandidate, StageProtectionCatalog, +}; +use crowdb_tree_ffi::chunk::stage::StageId; +use crowdb_tree_ffi::CtError; + +use super::{catalog_key, decode_authority, KvRootCatalogStore}; +mod candidate; +mod commit; +mod guards; +mod operation; +mod pin_owner; +mod process; +mod prune; +mod record; +mod release; +mod scan; +mod writer; +pub(super) use process::RecoveryReceipt; +use record::{operation_key, CandidateRecord}; + +// Native metadata proposals reserve progress for at most 64 distinct keys. +pub(super) const MAX_METADATA_MUTATIONS: usize = 64; + +pub(super) struct Authority { + key: Vec<u8>, + value: Bytes, + revision: u64, + pub(super) epoch: u64, + generation: u64, +} + +impl KvRootCatalogStore { + pub(super) async fn protection_authority(&self, require_owner: bool) -> Result<Authority, CtError> { + let key = catalog_key(self.tree_id, b"authority", 0); + let (value, revision) = self + .get(&key) + .await + .map_err(|_| CtError::Unavailable)? + .ok_or(CtError::Unavailable)?; + let (epoch, generation) = decode_authority(&value).ok_or(CtError::Corruption)?; + if require_owner { + self.check_registered_writer().await?; + } + if require_owner && epoch != self.owner_epoch { + return Err(CtError::Unavailable); + } + Ok(Authority { + key, + value, + revision, + epoch, + generation, + }) + } + + pub(super) async fn protection_write( + &self, + authority: Authority, + mut ops: Vec<BatchOp>, + ) -> super::transaction::MutationResult<()> { + ops.push(BatchOp::Put { + key: Bytes::from(authority.key.clone()), + value: authority.value, + }); + self.kv + .batch_write_cas( + self.store_id, + self.group_id, + &ops, + &authority.key, + authority.revision, + ) + .await + .map(|_| ()) + .map_err(|error| super::transaction::MutationError::from_cas(&error)) + } + + fn protection_identity(&self, tree_id: u64, id: StageId, owner: bool) -> Result<(), CtError> { + validate_operation(id)?; + if tree_id != self.tree_id { + return Err(CtError::InvalidArgument); + } + if owner && id.owner_epoch != self.owner_epoch { + return Err(CtError::Unavailable); + } + Ok(()) + } + + async fn candidate_record(&self, id: StageId) -> Result<Option<CandidateRecord>, CtError> { + let key = operation_key(self.tree_id, b"candidate", id); + self.get(&key) + .await + .map_err(|_| CtError::Unavailable)? + .map(|(bytes, _)| CandidateRecord::decode(id, &bytes)) + .transpose() + } +} + +impl StageProtectionCatalog for KvRootCatalogStore { + fn recover_manifest_candidate(&self, tree: u64, id: StageId) -> Result<(), CtError> { + self.protection_identity(tree, id, false)?; + self.wait(self.recover_candidate(id)) + } + fn writer_quiesced(&self, tree: u64, epoch: u64) -> Result<(), CtError> { + if tree != self.tree_id || epoch != self.owner_epoch { + return Err(CtError::InvalidArgument); + } + self.wait(self.publish_writer_quiescence()) + } + fn check_read_owner(&self) -> Result<(), CtError> { + self.check_process_owner() + } + fn prune_stage_pins( + &self, + tree: u64, + stage: StageId, + after: Option<StageId>, + limit: u32, + ) -> Result<crowdb_tree_ffi::chunk::protection::PinPruneProgress, CtError> { + self.protection_identity(tree, stage, false)?; + self.wait(self.prune_pins(stage, after, limit)) + } + + fn load_manifest_candidate(&self, tree: u64, id: StageId) -> Result<Option<CandidateStatus>, CtError> { + self.protection_identity(tree, id, false)?; + self.wait(self.candidate_record(id)) + .map(|record| record.map(|record| record.status)) + } + fn list_manifest_candidates( + &self, + tree: u64, + after: Option<StageId>, + limit: u32, + ) -> Result<Vec<CandidateStatus>, CtError> { + if tree != self.tree_id { + return Err(CtError::InvalidArgument); + } + self.wait(self.scan_candidates(after, limit)) + } + fn begin_manifest_candidate(&self, tree: u64, candidate: &ManifestCandidate) -> Result<(), CtError> { + candidate.validate()?; + self.protection_identity(tree, candidate.id, true)?; + self.wait(self.authorize_publication())?; + self.wait(self.begin_candidate(candidate)) + } + fn prepare_manifest_references( + &self, + tree: u64, + id: StageId, + window: u64, + refs: &[StageId], + ) -> Result<(), CtError> { + self.protection_identity(tree, id, true)?; + self.wait(self.prepare_candidate(id, window, refs)) + } + fn commit_manifest_candidate(&self, tree: u64, id: StageId) -> Result<(), CtError> { + self.protection_identity(tree, id, true)?; + self.wait(self.commit_candidate(id)) + } + fn abort_manifest_candidate(&self, tree: u64, id: StageId) -> Result<(), CtError> { + self.protection_identity(tree, id, true)?; + self.wait(self.abort_candidate(id)) + } + fn release_manifest_references(&self, tree: u64, id: StageId, window: u64) -> Result<(), CtError> { + self.protection_identity(tree, id, false)?; + self.wait(self.release_candidate(id, window)) + } + fn pin_stage(&self, tree: u64, stage: StageId, pin: StageId) -> Result<(), CtError> { + self.protection_identity(tree, stage, false)?; + self.protection_identity(tree, pin, true)?; + self.wait(self.change_stage_pin(stage, pin, true)) + } + fn unpin_stage(&self, tree: u64, stage: StageId, pin: StageId) -> Result<(), CtError> { + self.protection_identity(tree, stage, false)?; + self.protection_identity(tree, pin, true)?; + self.wait(self.change_stage_pin(stage, pin, false)) + } +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/candidate.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/candidate.rs new file mode 100644 index 000000000..8be2c423e --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/candidate.rs @@ -0,0 +1,158 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use super::record::{encode_references, operation_key, window_key, CandidateRecord}; +use super::KvRootCatalogStore; +use bytes::Bytes; +use crowdb_paxos_kv_client::BatchOp; +use crowdb_tree_ffi::chunk::protection::{ + reference_checksum, validate_operation, CandidateState, CandidateStatus, ManifestCandidate, + MAX_REFERENCE_WINDOW, +}; +use crowdb_tree_ffi::chunk::stage::StageId; +use crowdb_tree_ffi::{crc32c, CtError}; + +impl KvRootCatalogStore { + pub(super) async fn begin_candidate(&self, candidate: &ManifestCandidate) -> Result<(), CtError> { + self.authority_mutation(|| async { + let authority = self.protection_authority(true).await?; + let manifest_key = operation_key(self.tree_id, b"candidate-manifest", candidate.id); + if let Some(current) = self.candidate_record(candidate.id).await? { + let manifest = self.get(&manifest_key).await.map_err(|_| CtError::Unavailable)?; + if current.status.generation == candidate.expected_generation + 1 + && current.status.reference_count == candidate.reference_count + && current.expected_checksum == candidate.reference_checksum + && current.manifest_bytes == candidate.manifest.len() as u64 + && current.manifest_checksum == crc32c(&candidate.manifest) + && (manifest.as_ref().map(|(bytes, _)| bytes.as_ref()) + == Some(candidate.manifest.as_slice()) + || (manifest.is_none() + && matches!( + current.status.state, + CandidateState::Aborted | CandidateState::Reclaimed + ))) + { + return Ok(()); + } + return Err(CtError::InvalidArgument.into()); + } + if authority.generation != candidate.expected_generation { + return Err(CtError::Unavailable.into()); + } + let record = CandidateRecord { + status: CandidateStatus { + id: candidate.id, + state: if candidate.reference_count == 0 { + CandidateState::Ready + } else { + CandidateState::Preparing + }, + generation: candidate.expected_generation + 1, + reference_count: candidate.reference_count, + prepared_references: 0, + prepared_windows: 0, + released_windows: 0, + }, + expected_checksum: candidate.reference_checksum, + prepared_checksum: 0, + manifest_checksum: crc32c(&candidate.manifest), + manifest_bytes: candidate.manifest.len() as u64, + last_stage: StageId::default(), + }; + let mut ops = vec![ + BatchOp::Put { + key: Bytes::from(operation_key(self.tree_id, b"candidate", candidate.id)), + value: Bytes::from(record.encode()), + }, + BatchOp::Put { + key: Bytes::from(manifest_key), + value: Bytes::copy_from_slice(&candidate.manifest), + }, + ]; + if let Some(owner) = self.writer_owner()? { + ops.push(BatchOp::Put { + key: Bytes::from(operation_key(self.tree_id, b"candidate-owner", candidate.id)), + value: Bytes::from(owner.encode()), + }); + } + self.protection_write(authority, ops).await + }) + .await + } + + pub(super) async fn prepare_candidate( + &self, + id: StageId, + window: u64, + refs: &[StageId], + ) -> Result<(), CtError> { + self.authority_mutation(|| async { + if refs.is_empty() + || refs.len() > MAX_REFERENCE_WINDOW + || refs.windows(2).any(|pair| pair[0] >= pair[1]) + { + return Err(CtError::InvalidArgument.into()); + } + for reference in refs { + validate_operation(*reference)?; + } + let authority = self.protection_authority(true).await?; + let key = window_key(self.tree_id, id, window); + let encoded = encode_references(refs); + if let Some((current, _)) = self.get(&key).await.map_err(|_| CtError::Unavailable)? { + return if current.as_ref() == encoded { + Ok(()) + } else { + Err(CtError::InvalidArgument.into()) + }; + } + let mut record = self.candidate_record(id).await?.ok_or(CtError::NotFound)?; + if record.status.state != CandidateState::Preparing + || window != record.status.prepared_windows + || refs[0] <= record.last_stage + || authority.generation.checked_add(1) != Some(record.status.generation) + { + return Err(CtError::Unavailable.into()); + } + let total = record + .status + .prepared_references + .checked_add(refs.len() as u64) + .ok_or(CtError::ResourceExhausted)?; + if total > record.status.reference_count { + return Err(CtError::InvalidArgument.into()); + } + let checksum = reference_checksum(record.prepared_checksum, refs); + if total == record.status.reference_count && checksum != record.expected_checksum { + return Err(CtError::Corruption.into()); + } + let mut ops = Vec::with_capacity(refs.len() + 3); + for reference in refs { + self.readable_stage(*reference).await?; + let mut guard = self.load_stage_guard(*reference).await?; + guard.references = guard + .references + .checked_add(1) + .ok_or(CtError::ResourceExhausted)?; + ops.push(self.stage_guard_put(*reference, guard)); + } + record.status.prepared_references = total; + record.status.prepared_windows += 1; + record.prepared_checksum = checksum; + record.last_stage = *refs.last().ok_or(CtError::InvalidArgument)?; + if total == record.status.reference_count { + record.status.state = CandidateState::Ready; + } + ops.push(BatchOp::Put { + key: Bytes::from(key), + value: Bytes::from(encoded), + }); + ops.push(BatchOp::Put { + key: Bytes::from(operation_key(self.tree_id, b"candidate", id)), + value: Bytes::from(record.encode()), + }); + self.protection_write(authority, ops).await + }) + .await + } +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/commit.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/commit.rs new file mode 100644 index 000000000..d5cbd2c37 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/commit.rs @@ -0,0 +1,91 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use super::record::{encode_words, operation_key}; +use super::{catalog_key, KvRootCatalogStore}; +use bytes::Bytes; +use crowdb_paxos_kv_client::BatchOp; +use crowdb_tree_ffi::chunk::protection::CandidateState; +use crowdb_tree_ffi::chunk::stage::StageId; +use crowdb_tree_ffi::{crc32c, CtError}; + +impl KvRootCatalogStore { + pub(super) async fn commit_candidate(&self, id: StageId) -> Result<(), CtError> { + self.authority_mutation(|| async { + let mut authority = self.protection_authority(true).await?; + let mut record = self.candidate_record(id).await?.ok_or(CtError::NotFound)?; + if record.status.state == CandidateState::Committed { + return Ok(()); + } + if record.status.state != CandidateState::Ready + || authority.generation.checked_add(1) != Some(record.status.generation) + { + return Err(CtError::Unavailable.into()); + } + let (manifest, _) = self + .get(&operation_key(self.tree_id, b"candidate-manifest", id)) + .await + .map_err(|_| CtError::Unavailable)? + .ok_or(CtError::Corruption)?; + if manifest.len() as u64 != record.manifest_bytes || crc32c(&manifest) != record.manifest_checksum + { + return Err(CtError::Corruption.into()); + } + record.status.state = CandidateState::Committed; + authority.value = Bytes::copy_from_slice(&super::super::encode_authority( + authority.epoch, + record.status.generation, + )); + let ops = vec![ + BatchOp::Put { + key: Bytes::from(catalog_key(self.tree_id, b"current", 0)), + value: manifest.clone(), + }, + BatchOp::Put { + key: Bytes::from(catalog_key(self.tree_id, b"manifest", record.status.generation)), + value: manifest, + }, + BatchOp::Put { + key: Bytes::from(catalog_key( + self.tree_id, + b"manifest-candidate", + record.status.generation, + )), + value: Bytes::from(encode_words(&[ + id.owner_epoch, + id.operation_high, + id.operation_low, + ])), + }, + BatchOp::Put { + key: Bytes::from(operation_key(self.tree_id, b"candidate", id)), + value: Bytes::from(record.encode()), + }, + ]; + // Intents already own every reference. This CAS changes only the root + // and candidate state, so publication never creates an unprotected gap. + self.protection_write(authority, ops).await + }) + .await + } + + pub(super) async fn abort_candidate(&self, id: StageId) -> Result<(), CtError> { + self.authority_mutation(|| async { + let authority = self.protection_authority(true).await?; + let mut record = self.candidate_record(id).await?.ok_or(CtError::NotFound)?; + if record.status.state == CandidateState::Aborted { + return Ok(()); + } + if !matches!( + record.status.state, + CandidateState::Preparing | CandidateState::Ready + ) { + return Err(CtError::Unavailable.into()); + } + record.status.state = CandidateState::Aborted; + self.protection_write(authority, self.terminal_candidate_ops(&record)) + .await + }) + .await + } +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/guards.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/guards.rs new file mode 100644 index 000000000..e9106d609 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/guards.rs @@ -0,0 +1,260 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use super::record::{decode_words, encode_words, operation_key}; +use super::writer::WriterOwner; +use super::KvRootCatalogStore; +use bytes::Bytes; +use crowdb_paxos_kv_client::BatchOp; +use crowdb_tree_ffi::chunk::stage::{StageId, StageRecord, StageState}; +use crowdb_tree_ffi::CtError; + +#[derive(Clone, Copy, Default)] +pub(super) struct StageGuard { + pub references: u64, + pub pins: u64, + pub creator: Option<WriterOwner>, +} + +impl KvRootCatalogStore { + pub(super) async fn load_stage_guard(&self, id: StageId) -> Result<StageGuard, CtError> { + let key = operation_key(self.tree_id, b"stage-guard", id); + let (value, _) = self + .get(&key) + .await + .map_err(|_| CtError::Unavailable)? + .ok_or(CtError::Unavailable)?; + if value.len() == 28 { + let words = decode_words::<3>(&value)?; + if words[0] != 1 { + return Err(CtError::Corruption); + } + return Ok(StageGuard { + references: words[1], + pins: words[2], + creator: None, + }); + } + let words = decode_words::<12>(&value)?; + if words[0] != 2 || words[3] > 1 { + return Err(CtError::Corruption); + } + let creator = if words[3] == 0 { + if words[4..] != [0; 8] { + return Err(CtError::Corruption); + } + None + } else { + Some(WriterOwner::decode(&encode_words(&words[4..]))?) + }; + Ok(StageGuard { + references: words[1], + pins: words[2], + creator, + }) + } + pub(super) fn stage_guard_put(&self, id: StageId, guard: StageGuard) -> BatchOp { + let mut words = [0; 12]; + words[..3].copy_from_slice(&[2, guard.references, guard.pins]); + if let Some(owner) = guard.creator { + words[3] = 1; + words[4..].copy_from_slice(&decode_words::<8>(&owner.encode()).expect("valid writer encoding")); + } + BatchOp::Put { + key: Bytes::from(operation_key(self.tree_id, b"stage-guard", id)), + value: Bytes::from(encode_words(&words)), + } + } + pub(super) async fn readable_stage(&self, id: StageId) -> Result<StageRecord, CtError> { + let key = super::super::staging::stage_key(self.tree_id, id); + let (value, _) = self + .get(&key) + .await + .map_err(|_| CtError::Unavailable)? + .ok_or(CtError::NotFound)?; + let record = StageRecord::decode(&value)?; + if record.id != id || record.tree_id != self.tree_id { + return Err(CtError::Corruption); + } + if !matches!(record.state, StageState::Reloadable | StageState::Referenced) { + return Err(CtError::Unavailable); + } + Ok(record) + } + + pub(in super::super) async fn stage_guard_transition( + &self, + record: &StageRecord, + creating: bool, + next: Option<&StageRecord>, + ) -> Result<Vec<BatchOp>, CtError> { + let mut key = super::catalog_key(self.tree_id, b"chunk-stage", record.chunk_high); + key.extend_from_slice(&record.chunk_low.to_be_bytes()); + let current = self.get(&key).await.map_err(|_| CtError::Unavailable)?; + let identity = encode_words(&[ + record.id.owner_epoch, + record.id.operation_high, + record.id.operation_low, + ]); + if creating { + // The identity tombstone remains after stage cleanup. A fixed ChunkId + // cannot be allocated a second time under a different operation. + if current.is_some() { + return Err(CtError::Unavailable); + } + let creator = self.writer_owner()?; + let mut ops = vec![ + BatchOp::Put { + key: Bytes::from(key), + value: Bytes::from(identity), + }, + self.stage_guard_put( + record.id, + StageGuard { + creator, + pins: u64::from(creator.is_some()), + ..StageGuard::default() + }, + ), + ]; + if let Some(owner) = creator { + // Protect the output before Reloadable publication. The native + // operation transfers this exact pin to its returned backing. + let mut pin_key = operation_key(self.tree_id, b"stage-pin", record.id); + for part in [ + record.id.owner_epoch, + record.id.operation_high, + record.id.operation_low, + ] { + pin_key.extend_from_slice(&part.to_be_bytes()); + } + // A release that won before an unknown create reply leaves a + // tombstone under this same authority revision. Never resurrect it. + if self + .get(&pin_key) + .await + .map_err(|_| CtError::Unavailable)? + .is_some() + { + return Err(CtError::Unavailable); + } + let pin = super::pin_owner::PinOwner { + live: true, + process: Some(owner.process), + }; + ops.push(BatchOp::Put { + key: Bytes::from(pin_key), + value: Bytes::from(pin.encode()), + }); + } + return Ok(ops); + } + if current.as_ref().map(|(bytes, _)| bytes.as_ref()) != Some(identity.as_slice()) { + return Err(CtError::Unavailable); + } + if next.map_or(true, |next| { + matches!( + next.state, + StageState::Retired | StageState::Deleting | StageState::Reclaimed + ) + }) { + let guard = self.load_stage_guard(record.id).await?; + if let Some(creator) = guard.creator { + // Sealed packs have no outstanding disk writes. Unsealed packs + // remain protected until their exact writer has stopped. + if matches!(record.state, StageState::Reserved | StageState::Writing) { + self.require_completed_recovery().await?; + if !self.operation_is_excluded(record.id, creator).await? { + return Err(CtError::Unavailable); + } + } + } else if record.id.owner_epoch != self.owner_epoch + && matches!(record.state, StageState::Reserved | StageState::Writing) + { + // Old unsealed records have no durable creator identity from + // which a completed-write exclusion can be proved. + return Err(CtError::Unavailable); + } + if guard.references != 0 || guard.pins != 0 { + return Err(CtError::Unavailable); + } + } + Ok(Vec::new()) + } + + pub(super) async fn change_stage_pin( + &self, + stage: StageId, + pin: StageId, + acquire: bool, + ) -> Result<(), CtError> { + self.authority_mutation(|| async { + use super::pin_owner::PinOwner; + let owner = PinOwner { + live: acquire, + process: self.process_identity()?, + }; + if acquire && owner.process.is_some() { + self.check_process_owner()?; + } + // Reader ownership is process-scoped. A retained old reader need not + // be the current writer, but its pin still competes atomically with GC. + let authority = self + .protection_authority(acquire && owner.process.is_none()) + .await?; + let mut key = operation_key(self.tree_id, b"stage-pin", stage); + for value in [pin.owner_epoch, pin.operation_high, pin.operation_low] { + key.extend_from_slice(&value.to_be_bytes()); + } + let current = self.get(&key).await.map_err(|_| CtError::Unavailable)?; + let current = current.map(|(bytes, _)| PinOwner::decode(&bytes)).transpose()?; + if current == Some(owner) { + return Ok(()); + } + if current.is_some_and(|current| current.process != owner.process) { + return Err(CtError::Unavailable.into()); + } + if acquire { + if current.is_some() { + return Err(CtError::Unavailable.into()); + } + self.readable_stage(stage).await?; + } + let stage_key = super::super::staging::stage_key(self.tree_id, stage); + let stage_exists = self + .get(&stage_key) + .await + .map_err(|_| CtError::Unavailable)? + .is_some(); + let mut ops = Vec::with_capacity(3); + if stage_exists { + let mut guard = self.load_stage_guard(stage).await?; + if acquire { + guard.pins = guard.pins.checked_add(1).ok_or(CtError::ResourceExhausted)?; + } else if current.is_some_and(|current| current.live) { + guard.pins = guard.pins.checked_sub(1).ok_or(CtError::Corruption)?; + } + if !acquire && pin == stage { + if let Some(creator) = guard.creator { + ops.push(self.operation_quiescence(stage, creator).await?); + } + } + ops.push(self.stage_guard_put(stage, guard)); + } else if acquire || current.is_some_and(|value| value.live) { + return Err(CtError::Corruption.into()); + } + // Missing stages are not evidence that an unknown create can no + // longer commit. The inactive pin also fences any later exact retry. + ops.push(BatchOp::Put { + key: Bytes::from(key), + value: Bytes::from(owner.encode()), + }); + self.protection_write(authority, ops).await?; + if acquire && owner.process.is_some() { + self.check_process_owner()?; + } + Ok(()) + }) + .await + } +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/operation.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/operation.rs new file mode 100644 index 000000000..a11961317 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/operation.rs @@ -0,0 +1,69 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Initial-pin release is the terminal boundary of one accepted native write. +use super::record::{decode_words, encode_words, operation_key}; +use super::writer::WriterOwner; +use super::KvRootCatalogStore; +use bytes::Bytes; +use crowdb_paxos_kv_client::BatchOp; +use crowdb_protocol::timing::{MAX_WRITE_CLOCK_SKEW_MS, MAX_WRITE_REQUEST_AGE_MS}; +use crowdb_tree_ffi::chunk::stage::StageId; +use crowdb_tree_ffi::CtError; + +impl KvRootCatalogStore { + // The caller atomically includes this record with releasing the initial + // pin, after the native operation and every accepted local I/O tail ended. + // Independent reader pins never imply that a writer is quiescent. + pub(super) async fn operation_quiescence( + &self, + stage: StageId, + owner: WriterOwner, + ) -> Result<BatchOp, CtError> { + if self.writer_owner()? != Some(owner) { + return Err(CtError::Unavailable); + } + let key = operation_key(self.tree_id, b"stage-quiesced", stage); + let value = if let Some((bytes, _)) = self.get(&key).await.map_err(|_| CtError::Unavailable)? { + decode_receipt(&bytes, owner)?; + bytes + } else { + let mut words = [0; 10]; + words[..2].copy_from_slice(&[1, super::writer::wall_time_ms()?]); + words[2..].copy_from_slice(&decode_words::<8>(&owner.encode())?); + Bytes::from(encode_words(&words)) + }; + Ok(BatchOp::Put { + key: Bytes::from(key), + value, + }) + } + + pub(super) async fn operation_is_excluded( + &self, + stage: StageId, + owner: WriterOwner, + ) -> Result<bool, CtError> { + let key = operation_key(self.tree_id, b"stage-quiesced", stage); + let Some((bytes, _)) = self.get(&key).await.map_err(|_| CtError::Unavailable)? else { + return self.writer_is_excluded(stage.owner_epoch, owner).await; + }; + let released_at = decode_receipt(&bytes, owner)?; + let deadline = released_at + .checked_add(MAX_WRITE_CLOCK_SKEW_MS) + .and_then(|value| value.checked_add(MAX_WRITE_REQUEST_AGE_MS)) + .ok_or(CtError::Corruption)?; + if super::writer::wall_time_ms()? >= deadline { + return Ok(true); + } + self.writer_is_excluded(stage.owner_epoch, owner).await + } +} + +fn decode_receipt(bytes: &[u8], owner: WriterOwner) -> Result<u64, CtError> { + let words = decode_words::<10>(bytes)?; + if words[0] != 1 || words[1] == 0 || WriterOwner::decode(&encode_words(&words[2..]))? != owner { + return Err(CtError::Corruption); + } + Ok(words[1]) +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/pin_owner.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/pin_owner.rs new file mode 100644 index 000000000..fb92b933e --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/pin_owner.rs @@ -0,0 +1,64 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use super::record::{decode_words, encode_words}; +use crowdb_protocol::chunk_stream::{StreamProcessIdentity, StreamPurpose}; +use crowdb_tree_ffi::CtError; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) struct PinOwner { + pub live: bool, + pub process: Option<StreamProcessIdentity>, +} +impl PinOwner { + pub fn encode(self) -> Vec<u8> { + let mut words = [1, u64::from(self.live), 0, 0, 0, 0, 0, 0]; + if let Some(identity) = self.process { + words[2] = 1; + words[3] = identity.instance_id; + words[4] = identity.process_epoch; + words[5] = u64::from_le_bytes(identity.boot_nonce[..8].try_into().expect("nonce bounds")); + words[6] = u64::from_le_bytes(identity.boot_nonce[8..].try_into().expect("nonce bounds")); + } + encode_words(&words) + } + pub fn decode(bytes: &[u8]) -> Result<Self, CtError> { + if bytes.len() == 12 { + let [live] = decode_words::<1>(bytes)?; + if live > 1 { + return Err(CtError::Corruption); + } + return Ok(Self { + live: live != 0, + process: None, + }); + } + let words = decode_words::<8>(bytes)?; + if words[0] != 1 || words[1] > 1 || words[2] > 1 || words[7] != 0 { + return Err(CtError::Corruption); + } + let process = if words[2] == 0 { + if words[3..7] != [0, 0, 0, 0] { + return Err(CtError::Corruption); + } + None + } else { + if words[3] == 0 || words[4] == 0 || words[5] == 0 && words[6] == 0 { + return Err(CtError::Corruption); + } + let mut boot_nonce = [0; 16]; + boot_nonce[..8].copy_from_slice(&words[5].to_le_bytes()); + boot_nonce[8..].copy_from_slice(&words[6].to_le_bytes()); + Some(StreamProcessIdentity { + purpose: StreamPurpose::Wal, + instance_id: words[3], + process_epoch: words[4], + boot_nonce, + }) + }; + Ok(Self { + live: words[1] != 0, + process, + }) + } +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/process.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/process.rs new file mode 100644 index 000000000..7d42fa622 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/process.rs @@ -0,0 +1,169 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use super::record::{decode_words, encode_words}; +use super::{catalog_key, KvRootCatalogStore}; +use bytes::Bytes; +use crowdb_paxos_kv_client::{BatchOp, StreamProcessAuthority, StreamProcessLeaseClient}; +use crowdb_protocol::chunk_stream::{StreamProcessIdentity, StreamPurpose}; +use crowdb_tree_ffi::CtError; +use std::sync::Arc; +use std::time::{Instant, SystemTime, UNIX_EPOCH}; + +pub(in super::super) struct RecoveryReceipt { + root_generation: u64, + stream_high: u64, + stream_low: u64, + applied_seq: u64, + replay_end: u64, +} + +impl KvRootCatalogStore { + pub(in super::super::super) fn with_process( + mut self, + process: Option<Arc<StreamProcessAuthority>>, + ) -> Result<Self, CtError> { + self.writer_incarnation = Self::next_writer_incarnation()?; + self.process = process; + Ok(self) + } + + pub(in super::super) fn process_identity(&self) -> Result<Option<StreamProcessIdentity>, CtError> { + self.process + .as_ref() + .map(|process| { + let identity = process.status(Instant::now()).lease.identity; + if identity.purpose != StreamPurpose::Wal { + return Err(CtError::InvalidArgument); + } + Ok(identity) + }) + .transpose() + } + + pub(in super::super) fn check_process_owner(&self) -> Result<(), CtError> { + let process = self.process.as_ref().ok_or(CtError::NotSupported)?; + process.capture(Instant::now()).map(|_| ()).map_err(|error| { + tracing::warn!(tree_id = self.tree_id, error = %error, "tree process authority is unavailable"); + CtError::Unavailable + }) + } + + /// Called only after the real root and complete WAL replay have succeeded. + /// Prepared transfer roots retain the receipt locally until publication + /// authority has actually moved to this owner. + pub(in super::super::super) async fn complete_recovery( + &self, + partition: &crowdb_chunk_kv::Partition, + root_generation: u64, + ) -> Result<(), CtError> { + if self.process.is_none() { + return Ok(()); + } + self.check_process_owner()?; + let snapshot = partition.snapshot(); + if partition.tree_id() != self.tree_id + || snapshot.ownership_epoch != self.owner_epoch + || snapshot.applied_seq != snapshot.journal_durable_seq + { + return Err(CtError::Unavailable); + } + self.recovered.store(Some(Arc::new(RecoveryReceipt { + root_generation, + stream_high: snapshot.stream_name.high, + stream_low: snapshot.stream_name.low, + applied_seq: snapshot.applied_seq, + replay_end: snapshot.journal_durable_offset, + }))); + let authority = self.protection_authority(false).await?; + if authority.epoch < self.owner_epoch { + return Ok(()); + } + if authority.epoch != self.owner_epoch { + return Err(CtError::Unavailable); + } + self.register_writer().await?; + self.persist_completed_recovery().await + } + + pub(in super::super) async fn persist_completed_recovery(&self) -> Result<(), CtError> { + self.authority_mutation(|| async { + let Some(receipt) = self.recovered.load_full() else { + return Ok(()); + }; + self.check_process_owner()?; + let Some(identity) = self.process_identity()? else { + return Err(CtError::NotSupported.into()); + }; + let authority = self.protection_authority(true).await?; + if authority.generation < receipt.root_generation { + return Err(CtError::Corruption.into()); + } + let key = catalog_key(self.tree_id, b"completed-recovery", self.owner_epoch); + let words = [ + 1, + identity.instance_id, + identity.process_epoch, + u64::from_le_bytes(identity.boot_nonce[..8].try_into().expect("nonce bounds")), + u64::from_le_bytes(identity.boot_nonce[8..].try_into().expect("nonce bounds")), + self.writer_incarnation, + receipt.root_generation, + receipt.stream_high, + receipt.stream_low, + receipt.applied_seq, + receipt.replay_end, + ]; + let value = encode_words(&words); + if let Some((old, _)) = self.get(&key).await.map_err(|_| CtError::Unavailable)? { + if old.as_ref() == value { + return Ok(()); + } + // Another boot must prove its own recovery, even at the same Tree epoch. + } + self.protection_write( + authority, + vec![BatchOp::Put { + key: Bytes::from(key), + value: Bytes::from(value), + }], + ) + .await?; + self.check_process_owner().map_err(Into::into) + }) + .await + } + + pub(in super::super) async fn require_completed_recovery(&self) -> Result<(), CtError> { + self.check_process_owner()?; + let identity = self.process_identity()?.ok_or(CtError::NotSupported)?; + let key = catalog_key(self.tree_id, b"completed-recovery", self.owner_epoch); + let (value, _) = self + .get(&key) + .await + .map_err(|_| CtError::Unavailable)? + .ok_or(CtError::Unavailable)?; + let words = decode_words::<11>(&value)?; + if words[0] != 1 + || words[1] != identity.instance_id + || words[2] != identity.process_epoch + || words[3].to_le_bytes() != identity.boot_nonce[..8] + || words[4].to_le_bytes() != identity.boot_nonce[8..] + || words[5] != self.writer_incarnation + { + return Err(CtError::Unavailable); + } + Ok(()) + } + + pub(super) async fn process_is_excluded(&self, identity: StreamProcessIdentity) -> Result<bool, CtError> { + let now = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|_| CtError::Unavailable)?; + let now_ms = now.as_millis().try_into().map_err(|_| CtError::Unavailable)?; + StreamProcessLeaseClient::new(self.kv.clone()) + .confirm_exclusion(identity, now_ms) + .await + .map(|proof| proof.is_some()) + .map_err(|_| CtError::Unavailable) + } +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/prune.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/prune.rs new file mode 100644 index 000000000..c34e9e5bf --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/prune.rs @@ -0,0 +1,110 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use super::pin_owner::PinOwner; +use super::record::operation_key; +use super::KvRootCatalogStore; +use bytes::Bytes; +use crowdb_paxos_kv_client::{BatchOp, ReadMode}; +use crowdb_tree_ffi::chunk::protection::PinPruneProgress; +use crowdb_tree_ffi::chunk::stage::StageId; +use crowdb_tree_ffi::CtError; + +impl KvRootCatalogStore { + pub(super) async fn prune_pins( + &self, + stage: StageId, + after: Option<StageId>, + limit: u32, + ) -> Result<PinPruneProgress, CtError> { + self.authority_mutation(|| async { + if !(1..=64).contains(&limit) { + return Err(CtError::InvalidArgument.into()); + } + // Every pruning transaction also updates the stage guard and authority. + let limit = limit.min( + u32::try_from(super::MAX_METADATA_MUTATIONS - 2).map_err(|_| CtError::ResourceExhausted)?, + ); + self.require_completed_recovery().await?; + let authority = self.protection_authority(true).await?; + let prefix = operation_key(self.tree_id, b"stage-pin", stage); + let start = after + .map(|id| { + let mut key = prefix.clone(); + for value in [id.owner_epoch, id.operation_high, id.operation_low] { + key.extend_from_slice(&value.to_be_bytes()); + } + key + }) + .unwrap_or_default(); + let result = self + .kv + .scan( + self.store_id, + self.group_id, + &prefix, + &start, + &[], + limit, + ReadMode::Linearizable, + None, + false, + None, + ) + .await + .map_err(|_| CtError::Unavailable)?; + if result.timed_out { + return Err(CtError::Unavailable.into()); + } + let mut progress = PinPruneProgress { + more: result.truncated, + ..PinPruneProgress::default() + }; + let mut ops = Vec::with_capacity(result.items.len() + 2); + for (key, value) in result.items { + if key.len() != prefix.len() + 24 { + return Err(CtError::Corruption.into()); + } + let bytes = &key[prefix.len()..]; + let at = |offset| { + u64::from_be_bytes(bytes[offset..offset + 8].try_into().expect("pin key bounds")) + }; + let id = StageId { + owner_epoch: at(0), + operation_high: at(8), + operation_low: at(16), + }; + if after.is_some_and(|after| id <= after) { + return Err(CtError::Corruption.into()); + } + let mut owner = PinOwner::decode(&value)?; + progress.examined += 1; + progress.after = Some(id); + let Some(process) = owner.process else { + continue; + }; + if !owner.live || !self.process_is_excluded(process).await? { + continue; + } + owner.live = false; + progress.released += 1; + ops.push(BatchOp::Put { + key, + value: Bytes::from(owner.encode()), + }); + } + if progress.released != 0 { + let mut guard = self.load_stage_guard(stage).await?; + guard.pins = guard + .pins + .checked_sub(progress.released.into()) + .ok_or(CtError::Corruption)?; + ops.push(self.stage_guard_put(stage, guard)); + self.protection_write(authority, ops).await?; + } + self.check_process_owner()?; + Ok(progress) + }) + .await + } +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/record.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/record.rs new file mode 100644 index 000000000..564a33609 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/record.rs @@ -0,0 +1,156 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use crowdb_tree_ffi::chunk::protection::{CandidateState, CandidateStatus, MAX_CANDIDATE_REFERENCES}; +use crowdb_tree_ffi::chunk::stage::StageId; +use crowdb_tree_ffi::{crc32c, CtError}; + +#[derive(Clone, Debug, PartialEq, Eq)] +pub(super) struct CandidateRecord { + pub status: CandidateStatus, + pub expected_checksum: u32, + pub prepared_checksum: u32, + pub manifest_checksum: u32, + pub manifest_bytes: u64, + pub last_stage: StageId, +} + +impl CandidateRecord { + pub fn encode(&self) -> Vec<u8> { + encode_words(&[ + 1, + self.status.state as u64, + self.status.generation, + self.status.reference_count, + self.status.prepared_references, + self.status.prepared_windows, + self.status.released_windows, + self.expected_checksum.into(), + self.prepared_checksum.into(), + self.manifest_checksum.into(), + self.manifest_bytes, + self.last_stage.owner_epoch, + self.last_stage.operation_high, + self.last_stage.operation_low, + ]) + } + pub fn decode(id: StageId, bytes: &[u8]) -> Result<Self, CtError> { + let words = decode_words::<14>(bytes)?; + let state = match words[1] { + 1 => CandidateState::Preparing, + 2 => CandidateState::Ready, + 3 => CandidateState::Committed, + 4 => CandidateState::Aborted, + 5 => CandidateState::Reclaimed, + _ => return Err(CtError::Corruption), + }; + if words[0] != 1 + || words[2] == 0 + || words[3] > MAX_CANDIDATE_REFERENCES + || words[4] > words[3] + || words[5] > words[4] + || words[6] > words[5] + { + return Err(CtError::Corruption); + } + Ok(Self { + status: CandidateStatus { + id, + state, + generation: words[2], + reference_count: words[3], + prepared_references: words[4], + prepared_windows: words[5], + released_windows: words[6], + }, + expected_checksum: u32::try_from(words[7]).map_err(|_| CtError::Corruption)?, + prepared_checksum: u32::try_from(words[8]).map_err(|_| CtError::Corruption)?, + manifest_checksum: u32::try_from(words[9]).map_err(|_| CtError::Corruption)?, + manifest_bytes: words[10], + last_stage: StageId { + owner_epoch: words[11], + operation_high: words[12], + operation_low: words[13], + }, + }) + } +} + +pub(super) fn encode_words(words: &[u64]) -> Vec<u8> { + let mut bytes = Vec::with_capacity(8 * words.len() + 4); + for value in words { + bytes.extend_from_slice(&value.to_le_bytes()); + } + bytes.extend_from_slice(&crc32c(&bytes).to_le_bytes()); + bytes +} + +pub(super) fn decode_words<const N: usize>(bytes: &[u8]) -> Result<[u64; N], CtError> { + if bytes.len() != N * 8 + 4 { + return Err(CtError::Corruption); + } + let checksum = u32::from_le_bytes(bytes[N * 8..].try_into().map_err(|_| CtError::Corruption)?); + if crc32c(&bytes[..N * 8]) != checksum { + return Err(CtError::Corruption); + } + Ok(std::array::from_fn(|index| { + u64::from_le_bytes( + bytes[index * 8..index * 8 + 8] + .try_into() + .expect("fixed record bounds"), + ) + })) +} + +pub(super) fn operation_key(tree: u64, kind: &[u8], id: StageId) -> Vec<u8> { + let mut key = super::catalog_key(tree, kind, id.owner_epoch); + key.extend_from_slice(&id.operation_high.to_be_bytes()); + key.extend_from_slice(&id.operation_low.to_be_bytes()); + key +} + +pub(super) fn window_key(tree: u64, id: StageId, window: u64) -> Vec<u8> { + let mut key = operation_key(tree, b"candidate-window", id); + key.extend_from_slice(&window.to_be_bytes()); + key +} + +pub(super) fn encode_references(refs: &[StageId]) -> Vec<u8> { + let words = refs + .iter() + .flat_map(|id| [id.owner_epoch, id.operation_high, id.operation_low]) + .collect::<Vec<_>>(); + encode_words(&words) +} + +pub(super) fn decode_references(bytes: &[u8]) -> Result<Vec<StageId>, CtError> { + if bytes.len() < 28 || (bytes.len() - 4) % 24 != 0 || bytes.len() > 64 * 24 + 4 { + return Err(CtError::Corruption); + } + let payload = &bytes[..bytes.len() - 4]; + if crc32c(payload) + != u32::from_le_bytes( + bytes[bytes.len() - 4..] + .try_into() + .map_err(|_| CtError::Corruption)?, + ) + { + return Err(CtError::Corruption); + } + let refs = payload + .chunks_exact(24) + .map(|chunk| { + let at = + |offset| u64::from_le_bytes(chunk[offset..offset + 8].try_into().expect("reference bounds")); + StageId { + owner_epoch: at(0), + operation_high: at(8), + operation_low: at(16), + } + }) + .collect::<Vec<_>>(); + if refs.windows(2).any(|pair| pair[0] >= pair[1]) { + return Err(CtError::Corruption); + } + Ok(refs) +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/release.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/release.rs new file mode 100644 index 000000000..58ac6ba58 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/release.rs @@ -0,0 +1,139 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use super::record::{decode_references, decode_words, operation_key, window_key, CandidateRecord}; +use super::{catalog_key, KvRootCatalogStore}; +use bytes::Bytes; +use crowdb_paxos_kv_client::BatchOp; +use crowdb_tree_ffi::chunk::protection::CandidateState; +use crowdb_tree_ffi::chunk::stage::StageId; +use crowdb_tree_ffi::CtError; + +impl KvRootCatalogStore { + pub(super) fn terminal_candidate_ops(&self, record: &CandidateRecord) -> Vec<BatchOp> { + let mut ops = vec![BatchOp::Put { + key: Bytes::from(operation_key(self.tree_id, b"candidate", record.status.id)), + value: Bytes::from(record.encode()), + }]; + if record.status.released_windows == record.status.prepared_windows { + ops.push(BatchOp::Delete { + key: Bytes::from(operation_key( + self.tree_id, + b"candidate-manifest", + record.status.id, + )), + }); + } + ops + } + + pub(super) async fn recover_candidate(&self, id: StageId) -> Result<(), CtError> { + self.authority_mutation(|| async { + self.require_completed_recovery().await?; + let authority = self.protection_authority(true).await?; + let mut record = self.candidate_record(id).await?.ok_or(CtError::NotFound)?; + if matches!( + record.status.state, + CandidateState::Aborted | CandidateState::Reclaimed + ) { + return Ok(()); + } + if record.status.state == CandidateState::Committed || id.owner_epoch > self.owner_epoch { + return Err(CtError::Unavailable.into()); + } + if id.owner_epoch == self.owner_epoch { + let key = operation_key(self.tree_id, b"candidate-owner", id); + let (bytes, _) = self + .get(&key) + .await + .map_err(|_| CtError::Unavailable)? + .ok_or(CtError::Unavailable)?; + let creator = super::writer::WriterOwner::decode(&bytes)?; + if self.writer_owner()? == Some(creator) + || !self.writer_is_excluded(id.owner_epoch, creator).await? + { + return Err(CtError::Unavailable.into()); + } + } + record.status.state = CandidateState::Aborted; + self.protection_write(authority, self.terminal_candidate_ops(&record)) + .await + }) + .await + } + + pub(super) async fn release_candidate(&self, id: StageId, window: u64) -> Result<(), CtError> { + self.authority_mutation(|| async { + let authority = self.protection_authority(true).await?; + let mut record = self.candidate_record(id).await?.ok_or(CtError::NotFound)?; + if !matches!( + record.status.state, + CandidateState::Aborted | CandidateState::Reclaimed + ) { + return Err(CtError::Unavailable.into()); + } + if window < record.status.released_windows { + return Ok(()); + } + if window != record.status.released_windows || window >= record.status.prepared_windows { + return Err(CtError::InvalidArgument.into()); + } + let key = window_key(self.tree_id, id, window); + let (bytes, _) = self + .get(&key) + .await + .map_err(|_| CtError::Unavailable)? + .ok_or(CtError::Corruption)?; + let refs = decode_references(&bytes)?; + let mut ops = Vec::with_capacity(refs.len() + 4); + for reference in refs { + let mut guard = self.load_stage_guard(reference).await?; + guard.references = guard.references.checked_sub(1).ok_or(CtError::Corruption)?; + ops.push(self.stage_guard_put(reference, guard)); + } + record.status.released_windows += 1; + if record.status.released_windows == record.status.prepared_windows { + ops.push(BatchOp::Delete { + key: Bytes::from(operation_key(self.tree_id, b"candidate-manifest", id)), + }); + } + ops.push(BatchOp::Delete { + key: Bytes::from(key), + }); + ops.push(BatchOp::Put { + key: Bytes::from(operation_key(self.tree_id, b"candidate", id)), + value: Bytes::from(record.encode()), + }); + self.protection_write(authority, ops).await + }) + .await + } + + /// Called while the caller still owns the root authority revision used to + /// reclaim the matching immutable manifest and its generation-pin proof. + pub(in super::super) async fn reclaim_candidate_ops( + &self, + generation: u64, + ) -> Result<Vec<BatchOp>, CtError> { + let key = catalog_key(self.tree_id, b"manifest-candidate", generation); + let Some((value, _)) = self.get(&key).await.map_err(|_| CtError::Unavailable)? else { + return Ok(Vec::new()); + }; + let words = decode_words::<3>(&value)?; + let id = StageId { + owner_epoch: words[0], + operation_high: words[1], + operation_low: words[2], + }; + let mut record = self.candidate_record(id).await?.ok_or(CtError::Corruption)?; + if record.status.state != CandidateState::Committed || record.status.generation != generation { + return Err(CtError::Corruption); + } + record.status.state = CandidateState::Reclaimed; + let mut ops = self.terminal_candidate_ops(&record); + ops.push(BatchOp::Delete { + key: Bytes::from(key), + }); + Ok(ops) + } +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/scan.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/scan.rs new file mode 100644 index 000000000..15884dc18 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/scan.rs @@ -0,0 +1,71 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use super::record::{operation_key, CandidateRecord}; +use super::{catalog_key, KvRootCatalogStore}; +use crowdb_paxos_kv_client::ReadMode; +use crowdb_tree_ffi::chunk::protection::CandidateStatus; +use crowdb_tree_ffi::chunk::stage::StageId; +use crowdb_tree_ffi::CtError; + +impl KvRootCatalogStore { + pub(super) async fn scan_candidates( + &self, + after: Option<StageId>, + limit: u32, + ) -> Result<Vec<CandidateStatus>, CtError> { + if !(1..=64).contains(&limit) { + return Err(CtError::InvalidArgument); + } + let mut prefix = catalog_key(self.tree_id, b"candidate", 0); + prefix.truncate(prefix.len() - 8); + let start = after + .map(|id| operation_key(self.tree_id, b"candidate", id)) + .unwrap_or_default(); + let result = self + .kv + .scan( + self.store_id, + self.group_id, + &prefix, + &start, + &[], + limit, + ReadMode::Linearizable, + None, + false, + None, + ) + .await + .map_err(|_| CtError::Unavailable)?; + if result.timed_out { + return Err(CtError::Unavailable); + } + result + .items + .into_iter() + .map(|(key, value)| { + if key.len() != prefix.len() + 24 { + return Err(CtError::Corruption); + } + let bytes = &key[prefix.len()..]; + let at = |offset| { + u64::from_be_bytes( + bytes[offset..offset + 8] + .try_into() + .expect("candidate key bounds"), + ) + }; + let id = StageId { + owner_epoch: at(0), + operation_high: at(8), + operation_low: at(16), + }; + if after.is_some_and(|after| id <= after) { + return Err(CtError::Corruption); + } + Ok(CandidateRecord::decode(id, &value)?.status) + }) + .collect() + } +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/writer.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/writer.rs new file mode 100644 index 000000000..c0287704b --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/protection/writer.rs @@ -0,0 +1,234 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use super::pin_owner::PinOwner; +use super::record::{decode_words, encode_words}; +use super::{catalog_key, KvRootCatalogStore}; +use bytes::Bytes; +use crowdb_paxos_kv_client::BatchOp; +use crowdb_protocol::chunk_stream::StreamProcessIdentity; +use crowdb_protocol::timing::{MAX_WRITE_CLOCK_SKEW_MS, MAX_WRITE_REQUEST_AGE_MS}; +use crowdb_tree_ffi::CtError; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::{SystemTime, UNIX_EPOCH}; + +static NEXT_WRITER: AtomicU64 = AtomicU64::new(1); + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) struct WriterOwner { + pub process: StreamProcessIdentity, + pub incarnation: u64, +} + +impl WriterOwner { + pub(super) fn encode(self) -> Vec<u8> { + let owner = PinOwner { + live: true, + process: Some(self.process), + }; + let mut words = decode_words::<8>(&owner.encode()).expect("valid process encoding"); + words[7] = self.incarnation; + encode_words(&words) + } + pub(super) fn decode(bytes: &[u8]) -> Result<Self, CtError> { + let mut words = decode_words::<8>(bytes)?; + let incarnation = words[7]; + words[7] = 0; + let owner = PinOwner::decode(&encode_words(&words))?; + if !owner.live || incarnation == 0 { + return Err(CtError::Corruption); + } + Ok(Self { + process: owner.process.ok_or(CtError::Corruption)?, + incarnation, + }) + } +} + +impl KvRootCatalogStore { + pub(in super::super) fn next_writer_incarnation() -> Result<u64, CtError> { + NEXT_WRITER + .fetch_update(Ordering::Relaxed, Ordering::Relaxed, |value| value.checked_add(1)) + .map_err(|_| CtError::ResourceExhausted) + } + + pub(super) fn writer_owner(&self) -> Result<Option<WriterOwner>, CtError> { + Ok(self.process_identity()?.map(|process| WriterOwner { + process, + incarnation: self.writer_incarnation, + })) + } + + fn quiescence_key(&self, epoch: u64, owner: WriterOwner) -> Vec<u8> { + let mut key = catalog_key(self.tree_id, b"writer-quiesced", epoch); + key.extend_from_slice(&owner.encode()); + key + } + + async fn writer_was_closed(&self, epoch: u64, owner: WriterOwner) -> Result<bool, CtError> { + let Some((value, _)) = self + .get(&self.quiescence_key(epoch, owner)) + .await + .map_err(|_| CtError::Unavailable)? + else { + return Ok(false); + }; + let words = decode_words::<2>(&value)?; + if words[0] != 1 || words[1] == 0 { + return Err(CtError::Corruption); + } + Ok(true) + } + + pub(super) async fn writer_is_quiesced(&self, epoch: u64, owner: WriterOwner) -> Result<bool, CtError> { + match self + .get(&self.quiescence_key(epoch, owner)) + .await + .map_err(|_| CtError::Unavailable)? + { + None => Ok(false), + Some((value, _)) => { + let words = decode_words::<2>(&value)?; + if words[0] != 1 { + return Err(CtError::Corruption); + } + let deadline = words[1] + .checked_add(MAX_WRITE_CLOCK_SKEW_MS) + .and_then(|value| value.checked_add(MAX_WRITE_REQUEST_AGE_MS)) + .ok_or(CtError::Corruption)?; + Ok(wall_time_ms()? >= deadline) + } + } + } + + pub(super) async fn writer_is_excluded(&self, epoch: u64, owner: WriterOwner) -> Result<bool, CtError> { + if self.writer_is_quiesced(epoch, owner).await? { + return Ok(true); + } + self.process_is_excluded(owner.process).await + } + + // Reading the registration after authority is essential: replacement also + // advances that authority revision, fencing every already-prepared mutation. + pub(in super::super) async fn check_registered_writer(&self) -> Result<(), CtError> { + let Some(owner) = self.writer_owner()? else { + return Ok(()); + }; + self.check_process_owner()?; + if self.writer_quiesced.load(Ordering::Acquire) { + return Err(CtError::Unavailable); + } + let key = catalog_key(self.tree_id, b"writer-owner", self.owner_epoch); + let value = self + .get(&key) + .await + .map_err(|_| CtError::Unavailable)? + .ok_or(CtError::Unavailable)? + .0; + if WriterOwner::decode(&value)? != owner + || self + .get(&self.quiescence_key(self.owner_epoch, owner)) + .await + .map_err(|_| CtError::Unavailable)? + .is_some() + { + return Err(CtError::Unavailable); + } + Ok(()) + } + + pub(in super::super::super) async fn register_writer(&self) -> Result<(), CtError> { + self.authority_mutation(|| async { + let Some(owner) = self.writer_owner()? else { + return Ok(()); + }; + self.check_process_owner()?; + if self.writer_quiesced.load(Ordering::Acquire) { + return Err(CtError::Unavailable.into()); + } + let authority = self.protection_authority(false).await.map_err(|error| { + tracing::warn!(tree_id = self.tree_id, owner_epoch = self.owner_epoch, error = %error, + "writer registration could not load root authority"); + error + })?; + if authority.epoch != self.owner_epoch { + if authority.epoch > self.owner_epoch { + tracing::warn!(tree_id = self.tree_id, authority_epoch = authority.epoch, owner_epoch = self.owner_epoch, + "writer registration has a stale owner epoch"); + return Err(CtError::Unavailable.into()); + } + self.check_prepared_staging(authority.epoch).await.map_err(|error| { + tracing::warn!(tree_id = self.tree_id, authority_epoch = authority.epoch, owner_epoch = self.owner_epoch, + prepared = self.prepared_assignment.is_some(), error = %error, + "writer registration does not match prepared staging"); + error + })?; + } + let key = catalog_key(self.tree_id, b"writer-owner", self.owner_epoch); + if let Some((bytes, _)) = self.get(&key).await.map_err(|_| CtError::Unavailable)? { + let previous = WriterOwner::decode(&bytes)?; + if previous == owner { + return self.check_registered_writer().await.map_err(Into::into); + } + // Writer replacement is fenced by this authority CAS immediately. + // Reclaiming an old unsealed pack still waits for the separate + // server request-age exclusion in writer_is_excluded. + if !self.writer_was_closed(self.owner_epoch, previous).await? + && !self.process_is_excluded(previous.process).await? + { + tracing::warn!( + tree_id = self.tree_id, + previous_incarnation = previous.incarnation, + "tree writer replacement waits for the previous writer to close" + ); + return Err(CtError::Unavailable.into()); + } + } + self.protection_write( + authority, + vec![BatchOp::Put { + key: Bytes::from(key), + value: Bytes::from(owner.encode()), + }], + ) + .await?; + self.check_process_owner().map_err(Into::into) + }) + .await + } + + // The caller must have closed write admission and drained accepted native + // backend work first. Reader/export pins are deliberately untouched. + pub(super) async fn publish_writer_quiescence(&self) -> Result<(), CtError> { + self.authority_mutation(|| async { + let owner = self.writer_owner()?.ok_or(CtError::NotSupported)?; + self.writer_quiesced.store(true, Ordering::Release); + let authority = self.protection_authority(false).await?; + let key = self.quiescence_key(self.owner_epoch, owner); + if let Some((value, _)) = self.get(&key).await.map_err(|_| CtError::Unavailable)? { + let words = decode_words::<2>(&value)?; + return if words[0] == 1 { + Ok(()) + } else { + Err(CtError::Corruption.into()) + }; + } + self.protection_write( + authority, + vec![BatchOp::Put { + key: Bytes::from(key), + value: Bytes::from(encode_words(&[1, wall_time_ms()?])), + }], + ) + .await + }) + .await + } +} + +pub(super) fn wall_time_ms() -> Result<u64, CtError> { + let time = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|_| CtError::Unavailable)?; + time.as_millis().try_into().map_err(|_| CtError::Unavailable) +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/retention.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/retention.rs new file mode 100644 index 000000000..65c6d4562 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/retention.rs @@ -0,0 +1,201 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Pins and reclamation mutate the same authority revision as root publication. + +use super::{ + catalog_key, catalog_pin_key, catalog_pin_prefix, decode_authority, decode_u64, KvRootCatalogStore, +}; +use bytes::Bytes; +use crowdb_paxos_kv_client::{BatchOp, ReadMode}; +use crowdb_tree_ffi::CtError; + +impl KvRootCatalogStore { + async fn retention_authority(&self) -> Result<(Vec<u8>, Bytes, u64), CtError> { + let key = catalog_key(self.tree_id, b"authority", 0); + let (value, revision) = self + .get(&key) + .await + .map_err(|_| CtError::Unavailable)? + .ok_or(CtError::Unavailable)?; + decode_authority(&value).ok_or(CtError::Corruption)?; + Ok((key, value, revision)) + } + + pub(super) fn retain_generation( + &self, + tree_id: u64, + high: u64, + low: u64, + generation: u64, + ) -> Result<(), CtError> { + if tree_id != self.tree_id || (high == 0 && low == 0) || generation == 0 { + return Err(CtError::InvalidArgument); + } + self.wait(self.authority_mutation(|| async { + let (authority_key, authority, revision) = self.retention_authority().await?; + if self.process.is_some() { + self.check_process_owner()?; + } + if self + .get(&catalog_key(tree_id, b"manifest", generation)) + .await + .map_err(|_| CtError::Unavailable)? + .is_none() + { + return Err(CtError::NotFound.into()); + } + self.check_legacy_pin_admission(generation).await?; + let key = catalog_pin_key(tree_id, high, low); + if let Some((value, _)) = self.get(&key).await.map_err(|_| CtError::Unavailable)? { + return if decode_u64(&value) == Some(generation) { + Ok(()) + } else { + Err(CtError::InvalidArgument.into()) + }; + } + let ops = [ + BatchOp::Put { + key: Bytes::from(authority_key.clone()), + value: authority, + }, + BatchOp::Put { + key: Bytes::from(key), + value: Bytes::copy_from_slice(&generation.to_be_bytes()), + }, + ]; + self.kv + .batch_write_cas(self.store_id, self.group_id, &ops, &authority_key, revision) + .await + .map(|_| ()) + .map_err(|error| super::transaction::MutationError::from_cas(&error)) + })) + } + + pub(super) fn release_generation(&self, tree_id: u64, high: u64, low: u64) -> Result<(), CtError> { + if tree_id != self.tree_id || (high == 0 && low == 0) { + return Err(CtError::InvalidArgument); + } + self.wait(self.authority_mutation(|| async { + let (authority_key, authority, revision) = self.retention_authority().await?; + let key = catalog_pin_key(tree_id, high, low); + if self.get(&key).await.map_err(|_| CtError::Unavailable)?.is_none() { + return Ok(()); + } + let ops = [ + BatchOp::Put { + key: Bytes::from(authority_key.clone()), + value: authority, + }, + BatchOp::Delete { + key: Bytes::from(key), + }, + ]; + self.kv + .batch_write_cas(self.store_id, self.group_id, &ops, &authority_key, revision) + .await + .map(|_| ()) + .map_err(|error| super::transaction::MutationError::from_cas(&error)) + })) + } + + pub(super) fn reclaim_generations(&self, tree_id: u64, generation: u64) -> u64 { + if tree_id != self.tree_id || generation <= 2 { + return 0; + } + self.wait(self.reclaim_generations_inner(generation)).unwrap_or(0) + } + + async fn reclaim_generations_inner(&self, generation: u64) -> Result<u64, CtError> { + self.authority_mutation(|| async { + let (authority_key, authority, revision) = self.retention_authority().await?; + self.check_registered_writer().await?; + let (epoch, current_generation) = decode_authority(&authority).ok_or(CtError::Corruption)?; + if epoch != self.owner_epoch { + return Err(CtError::Unavailable.into()); + } + let prefix = catalog_pin_prefix(self.tree_id); + // A saturated pin window conservatively retains history for this pass. + let pins = self + .kv + .scan( + self.store_id, + self.group_id, + &prefix, + &[], + &[], + 1024, + ReadMode::Linearizable, + None, + false, + None, + ) + .await + .map_err(|_| CtError::Unavailable)?; + if pins.truncated || pins.timed_out { + return Err(CtError::ResourceExhausted.into()); + } + let mut generation = generation.min(current_generation); + for (_, value) in pins.items { + generation = generation.min(decode_u64(&value).ok_or(CtError::Corruption)?); + } + if generation <= 2 { + return Ok(0); + } + let floor_key = catalog_key(self.tree_id, b"reclaim-floor", 0); + let floor = match self.get(&floor_key).await.map_err(|_| CtError::Unavailable)? { + Some((value, _)) => decode_u64(&value).ok_or(CtError::Corruption)?, + None => 1, + }; + let limit = generation.saturating_sub(1).min(floor.saturating_add(128)); + if floor >= limit { + return Ok(0); + } + let mut end = floor; + let mut ops = Vec::with_capacity(super::protection::MAX_METADATA_MUTATIONS); + for old in floor..limit { + let preserved = self.legacy_history_ops(old).await?; + let candidate = self.reclaim_candidate_ops(old).await?; + // Reserve the floor and authority keys; never split one + // generation's manifest, candidate and legacy-history mutation. + if ops.len() + 1 + preserved.len() + candidate.len() + 2 + > super::protection::MAX_METADATA_MUTATIONS + { + break; + } + ops.push(BatchOp::Delete { + key: Bytes::from(catalog_key(self.tree_id, b"manifest", old)), + }); + let copied_legacy = !preserved.is_empty(); + ops.extend(preserved); + ops.extend(candidate); + end = old + 1; + // Retain at most one newly copied legacy image in this pass. + if copied_legacy { + break; + } + } + ops.push(BatchOp::Put { + key: Bytes::from(floor_key), + value: Bytes::copy_from_slice(&end.to_be_bytes()), + }); + ops.push(BatchOp::Put { + key: Bytes::from(authority_key.clone()), + value: authority, + }); + self.kv + .batch_write_cas(self.store_id, self.group_id, &ops, &authority_key, revision) + .await + .map_err(|error| super::transaction::MutationError::from_cas(&error))?; + self.published_objects.rcu(|current| { + let mut next = (**current).clone(); + for old in floor..end { + next.remove(&catalog_key(self.tree_id, b"manifest", old)); + } + next + }); + Ok(end - floor) + }) + .await + } +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/staging.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/staging.rs new file mode 100644 index 000000000..3d7ea2dbe --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/staging.rs @@ -0,0 +1,194 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Conditional staging mutations share the root authority revision. + +use bytes::Bytes; +use crowdb_paxos_kv_client::{BatchOp, ReadMode}; +use crowdb_tree_ffi::chunk::stage::{validate_stage_transition, StageId, StageRecord, MAX_STAGE_SCAN}; +use crowdb_tree_ffi::CtError; + +use super::{catalog_key, decode_authority, KvRootCatalogStore}; + +impl KvRootCatalogStore { + pub(super) fn stage_load(&self, tree_id: u64, id: StageId) -> Result<Option<StageRecord>, CtError> { + if tree_id != self.tree_id || id.owner_epoch == 0 { + return Err(CtError::InvalidArgument); + } + let value = self + .wait(self.get(&stage_key(tree_id, id))) + .map_err(|_| CtError::Unavailable)?; + value + .map(|(bytes, _)| { + let record = StageRecord::decode(&bytes)?; + if record.tree_id != tree_id || record.id != id { + return Err(CtError::Corruption); + } + Ok(record) + }) + .transpose() + } + + pub(super) fn stage_compare( + &self, + tree_id: u64, + owner_epoch: u64, + expected: Option<&StageRecord>, + next: Option<&StageRecord>, + ) -> Result<(), CtError> { + validate_stage_transition(expected, next)?; + let record = next.or(expected).ok_or(CtError::InvalidArgument)?; + if tree_id != self.tree_id || record.tree_id != tree_id { + return Err(CtError::InvalidArgument); + } + if owner_epoch != self.owner_epoch || record.id.owner_epoch > owner_epoch { + return Err(CtError::Unavailable); + } + if record.id.owner_epoch != owner_epoch { + use crowdb_tree_ffi::chunk::stage::StageState; + if expected.is_none() + || next.is_some_and(|value| { + !matches!( + value.state, + StageState::Retired | StageState::Deleting | StageState::Reclaimed + ) + }) + { + return Err(CtError::Unavailable); + } + self.wait(self.require_completed_recovery())?; + } + if record.id.owner_epoch == owner_epoch { + self.wait(self.authorize_staging())?; + } else { + self.wait(self.authorize_publication())?; + } + self.wait(self.stage_compare_authorized(owner_epoch, record.id, expected, next)) + } + + async fn stage_compare_authorized( + &self, + owner_epoch: u64, + id: StageId, + expected: Option<&StageRecord>, + next: Option<&StageRecord>, + ) -> Result<(), CtError> { + self.authority_mutation(|| async { + let authority_key = catalog_key(self.tree_id, b"authority", 0); + let (authority, revision) = self + .get(&authority_key) + .await + .map_err(|_| CtError::Unavailable)? + .ok_or(CtError::Unavailable)?; + let (epoch, _) = decode_authority(&authority).ok_or(CtError::Corruption)?; + self.check_registered_writer().await?; + if epoch != owner_epoch { + if epoch > owner_epoch || id.owner_epoch != owner_epoch { + return Err(CtError::Unavailable.into()); + } + self.check_prepared_staging(epoch).await?; + } + let key = stage_key(self.tree_id, id); + let current = self.get(&key).await.map_err(|_| CtError::Unavailable)?; + let expected_record = expected; + let next_record = next; + let expected = expected.map(StageRecord::encode).transpose()?; + let next = next.map(StageRecord::encode).transpose()?; + let current = current.as_ref().map(|(bytes, _)| bytes.as_ref()); + if current == next.as_deref() { + return Ok(()); + } + if current != expected.as_deref() { + return Err(CtError::Unavailable.into()); + } + let update = match next { + Some(bytes) => BatchOp::Put { + key: Bytes::from(key), + value: Bytes::from(bytes), + }, + None => BatchOp::Delete { + key: Bytes::from(key), + }, + }; + // Writing the unchanged authority value advances its KV revision. Root + // publication, stage transitions, pinning, and reclamation serialize here. + let mut ops = vec![ + BatchOp::Put { + key: Bytes::from(authority_key.clone()), + value: authority, + }, + update, + ]; + let record = expected_record.or(next_record).ok_or(CtError::InvalidArgument)?; + ops.extend( + self.stage_guard_transition(record, expected_record.is_none(), next_record) + .await?, + ); + self.kv + .batch_write_cas(self.store_id, self.group_id, &ops, &authority_key, revision) + .await + .map(|_| ()) + .map_err(|error| super::transaction::MutationError::from_cas(&error)) + }) + .await + } + + pub(super) fn stage_list( + &self, + tree_id: u64, + after: Option<StageId>, + limit: u32, + ) -> Result<Vec<StageRecord>, CtError> { + if tree_id != self.tree_id || !(1..=MAX_STAGE_SCAN).contains(&limit) { + return Err(CtError::InvalidArgument); + } + let prefix = stage_prefix(tree_id); + let start = after.map(|id| stage_key(tree_id, id)).unwrap_or_default(); + let result = self + .wait(self.kv.scan( + self.store_id, + self.group_id, + &prefix, + &start, + &[], + limit, + ReadMode::Linearizable, + None, + false, + None, + )) + .map_err(|_| CtError::Unavailable)?; + if result.timed_out { + return Err(CtError::Unavailable); + } + result + .items + .into_iter() + .map(|(key, value)| { + let record = StageRecord::decode(&value)?; + if record.tree_id != tree_id + || key.as_ref() != stage_key(tree_id, record.id).as_slice() + || after.is_some_and(|id| record.id <= id) + { + return Err(CtError::Corruption); + } + Ok(record) + }) + .collect() + } +} + +fn stage_prefix(tree_id: u64) -> Vec<u8> { + let mut key = b"\0crowdb/chunk-kv/root/v1/".to_vec(); + key.extend_from_slice(&tree_id.to_be_bytes()); + key.extend_from_slice(b"/stage/v1/"); + key +} + +pub(super) fn stage_key(tree_id: u64, id: StageId) -> Vec<u8> { + let mut key = stage_prefix(tree_id); + for value in [id.owner_epoch, id.operation_high, id.operation_low] { + key.extend_from_slice(&value.to_be_bytes()); + } + key +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog/transaction.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog/transaction.rs new file mode 100644 index 000000000..451a8c58b --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog/transaction.rs @@ -0,0 +1,53 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Rebuild authority mutations after a definite revision conflict. + +use super::KvRootCatalogStore; +use crowdb_paxos_kv_client::Error; +use crowdb_tree_ffi::CtError; +use std::future::Future; + +pub(super) enum MutationError { + Conflict, + Status(CtError), +} +pub(super) type MutationResult<T> = Result<T, MutationError>; + +impl From<CtError> for MutationError { + fn from(status: CtError) -> Self { + Self::Status(status) + } +} +impl MutationError { + pub(super) fn from_cas(error: &Error) -> Self { + match error { + Error::CasFailed { .. } | Error::CasBusy => Self::Conflict, + // An unknown reply requires the caller's durable operation lookup. + // Do not treat transport or ownership failures as revision conflicts. + _ => { + tracing::warn!(error = %error, "tree authority conditional write failed"); + Self::Status(CtError::Unavailable) + } + } + } +} +impl KvRootCatalogStore { + pub(super) async fn authority_mutation<T, F, Fut>(&self, mut prepare: F) -> Result<T, CtError> + where + F: FnMut() -> Fut, + Fut: Future<Output = MutationResult<T>>, + { + // Pins, root publication and GC deliberately share one revision. A + // definite conflict discards all derived counters and proofs; each + // attempt rereads authority, revalidates ownership and rebuilds ops. + // Definite conflicts require a fresh derivation, not a storage failure. + loop { + match prepare().await { + Ok(value) => return Ok(value), + Err(MutationError::Status(status)) => return Err(status), + Err(MutationError::Conflict) => tokio::task::yield_now().await, + } + } + } +} diff --git a/app/crowdb-chunk-kv-server/src/storage/root_catalog_test.rs b/app/crowdb-chunk-kv-server/src/storage/root_catalog_test.rs new file mode 100644 index 000000000..3b55ff65e --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/root_catalog_test.rs @@ -0,0 +1,60 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Feature-gated access to the production process-bound root authority. +use super::{KvRootCatalogStore, StorageRuntimeError}; +use crowdb_chunk_kv::Partition; +use crowdb_paxos_kv_client::{PaxosKvClient, StreamProcessAuthority}; +use crowdb_protocol::chunk_kv::ChunkKvRangeCatalogEntry; +use crowdb_tree_ffi::{CtError, RootCatalogStore}; +use std::sync::Arc; + +pub struct TestOwnedRootCatalog { + catalog: Arc<KvRootCatalogStore>, +} +impl TestOwnedRootCatalog { + #[must_use] + pub fn store(&self) -> &dyn RootCatalogStore { + self.catalog.as_ref() + } + /// # Errors + /// Requires matching, actually recovered root and replay frontiers. + pub async fn complete_recovery( + &self, + partition: &Partition, + root_generation: u64, + ) -> Result<(), CtError> { + self.catalog.complete_recovery(partition, root_generation).await + } + + /// # Errors + /// Returns native manifest inspection or recovery receipt errors. + pub async fn complete_tree_recovery_for_tests( + &self, + partition: &Partition, + store: &crowdb_tree_ffi::PageStore, + ) -> Result<(), StorageRuntimeError> { + super::ChunkKvStorage::complete_tree_recovery(partition, store, &self.catalog).await + } +} +/// # Errors +/// Returns identity, process registration or KV availability errors. +pub async fn open_root_catalog_with_process_for_tests( + kv: Arc<PaxosKvClient>, + store_id: u64, + group_id: u64, + entry: &ChunkKvRangeCatalogEntry, + process: Arc<StreamProcessAuthority>, +) -> Result<TestOwnedRootCatalog, StorageRuntimeError> { + let catalog = KvRootCatalogStore::for_assignment(kv, store_id, group_id, entry) + .await? + .with_process(Some(process)) + .map_err(|error| StorageRuntimeError::Tree(error.to_string()))?; + catalog + .register_writer() + .await + .map_err(|error| StorageRuntimeError::Tree(error.to_string()))?; + Ok(TestOwnedRootCatalog { + catalog: Arc::new(catalog), + }) +} diff --git a/app/crowdb-chunk-kv-server/src/storage/split_recovery.rs b/app/crowdb-chunk-kv-server/src/storage/split_recovery.rs index 74213f740..c57030e08 100644 --- a/app/crowdb-chunk-kv-server/src/storage/split_recovery.rs +++ b/app/crowdb-chunk-kv-server/src/storage/split_recovery.rs @@ -5,7 +5,7 @@ use super::{prepared_overlay_artifact, split_plan, storage_plan_error, ChunkKvStorage}; use crate::MonitorError; -use crowdb_chunk_kv::{Partition, PartitionConfig, PreparedSplit}; +use crowdb_chunk_kv::{Partition, PreparedSplit}; use crowdb_chunk_stream::StreamRegistry; use crowdb_protocol::chunk_kv::{ ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogPartitionState, SplitTransition, @@ -42,13 +42,19 @@ impl ChunkKvStorage { ) .await .map_err(|error| storage_plan_error(&error.to_string()))?; + // A live retry may still have a synchronized but unpublished parent + // suffix. Publish it before opening the independent recovery reader. + parent + .publish_split_journal_prefix(proof.cutover_offset) + .await + .map_err(|error| storage_plan_error(&error.to_string()))?; let source_stream = self .streams .open_read_only_current(proof.source_stream_name, self.metadata_store_id) .await .map_err(|error| storage_plan_error(&error.to_string()))?; - let page_store = self - .open_durable_tree_page_store( + let (page_store, catalog) = self + .open_durable_tree( ChunkPageStoreOptions { tree_id: transition.child.artifact.tree_id, owner_epoch: transition.child.owner_epoch, @@ -75,14 +81,17 @@ impl ChunkKvStorage { }; let (child, artifact) = Partition::recover_native_split_handoff( prepared_overlay_artifact(&entry, proof), - PartitionConfig::default(), - crowdb_tree_ffi::Config::default(), - page_store, + self.partition_config.clone(), + self.tree_config(), + page_store.clone(), child_stream, source_stream, ) .await .map_err(|error| storage_plan_error(&error.to_string()))?; + Self::complete_tree_recovery(&child, &page_store, &catalog) + .await + .map_err(|error| storage_plan_error(&error.to_string()))?; parent .resume_split_child_session(split_plan(transition), child, artifact) .await diff --git a/app/crowdb-chunk-kv-server/src/storage/tree_owner.rs b/app/crowdb-chunk-kv-server/src/storage/tree_owner.rs new file mode 100644 index 000000000..15039a6c7 --- /dev/null +++ b/app/crowdb-chunk-kv-server/src/storage/tree_owner.rs @@ -0,0 +1,187 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! One fixed process lease covers tree writers and independent read windows. +use super::{ChunkKvStorage, KvRootCatalogStore, StorageRuntimeError}; +use crowdb_chunk_kv::Partition; +use crowdb_paxos_kv_client::StreamProcessAuthority; +use crowdb_tree_ffi::{ChunkPageStoreOptions, ChunkRootCatalog, CtError, PageStore}; +use std::sync::Arc; + +impl ChunkKvStorage { + pub(super) fn tree_config(&self) -> crowdb_tree_ffi::Config { + crowdb_tree_ffi::Config { + runtime: Some(self.tree_runtime.clone()), + ..self.tree_options.clone() + } + } + + /// Bind the same fixed process authority used by serving grants and WAL. + /// Must run before opening any hosted tree. + /// # Errors + /// Rejects a conflicting process binding or invalid WAL authority. + pub fn bind_owner_process( + &self, + process: Arc<StreamProcessAuthority>, + ) -> Result<(), StorageRuntimeError> { + self.streams + .registry() + .bind_owner_process(process.clone()) + .map_err(|error| StorageRuntimeError::Stream(error.to_string()))?; + let expected = process.clone(); + let previous = self + .owner_process + .compare_and_swap(&None::<Arc<StreamProcessAuthority>>, Some(process)); + if previous.as_ref().is_some_and(|old| !Arc::ptr_eq(old, &expected)) { + return Err(StorageRuntimeError::Tree( + "tree process authority is already bound".into(), + )); + } + Ok(()) + } + + pub(super) fn bind_catalog( + &self, + catalog: KvRootCatalogStore, + ) -> Result<Arc<KvRootCatalogStore>, StorageRuntimeError> { + let process = Some( + self.owner_process + .load_full() + .ok_or_else(|| StorageRuntimeError::Tree("tree process authority is not bound".into()))?, + ); + catalog + .with_process(process) + .map(Arc::new) + .map_err(|error| StorageRuntimeError::Tree(error.to_string())) + } + + pub(super) async fn open_durable_tree( + &self, + options: ChunkPageStoreOptions, + metadata_group_id: u64, + ) -> Result<(Arc<PageStore>, Arc<KvRootCatalogStore>), StorageRuntimeError> { + let catalog = KvRootCatalogStore::claim( + self.kv.clone(), + self.metadata_store_id, + metadata_group_id, + options.tree_id, + options.owner_epoch, + ) + .await?; + let catalog = self.bind_catalog(catalog)?; + let store = self.open_tree_page_store( + options, + Arc::new( + ChunkRootCatalog::open_callback(catalog.clone()) + .map_err(|error| StorageRuntimeError::Tree(error.to_string()))?, + ), + )?; + catalog + .register_writer() + .await + .map_err(|error| StorageRuntimeError::Tree(error.to_string()))?; + Ok((store, catalog)) + } + + pub(super) async fn open_assignment_tree( + &self, + entry: &crowdb_protocol::chunk_kv::ChunkKvRangeCatalogEntry, + metadata_group_id: u64, + ) -> Result<(Arc<PageStore>, Arc<KvRootCatalogStore>), StorageRuntimeError> { + let open_generation = entry + .artifact + .tail_overlay + .as_ref() + .map_or(0, |overlay| overlay.base_root_manifest_generation); + let options = ChunkPageStoreOptions { + tree_id: entry.artifact.tree_id, + owner_epoch: entry.owner_epoch, + open_generation, + pack_bytes: 0, + iu_size: 0, + max_concurrent_packs: 0, + materialization_bytes_per_pass: 0, + mirror_copies: 0, + max_chunk_bytes: 0, + }; + let catalog = KvRootCatalogStore::for_assignment( + self.kv.clone(), + self.metadata_store_id, + metadata_group_id, + entry, + ) + .await + .map_err(|error| { + tracing::warn!(tree_id = entry.artifact.tree_id, error = %error, "assignment root claim failed"); + error + })?; + let catalog = self.bind_catalog(catalog)?; + let page_store = self.open_tree_page_store( + options, + Arc::new( + ChunkRootCatalog::open_callback(catalog.clone()) + .map_err(|error| StorageRuntimeError::Tree(error.to_string()))?, + ), + ).map_err(|error| { + tracing::warn!(tree_id = entry.artifact.tree_id, error = %error, "assignment PageStore open failed"); + error + })?; + catalog + .register_writer() + .await + .map_err(|error| { + tracing::warn!(tree_id = entry.artifact.tree_id, error = %error, "assignment writer registration failed"); + StorageRuntimeError::Tree(error.to_string()) + })?; + Ok((page_store, catalog)) + } + + pub(super) async fn complete_tree_recovery( + partition: &Partition, + store: &PageStore, + catalog: &KvRootCatalogStore, + ) -> Result<(), StorageRuntimeError> { + // Each caller has already awaited root and WAL recovery. A prepared + // split retains an overlay artifact but has no live transfer catch-up + // request to complete its separate initialization future. + let generation = match store.chunk_manifest_generation() { + Ok(value) => value, + Err(CtError::NotFound) => 0, + Err(error) => return Err(StorageRuntimeError::Tree(error.to_string())), + }; + catalog + .complete_recovery(partition, generation) + .await + .map_err(|error| StorageRuntimeError::Tree(error.to_string())) + } +} + +#[cfg(feature = "test-util")] +impl ChunkKvStorage { + /// Assemble the production runtime backend with already connected test services. + /// # Errors + /// Returns native route, shared host budget, or storage initialization errors. + pub async fn from_parts_parallel_for_tests( + kv: Arc<crowdb_paxos_kv_client::PaxosKvClient>, + chunk_io: crowdb_chunk_client::ChunkIoClient, + metadata_store_id: u64, + writer_lease_ms: u64, + mirrors: u32, + ) -> Result<Self, StorageRuntimeError> { + Self::from_parts_with_options( + kv, + chunk_io, + metadata_store_id, + writer_lease_ms, + super::TreeStorageOptions { + mirror_copies: mirrors, + chunk_capacity_bytes: 256 * 1024 * 1024, + strip_capacity_bytes: 0, + ..super::TreeStorageOptions::default() + }, + crowdb_chunk_stream::StreamConfig::default(), + crowdb_chunk_client::ChunkReadPolicy::default(), + ) + .await + } +} diff --git a/app/crowdb-chunk-kv-server/tests/backing_observation_test.rs b/app/crowdb-chunk-kv-server/tests/backing_observation_test.rs new file mode 100644 index 000000000..fc494ca7f --- /dev/null +++ b/app/crowdb-chunk-kv-server/tests/backing_observation_test.rs @@ -0,0 +1,107 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +#[path = "common/process.rs"] +mod test_process; + +use std::sync::Arc; + +use crowdb_chunk_kv::{ + CrowdbPartitionTree, MutationOperation, Partition, PartitionConfig, PartitionId, PartitionRange, + PartitionTree, StreamPartitionJournal, +}; +use crowdb_chunk_kv_server::ChunkKvService; +use crowdb_chunk_stream::{ + memory::MemoryStreamStore, ChunkStream, StreamConfig, StreamName, StreamPurpose, StreamRecord, + StreamRecordState, +}; +use crowdb_tree_ffi::{chunk::test_util::TestChunkBackend, Config}; + +async fn partition(id: u64, tree: Arc<dyn PartitionTree>) -> Partition { + let store = Arc::new(MemoryStreamStore::new(4096)); + let name = StreamName { high: 1, low: id }; + let stream = ChunkStream::create( + StreamRecord { + purpose: StreamPurpose::Wal, + stream_name: name, + metadata_group_id: 1, + record_version: 1, + state: StreamRecordState::Active, + consumer_kind: Some("chunk-kv-partition".into()), + consumer_id: None, + created_at_ms: StreamRecord::creation_time_ms(), + }, + 1, + StreamConfig::default(), + store.clone(), + store.clone(), + store, + ) + .await + .unwrap(); + Partition::open( + PartitionId { high: 1, low: id }, + PartitionRange::default(), + 1, + PartitionConfig::default(), + tree, + Arc::new(StreamPartitionJournal::new(stream, name).unwrap()), + ) + .unwrap() +} + +#[tokio::test] +async fn registry_counts_one_physical_ledger_once_across_partition_views() { + let backend = TestChunkBackend::new(82, 1).unwrap(); + let tree = Arc::new( + CrowdbPartitionTree::open( + 82, + &Config { + page_store: Some(backend.page_store(0).unwrap()), + ..backend.config() + }, + ) + .unwrap(), + ); + tree.apply( + 1, + &MutationOperation::Put { + key: b"key".to_vec(), + value: vec![1; 8192], + }, + ) + .await + .unwrap(); + tree.checkpoint(0).await.unwrap(); + let expected = tree.backing_stats().unwrap().unwrap(); + assert!(expected.allocated_bytes >= 8192); + assert_eq!(tree.estimated_bytes().unwrap(), expected.allocated_bytes); + assert!(tree.chunk_stats().is_err()); + let first = partition(1, tree.clone()).await; + let second = partition(2, tree.clone()).await; + assert_eq!( + first.backing_storage_stats().unwrap(), + second.backing_storage_stats().unwrap() + ); + let service = ChunkKvService::new(test_process::TestProcess::authority(1), 8).unwrap(); + service.install_partition(&first).unwrap(); + service.install_partition(&second).unwrap(); + assert_eq!( + service.registry_observation(1 << 20, 0).durable_bytes, + expected.allocated_bytes + ); + let observed = service + .registry_observation_with_load_samples(1 << 20, 0, 0) + .await + .unwrap(); + assert_eq!(observed.partition_loads.len(), 2); + assert!(observed + .partition_loads + .iter() + .all(|load| load.durable_bytes == expected.allocated_bytes)); + first.suspend_for_shutdown(1).await.unwrap(); + second.suspend_for_shutdown(1).await.unwrap(); + drop(service); + first.close().await.unwrap(); + second.close().await.unwrap(); +} diff --git a/app/crowdb-chunk-kv-server/tests/bootstrap_recovery_test.rs b/app/crowdb-chunk-kv-server/tests/bootstrap_recovery_test.rs index 348a6bfed..dd541fa16 100644 --- a/app/crowdb-chunk-kv-server/tests/bootstrap_recovery_test.rs +++ b/app/crowdb-chunk-kv-server/tests/bootstrap_recovery_test.rs @@ -1,20 +1,44 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -use crowdb_chunk_kv::{ - MutationOperation, Partition, PartitionConfig, PartitionId, PartitionRange, RequestId, -}; -use crowdb_chunk_kv_server::{BootstrapPartitionConfig, ChunkKvStorage}; +use crowdb_chunk_kv::{ChunkKvError, MutationOperation, Partition, RequestId}; +use crowdb_chunk_kv_server::{BootstrapPartitionConfig, ChunkKvServerConfig, ChunkKvStorage}; use crowdb_chunk_stream::memory::MemoryStreamStore; use crowdb_chunk_stream::{ ChunkStream, StreamConfig, StreamName, StreamPurpose, StreamRecord, StreamRecordState, }; use crowdb_protocol::chunk_kv::Id128; -use crowdb_tree_ffi::{ChunkPageStoreOptions, ChunkRootCatalog, PageStore}; +use crowdb_tree_ffi::chunk::test_util::TestChunkBackend; use std::sync::Arc; +#[path = "common/process.rs"] +mod test_process; + +fn assert_resource_log(partition: &Partition, runtime: &Arc<crowdb_tree_ffi::TreeRuntime>) { + use crowdb_common::metrics::MetricsSampler; + let service = Arc::new( + crowdb_chunk_kv_server::ChunkKvService::new(test_process::TestProcess::authority(7), 8).unwrap(), + ); + service.install_partition(partition).unwrap(); + let mut sampler = crowdb_chunk_kv_server::metrics::TreeResources::new(runtime.clone(), &service); + sampler.sample(); + let mut bytes = Vec::new(); + sampler.flush(&mut bytes, "test").unwrap(); + let log = String::from_utf8(bytes).unwrap(); + assert!(log.contains("host.tree.resource.cpu_workers"), "{log}"); + assert!(log.contains("chunk.tree.45.resource.writer"), "{log}"); + assert!(log.contains("allocated_sample_max="), "{log}"); +} + #[tokio::test] async fn bootstrap_resume_finishes_its_first_checkpoint_and_does_not_reinitialize_it() { + let mut server_config = ChunkKvServerConfig::load( + &std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("conf/crowdb_chunk_kv_server_config.toml"), + ) + .unwrap(); + server_config.partition.max_key_bytes = 3; + server_config.partition.batch_requests = 2; + server_config.validate().unwrap(); let config = BootstrapPartitionConfig { partition_id: Id128 { high: 1, low: 11 }, tree_id: 45, @@ -24,52 +48,24 @@ async fn bootstrap_resume_finishes_its_first_checkpoint_and_does_not_reinitializ }; let name = StreamName { high: 2, low: 11 }; let store = Arc::new(MemoryStreamStore::new(4096)); - let stream = ChunkStream::create( - StreamRecord { - purpose: StreamPurpose::Wal, - stream_name: name, - metadata_group_id: 7, - record_version: 1, - state: StreamRecordState::Active, - consumer_kind: Some("chunk-kv-partition".into()), - consumer_id: None, - created_at_ms: StreamRecord::creation_time_ms(), - }, - 4, - StreamConfig::default(), - store.clone(), - store.clone(), - store.clone(), - ) - .await - .unwrap(); - let catalog = Arc::new(ChunkRootCatalog::open_memory(4).unwrap()); - let options = ChunkPageStoreOptions { - tree_id: 45, - owner_epoch: 4, - open_generation: 0, - pack_bytes: 4096, - iu_size: 1, - max_concurrent_packs: 2, - max_chunk_bytes: 0, - mirror_copies: 0, - materialization_bytes_per_pass: 4096, + let stream = create_stream(name, &store).await; + let mut backend = TestChunkBackend::new(45, 4).unwrap(); + let pages = backend.page_store(0).unwrap(); + let tree_config = crowdb_tree_ffi::Config { + runtime: Some(backend.runtime.clone()), + ..server_config.native_tree_options() }; - let pages = Arc::new(PageStore::open_chunk(options, catalog.clone(), None).unwrap()); - let partition = Partition::open_native( - PartitionId { high: 1, low: 11 }, - PartitionRange { - start: Some(Vec::new()), - end: None, - }, - 4, - PartitionConfig::default(), - 45, - crowdb_tree_ffi::Config::default(), + let partition = ChunkKvStorage::open_bootstrap_partition_for_tests( + &config, + server_config.partition.clone(), + tree_config.clone(), pages, stream, ) + .await .unwrap(); + assert_resource_log(&partition, &backend.runtime); + assert_partition_limit(&partition).await; partition .mutate( 4, @@ -86,25 +82,43 @@ async fn bootstrap_resume_finishes_its_first_checkpoint_and_does_not_reinitializ .await .unwrap(); partition.suspend_for_transfer(4).await.unwrap(); - drop(partition); + partition.close().await.unwrap(); let stream = reopen(name, &store).await; - let pages = Arc::new(PageStore::open_chunk(options, catalog.clone(), None).unwrap()); - let recovered = ChunkKvStorage::recover_bootstrap_partition_for_tests(&config, pages.clone(), stream) - .await - .unwrap(); + backend.next_writer().unwrap(); + let pages = backend.page_store(0).unwrap(); + let recovered = ChunkKvStorage::recover_bootstrap_partition_for_tests( + &config, + server_config.partition.clone(), + tree_config.clone(), + pages.clone(), + stream, + ) + .await + .unwrap(); assert!(recovered.get(4, b"", None).await.unwrap().is_none()); + assert_partition_limit(&recovered).await; let generation = pages.chunk_manifest_generation().unwrap(); assert!(generation > 0); recovered.suspend_for_transfer(4).await.unwrap(); - drop(recovered); + recovered.close().await.unwrap(); let stream = reopen(name, &store).await; - let pages = Arc::new(PageStore::open_chunk(options, catalog, None).unwrap()); - let recovered = ChunkKvStorage::recover_bootstrap_partition_for_tests(&config, pages.clone(), stream) - .await - .unwrap(); + backend.next_writer().unwrap(); + let pages = backend.page_store(0).unwrap(); + let recovered = ChunkKvStorage::recover_bootstrap_partition_for_tests( + &config, + server_config.partition.clone(), + tree_config, + pages.clone(), + stream, + ) + .await + .unwrap(); assert_eq!(pages.chunk_manifest_generation().unwrap(), generation); recovered.activate_recovered(4).unwrap(); + assert_partition_limit(&recovered).await; assert!(recovered.get(4, b"", None).await.unwrap().is_none()); + recovered.suspend_for_shutdown(4).await.unwrap(); + recovered.close().await.unwrap(); } async fn reopen(name: StreamName, store: &Arc<MemoryStreamStore>) -> ChunkStream { @@ -119,3 +133,43 @@ async fn reopen(name: StreamName, store: &Arc<MemoryStreamStore>) -> ChunkStream .await .unwrap() } + +async fn assert_partition_limit(partition: &Partition) { + let result = partition + .mutate( + 4, + RequestId { + client_high: 1, + client_low: 12, + client_sequence: 1, + }, + MutationOperation::Put { + key: b"four".to_vec(), + value: b"value".to_vec(), + }, + ) + .await; + assert!(matches!(result, Err(ChunkKvError::InvalidRequest(_)))); +} + +async fn create_stream(name: StreamName, store: &Arc<MemoryStreamStore>) -> ChunkStream { + ChunkStream::create( + StreamRecord { + purpose: StreamPurpose::Wal, + stream_name: name, + metadata_group_id: 7, + record_version: 1, + state: StreamRecordState::Active, + consumer_kind: Some("chunk-kv-partition".into()), + consumer_id: None, + created_at_ms: StreamRecord::creation_time_ms(), + }, + 4, + StreamConfig::default(), + store.clone(), + store.clone(), + store.clone(), + ) + .await + .unwrap() +} diff --git a/app/crowdb-chunk-kv-server/tests/common/legacy_retirement.rs b/app/crowdb-chunk-kv-server/tests/common/legacy_retirement.rs new file mode 100644 index 000000000..a5f3c0ce2 --- /dev/null +++ b/app/crowdb-chunk-kv-server/tests/common/legacy_retirement.rs @@ -0,0 +1,265 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +use async_trait::async_trait; +use bytes::Bytes; +use crowdb_chunk_client::{ChunkAllocator, DiskWriter, IoError, Result}; +use crowdb_protocol::{ + chunkdb::rpc::*, + common::{ChunkId, DiskId}, + diskdb::rpc::Segment, +}; +use crowdb_tree_ffi::chunk::legacy::LegacyManifest; +use std::{ + collections::HashMap, + sync::{ + atomic::{AtomicBool, AtomicUsize, Ordering}, + Mutex, + }, + time::Instant, +}; + +pub fn manifest() -> &'static [u8] { + let fixture = include_bytes!("../../../../lib/crowdb-tree/tests/data/legacy-chunk-v7.bin"); + let length = usize::try_from(u64::from_le_bytes(fixture[8..16].try_into().unwrap())).unwrap(); + &fixture[16..16 + length] +} +pub fn typed_root() -> Vec<u8> { + let mut bytes = vec![0; 132]; + for (index, value) in [ + (0, 0x3354_4F4F_524B_4843_u64), + (1, 1), + (2, 42), + (3, 2), + (4, 7), + (5, 1), + (6, 1), + (9, u64::MAX), + ] { + bytes[index * 8..index * 8 + 8].copy_from_slice(&value.to_le_bytes()); + } + let crc = crowdb_tree_ffi::crc32c(&bytes[..128]); + bytes[128..].copy_from_slice(&crc.to_le_bytes()); + bytes +} +#[derive(Default)] +pub struct TestPhysical { + chunks: Mutex<HashMap<(u64, u64), Chunk>>, + pub deletes: AtomicUsize, + pub drains: AtomicUsize, + pub lose_reply: AtomicBool, +} +impl TestPhysical { + pub fn new() -> Self { + let mut chunks = HashMap::new(); + for pack in LegacyManifest::decode(manifest()).unwrap().packs() { + let capacity = + u32::try_from((pack.offset + u64::from(pack.length)).div_ceil(1024) * 1024).unwrap(); + let id = ChunkId { + high: pack.chunk_high, + low: pack.chunk_low, + }; + let chunk = chunks.entry((id.high, id.low)).or_insert_with(|| Chunk { + id: Some(id), + create_ts_ms: 10, + modify_ts: 1, + chunk_type: i32::try_from(id.high >> 56).unwrap(), + state: ChunkState::Sealed as i32, + ..Chunk::default() + }); + chunk.capacity = chunk.capacity.max(capacity); + } + for chunk in chunks.values_mut() { + chunk.strips.push(ChunkStrip { + unit_kb: 1, + capacity: chunk.capacity, + strip_type: StripType::Mirror as i32, + strip: Some(Strip::MirrorStrip(MirrorStrip { + segments: vec![Segment { + disk_id: Some(DiskId { high: 1, low: 2 }), + zone_index: 0, + unit_offset: 0, + unit_count: chunk.capacity / 1024, + ..Segment::default() + }], + })), + ..ChunkStrip::default() + }); + } + Self { + chunks: Mutex::new(chunks), + ..Self::default() + } + } + pub fn set_physical_layout(&self, unit_kb: u32, unit_count: u32) { + for chunk in self.chunks.lock().unwrap().values_mut() { + for strip in &mut chunk.strips { + strip.unit_kb = unit_kb; + let Some(Strip::MirrorStrip(mirror)) = &mut strip.strip else { + panic!("fixture requires mirror strips"); + }; + for segment in &mut mirror.segments { + segment.unit_count = unit_count; + } + } + } + } + pub fn count(&self) -> usize { + self.chunks.lock().unwrap().len() + } + pub fn physical_bytes(&self) -> u64 { + self.chunks + .lock() + .unwrap() + .values() + .map(|chunk| u64::from(chunk.capacity)) + .sum() + } + pub fn all_deleted(&self) -> bool { + self.chunks.lock().unwrap().values().all(|chunk| { + chunk.state == ChunkState::Deleted as i32 && chunk.capacity == 0 && chunk.strips.is_empty() + }) + } +} +#[async_trait] +impl ChunkAllocator for TestPhysical { + async fn allocate_chunk(&self, _: AllocateChunkRequest) -> Result<AllocateChunkResponse> { + Err(IoError::Unsupported("fixture allocation".into())) + } + async fn append_chunk(&self, _: AppendChunkRequest) -> Result<AppendChunkResponse> { + Err(IoError::Unsupported("fixture append".into())) + } + async fn seal_chunk(&self, _: SealChunkRequest) -> Result<SealChunkResponse> { + Err(IoError::Unsupported("fixture seal".into())) + } + async fn query_chunk(&self, request: QueryChunkRequest) -> Result<QueryChunkResponse> { + let id = request.chunk_id.unwrap(); + let chunk = self + .chunks + .lock() + .unwrap() + .get(&(id.high, id.low)) + .cloned() + .ok_or_else(|| IoError::ChunkNotFound("fixture absent".into()))?; + Ok(QueryChunkResponse { + chunk: Some(chunk), + ..QueryChunkResponse::default() + }) + } + async fn delete_chunk(&self, request: DeleteChunkRequest) -> Result<DeleteChunkResponse> { + let id = request.chunk_id.unwrap(); + let mut chunks = self.chunks.lock().unwrap(); + let chunk = chunks.get_mut(&(id.high, id.low)).unwrap(); + assert!(self.drains.load(Ordering::Relaxed) > self.deletes.load(Ordering::Relaxed)); + self.deletes.fetch_add(1, Ordering::Relaxed); + chunk.state = ChunkState::Deleted as i32; + chunk.capacity = 0; + chunk.strips.clear(); + if self.lose_reply.swap(false, Ordering::Relaxed) { + return Err(IoError::WriteFailed("delete committed but reply lost".into())); + } + Ok(DeleteChunkResponse { + chunk: Some(chunk.clone()), + }) + } +} +#[async_trait] +impl DiskWriter for TestPhysical { + async fn write(&self, _: &Segment, _: u64, _: Bytes) -> Result<()> { + Err(IoError::Unsupported("fixture write".into())) + } + async fn write_at_byte_offset(&self, _: &Segment, _: u64, _: u64, _: Bytes) -> Result<()> { + Err(IoError::Unsupported("fixture write".into())) + } + async fn drain_segment(&self, _: &Segment, _: u64, deadline: Instant) -> Result<()> { + assert!(deadline > Instant::now()); + self.drains.fetch_add(1, Ordering::Relaxed); + Ok(()) + } +} + +use crowdb_paxos_kv_client::{ + PaxosKvClient, StreamProcessAuthority, StreamProcessLeaseClient, StreamProcessLeaseTiming, +}; +use crowdb_protocol::{ + chunk_kv::{ + ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogPartitionState, Id128, KeyRange, OwnerDescriptor, + PartitionArtifact, + }, + chunk_stream::{StreamName, StreamPurpose}, + key::{StreamProcessLeaseKey, TextKey}, +}; +use std::{ + sync::Arc, + time::{SystemTime, UNIX_EPOCH}, +}; +pub fn entry(epoch: u64) -> ChunkKvRangeCatalogEntry { + ChunkKvRangeCatalogEntry { + partition_id: Id128 { high: 1, low: 1 }, + range: KeyRange { + start: vec![], + end: None, + }, + owner: OwnerDescriptor { + instance_id: epoch, + rpc_endpoint: "127.0.0.1:1".into(), + }, + owner_epoch: epoch, + state: ChunkKvRangeCatalogPartitionState::Serving, + artifact: PartitionArtifact { + tree_id: 42, + stream_name: StreamName { high: 1, low: 2 }, + tail_overlay: None, + }, + transition_id: None, + } +} +pub fn now() -> u64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_millis() + .try_into() + .unwrap() +} +pub async fn process(kv: &Arc<PaxosKvClient>, instance: u64) -> Arc<StreamProcessAuthority> { + let now = now(); + let lease = StreamProcessLeaseClient::new(kv.clone()) + .claim(StreamPurpose::Wal, instance, [1; 16], None, now, 300_000) + .await + .unwrap(); + Arc::new( + StreamProcessAuthority::new(lease, StreamProcessLeaseTiming::default(), now, Instant::now()).unwrap(), + ) +} +pub async fn expire(kv: &Arc<PaxosKvClient>, process: &StreamProcessAuthority) { + process.fence(); + let mut lease = process.status(Instant::now()).lease; + lease.expires_at_ms = now() - 60_000; + let key = StreamProcessLeaseKey { + purpose: StreamPurpose::Wal, + instance_id: lease.identity.instance_id, + } + .to_path(); + kv.put(0, 0, key.as_bytes(), &serde_json::to_vec(&lease).unwrap(), None) + .await + .unwrap(); +} + +pub fn foreign_manifest(tree: u64) -> Vec<u8> { + let mut bytes = manifest().to_vec(); + bytes[4..12].copy_from_slice(&tree.to_le_bytes()); + let segments = usize::try_from(u64::from_le_bytes(bytes[72..80].try_into().unwrap())).unwrap(); + let packs = 80 + segments * 33 + 8; + let crc = crowdb_tree_ffi::crc32c_update(crowdb_tree_ffi::crc32c(&bytes[..68]), &bytes[packs..]); + let crc = crowdb_tree_ffi::crc32c_update(crc, &bytes[80..packs - 8]); + bytes[68..72].copy_from_slice(&crc.to_le_bytes()); + LegacyManifest::decode(&bytes).unwrap(); + bytes +} +pub fn foreign_typed_root(tree: u64) -> Vec<u8> { + let mut bytes = typed_root(); + bytes[16..24].copy_from_slice(&tree.to_le_bytes()); + let crc = crowdb_tree_ffi::crc32c(&bytes[..128]); + bytes[128..].copy_from_slice(&crc.to_le_bytes()); + bytes +} diff --git a/app/crowdb-chunk-kv-server/tests/common/metadata.rs b/app/crowdb-chunk-kv-server/tests/common/metadata.rs index f6887f81a..a69d066a6 100644 --- a/app/crowdb-chunk-kv-server/tests/common/metadata.rs +++ b/app/crowdb-chunk-kv-server/tests/common/metadata.rs @@ -5,28 +5,46 @@ use std::sync::Arc; -use crowdb_kv::cluster::group::PxGroup; -use crowdb_kv::cluster::kv_server::KvServer; -use crowdb_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient}; +use crowdb_paxos_kv::cluster::group::PxGroup; +use crowdb_paxos_kv::cluster::paxos_kv_server::PaxosKvServer; +use crowdb_paxos_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; +use crowdb_paxos_kv::kv::{CrowdbTreeConfig, CrowdbTreeEngine}; +use crowdb_paxos_kv::wal::replay::replay_group; +use crowdb_paxos_kv::wal::IoBackend; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient}; +use crowdb_tree_ffi::{TreeRuntime, TreeRuntimeOptions}; pub struct TestMetadata { store: Arc<PxKvStore>, - pub client: Arc<CrowdbKvClient>, + pub client: Arc<PaxosKvClient>, } impl TestMetadata { pub async fn start() -> Self { let store = Arc::new(PxKvStore::new(0, "127.0.0.1:0".parse().unwrap())); + // Parallel fixtures are independent hosts. Their two real Tree engines + // share one bounded runtime, without competing with unrelated fixtures. + let runtime = Arc::new(TreeRuntime::new(TreeRuntimeOptions::default()).unwrap()); + let replay = replay_group(&Arc::new(IoBackend::File), &[], 1).await.unwrap(); for id in [0, 1] { - store.add_group(PxGroup::new( - id, - PxLocalReplica::new(1, PxLocalReplicaRole::Leader), - )); + let engine = CrowdbTreeEngine::open(&CrowdbTreeConfig { + runtime: Some(runtime.clone()), + ..Default::default() + }) + .unwrap(); + let replica = PxLocalReplica::restore_from_replay_with_engine( + 1, + PxLocalReplicaRole::Leader, + &replay, + Box::new(engine), + ) + .await + .unwrap(); + store.add_group(PxGroup::new(id, replica)); } store.start().await.expect("start pinned memory metadata"); let endpoint = store.listen_addr().unwrap().to_string(); - let client = Arc::new(CrowdbKvClient::new(ClientConfig::new(vec![]))); + let client = Arc::new(PaxosKvClient::new(ClientConfig::new(vec![]))); for id in [0, 1] { client.seed_leader(0, id, endpoint.clone()); } diff --git a/app/crowdb-chunk-kv-server/tests/common/process.rs b/app/crowdb-chunk-kv-server/tests/common/process.rs index c0c9885b8..0f63a621a 100644 --- a/app/crowdb-chunk-kv-server/tests/common/process.rs +++ b/app/crowdb-chunk-kv-server/tests/common/process.rs @@ -1,7 +1,7 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -use crowdb_kv_client::{StreamProcessAuthority, StreamProcessLeaseTiming}; +use crowdb_paxos_kv_client::{StreamProcessAuthority, StreamProcessLeaseTiming}; use crowdb_protocol::chunk_stream::{StreamProcessIdentity, StreamProcessLease, StreamPurpose}; use std::{sync::Arc, time::Instant}; @@ -22,6 +22,7 @@ impl TestProcess { prior_exclusion_deadline_ms: 0, grant_sequence: 0, grant_expires_at_ms: 0, + revoked: false, }, StreamProcessLeaseTiming::default(), 1_000, diff --git a/app/crowdb-chunk-kv-server/tests/common/process_kv.rs b/app/crowdb-chunk-kv-server/tests/common/process_kv.rs index 99e893b27..1954ef309 100644 --- a/app/crowdb-chunk-kv-server/tests/common/process_kv.rs +++ b/app/crowdb-chunk-kv-server/tests/common/process_kv.rs @@ -1,16 +1,16 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -use crowdb_kv::cluster::{ - group::PxGroup, kv_server::KvServer, PxKvStore, PxLocalReplica, PxLocalReplicaRole, +use crowdb_paxos_kv::cluster::{ + group::PxGroup, paxos_kv_server::PaxosKvServer, PxKvStore, PxLocalReplica, PxLocalReplicaRole, }; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient}; use crowdb_protocol::chunk_kv::{DomainFailurePolicy, DomainMonitorDescriptor}; use std::sync::Arc; pub struct TestProcessKv { pub store: Arc<PxKvStore>, - pub kv: Arc<CrowdbKvClient>, + pub kv: Arc<PaxosKvClient>, } impl TestProcessKv { @@ -21,7 +21,7 @@ impl TestProcessKv { PxLocalReplica::new(1, PxLocalReplicaRole::Leader), )); store.start().await.unwrap(); - let kv = Arc::new(CrowdbKvClient::new(ClientConfig::new(vec![]))); + let kv = Arc::new(PaxosKvClient::new(ClientConfig::new(vec![]))); kv.seed_leader(0, 0, store.listen_addr().unwrap().to_string()); Self { store, kv } } diff --git a/app/crowdb-chunk-kv-server/tests/common/root_process.rs b/app/crowdb-chunk-kv-server/tests/common/root_process.rs new file mode 100644 index 000000000..75537ffdd --- /dev/null +++ b/app/crowdb-chunk-kv-server/tests/common/root_process.rs @@ -0,0 +1,167 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +use crowdb_chunk_kv::{ + memory::MemoryPartitionTree, Checkpoint, Partition, PartitionConfig, PartitionId, PartitionRange, + StreamPartitionJournal, +}; +use crowdb_chunk_stream::{ + memory::MemoryStreamStore, ChunkStream, StreamConfig, StreamRecord, StreamRecordState, +}; +use crowdb_paxos_kv_client::{ + PaxosKvClient, StreamProcessAuthority, StreamProcessLeaseClient, StreamProcessLeaseTiming, +}; +use crowdb_protocol::chunk_kv::{ + ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogPartitionState, Id128, KeyRange, OwnerDescriptor, + PartitionArtifact, +}; +use crowdb_protocol::chunk_stream::{StreamName, StreamPurpose}; +use crowdb_protocol::key::{StreamProcessLeaseKey, TextKey}; +use crowdb_tree_ffi::chunk::stage::{StageId, StageRecord, StageState}; +use std::{ + sync::Arc, + time::{Instant, SystemTime, UNIX_EPOCH}, +}; + +pub fn entry(epoch: u64) -> ChunkKvRangeCatalogEntry { + ChunkKvRangeCatalogEntry { + partition_id: Id128 { high: 1, low: 1 }, + range: KeyRange { + start: vec![], + end: None, + }, + owner: OwnerDescriptor { + instance_id: epoch, + rpc_endpoint: "127.0.0.1:1".into(), + }, + owner_epoch: epoch, + state: ChunkKvRangeCatalogPartitionState::Serving, + artifact: PartitionArtifact { + tree_id: 42, + stream_name: StreamName { high: 1, low: 2 }, + tail_overlay: None, + }, + transition_id: None, + } +} +pub fn stage() -> StageRecord { + StageRecord { + tree_id: 42, + id: StageId { + owner_epoch: 1, + operation_high: 3, + operation_low: 4, + }, + revision: 1, + chunk_high: 2_u64 << 56, + chunk_low: 10, + capacity: 4096, + purpose: 2, + state: StageState::Reserved, + manifest_generation: 0, + packs: vec![], + } +} +pub fn next(old: &StageRecord, state: StageState) -> StageRecord { + let mut next = old.clone(); + next.state = state; + next.revision += 1; + next +} +pub fn now() -> u64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_millis() + .try_into() + .unwrap() +} +pub async fn process(kv: &Arc<PaxosKvClient>, instance: u64) -> Arc<StreamProcessAuthority> { + let now = now(); + let lease = StreamProcessLeaseClient::new(kv.clone()) + .claim(StreamPurpose::Wal, instance, [1; 16], None, now, 300_000) + .await + .unwrap(); + Arc::new( + StreamProcessAuthority::new(lease, StreamProcessLeaseTiming::default(), now, Instant::now()).unwrap(), + ) +} +// Deterministically install an already elapsed authoritative crash exclusion boundary. +// The old local process is fenced before the metadata fixture changes the clock record. +pub async fn expire(kv: &Arc<PaxosKvClient>, process: &StreamProcessAuthority) { + process.fence(); + let mut lease = process.status(Instant::now()).lease; + lease.expires_at_ms = now() - 60_000; + let key = StreamProcessLeaseKey { + purpose: StreamPurpose::Wal, + instance_id: lease.identity.instance_id, + } + .to_path(); + kv.put(0, 0, key.as_bytes(), &serde_json::to_vec(&lease).unwrap(), None) + .await + .unwrap(); +} +pub async fn recovered(epoch: u64) -> Partition { + let store = Arc::new(MemoryStreamStore::new(4096)); + let name = entry(epoch).artifact.stream_name; + let stream = ChunkStream::create( + StreamRecord { + purpose: StreamPurpose::Wal, + stream_name: name, + metadata_group_id: 1, + record_version: 1, + state: StreamRecordState::Active, + consumer_kind: Some("chunk-kv-partition".into()), + consumer_id: None, + created_at_ms: now(), + }, + epoch, + StreamConfig::default(), + store.clone(), + store.clone(), + store, + ) + .await + .unwrap(); + Partition::recover_prepared_assignment( + PartitionId { high: 1, low: 1 }, + PartitionRange::default(), + epoch, + Checkpoint { + tree_id: 42, + tree_manifest: 0, + root_manifest_generation: 0, + applied_seq: 0, + stream_name: name, + stream_generation: 1, + replay_offset: 0, + }, + PartitionConfig::default(), + Arc::new(MemoryPartitionTree::with_tree_id(42)), + Arc::new(StreamPartitionJournal::new(stream, name).unwrap()), + ) + .await + .unwrap() +} + +// Age the already committed operation receipt without waiting for wall time. +// The native final-owner release must have created this exact proof first. +pub async fn age_operation_release(kv: &PaxosKvClient, stage: StageId) { + use crowdb_paxos_kv_client::{GetOutcome, ReadMode}; + let mut key = b"\0crowdb/chunk-kv/root/v1/".to_vec(); + key.extend_from_slice(&42_u64.to_be_bytes()); + key.extend_from_slice(b"/stage-quiesced/"); + for word in [stage.owner_epoch, stage.operation_high, stage.operation_low] { + key.extend_from_slice(&word.to_be_bytes()); + } + let GetOutcome::Found { value, revision } = + kv.get(0, 1, &key, ReadMode::Linearizable, None).await.unwrap() + else { + panic!("native operation release has no durable receipt"); + }; + let mut bytes = value.to_vec(); + assert_eq!(bytes.len(), 84); + bytes[8..16].copy_from_slice(&(now() - 60_000).to_le_bytes()); + let checksum = crowdb_tree_ffi::crc32c(&bytes[..80]); + bytes[80..].copy_from_slice(&checksum.to_le_bytes()); + kv.put_cas(0, 1, &key, &bytes, revision).await.unwrap(); +} diff --git a/app/crowdb-chunk-kv-server/tests/config_test.rs b/app/crowdb-chunk-kv-server/tests/config_test.rs index b39d5906f..072e2209d 100644 --- a/app/crowdb-chunk-kv-server/tests/config_test.rs +++ b/app/crowdb-chunk-kv-server/tests/config_test.rs @@ -57,11 +57,15 @@ fn defaults_close_the_documented_timing_contract() { assert_eq!(config.storage.stream_writer_lease_ms, 30_000); assert_eq!(config.storage.tree_chunk_capacity_bytes, 256 * 1024 * 1024); assert_eq!(config.storage.tree_strip_capacity_bytes, 0); + assert_eq!(config.storage.tree_pin_release_slots, 1_048_576); assert_eq!(config.storage.stream_chunk_capacity_bytes, 256 * 1024 * 1024); assert_eq!(config.storage.diskio_connections_per_endpoint, 1); assert_eq!(config.storage.diskio_rpc_workers, 2); assert_eq!(config.rpc_workers, 2); - assert_eq!(config.rpc_advertise_addr, "127.0.0.1:15200"); + assert_eq!( + config.rpc_advertise_addr, + format!("127.0.0.1:{}", crowdb_protocol::CHUNK_KV_RPC_BASE) + ); config.monitor.self_fence_margin_ms = 3_000; assert!(matches!(config.validate(), Err(ConfigError::Invalid(_)))); @@ -179,3 +183,32 @@ fn data_weight_is_reserved_until_tree_range_metrics_are_available() { config.balance.byte_weight_percent = 80; assert!(config.validate().is_err()); } + +#[test] +fn storage_defaults_round_trip_without_a_tree_mode_switch() { + let config = ChunkKvServerConfig { + instance_id: 1, + ..Default::default() + }; + config.validate().unwrap(); + let encoded = toml::to_string(&config).unwrap(); + let restored: ChunkKvServerConfig = toml::from_str(&encoded).unwrap(); + assert_eq!(config, restored); +} + +#[test] +fn pin_release_capacity_rejects_zero_and_the_native_closed_bit() { + let mut config = ChunkKvServerConfig { + instance_id: 1, + ..Default::default() + }; + for capacity in [0, usize::MAX] { + config.storage.tree_pin_release_slots = capacity; + let error = config.validate().unwrap_err(); + assert!(error.to_string().contains("tree_pin_release_slots"), "{error}"); + } + config.storage.tree_pin_release_slots = 64; + config.validate().unwrap(); + let restored: ChunkKvServerConfig = toml::from_str(&toml::to_string(&config).unwrap()).unwrap(); + assert_eq!(restored.storage.tree_pin_release_slots, 64); +} diff --git a/app/crowdb-chunk-kv-server/tests/legacy_retirement/allocation.rs b/app/crowdb-chunk-kv-server/tests/legacy_retirement/allocation.rs new file mode 100644 index 000000000..364d5fca7 --- /dev/null +++ b/app/crowdb-chunk-kv-server/tests/legacy_retirement/allocation.rs @@ -0,0 +1,36 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +use super::*; + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn overflowing_physical_layout_is_corruption_before_drain_or_delete() { + let metadata = metadata::TestMetadata::start().await; + let owner = process(&metadata.client, 10).await; + let old = open(metadata.client.clone(), 0, 1, &entry(7), owner.clone()) + .await + .unwrap(); + old.store().publish(42, 0, 7, 1, manifest()).unwrap(); + old.store().publish(42, 1, 7, 2, &typed_root()).unwrap(); + expire(&metadata.client, &owner).await; + let current = process(&metadata.client, 20).await; + let physical = Arc::new(TestPhysical::new()); + physical.set_physical_layout(u32::MAX, u32::MAX); + assert_eq!( + collect( + metadata.client.clone(), + current, + physical.clone(), + physical.clone(), + 64, + ) + .await, + Err(CtError::Corruption) + ); + assert_eq!(physical.drains.load(Ordering::Relaxed), 0); + assert_eq!(physical.deletes.load(Ordering::Relaxed), 0); + assert!(!physical.all_deleted()); + assert_eq!( + old.store().load_legacy_migration(42, 1).unwrap().unwrap(), + manifest() + ); +} diff --git a/app/crowdb-chunk-kv-server/tests/legacy_retirement/readers.rs b/app/crowdb-chunk-kv-server/tests/legacy_retirement/readers.rs new file mode 100644 index 000000000..2104c0859 --- /dev/null +++ b/app/crowdb-chunk-kv-server/tests/legacy_retirement/readers.rs @@ -0,0 +1,101 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +use super::*; + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_later_registered_reader_survives_publication_writer_exclusion() { + let metadata = metadata::TestMetadata::start().await; + let owner = process(&metadata.client, 10).await; + let original = open(metadata.client.clone(), 0, 1, &entry(7), owner.clone()) + .await + .unwrap(); + original.store().publish(42, 0, 7, 1, manifest()).unwrap(); + original + .store() + .stage_protection() + .unwrap() + .writer_quiesced(42, 7) + .unwrap(); + let reader_process = process(&metadata.client, 30).await; + let reader = open(metadata.client.clone(), 0, 1, &entry(7), reader_process.clone()) + .await + .unwrap(); + assert_eq!( + reader + .store() + .load(42, crowdb_tree_ffi::RootCatalogObject::Manifest(1)) + .unwrap() + .unwrap(), + manifest() + ); + reader.store().publish(42, 1, 7, 2, &typed_root()).unwrap(); + reader + .store() + .stage_protection() + .unwrap() + .writer_quiesced(42, 7) + .unwrap(); + expire(&metadata.client, &owner).await; + let current = process(&metadata.client, 20).await; + let physical = Arc::new(TestPhysical::new()); + let progress = collect( + metadata.client.clone(), + current.clone(), + physical.clone(), + physical.clone(), + 64, + ) + .await + .unwrap(); + assert!(matches!( + progress.pending, + Some(Pending::LiveProcess { instance: 30, .. }) + )); + assert_eq!(physical.deletes.load(Ordering::Relaxed), 0); + assert_eq!( + reader + .store() + .load(42, crowdb_tree_ffi::RootCatalogObject::Manifest(1)), + Err(CtError::Unavailable) + ); + expire(&metadata.client, &reader_process).await; + for step in 0..16 { + let progress = collect( + metadata.client.clone(), + current.clone(), + physical.clone(), + physical.clone(), + 64, + ) + .await + .unwrap(); + assert!(progress.pending.is_none(), "{progress:?}"); + if progress.complete { + break; + } + assert!(step < 15); + } + assert!(physical.all_deleted()); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn scheduled_resume_does_not_start_a_namespace_upgrade() { + let metadata = metadata::TestMetadata::start().await; + let current = process(&metadata.client, 20).await; + let physical = Arc::new(TestPhysical::new()); + let progress = crowdb_chunk_kv_server::storage::legacy_retirement::resume_legacy_storage_for_tests( + metadata.client.clone(), + current.clone(), + physical.clone(), + physical.clone(), + 64, + ) + .await + .unwrap(); + assert!(progress.is_none()); + let old = open(metadata.client.clone(), 0, 1, &entry(7), current.clone()) + .await + .unwrap(); + old.store().publish(42, 0, 7, 1, manifest()).unwrap(); + assert_eq!(physical.deletes.load(Ordering::Relaxed), 0); +} diff --git a/app/crowdb-chunk-kv-server/tests/legacy_retirement_test.rs b/app/crowdb-chunk-kv-server/tests/legacy_retirement_test.rs new file mode 100644 index 000000000..c6fdfea96 --- /dev/null +++ b/app/crowdb-chunk-kv-server/tests/legacy_retirement_test.rs @@ -0,0 +1,288 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +use crowdb_chunk_kv_server::storage::{ + legacy_retirement::{collect_legacy_storage_for_tests as collect, LegacyRetirementPending as Pending}, + open_root_catalog_with_process_for_tests as open, +}; +use crowdb_tree_ffi::CtError; +use std::sync::{atomic::Ordering, Arc}; +#[path = "common/legacy_retirement.rs"] +mod fixture; +#[path = "common/metadata.rs"] +mod metadata; +use fixture::{entry, expire, process}; +use fixture::{manifest, typed_root, TestPhysical}; + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn quiesced_writer_is_not_a_reader_exclusion_and_deleted_bytes_are_confirmed() { + let metadata = metadata::TestMetadata::start().await; + let old_process = process(&metadata.client, 10).await; + let old = open(metadata.client.clone(), 0, 1, &entry(7), old_process.clone()) + .await + .unwrap(); + old.store().publish(42, 0, 7, 1, manifest()).unwrap(); + old.store().prepare_legacy_migration(42, 1, manifest()).unwrap(); + old.store().publish(42, 1, 7, 2, &typed_root()).unwrap(); + old.store() + .stage_protection() + .unwrap() + .writer_quiesced(42, 7) + .unwrap(); + let current = process(&metadata.client, 20).await; + let physical = Arc::new(TestPhysical::new()); + let bytes = physical.physical_bytes(); + let count = physical.count(); + let progress = collect( + metadata.client.clone(), + current.clone(), + physical.clone(), + physical.clone(), + 64, + ) + .await + .unwrap(); + assert!(matches!( + progress.pending, + Some(Pending::LiveProcess { + tree: 42, + generation: 1, + .. + }) + )); + assert_eq!(physical.deletes.load(Ordering::Relaxed), 0); + expire(&metadata.client, &old_process).await; + let mut reclaimed = 0; + for step in 0..128 { + let progress = collect( + metadata.client.clone(), + current.clone(), + physical.clone(), + physical.clone(), + 4, + ) + .await + .unwrap(); + assert!(progress.pending.is_none(), "{progress:?}"); + reclaimed += progress.reclaimed_bytes; + if progress.complete { + break; + } + assert!(step < 127, "bounded invocations did not finish"); + } + assert!(physical.all_deleted()); + assert_eq!(physical.deletes.load(Ordering::Relaxed), count); + assert_eq!(physical.drains.load(Ordering::Relaxed), count); + assert_eq!(reclaimed, bytes); + assert_eq!( + old.store().load_legacy_migration(42, 1).unwrap().unwrap(), + manifest() + ); + let again = collect( + metadata.client.clone(), + current, + physical.clone(), + physical.clone(), + 4, + ) + .await + .unwrap(); + assert!(again.complete); + assert_eq!(again.reclaimed_bytes, 0); + assert_eq!(physical.deletes.load(Ordering::Relaxed), count); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn legacy_generation_pin_must_be_released_and_new_pins_are_fenced() { + let metadata = metadata::TestMetadata::start().await; + let owner = process(&metadata.client, 10).await; + let old = open(metadata.client.clone(), 0, 1, &entry(7), owner.clone()) + .await + .unwrap(); + old.store().publish(42, 0, 7, 1, manifest()).unwrap(); + old.store().pin_generation(42, 9, 3, 1).unwrap(); + old.store().publish(42, 1, 7, 2, &typed_root()).unwrap(); + let current = process(&metadata.client, 20).await; + let physical = Arc::new(TestPhysical::new()); + let progress = collect( + metadata.client.clone(), + current.clone(), + physical.clone(), + physical.clone(), + 64, + ) + .await + .unwrap(); + assert_eq!( + progress.pending, + Some(Pending::GenerationPin { + tree: 42, + generation: 1 + }) + ); + assert_eq!(old.store().pin_generation(42, 9, 4, 1), Err(CtError::Unavailable)); + old.store().unpin_generation(42, 9, 3).unwrap(); + expire(&metadata.client, &owner).await; + for step in 0..16 { + let progress = collect( + metadata.client.clone(), + current.clone(), + physical.clone(), + physical.clone(), + 64, + ) + .await + .unwrap(); + assert!(progress.pending.is_none(), "{progress:?}"); + if progress.complete { + break; + } + assert!(step < 15); + } + assert!(physical.all_deleted()); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn unknown_delete_reply_resumes_the_exact_claim_without_a_second_delete() { + let metadata = metadata::TestMetadata::start().await; + let owner = process(&metadata.client, 10).await; + let old = open(metadata.client.clone(), 0, 1, &entry(7), owner.clone()) + .await + .unwrap(); + old.store().publish(42, 0, 7, 1, manifest()).unwrap(); + old.store().publish(42, 1, 7, 2, &typed_root()).unwrap(); + expire(&metadata.client, &owner).await; + let current = process(&metadata.client, 20).await; + let physical = Arc::new(TestPhysical::new()); + physical.lose_reply.store(true, Ordering::Relaxed); + assert_eq!( + collect( + metadata.client.clone(), + current.clone(), + physical.clone(), + physical.clone(), + 64 + ) + .await, + Err(CtError::Unavailable) + ); + assert_eq!(physical.deletes.load(Ordering::Relaxed), 1); + for step in 0..16 { + let progress = collect( + metadata.client.clone(), + current.clone(), + physical.clone(), + physical.clone(), + 64, + ) + .await + .unwrap(); + assert!(progress.pending.is_none(), "{progress:?}"); + if progress.complete { + break; + } + assert!(step < 15); + } + assert!(physical.all_deleted()); + assert_eq!(physical.deletes.load(Ordering::Relaxed), physical.count()); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn historical_manifest_without_publication_provenance_stays_durably_pending() { + let metadata = metadata::TestMetadata::start().await; + let old = crowdb_chunk_kv_server::storage::open_root_catalog_for_tests( + metadata.client.clone(), + 0, + 1, + &entry(7), + ) + .await + .unwrap(); + old.publish(42, 0, 7, 1, manifest()).unwrap(); + old.publish(42, 1, 7, 2, &typed_root()).unwrap(); + let current = process(&metadata.client, 20).await; + let physical = Arc::new(TestPhysical::new()); + for _ in 0..2 { + let progress = collect( + metadata.client.clone(), + current.clone(), + physical.clone(), + physical.clone(), + 64, + ) + .await + .unwrap(); + assert_eq!( + progress.pending, + Some(Pending::MissingOwnerProof { + tree: 42, + generation: 1 + }) + ); + assert!(!progress.complete); + } + assert_eq!(physical.deletes.load(Ordering::Relaxed), 0); + assert_eq!(old.load_legacy_migration(42, 1).unwrap().unwrap(), manifest()); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn foreign_legacy_root_blocks_the_namespace_and_shared_chunks_delete_once() { + let metadata = metadata::TestMetadata::start().await; + let owner = process(&metadata.client, 10).await; + let parent = open(metadata.client.clone(), 0, 1, &entry(7), owner.clone()) + .await + .unwrap(); + parent.store().publish(42, 0, 7, 1, manifest()).unwrap(); + let mut child_entry = entry(7); + child_entry.artifact.tree_id = 43; + let child = open(metadata.client.clone(), 0, 1, &child_entry, owner.clone()) + .await + .unwrap(); + let foreign = fixture::foreign_manifest(43); + child.store().publish(43, 0, 7, 1, &foreign).unwrap(); + parent.store().publish(42, 1, 7, 2, &typed_root()).unwrap(); + expire(&metadata.client, &owner).await; + let current = process(&metadata.client, 20).await; + let physical = Arc::new(TestPhysical::new()); + let progress = collect( + metadata.client.clone(), + current.clone(), + physical.clone(), + physical.clone(), + 64, + ) + .await + .unwrap(); + assert_eq!(progress.pending, Some(Pending::LegacyRoot { tree: 43 })); + assert_eq!(physical.deletes.load(Ordering::Relaxed), 0); + let replacement = open(metadata.client.clone(), 0, 1, &child_entry, current.clone()) + .await + .unwrap(); + replacement + .store() + .publish(43, 1, 7, 2, &fixture::foreign_typed_root(43)) + .unwrap(); + for step in 0..16 { + let progress = collect( + metadata.client.clone(), + current.clone(), + physical.clone(), + physical.clone(), + 64, + ) + .await + .unwrap(); + assert!(progress.pending.is_none(), "{progress:?}"); + if progress.complete { + break; + } + assert!(step < 15); + } + assert!(physical.all_deleted()); + assert_eq!(physical.deletes.load(Ordering::Relaxed), physical.count()); +} + +#[path = "legacy_retirement/readers.rs"] +mod readers; + +#[path = "legacy_retirement/allocation.rs"] +mod allocation; diff --git a/app/crowdb-chunk-kv-server/tests/perf_log_test.rs b/app/crowdb-chunk-kv-server/tests/perf_log_test.rs index da464ab31..2ec54eb03 100644 --- a/app/crowdb-chunk-kv-server/tests/perf_log_test.rs +++ b/app/crowdb-chunk-kv-server/tests/perf_log_test.rs @@ -1,16 +1,17 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -use crowdb_chunk_kv_server::metrics::start_stream_perf_log; +use crowdb_chunk_kv_server::metrics::create_perf_log; use crowdb_test_harness::test_dirs::tempdir_in_test_data; #[tokio::test] async fn service_perf_log_flushes_registered_stream_histograms_on_shutdown() { let dir = tempdir_in_test_data("stream-perf-log"); let path = dir.path().to_str().unwrap(); - assert!(start_stream_perf_log(path, 0, 10, 2).unwrap().is_none()); + assert!(create_perf_log(path, 0, 10, 2).unwrap().is_none()); assert_eq!(std::fs::read_dir(dir.path()).unwrap().count(), 0); - let mut runner = start_stream_perf_log(path, 1, 10, 2).unwrap().unwrap(); + let mut runner = create_perf_log(path, 1, 10, 2).unwrap().unwrap(); + runner.start(); assert!(runner .registry() .lock() diff --git a/app/crowdb-chunk-kv-server/tests/process_session_test.rs b/app/crowdb-chunk-kv-server/tests/process_session_test.rs index f1b363ec1..a68d5fda8 100644 --- a/app/crowdb-chunk-kv-server/tests/process_session_test.rs +++ b/app/crowdb-chunk-kv-server/tests/process_session_test.rs @@ -5,7 +5,7 @@ mod fixture; use crowdb_chunk_kv_server::serving::process::ProcessLeaseSession; -use crowdb_kv_client::{GetOutcome, ReadMode, StreamProcessLeaseClient, StreamProcessLeaseError}; +use crowdb_paxos_kv_client::{GetOutcome, ReadMode, StreamProcessLeaseClient, StreamProcessLeaseError}; use crowdb_protocol::{ chunk_stream::StreamPurpose, common::{ChunkKvExtra, InstanceValue}, @@ -280,7 +280,7 @@ async fn queued_heartbeat_cannot_extend_the_original_fence_deadline() { assert!(authority.status(Instant::now()).fenced); } -async fn wait_for_pending_renewal(group: &crowdb_kv::cluster::group::PxGroup) { +async fn wait_for_pending_renewal(group: &crowdb_paxos_kv::cluster::group::PxGroup) { let lease_key = StreamProcessLeaseKey { purpose: StreamPurpose::Wal, instance_id: 7, diff --git a/app/crowdb-chunk-kv-server/tests/root_catalog_legacy_migration_test.rs b/app/crowdb-chunk-kv-server/tests/root_catalog_legacy_migration_test.rs new file mode 100644 index 000000000..eed566ff4 --- /dev/null +++ b/app/crowdb-chunk-kv-server/tests/root_catalog_legacy_migration_test.rs @@ -0,0 +1,102 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +use crowdb_chunk_kv_server::storage::open_root_catalog_for_tests; +use crowdb_protocol::chunk_kv::{ + ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogPartitionState, Id128, KeyRange, OwnerDescriptor, + PartitionArtifact, +}; +use crowdb_protocol::chunk_stream::StreamName; +use crowdb_tree_ffi::{CtError, RootCatalogObject}; +#[path = "common/metadata.rs"] +mod metadata; + +fn entry() -> ChunkKvRangeCatalogEntry { + ChunkKvRangeCatalogEntry { + partition_id: Id128 { high: 1, low: 1 }, + range: KeyRange { + start: vec![], + end: None, + }, + owner: OwnerDescriptor { + instance_id: 7, + rpc_endpoint: "127.0.0.1:1".into(), + }, + owner_epoch: 7, + state: ChunkKvRangeCatalogPartitionState::Serving, + artifact: PartitionArtifact { + tree_id: 42, + stream_name: StreamName { high: 1, low: 2 }, + tail_overlay: None, + }, + transition_id: None, + } +} +fn manifest() -> &'static [u8] { + let fixture = include_bytes!("../../../lib/crowdb-tree/tests/data/legacy-chunk-v7.bin"); + let length = usize::try_from(u64::from_le_bytes(fixture[8..16].try_into().unwrap())).unwrap(); + &fixture[16..16 + length] +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn receipt_is_idempotent_and_survives_old_manifest_retention() { + let metadata = metadata::TestMetadata::start().await; + let store = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry()) + .await + .unwrap(); + store.publish(42, 0, 7, 1, manifest()).unwrap(); + store.prepare_legacy_migration(42, 1, manifest()).unwrap(); + store.prepare_legacy_migration(42, 1, manifest()).unwrap(); + for generation in 2..=4 { + store + .publish(42, generation - 1, 7, generation, b"new root") + .unwrap(); + } + assert_eq!(store.reclaim_before(42, 4), 2); + assert!(store.load(42, RootCatalogObject::Manifest(1)).unwrap().is_none()); + assert_eq!(store.load_legacy_migration(42, 1).unwrap().unwrap(), manifest()); + store.prepare_legacy_migration(42, 1, manifest()).unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn changed_generation_or_corrupt_bytes_cannot_prepare_a_receipt() { + let metadata = metadata::TestMetadata::start().await; + let store = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry()) + .await + .unwrap(); + store.publish(42, 0, 7, 1, manifest()).unwrap(); + assert_eq!( + store.prepare_legacy_migration(42, 2, manifest()), + Err(CtError::InvalidArgument) + ); + let mut damaged = manifest().to_vec(); + let last = damaged.len() - 1; + damaged[last] ^= 1; + assert_eq!( + store.prepare_legacy_migration(42, 1, &damaged), + Err(CtError::Corruption) + ); + store.publish(42, 1, 7, 2, b"replacement root").unwrap(); + assert_eq!( + store.prepare_legacy_migration(42, 1, manifest()), + Err(CtError::Unavailable) + ); + assert!(store.load_legacy_migration(42, 1).unwrap().is_none()); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn concurrent_generation_pin_and_receipt_preserve_both_authority_updates() { + let metadata = metadata::TestMetadata::start().await; + let store = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry()) + .await + .unwrap(); + store.publish(42, 0, 7, 1, manifest()).unwrap(); + let writer = store.clone(); + let receipt = tokio::task::spawn_blocking(move || writer.prepare_legacy_migration(42, 1, manifest())); + let reader = store.clone(); + let pin = tokio::task::spawn_blocking(move || reader.pin_generation(42, 9, 3, 1)); + receipt.await.unwrap().unwrap(); + pin.await.unwrap().unwrap(); + assert_eq!(store.load_legacy_migration(42, 1).unwrap().unwrap(), manifest()); + assert_eq!(store.reclaim_before(42, 4), 0); + store.unpin_generation(42, 9, 3).unwrap(); +} diff --git a/app/crowdb-chunk-kv-server/tests/root_catalog_prepared_staging_test.rs b/app/crowdb-chunk-kv-server/tests/root_catalog_prepared_staging_test.rs new file mode 100644 index 000000000..87a442494 --- /dev/null +++ b/app/crowdb-chunk-kv-server/tests/root_catalog_prepared_staging_test.rs @@ -0,0 +1,212 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use crowdb_chunk_kv_server::{storage::open_root_catalog_with_process_for_tests, Group0ControlStore}; +use crowdb_protocol::chunk_kv::{ + ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogPartitionState, Id128, TailOverlayArtifact, TransferPhase, + TransferReadinessLimits, TransferTransition, +}; +use crowdb_tree_ffi::{ + chunk::stage::{StagePack, StageState}, + CtError, +}; + +#[path = "common/metadata.rs"] +mod metadata; +#[allow(dead_code)] +#[path = "common/root_process.rs"] +mod root_process; + +fn preparation() -> (TransferTransition, ChunkKvRangeCatalogEntry) { + let source = root_process::entry(1); + let mut target = root_process::entry(2); + target.state = ChunkKvRangeCatalogPartitionState::Prepared; + target.transition_id = Some(Id128 { high: 3, low: 4 }); + target.artifact.stream_name.low = 3; + target.artifact.tail_overlay = Some(TailOverlayArtifact { + source_partition_id: source.partition_id, + source_epoch: 1, + source_stream_name: source.artifact.stream_name, + source_stream_generation: 1, + replay_offset: 0, + cutover_offset: 0, + base_root_manifest_generation: 1, + base_tree_manifest: 1, + base_applied_seq: 0, + cutover_seq: 0, + target_stream_start_seq: 1, + }); + let transition = TransferTransition { + transition_id: target.transition_id.unwrap(), + partition_id: target.partition_id, + range: target.range.clone(), + source: source.owner, + source_epoch: source.owner_epoch, + target: target.owner.clone(), + target_epoch: target.owner_epoch, + artifact: source.artifact, + target_artifact: target.artifact.clone(), + readiness_limits: TransferReadinessLimits { + max_tail_records: 100, + max_tail_bytes: 1_000_000, + max_estimated_catchup_ms: 1000, + prepare_deadline_ms: 1000, + forwarding_grace_ms: 1000, + }, + planned_at_ms: 0, + old_grant_expires_at_ms: 100, + phase: TransferPhase::TargetPreparing, + release_proof: None, + readiness_proof: None, + catchup_proof: None, + failure: None, + }; + (transition, target) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn durable_preparation_stages_without_fencing_source_and_abort_stops_next_window() { + let metadata = metadata::TestMetadata::start().await; + let source_process = root_process::process(&metadata.client, 1).await; + let source = open_root_catalog_with_process_for_tests( + metadata.client.clone(), + 0, + 1, + &root_process::entry(1), + source_process, + ) + .await + .unwrap(); + source.store().publish(42, 0, 1, 1, b"source-one").unwrap(); + let (mut transition, entry) = preparation(); + let control = Group0ControlStore::from_client(metadata.client.clone()); + let revision = control.persist_transfer_transition(&transition, 0).await.unwrap(); + let target_process = root_process::process(&metadata.client, 2).await; + let target = + open_root_catalog_with_process_for_tests(metadata.client.clone(), 0, 1, &entry, target_process) + .await + .unwrap(); + let mut stage = root_process::stage(); + stage.id.owner_epoch = 2; + target.store().compare_stage(42, 2, None, Some(&stage)).unwrap(); + source.store().publish(42, 1, 1, 2, b"source-two").unwrap(); + assert_eq!( + target.store().publish(42, 2, 2, 3, b"premature-target"), + Err(CtError::Unavailable) + ); + let mut writing = root_process::next(&stage, StageState::Writing); + writing.packs.push(StagePack { + offset: 0, + length: 4096, + checksum: 1, + }); + target + .store() + .compare_stage(42, 2, Some(&stage), Some(&writing)) + .unwrap(); + transition.phase = TransferPhase::Aborted; + transition.failure = Some("preparation cancelled".into()); + control + .persist_transfer_transition(&transition, revision) + .await + .unwrap(); + let reloadable = root_process::next(&writing, StageState::Reloadable); + assert_eq!( + target + .store() + .compare_stage(42, 2, Some(&writing), Some(&reloadable)), + Err(CtError::Unavailable) + ); + source.store().publish(42, 2, 1, 3, b"source-three").unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn preparation_requires_exact_recovery_base_and_fixed_target_process() { + let metadata = metadata::TestMetadata::start().await; + let source_process = root_process::process(&metadata.client, 1).await; + let source = open_root_catalog_with_process_for_tests( + metadata.client.clone(), + 0, + 1, + &root_process::entry(1), + source_process, + ) + .await + .unwrap(); + let (transition, entry) = preparation(); + Group0ControlStore::from_client(metadata.client.clone()) + .persist_transfer_transition(&transition, 0) + .await + .unwrap(); + let target_process = root_process::process(&metadata.client, 2).await; + let mut wrong = entry.clone(); + wrong.artifact.tail_overlay.as_mut().unwrap().base_tree_manifest += 1; + assert!( + open_root_catalog_with_process_for_tests(metadata.client.clone(), 0, 1, &wrong, target_process,) + .await + .is_err() + ); + let foreign_process = root_process::process(&metadata.client, 3).await; + assert!( + open_root_catalog_with_process_for_tests(metadata.client.clone(), 0, 1, &entry, foreign_process,) + .await + .is_err() + ); + source.store().publish(42, 0, 1, 1, b"still-source").unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn retained_split_tree_can_prepare_transfer_at_a_newer_partition_epoch() { + let metadata = metadata::TestMetadata::start().await; + let source_process = root_process::process(&metadata.client, 1).await; + let source = open_root_catalog_with_process_for_tests( + metadata.client.clone(), + 0, + 1, + &root_process::entry(1), + source_process, + ) + .await + .unwrap(); + source.store().publish(42, 0, 1, 1, b"retained-source").unwrap(); + // A retained split changes partition ownership epoch, keeping its Tree writer. + let (mut transition, mut entry) = preparation(); + transition.source_epoch = 2; + transition.target_epoch = 3; + entry.owner_epoch = 3; + entry.artifact.tail_overlay.as_mut().unwrap().source_epoch = 2; + transition.target_artifact = entry.artifact.clone(); + let control = Group0ControlStore::from_client(metadata.client.clone()); + control.persist_transfer_transition(&transition, 0).await.unwrap(); + let target_process = root_process::process(&metadata.client, 2).await; + let target = + open_root_catalog_with_process_for_tests(metadata.client.clone(), 0, 1, &entry, target_process) + .await + .unwrap(); + let mut stage = root_process::stage(); + stage.id.owner_epoch = 3; + target.store().compare_stage(42, 3, None, Some(&stage)).unwrap(); + source + .store() + .publish(42, 1, 1, 2, b"source-still-writes") + .unwrap(); + assert_eq!( + target.store().publish(42, 2, 3, 3, b"premature-target"), + Err(CtError::Unavailable) + ); + let successor_process = root_process::process(&metadata.client, 4).await; + let _successor = open_root_catalog_with_process_for_tests( + metadata.client.clone(), + 0, + 1, + &root_process::entry(4), + successor_process, + ) + .await + .unwrap(); + let writing = root_process::next(&stage, StageState::Writing); + assert_eq!( + target.store().compare_stage(42, 3, Some(&stage), Some(&writing)), + Err(CtError::Unavailable) + ); +} diff --git a/app/crowdb-chunk-kv-server/tests/root_catalog_process_test.rs b/app/crowdb-chunk-kv-server/tests/root_catalog_process_test.rs new file mode 100644 index 000000000..8709415b1 --- /dev/null +++ b/app/crowdb-chunk-kv-server/tests/root_catalog_process_test.rs @@ -0,0 +1,555 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +use crowdb_chunk_kv_server::storage::open_root_catalog_with_process_for_tests as open; +use crowdb_tree_ffi::chunk::protection::{reference_checksum, CandidateState, ManifestCandidate}; +use crowdb_tree_ffi::chunk::stage::{StageId, StagePack, StageState}; +use crowdb_tree_ffi::{CtError, RootCatalogObject}; +#[path = "common/root_process.rs"] +mod fixture; +#[path = "common/metadata.rs"] +mod metadata; +use fixture::{entry, expire, next, process, recovered, stage}; + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn saturated_pin_pruning_commits_bounded_progress() { + let metadata = metadata::TestMetadata::start().await; + let old_process = process(&metadata.client, 10).await; + let old = open(metadata.client.clone(), 0, 1, &entry(1), old_process.clone()) + .await + .unwrap(); + let reserved = stage(); + old.store().compare_stage(42, 1, None, Some(&reserved)).unwrap(); + let mut writing = next(&reserved, StageState::Writing); + writing.packs.push(StagePack { + offset: 0, + length: 100, + checksum: 7, + }); + old.store() + .compare_stage(42, 1, Some(&reserved), Some(&writing)) + .unwrap(); + let ready = next(&writing, StageState::Reloadable); + old.store() + .compare_stage(42, 1, Some(&writing), Some(&ready)) + .unwrap(); + let protection = old.store().stage_protection().unwrap(); + for low in 100..164 { + protection + .pin_stage( + 42, + ready.id, + StageId { + operation_low: low, + ..ready.id + }, + ) + .unwrap(); + } + let replacement = open( + metadata.client.clone(), + 0, + 1, + &entry(2), + process(&metadata.client, 20).await, + ) + .await + .unwrap(); + replacement + .complete_recovery(&recovered(2).await, 0) + .await + .unwrap(); + expire(&metadata.client, &old_process).await; + let protection = replacement.store().stage_protection().unwrap(); + let first = protection.prune_stage_pins(42, ready.id, None, 64).unwrap(); + assert_eq!(first.released, 62); + assert!(first.more); + let second = protection + .prune_stage_pins(42, ready.id, first.after, 64) + .unwrap(); + assert_eq!(second.released, 3); + assert!(!second.more); + replacement + .store() + .compare_stage(42, 2, Some(&ready), Some(&next(&ready, StageState::Retired))) + .unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn new_epoch_and_completed_recovery_do_not_erase_a_live_old_reader() { + let metadata = metadata::TestMetadata::start().await; + let old_process = process(&metadata.client, 10).await; + let old = open(metadata.client.clone(), 0, 1, &entry(1), old_process.clone()) + .await + .unwrap(); + let reserved = stage(); + old.store().compare_stage(42, 1, None, Some(&reserved)).unwrap(); + let mut writing = next(&reserved, StageState::Writing); + writing.packs.push(StagePack { + offset: 0, + length: 100, + checksum: 7, + }); + old.store() + .compare_stage(42, 1, Some(&reserved), Some(&writing)) + .unwrap(); + let ready = next(&writing, StageState::Reloadable); + old.store() + .compare_stage(42, 1, Some(&writing), Some(&ready)) + .unwrap(); + let pin = StageId { + operation_low: 99, + ..ready.id + }; + let protection = old.store().stage_protection().unwrap(); + protection.pin_stage(42, ready.id, pin).unwrap(); + protection.unpin_stage(42, ready.id, ready.id).unwrap(); + protection.writer_quiesced(42, 1).unwrap(); + assert_eq!(protection.check_read_owner(), Ok(())); + let replacement = open( + metadata.client.clone(), + 0, + 1, + &entry(2), + process(&metadata.client, 20).await, + ) + .await + .unwrap(); + replacement + .complete_recovery(&recovered(2).await, 0) + .await + .unwrap(); + let reader = replacement.store().stage_protection().unwrap(); + let progress = reader.prune_stage_pins(42, ready.id, None, 64).unwrap(); + assert_eq!(progress.released, 0); + let retired = next(&ready, StageState::Retired); + assert_eq!( + replacement + .store() + .compare_stage(42, 2, Some(&ready), Some(&retired)), + Err(CtError::Unavailable) + ); + protection.unpin_stage(42, ready.id, pin).unwrap(); + replacement + .store() + .compare_stage(42, 2, Some(&ready), Some(&retired)) + .unwrap(); + assert_eq!( + protection.pin_stage( + 42, + ready.id, + StageId { + operation_low: 100, + ..pin + } + ), + Err(CtError::Unavailable) + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn actual_recovery_and_authoritative_process_exclusion_release_crash_pins() { + let metadata = metadata::TestMetadata::start().await; + let old_process = process(&metadata.client, 10).await; + let old = open(metadata.client.clone(), 0, 1, &entry(1), old_process.clone()) + .await + .unwrap(); + let reserved = stage(); + old.store().compare_stage(42, 1, None, Some(&reserved)).unwrap(); + let replacement = open( + metadata.client.clone(), + 0, + 1, + &entry(2), + process(&metadata.client, 20).await, + ) + .await + .unwrap(); + let protection = replacement.store().stage_protection().unwrap(); + assert_eq!( + protection.prune_stage_pins(42, reserved.id, None, 64), + Err(CtError::Unavailable) + ); + replacement + .complete_recovery(&recovered(2).await, 0) + .await + .unwrap(); + assert_eq!( + protection + .prune_stage_pins(42, reserved.id, None, 64) + .unwrap() + .released, + 0 + ); + expire(&metadata.client, &old_process).await; + assert_eq!( + old.store().stage_protection().unwrap().check_read_owner(), + Err(CtError::Unavailable) + ); + let progress = protection.prune_stage_pins(42, reserved.id, None, 64).unwrap(); + assert_eq!(progress.examined, 1); + assert_eq!(progress.released, 1); + assert_eq!( + protection + .prune_stage_pins(42, reserved.id, None, 64) + .unwrap() + .released, + 0 + ); + replacement + .store() + .compare_stage( + 42, + 2, + Some(&reserved), + Some(&next(&reserved, StageState::Retired)), + ) + .unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn same_epoch_reopen_cannot_steal_a_live_writer_or_reuse_its_recovery_receipt() { + let metadata = metadata::TestMetadata::start().await; + let old_process = process(&metadata.client, 10).await; + let old = open(metadata.client.clone(), 0, 1, &entry(1), old_process.clone()) + .await + .unwrap(); + old.complete_recovery(&recovered(1).await, 0).await.unwrap(); + for generation in 1..=3 { + old.store() + .publish(42, generation - 1, 1, generation, b"root") + .unwrap(); + } + assert!( + open(metadata.client.clone(), 0, 1, &entry(1), old_process.clone()) + .await + .is_err() + ); + let reserved = stage(); + old.store().compare_stage(42, 1, None, Some(&reserved)).unwrap(); + expire(&metadata.client, &old_process).await; + let replacement = open( + metadata.client.clone(), + 0, + 1, + &entry(1), + process(&metadata.client, 20).await, + ) + .await + .unwrap(); + assert_eq!( + replacement + .store() + .stage_protection() + .unwrap() + .prune_stage_pins(42, reserved.id, None, 64), + Err(CtError::Unavailable) + ); + assert_eq!(old.store().reclaim_before(42, 3), 0); + assert_eq!(old.store().pin_generation(42, 9, 9, 1), Err(CtError::Unavailable)); + assert_eq!( + replacement + .store() + .load(42, RootCatalogObject::Manifest(1)) + .unwrap(), + Some(b"root".to_vec()) + ); + replacement + .complete_recovery(&recovered(1).await, 3) + .await + .unwrap(); + assert_eq!(replacement.store().reclaim_before(42, 3), 1); + assert_eq!( + replacement + .store() + .stage_protection() + .unwrap() + .prune_stage_pins(42, reserved.id, None, 64) + .unwrap() + .released, + 1 + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn recovered_new_owner_aborts_old_candidate_before_releasing_its_references() { + let metadata = metadata::TestMetadata::start().await; + let old_process = process(&metadata.client, 10).await; + let old = open(metadata.client.clone(), 0, 1, &entry(1), old_process.clone()) + .await + .unwrap(); + let reserved = stage(); + old.store().compare_stage(42, 1, None, Some(&reserved)).unwrap(); + let mut writing = next(&reserved, StageState::Writing); + writing.packs.push(StagePack { + offset: 0, + length: 100, + checksum: 7, + }); + old.store() + .compare_stage(42, 1, Some(&reserved), Some(&writing)) + .unwrap(); + let ready = next(&writing, StageState::Reloadable); + old.store() + .compare_stage(42, 1, Some(&writing), Some(&ready)) + .unwrap(); + let id = StageId { + operation_low: 66, + ..ready.id + }; + let candidate = ManifestCandidate { + id, + expected_generation: 0, + reference_count: 1, + reference_checksum: reference_checksum(0, &[ready.id]), + manifest: b"pending-root".to_vec(), + }; + let protection = old.store().stage_protection().unwrap(); + protection.begin_manifest_candidate(42, &candidate).unwrap(); + protection + .prepare_manifest_references(42, id, 0, &[ready.id]) + .unwrap(); + let replacement = open( + metadata.client.clone(), + 0, + 1, + &entry(2), + process(&metadata.client, 20).await, + ) + .await + .unwrap(); + let next_protection = replacement.store().stage_protection().unwrap(); + assert_eq!( + next_protection.recover_manifest_candidate(42, id), + Err(CtError::Unavailable) + ); + replacement + .complete_recovery(&recovered(2).await, 0) + .await + .unwrap(); + next_protection.recover_manifest_candidate(42, id).unwrap(); + assert_eq!( + next_protection + .load_manifest_candidate(42, id) + .unwrap() + .unwrap() + .state, + CandidateState::Aborted + ); + assert_eq!( + protection.commit_manifest_candidate(42, id), + Err(CtError::Unavailable) + ); + next_protection.release_manifest_references(42, id, 0).unwrap(); + next_protection.release_manifest_references(42, id, 0).unwrap(); + let retired = next(&ready, StageState::Retired); + assert_eq!( + replacement + .store() + .compare_stage(42, 2, Some(&ready), Some(&retired)), + Err(CtError::Unavailable) + ); + expire(&metadata.client, &old_process).await; + assert_eq!( + next_protection + .prune_stage_pins(42, ready.id, None, 64) + .unwrap() + .released, + 1 + ); + replacement + .store() + .compare_stage(42, 2, Some(&ready), Some(&retired)) + .unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn unsealed_stage_requires_creator_exclusion_even_after_the_last_pin() { + let metadata = metadata::TestMetadata::start().await; + let old_process = process(&metadata.client, 10).await; + let old = open(metadata.client.clone(), 0, 1, &entry(1), old_process.clone()) + .await + .unwrap(); + old.complete_recovery(&recovered(1).await, 0).await.unwrap(); + let reserved = stage(); + old.store().compare_stage(42, 1, None, Some(&reserved)).unwrap(); + old.store() + .stage_protection() + .unwrap() + .unpin_stage(42, reserved.id, reserved.id) + .unwrap(); + let retired = next(&reserved, StageState::Retired); + assert_eq!( + old.store().compare_stage(42, 1, Some(&reserved), Some(&retired)), + Err(CtError::Unavailable) + ); + let replacement = open( + metadata.client.clone(), + 0, + 1, + &entry(2), + process(&metadata.client, 20).await, + ) + .await + .unwrap(); + replacement + .complete_recovery(&recovered(2).await, 0) + .await + .unwrap(); + assert_eq!( + replacement + .store() + .compare_stage(42, 2, Some(&reserved), Some(&retired)), + Err(CtError::Unavailable) + ); + expire(&metadata.client, &old_process).await; + replacement + .store() + .compare_stage(42, 2, Some(&reserved), Some(&retired)) + .unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn release_before_unknown_stage_creation_fences_its_late_retry() { + let metadata = metadata::TestMetadata::start().await; + let owner = open( + metadata.client.clone(), + 0, + 1, + &entry(1), + process(&metadata.client, 10).await, + ) + .await + .unwrap(); + let reserved = stage(); + let protection = owner.store().stage_protection().unwrap(); + protection.unpin_stage(42, reserved.id, reserved.id).unwrap(); + protection.unpin_stage(42, reserved.id, reserved.id).unwrap(); + assert_eq!( + owner.store().compare_stage(42, 1, None, Some(&reserved)), + Err(CtError::Unavailable) + ); + assert_eq!(owner.store().load_stage(42, reserved.id).unwrap(), None); + let mut created_first = reserved.clone(); + created_first.id.operation_low += 1; + created_first.chunk_low += 1; + owner + .store() + .compare_stage(42, 1, None, Some(&created_first)) + .unwrap(); + protection + .unpin_stage(42, created_first.id, created_first.id) + .unwrap(); + protection + .unpin_stage(42, created_first.id, created_first.id) + .unwrap(); + assert_eq!( + protection.prune_stage_pins(42, created_first.id, None, 64), + Err(CtError::Unavailable) + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn terminal_unknown_operation_can_retire_while_its_process_and_tree_keep_serving() { + let metadata = metadata::TestMetadata::start().await; + let process = process(&metadata.client, 10).await; + let owner = open(metadata.client.clone(), 0, 1, &entry(1), process.clone()) + .await + .unwrap(); + owner.complete_recovery(&recovered(1).await, 0).await.unwrap(); + let reserved = stage(); + owner.store().compare_stage(42, 1, None, Some(&reserved)).unwrap(); + let protection = owner.store().stage_protection().unwrap(); + protection.unpin_stage(42, reserved.id, reserved.id).unwrap(); + let retired = next(&reserved, StageState::Retired); + assert_eq!( + owner + .store() + .compare_stage(42, 1, Some(&reserved), Some(&retired)), + Err(CtError::Unavailable) + ); + fixture::age_operation_release(&metadata.client, reserved.id).await; + protection.unpin_stage(42, reserved.id, reserved.id).unwrap(); + assert_eq!(protection.check_read_owner(), Ok(())); + owner + .store() + .compare_stage(42, 1, Some(&reserved), Some(&retired)) + .unwrap(); + let mut ongoing = reserved; + ongoing.id.operation_low += 1; + ongoing.chunk_low += 1; + owner.store().compare_stage(42, 1, None, Some(&ongoing)).unwrap(); + assert_eq!(protection.check_read_owner(), Ok(())); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn drained_writer_reopens_immediately_without_expiring_its_old_readers() { + let metadata = metadata::TestMetadata::start().await; + let process = process(&metadata.client, 10).await; + let old = open(metadata.client.clone(), 0, 1, &entry(1), process.clone()) + .await + .unwrap(); + let old_reader = old.store().stage_protection().unwrap(); + old_reader.writer_quiesced(42, 1).unwrap(); + let replacement = open(metadata.client.clone(), 0, 1, &entry(1), process) + .await + .unwrap(); + assert_eq!(old_reader.check_read_owner(), Ok(())); + assert_eq!( + old.store().compare_stage(42, 1, None, Some(&stage())), + Err(CtError::Unavailable) + ); + replacement + .store() + .compare_stage(42, 1, None, Some(&stage())) + .unwrap(); + // An idempotent late release still refers only to the old incarnation. + old_reader.writer_quiesced(42, 1).unwrap(); + let mut another = stage(); + another.id.operation_low += 1; + another.chunk_low += 1; + replacement + .store() + .compare_stage(42, 1, None, Some(&another)) + .unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn sealed_legacy_stage_can_retire_after_recovery_without_an_invented_process_owner() { + let metadata = metadata::TestMetadata::start().await; + let old = crowdb_chunk_kv_server::storage::open_root_catalog_for_tests( + metadata.client.clone(), + 0, + 1, + &entry(1), + ) + .await + .unwrap(); + let reserved = stage(); + old.compare_stage(42, 1, None, Some(&reserved)).unwrap(); + let mut writing = next(&reserved, StageState::Writing); + writing.packs.push(StagePack { + offset: 0, + length: 100, + checksum: 7, + }); + old.compare_stage(42, 1, Some(&reserved), Some(&writing)).unwrap(); + let ready = next(&writing, StageState::Reloadable); + old.compare_stage(42, 1, Some(&writing), Some(&ready)).unwrap(); + let replacement = open( + metadata.client.clone(), + 0, + 1, + &entry(2), + process(&metadata.client, 20).await, + ) + .await + .unwrap(); + replacement + .complete_recovery(&recovered(2).await, 0) + .await + .unwrap(); + replacement + .store() + .compare_stage(42, 2, Some(&ready), Some(&next(&ready, StageState::Retired))) + .unwrap(); +} diff --git a/app/crowdb-chunk-kv-server/tests/root_catalog_reference_test.rs b/app/crowdb-chunk-kv-server/tests/root_catalog_reference_test.rs new file mode 100644 index 000000000..295ac9b16 --- /dev/null +++ b/app/crowdb-chunk-kv-server/tests/root_catalog_reference_test.rs @@ -0,0 +1,462 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use crowdb_chunk_kv_server::storage::open_root_catalog_for_tests; +use crowdb_protocol::chunk_kv::{ + ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogPartitionState, Id128, KeyRange, OwnerDescriptor, + PartitionArtifact, +}; +use crowdb_protocol::chunk_stream::StreamName; +use crowdb_tree_ffi::chunk::protection::{ + reference_checksum, CandidateState, ManifestCandidate, MAX_REFERENCE_WINDOW, +}; +use crowdb_tree_ffi::chunk::stage::{StageId, StagePack, StageRecord, StageState}; +use crowdb_tree_ffi::{CtError, RootCatalogObject, RootCatalogStore}; +#[path = "common/metadata.rs"] +mod metadata; + +fn entry(epoch: u64) -> ChunkKvRangeCatalogEntry { + ChunkKvRangeCatalogEntry { + partition_id: Id128 { high: 1, low: 1 }, + range: KeyRange { + start: vec![], + end: None, + }, + owner: OwnerDescriptor { + instance_id: epoch, + rpc_endpoint: "127.0.0.1:1".into(), + }, + owner_epoch: epoch, + state: ChunkKvRangeCatalogPartitionState::Serving, + artifact: PartitionArtifact { + tree_id: 42, + stream_name: StreamName { high: 1, low: 2 }, + tail_overlay: None, + }, + transition_id: None, + } +} +fn id(low: u64) -> StageId { + StageId { + owner_epoch: 1, + operation_high: 100, + operation_low: low, + } +} +fn next(old: &StageRecord, state: StageState) -> StageRecord { + let mut value = old.clone(); + value.state = state; + value.revision += 1; + value +} +fn ready(store: &dyn RootCatalogStore, low: u64) -> StageRecord { + let reserved = StageRecord { + tree_id: 42, + id: id(low), + revision: 1, + chunk_high: 2_u64 << 56, + chunk_low: low, + capacity: 4096, + purpose: 2, + state: StageState::Reserved, + manifest_generation: 0, + packs: vec![], + }; + store.compare_stage(42, 1, None, Some(&reserved)).unwrap(); + let mut writing = next(&reserved, StageState::Writing); + writing.packs.push(StagePack { + offset: 0, + length: 4000, + checksum: 7, + }); + store + .compare_stage(42, 1, Some(&reserved), Some(&writing)) + .unwrap(); + let ready = next(&writing, StageState::Reloadable); + store.compare_stage(42, 1, Some(&writing), Some(&ready)).unwrap(); + ready +} +fn candidate(low: u64, generation: u64, refs: &[StageId]) -> ManifestCandidate { + ManifestCandidate { + id: id(low), + expected_generation: generation - 1, + reference_count: refs.len() as u64, + reference_checksum: reference_checksum(0, refs), + manifest: format!("fixed manifest {low}").into_bytes(), + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn full_reference_windows_prepare_and_release_with_native_metadata_admission() { + let metadata = metadata::TestMetadata::start().await; + let store = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry(1)) + .await + .unwrap(); + let stages: Vec<_> = (1..=MAX_REFERENCE_WINDOW as u64 + 1) + .map(|low| ready(store.as_ref(), low)) + .collect(); + let refs: Vec<_> = stages.iter().map(|stage| stage.id).collect(); + let protection = store.stage_protection().unwrap(); + // Exercise both a full final window and a full non-final window. + for (index, refs) in [&refs[..MAX_REFERENCE_WINDOW], refs.as_slice()] + .into_iter() + .enumerate() + { + let plan = candidate(100 + index as u64, 1, refs); + protection.begin_manifest_candidate(42, &plan).unwrap(); + for (window, refs) in refs.chunks(MAX_REFERENCE_WINDOW).enumerate() { + protection + .prepare_manifest_references(42, plan.id, window as u64, refs) + .unwrap(); + } + protection.abort_manifest_candidate(42, plan.id).unwrap(); + for window in 0..refs.len().div_ceil(MAX_REFERENCE_WINDOW) { + protection + .release_manifest_references(42, plan.id, window as u64) + .unwrap(); + } + } + for stage in stages { + let retired = next(&stage, StageState::Retired); + store.compare_stage(42, 1, Some(&stage), Some(&retired)).unwrap(); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn generation_reclamation_advances_within_complete_transaction_limit() { + let metadata = metadata::TestMetadata::start().await; + let store = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry(1)) + .await + .unwrap(); + let protection = store.stage_protection().unwrap(); + for generation in 1..=70 { + let plan = candidate(100 + generation, generation, &[]); + protection.begin_manifest_candidate(42, &plan).unwrap(); + protection.commit_manifest_candidate(42, plan.id).unwrap(); + } + let mut reclaimed = 0; + while reclaimed < 68 { + let count = store.reclaim_before(42, 70); + assert!(count > 0, "reclamation stopped at generation {reclaimed}"); + reclaimed += count; + } + assert_eq!(reclaimed, 68); + assert_eq!(store.reclaim_before(42, 70), 0); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn prepared_references_protect_before_commit_and_survive_unknown_commit_reply() { + let metadata = metadata::TestMetadata::start().await; + let store = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry(1)) + .await + .unwrap(); + let first = ready(store.as_ref(), 1); + let second = ready(store.as_ref(), 2); + let refs = [first.id, second.id]; + let plan = candidate(10, 1, &refs); + let protection = store.stage_protection().unwrap(); + protection.begin_manifest_candidate(42, &plan).unwrap(); + protection + .prepare_manifest_references(42, plan.id, 0, &refs[..1]) + .unwrap(); + let retired = next(&first, StageState::Retired); + assert_eq!( + store.compare_stage(42, 1, Some(&first), Some(&retired)), + Err(CtError::Unavailable) + ); + assert_eq!( + protection.commit_manifest_candidate(42, plan.id), + Err(CtError::Unavailable) + ); + protection + .prepare_manifest_references(42, plan.id, 1, &refs[1..]) + .unwrap(); + protection + .prepare_manifest_references(42, plan.id, 1, &refs[1..]) + .unwrap(); + // Discard the result, query exactly this operation, then repeat it. + protection.commit_manifest_candidate(42, plan.id).unwrap(); + assert_eq!( + protection + .load_manifest_candidate(42, plan.id) + .unwrap() + .unwrap() + .state, + CandidateState::Committed + ); + protection.commit_manifest_candidate(42, plan.id).unwrap(); + assert_eq!( + protection.abort_manifest_candidate(42, plan.id), + Err(CtError::Unavailable) + ); + assert_eq!( + store.load(42, RootCatalogObject::CurrentManifest).unwrap(), + Some(plan.manifest) + ); + assert_eq!(protection.list_manifest_candidates(42, None, 1).unwrap().len(), 1); + assert!(protection + .list_manifest_candidates(42, Some(plan.id), 1) + .unwrap() + .is_empty()); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn generation_pin_and_live_pin_both_prevent_pack_deletion() { + let metadata = metadata::TestMetadata::start().await; + let store = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry(1)) + .await + .unwrap(); + let first = ready(store.as_ref(), 1); + let protection = store.stage_protection().unwrap(); + let plan = candidate(10, 1, &[first.id]); + protection.begin_manifest_candidate(42, &plan).unwrap(); + protection + .prepare_manifest_references(42, plan.id, 0, &[first.id]) + .unwrap(); + protection.commit_manifest_candidate(42, plan.id).unwrap(); + store.pin_generation(42, 1, 99, 1).unwrap(); + protection.pin_stage(42, first.id, id(200)).unwrap(); + for generation in 2..=4 { + store + .publish(42, generation - 1, 1, generation, b"other root") + .unwrap(); + } + assert_eq!(store.reclaim_before(42, 4), 0); + assert_eq!( + protection.release_manifest_references(42, plan.id, 0), + Err(CtError::Unavailable) + ); + store.unpin_generation(42, 1, 99).unwrap(); + assert_eq!(store.reclaim_before(42, 4), 2); + assert_eq!( + protection + .load_manifest_candidate(42, plan.id) + .unwrap() + .unwrap() + .state, + CandidateState::Reclaimed + ); + protection.release_manifest_references(42, plan.id, 0).unwrap(); + protection.release_manifest_references(42, plan.id, 0).unwrap(); + let retired = next(&first, StageState::Retired); + assert_eq!( + store.compare_stage(42, 1, Some(&first), Some(&retired)), + Err(CtError::Unavailable) + ); + protection.unpin_stage(42, first.id, id(200)).unwrap(); + store.compare_stage(42, 1, Some(&first), Some(&retired)).unwrap(); + let deleting = next(&retired, StageState::Deleting); + store + .compare_stage(42, 1, Some(&retired), Some(&deleting)) + .unwrap(); + assert_eq!( + protection.pin_stage(42, first.id, id(201)), + Err(CtError::Unavailable) + ); + let new_plan = candidate(11, 5, &[first.id]); + protection.begin_manifest_candidate(42, &new_plan).unwrap(); + assert_eq!( + protection.prepare_manifest_references(42, new_plan.id, 0, &[first.id]), + Err(CtError::Unavailable) + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn abort_releases_only_prepared_windows_and_fixed_identity_cannot_be_reallocated() { + let metadata = metadata::TestMetadata::start().await; + let store = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry(1)) + .await + .unwrap(); + let first = ready(store.as_ref(), 1); + let second = ready(store.as_ref(), 2); + let protection = store.stage_protection().unwrap(); + let plan = candidate(10, 1, &[first.id, second.id]); + protection.begin_manifest_candidate(42, &plan).unwrap(); + protection + .prepare_manifest_references(42, plan.id, 0, &[first.id]) + .unwrap(); + protection.abort_manifest_candidate(42, plan.id).unwrap(); + assert_eq!( + protection.prepare_manifest_references(42, plan.id, 1, &[second.id]), + Err(CtError::Unavailable) + ); + protection.release_manifest_references(42, plan.id, 0).unwrap(); + let mut last = first; + for state in [StageState::Retired, StageState::Deleting, StageState::Reclaimed] { + let value = next(&last, state); + store.compare_stage(42, 1, Some(&last), Some(&value)).unwrap(); + last = value; + } + store.compare_stage(42, 1, Some(&last), None).unwrap(); + let duplicate = StageRecord { + id: id(999), + revision: 1, + state: StageState::Reserved, + packs: vec![], + ..last + }; + assert_eq!( + store.compare_stage(42, 1, None, Some(&duplicate)), + Err(CtError::Unavailable) + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn checksum_failure_does_not_publish_protection_and_released_pin_never_resurrects() { + let metadata = metadata::TestMetadata::start().await; + let store = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry(1)) + .await + .unwrap(); + let first = ready(store.as_ref(), 1); + let protection = store.stage_protection().unwrap(); + let mut plan = candidate(10, 1, &[first.id]); + plan.reference_checksum ^= 1; + protection.begin_manifest_candidate(42, &plan).unwrap(); + assert_eq!( + protection.prepare_manifest_references(42, plan.id, 0, &[first.id]), + Err(CtError::Corruption) + ); + protection.unpin_stage(42, first.id, id(100)).unwrap(); + assert_eq!( + protection.pin_stage(42, first.id, id(100)), + Err(CtError::Unavailable) + ); + let retired = next(&first, StageState::Retired); + store.compare_stage(42, 1, Some(&first), Some(&retired)).unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn live_pin_invalidates_an_observed_deletion_proof_at_the_authority_revision() { + use bytes::Bytes; + use crowdb_paxos_kv_client::{BatchOp, GetOutcome, ReadMode}; + let metadata = metadata::TestMetadata::start().await; + let store = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry(1)) + .await + .unwrap(); + let first = ready(store.as_ref(), 1); + let mut prefix = b"\0crowdb/chunk-kv/root/v1/".to_vec(); + prefix.extend_from_slice(&42_u64.to_be_bytes()); + let mut authority = prefix.clone(); + authority.extend_from_slice(b"/authority/"); + authority.extend_from_slice(&0_u64.to_be_bytes()); + let GetOutcome::Found { value, revision } = metadata + .client + .get(0, 1, &authority, ReadMode::Linearizable, None) + .await + .unwrap() + else { + panic!("authority must exist"); + }; + store + .stage_protection() + .unwrap() + .pin_stage(42, first.id, id(200)) + .unwrap(); + let mut stage_key = prefix; + stage_key.extend_from_slice(b"/stage/v1/"); + for part in [ + first.id.owner_epoch, + first.id.operation_high, + first.id.operation_low, + ] { + stage_key.extend_from_slice(&part.to_be_bytes()); + } + let ops = [ + BatchOp::Put { + key: Bytes::from(authority.clone()), + value, + }, + BatchOp::Put { + key: Bytes::from(stage_key), + value: Bytes::from(next(&first, StageState::Retired).encode().unwrap()), + }, + ]; + assert!(matches!( + metadata + .client + .batch_write_cas(0, 1, &ops, &authority, revision) + .await, + Err(crowdb_paxos_kv_client::Error::CasFailed { .. } | crowdb_paxos_kv_client::Error::CasBusy) + )); + assert_eq!(store.load_stage(42, first.id).unwrap(), Some(first)); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn candidate_decode_rejects_oversized_checksum_words_even_with_valid_record_crc() { + let metadata = metadata::TestMetadata::start().await; + let store = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry(1)) + .await + .unwrap(); + let candidate_id = id(50); + let mut key = b"\0crowdb/chunk-kv/root/v1/".to_vec(); + key.extend_from_slice(&42_u64.to_be_bytes()); + key.extend_from_slice(b"/candidate/"); + for value in [ + candidate_id.owner_epoch, + candidate_id.operation_high, + candidate_id.operation_low, + ] { + key.extend_from_slice(&value.to_be_bytes()); + } + for checksum_index in 7..10 { + let mut words = [0_u64; 14]; + words[..3].copy_from_slice(&[1, 2, 1]); + words[checksum_index] = u64::from(u32::MAX) + 1; + let mut bytes = words.into_iter().flat_map(u64::to_le_bytes).collect::<Vec<_>>(); + bytes.extend_from_slice(&crowdb_tree_ffi::crc32c(&bytes).to_le_bytes()); + metadata.client.put(0, 1, &key, &bytes, None).await.unwrap(); + assert_eq!( + store + .stage_protection() + .unwrap() + .load_manifest_candidate(42, candidate_id), + Err(CtError::Corruption), + "oversized checksum word {checksum_index} must not truncate" + ); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn concurrent_pin_mutations_rebuild_guard_counts_before_gc() { + let metadata = metadata::TestMetadata::start().await; + let store = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry(1)) + .await + .unwrap(); + let stage = ready(store.as_ref(), 1); + let retired = next(&stage, StageState::Retired); + for turn in 0..4 { + let pins: [_; 32] = std::array::from_fn(|index| id(100 + 32 * turn + u64::try_from(index).unwrap())); + let start = std::sync::Arc::new(std::sync::Barrier::new(pins.len())); + let jobs = pins.map(|pin| { + let catalog = store.clone(); + let start = start.clone(); + let stage_id = stage.id; + tokio::task::spawn_blocking(move || { + start.wait(); + catalog.stage_protection().unwrap().pin_stage(42, stage_id, pin) + }) + }); + for job in jobs { + job.await.unwrap().unwrap(); + } + for pin in &pins[..pins.len() - 1] { + store + .stage_protection() + .unwrap() + .unpin_stage(42, stage.id, *pin) + .unwrap(); + } + assert_eq!( + store.compare_stage(42, 1, Some(&stage), Some(&retired)), + Err(CtError::Unavailable), + "one remaining live pin must retain its exact guard count" + ); + store + .stage_protection() + .unwrap() + .unpin_stage(42, stage.id, *pins.last().unwrap()) + .unwrap(); + } + store.compare_stage(42, 1, Some(&stage), Some(&retired)).unwrap(); + assert_eq!(store.load_stage(42, stage.id).unwrap(), Some(retired)); +} diff --git a/app/crowdb-chunk-kv-server/tests/root_catalog_staging_test.rs b/app/crowdb-chunk-kv-server/tests/root_catalog_staging_test.rs new file mode 100644 index 000000000..bf59bbcb7 --- /dev/null +++ b/app/crowdb-chunk-kv-server/tests/root_catalog_staging_test.rs @@ -0,0 +1,207 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use bytes::Bytes; +use crowdb_chunk_kv_server::storage::open_root_catalog_for_tests; +use crowdb_paxos_kv_client::{BatchOp, GetOutcome, ReadMode}; +use crowdb_protocol::chunk_kv::{ + ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogPartitionState, Id128, KeyRange, OwnerDescriptor, + PartitionArtifact, +}; +use crowdb_protocol::chunk_stream::StreamName; +use crowdb_tree_ffi::chunk::stage::{StageId, StagePack, StageRecord, StageState}; +use crowdb_tree_ffi::{ChunkRootCatalog, CtError, RootCatalogObject}; +use std::sync::Arc; +#[path = "common/metadata.rs"] +mod metadata; + +fn entry(epoch: u64) -> ChunkKvRangeCatalogEntry { + ChunkKvRangeCatalogEntry { + partition_id: Id128 { high: 1, low: 1 }, + range: KeyRange { + start: vec![], + end: None, + }, + owner: OwnerDescriptor { + instance_id: epoch, + rpc_endpoint: "127.0.0.1:1".into(), + }, + owner_epoch: epoch, + state: ChunkKvRangeCatalogPartitionState::Serving, + artifact: PartitionArtifact { + tree_id: 42, + stream_name: StreamName { high: 1, low: 2 }, + tail_overlay: None, + }, + transition_id: None, + } +} +fn record(low: u64) -> StageRecord { + StageRecord { + tree_id: 42, + id: StageId { + owner_epoch: 1, + operation_high: 100, + operation_low: low, + }, + revision: 1, + chunk_high: 2_u64 << 56, + chunk_low: low, + capacity: 4096, + purpose: 2, + state: StageState::Reserved, + manifest_generation: 0, + packs: vec![], + } +} +fn next(old: &StageRecord, state: StageState) -> StageRecord { + let mut out = old.clone(); + out.revision += 1; + out.state = state; + out +} +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn callback_stage_roundtrip_survives_reopen_and_retries_the_same_operation() { + let metadata = metadata::TestMetadata::start().await; + let store = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry(1)) + .await + .unwrap(); + let catalog = ChunkRootCatalog::open_callback(Arc::clone(&store)).unwrap(); + assert!(catalog.supports_staging()); + let first = record(1); + catalog.compare_stage(42, 1, None, Some(&first)).unwrap(); + // Discarding the first result models a successful commit whose reply was lost. + catalog.compare_stage(42, 1, None, Some(&first)).unwrap(); + assert_eq!(catalog.load_stage(42, first.id).unwrap(), Some(first.clone())); + let mut writing = next(&first, StageState::Writing); + writing.packs.push(StagePack { + offset: 0, + length: 4096, + checksum: 7, + }); + catalog + .compare_stage(42, 1, Some(&first), Some(&writing)) + .unwrap(); + catalog + .compare_stage(42, 1, Some(&first), Some(&writing)) + .unwrap(); + let second = record(2); + catalog.compare_stage(42, 1, None, Some(&second)).unwrap(); + assert_eq!(catalog.list_stages(42, None, 1).unwrap(), vec![writing.clone()]); + assert_eq!(catalog.list_stages(42, Some(first.id), 1).unwrap(), vec![second]); + drop(catalog); + drop(store); + let reopened = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry(1)) + .await + .unwrap(); + assert_eq!(reopened.load_stage(42, first.id).unwrap(), Some(writing)); +} +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn epoch_fencing_rejects_stale_transitions_and_keeps_unresolved_allocations() { + let metadata = metadata::TestMetadata::start().await; + let old = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry(1)) + .await + .unwrap(); + let first = record(1); + old.compare_stage(42, 1, None, Some(&first)).unwrap(); + let writing = next(&first, StageState::Writing); + let current = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry(2)) + .await + .unwrap(); + assert_eq!( + old.compare_stage(42, 1, Some(&first), Some(&writing)), + Err(CtError::Unavailable) + ); + assert_eq!( + current.compare_stage(42, 2, Some(&first), Some(&writing)), + Err(CtError::Unavailable) + ); + assert_eq!(current.load_stage(42, first.id).unwrap(), Some(first)); + assert_eq!(current.list_stages(42, None, 64).unwrap().len(), 1); +} +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn stale_cas_and_cleanup_cannot_skip_reclamation_completion() { + let metadata = metadata::TestMetadata::start().await; + let store = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry(1)) + .await + .unwrap(); + let catalog = ChunkRootCatalog::open_callback(store).unwrap(); + let first = record(1); + catalog.compare_stage(42, 1, None, Some(&first)).unwrap(); + let writing = next(&first, StageState::Writing); + catalog + .compare_stage(42, 1, Some(&first), Some(&writing)) + .unwrap(); + let stale_retired = next(&first, StageState::Retired); + assert_eq!( + catalog.compare_stage(42, 1, Some(&first), Some(&stale_retired)), + Err(CtError::Unavailable) + ); + assert_eq!( + catalog.compare_stage(42, 1, Some(&writing), None), + Err(CtError::InvalidArgument) + ); + let mut last = writing; + for state in [StageState::Retired, StageState::Deleting, StageState::Reclaimed] { + let value = next(&last, state); + catalog.compare_stage(42, 1, Some(&last), Some(&value)).unwrap(); + last = value; + } + catalog.compare_stage(42, 1, Some(&last), None).unwrap(); + catalog.compare_stage(42, 1, Some(&last), None).unwrap(); + assert_eq!(catalog.load_stage(42, last.id).unwrap(), None); + assert!(catalog.list_stages(42, None, 1).unwrap().is_empty()); +} +fn key(kind: &[u8], id: u64) -> Vec<u8> { + let mut key = b"\0crowdb/chunk-kv/root/v1/".to_vec(); + key.extend_from_slice(&42_u64.to_be_bytes()); + key.push(b'/'); + key.extend_from_slice(kind); + key.push(b'/'); + key.extend_from_slice(&id.to_be_bytes()); + key +} +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn pin_invalidates_a_reclamation_proof_at_the_shared_authority_key() { + let metadata = metadata::TestMetadata::start().await; + let store = open_root_catalog_for_tests(metadata.client.clone(), 0, 1, &entry(1)) + .await + .unwrap(); + for generation in 1..=4 { + store + .publish(42, generation - 1, 1, generation, b"manifest") + .unwrap(); + } + let authority = key(b"authority", 0); + let GetOutcome::Found { value, revision } = metadata + .client + .get(0, 1, &authority, ReadMode::Linearizable, None) + .await + .unwrap() + else { + panic!("authority must exist"); + }; + store.pin_generation(42, 1, 1, 1).unwrap(); + let ops = [ + BatchOp::Put { + key: Bytes::from(authority.clone()), + value, + }, + BatchOp::Delete { + key: Bytes::from(key(b"manifest", 1)), + }, + ]; + assert!(matches!( + metadata + .client + .batch_write_cas(0, 1, &ops, &authority, revision) + .await, + Err(crowdb_paxos_kv_client::Error::CasFailed { .. } | crowdb_paxos_kv_client::Error::CasBusy) + )); + assert_eq!(store.reclaim_before(42, 4), 0); + store.unpin_generation(42, 1, 1).unwrap(); + assert_eq!(store.reclaim_before(42, u64::MAX), 2); + assert_eq!(store.load(42, RootCatalogObject::Manifest(1)).unwrap(), None); + assert_eq!(store.pin_generation(42, 2, 2, 1), Err(CtError::NotFound)); + assert!(store.load(42, RootCatalogObject::Manifest(3)).unwrap().is_some()); +} diff --git a/app/crowdb-chunk-kv-server/tests/rpc_test.rs b/app/crowdb-chunk-kv-server/tests/rpc_test.rs index 249e8d3aa..472036199 100644 --- a/app/crowdb-chunk-kv-server/tests/rpc_test.rs +++ b/app/crowdb-chunk-kv-server/tests/rpc_test.rs @@ -97,3 +97,22 @@ async fn point_request_crosses_real_rpc_boundary_with_typed_response() { server.stop(); } + +#[tokio::test] +async fn shutdown_drains_owned_rpc_tail_and_permanently_closes_admission() { + let service = Arc::new(ChunkKvService::new(test_process::TestProcess::authority(7), 8).unwrap()); + let rpc = Arc::new(ChunkKvRpcService::new(service, tokio::runtime::Handle::current())); + let held = rpc.hold_request_for_tests().unwrap(); + let shutdown = rpc.close_and_drain(); + tokio::pin!(shutdown); + std::future::poll_fn(|context| { + assert!(std::future::Future::poll(shutdown.as_mut(), context).is_pending()); + std::task::Poll::Ready(()) + }) + .await; + assert!(rpc.hold_request_for_tests().is_none()); + drop(held); + shutdown.await; + assert!(rpc.hold_request_for_tests().is_none()); + rpc.close_and_drain().await; +} diff --git a/app/crowdb-chunk-kv-server/tests/split_recovery_test.rs b/app/crowdb-chunk-kv-server/tests/split_recovery_test.rs index c2734a60d..8ac8e543f 100644 --- a/app/crowdb-chunk-kv-server/tests/split_recovery_test.rs +++ b/app/crowdb-chunk-kv-server/tests/split_recovery_test.rs @@ -1,6 +1,8 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. +#[path = "common/metadata.rs"] +mod metadata; #[path = "common/process.rs"] mod test_process; @@ -19,6 +21,44 @@ use crowdb_protocol::chunk_kv::{ }; use std::sync::Arc; +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn recovered_split_records_tree_completion_without_a_live_transfer() { + use crowdb_chunk_kv_server::storage::open_root_catalog_with_process_for_tests; + use crowdb_tree_ffi::chunk::test_util::TestChunkBackend; + + let metadata = metadata::TestMetadata::start().await; + let entry = entry(&transition(), false); + let partition = recovered(&entry).await; + assert!(transfer_initialization_pending(&partition).await); + let catalog = open_root_catalog_with_process_for_tests( + metadata.client.clone(), + 0, + 1, + &entry, + test_process::TestProcess::authority(1), + ) + .await + .unwrap(); + let backend = TestChunkBackend::new(entry.artifact.tree_id, entry.owner_epoch).unwrap(); + let pages = backend.page_store(0).unwrap(); + tokio::time::timeout( + std::time::Duration::from_secs(1), + catalog.complete_tree_recovery_for_tests(&partition, &pages), + ) + .await + .expect("completed split replay must not wait for a transfer command") + .unwrap(); + assert!(transfer_initialization_pending(&partition).await); + partition.close().await.unwrap(); +} + +async fn transfer_initialization_pending(partition: &Partition) -> bool { + use std::future::Future; + let mut completion = std::pin::pin!(partition.await_transfer_initialization()); + std::future::poll_fn(|context| std::task::Poll::Ready(completion.as_mut().poll(context).is_pending())) + .await +} + #[tokio::test] async fn committed_split_proves_both_recovered_writers_and_rejects_mismatches() { let transition = transition(); diff --git a/app/crowdb-chunk-kv-server/tests/stream_metadata_reclaim_test.rs b/app/crowdb-chunk-kv-server/tests/stream_metadata_reclaim_test.rs index 1d1fdbac3..ff6bb0e11 100644 --- a/app/crowdb-chunk-kv-server/tests/stream_metadata_reclaim_test.rs +++ b/app/crowdb-chunk-kv-server/tests/stream_metadata_reclaim_test.rs @@ -8,14 +8,14 @@ use crowdb_chunk_stream::{ memory::MemoryStreamStore, ChunkStream, KvStreamMetadataStore, StreamConfig, StreamMetadataStore, StreamName, StreamPurpose, StreamRecord, StreamRecordState, }; -use crowdb_kv_client::{CrowdbKvClient, ReadMode}; +use crowdb_paxos_kv_client::{PaxosKvClient, ReadMode}; use crowdb_protocol::chunk_stream::decode_stream_chunk_record; use crowdb_protocol::key::{BinaryKey, StreamChunkRecordKey}; #[path = "common/metadata.rs"] mod metadata; -async fn pages(kv: &CrowdbKvClient, name: StreamName) -> Vec<StreamChunkRecordKey> { +async fn pages(kv: &PaxosKvClient, name: StreamName) -> Vec<StreamChunkRecordKey> { kv.scan( 0, 1, diff --git a/app/crowdb-chunk-kv-server/tests/transfer_storage_test.rs b/app/crowdb-chunk-kv-server/tests/transfer_storage_test.rs index ddadaca8d..a7ff5b1c3 100644 --- a/app/crowdb-chunk-kv-server/tests/transfer_storage_test.rs +++ b/app/crowdb-chunk-kv-server/tests/transfer_storage_test.rs @@ -5,7 +5,7 @@ use std::sync::Arc; use crowdb_chunk_kv_server::storage::open_root_catalog_for_tests; use crowdb_chunk_kv_server::{ChunkKvRangeCatalogPublisher, Group0ControlStore}; -use crowdb_kv_client::CrowdbKvClient; +use crowdb_paxos_kv_client::PaxosKvClient; use crowdb_protocol::chunk_kv::{ ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogHead, ChunkKvRangeCatalogPage, ChunkKvRangeCatalogPageRef, ChunkKvRangeCatalogPartitionState, Id128, KeyRange, OwnerDescriptor, PartitionArtifact, @@ -88,7 +88,7 @@ async fn preparing_target_does_not_fence_source_or_mutate_shared_root() { recovered.publish(42, 2, 1, 3, b"source-checkpoint-3").unwrap(); } -async fn publish_assignment(kv: Arc<CrowdbKvClient>, entry: ChunkKvRangeCatalogEntry, generation: u64) { +async fn publish_assignment(kv: Arc<PaxosKvClient>, entry: ChunkKvRangeCatalogEntry, generation: u64) { let mut page = ChunkKvRangeCatalogPage { generation, page_index: 0, diff --git a/app/crowdb-chunk-kv-server/tests/tree_config_test.rs b/app/crowdb-chunk-kv-server/tests/tree_config_test.rs new file mode 100644 index 000000000..84f62f483 --- /dev/null +++ b/app/crowdb-chunk-kv-server/tests/tree_config_test.rs @@ -0,0 +1,173 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use std::collections::BTreeSet; +use std::sync::Arc; + +use crowdb_chunk_kv_server::ChunkKvServerConfig; +use crowdb_tree_ffi::{Crowdbtree, CtError, TreeRuntime}; + +const CANONICAL: &str = include_str!("../conf/crowdb_chunk_kv_server_config.toml"); +const PAXOS: &str = include_str!("../../crowdb-paxos-kv-server/conf/crowdb_paxos_kv_server_config.toml"); + +fn load(text: &str) -> Result<ChunkKvServerConfig, String> { + let root = crowdb_test_harness::test_dirs::tempdir_in_test_data("chunk-tree-config"); + let path = root.path().join("server.toml"); + std::fs::write(&path, text).unwrap(); + ChunkKvServerConfig::load(&path).map_err(|error| error.to_string()) +} + +#[test] +fn canonical_config_loads_all_service_defaults() { + let config = load(CANONICAL).unwrap(); + assert_eq!( + config, + ChunkKvServerConfig { + instance_id: 1, + tree_runtime: config.tree_runtime.clone(), + ..Default::default() + } + ); + let expected = ChunkKvServerConfig::default().tree_runtime.runtime_options(); + let host = config.tree_runtime.runtime_options(); + assert_eq!(host.workers, expected.workers); + assert_eq!(host.backend_workers, expected.backend_workers); + assert_eq!(host.limits, expected.limits); +} + +fn field_paths(value: &toml::Value, prefix: &str, paths: &mut BTreeSet<String>) { + if let Some(table) = value.as_table() { + for (key, child) in table { + field_paths(child, &format!("{prefix}.{key}"), paths); + } + } else { + paths.insert(prefix.to_owned()); + } +} + +#[test] +fn independent_server_files_keep_the_same_tree_fields() { + let chunk: toml::Value = toml::from_str(CANONICAL).unwrap(); + let paxos: toml::Value = toml::from_str(PAXOS).unwrap(); + for section in ["tree", "tree_runtime"] { + let mut chunk_fields = BTreeSet::new(); + let mut paxos_fields = BTreeSet::new(); + field_paths(&chunk[section], section, &mut chunk_fields); + field_paths(&paxos[section], section, &mut paxos_fields); + assert_eq!(chunk_fields, paxos_fields, "{section}"); + } +} + +#[test] +fn configured_native_engine_and_shared_runtime_are_used() { + let mut config = load(&CANONICAL.replace("max_key_size = 0", "max_key_size = 4")).unwrap(); + config.tree_runtime.host.workers = Some(2); + config.tree_runtime.host.backend_workers = Some(1); + config.tree_runtime.tree.cpu_queue_limit = Some(32); + config.tree.tuning.flush_parallelism = 2; + config.validate().unwrap(); + let runtime = Arc::new(TreeRuntime::new(config.tree_runtime.runtime_options()).unwrap()); + let mut options = config.native_tree_options(); + assert_eq!(options.runtime_binding.cpu_queue_limit, 32); + options.runtime = Some(runtime.clone()); + let first = Crowdbtree::open(&options).unwrap(); + let second = Crowdbtree::open(&options).unwrap(); + assert_eq!(runtime.snapshot().unwrap().bindings, 2); + for tree in [&first, &second] { + assert_eq!(tree.put(b"abcde", b"value"), Err(CtError::InvalidArgument)); + tree.put(b"abcd", b"value").unwrap(); + assert_eq!(tree.get(b"abcd").unwrap().unwrap().1, b"value"); + } +} + +#[test] +fn configured_chunk_tree_survives_range_rebuild() { + use crowdb_tree_ffi::chunk::test_util::TestChunkBackend; + + let config = load(&CANONICAL.replace("max_key_size = 0", "max_key_size = 4")).unwrap(); + let source_backend = TestChunkBackend::new(84, 1).unwrap(); + let child_backend = TestChunkBackend::new(85, 1).unwrap(); + let source_options = crowdb_tree_ffi::Config { + runtime: Some(source_backend.runtime.clone()), + page_store: Some(source_backend.page_store(0).unwrap()), + ..config.native_tree_options() + }; + let source = Crowdbtree::open(&source_options).unwrap(); + source.put(b"abcd", b"value").unwrap(); + let destination = crowdb_tree_ffi::Config { + runtime: Some(child_backend.runtime.clone()), + page_store: Some(child_backend.page_store(0).unwrap()), + ..config.native_tree_options() + }; + let (child, stats) = source.rebuild_range(&destination).unwrap(); + assert_eq!(stats.entries_emitted, 1); + assert_eq!(child.get(b"abcd").unwrap().unwrap().1, b"value"); + assert_eq!(child.put(b"abcde", b"value"), Err(CtError::InvalidArgument)); +} + +#[test] +fn chunk_backend_alignment_is_validated_before_open() { + let mut config = load(CANONICAL).unwrap(); + config.tree.frame_bytes = 16 * 1024; + assert!(config + .validate() + .unwrap_err() + .to_string() + .contains("Chunk backend IU")); + config.tree.iu_size = 4096; + config.validate().unwrap(); + assert_eq!(config.native_tree_options().iu_size, 4096); +} + +#[test] +fn invalid_runtime_engine_and_partition_bounds_fail_on_load() { + for (from, to, field) in [ + ( + "flush_parallelism = 0", + "flush_parallelism = 5", + "flush_parallelism", + ), + ("cache_max_pages = 256", "cache_max_pages = 0", "cache_max_pages"), + ("queue_requests = 1_024", "queue_requests = 0", "partition bounds"), + ("batch_bytes = 4_194_304", "batch_bytes = 0", "partition bounds"), + ("backend_workers = 2", "backend_workers = 0", "backend workers"), + ( + "run_max_entries = 128", + "run_entries_typo = 128", + "run_entries_typo", + ), + ("queue_requests = 1_024", "queue_typo = 1_024", "queue_typo"), + ] { + assert!( + load(&CANONICAL.replace(from, to)).unwrap_err().contains(field), + "{field}" + ); + } +} + +#[test] +fn partial_partition_table_keeps_native_defaults() { + let config = load("instance_id = 1\n[partition]\nbatch_requests = 16\n").unwrap(); + assert_eq!(config.partition.batch_requests, 16); + assert_eq!(config.partition.queue_requests, 1024); + assert_eq!(config.partition.max_value_bytes, 16 * 1024 * 1024); +} + +#[test] +fn commented_optional_examples_are_loadable() { + let start = CANONICAL.find("# [bootstrap_partition]").unwrap(); + let end = CANONICAL.find("# Keep Tree field additions").unwrap(); + let examples = CANONICAL[start..end] + .lines() + .filter_map(|line| line.strip_prefix("# ")) + .filter(|line| line.starts_with('[') || line.contains(" = ")) + .collect::<Vec<_>>() + .join("\n"); + let config = load(&format!("{CANONICAL}\n{examples}")).unwrap(); + assert_eq!(config.bootstrap_partition.unwrap().tree_id, 1); + let mut rdma = config.rpc_rdma.unwrap(); + assert_eq!(rdma.control_data_capacity(), Some(262_144)); + // Exercise explicit server configuration without requiring an RDMA device. + rdma.enabled = false; + crowdb_chunk_kv_server::ChunkKvRpcService::create_server(1, Some(&rdma)).unwrap(); +} diff --git a/app/crowdb-chunkdb/Cargo.toml b/app/crowdb-chunkdb/Cargo.toml index a7b7bfffd..69fca90a3 100644 --- a/app/crowdb-chunkdb/Cargo.toml +++ b/app/crowdb-chunkdb/Cargo.toml @@ -26,7 +26,7 @@ workspace = true [dependencies] crowdb-protocol = { path = "../../lib/crowdb-protocol" } -crowdb-kv-client = { path = "../../lib/crowdb-kv-client" } +crowdb-paxos-kv-client = { path = "../../lib/crowdb-paxos-kv-client" } crowdb-diskdb-client = { path = "../../lib/crowdb-diskdb-client" } crowdb-diskio-client = { path = "../../lib/crowdb-diskio-client" } crowdb-chunk-client = { path = "../../lib/crowdb-chunk-client" } @@ -55,7 +55,7 @@ futures = "0.3" [dev-dependencies] tokio = { workspace = true, features = ["test-util"] } crowdb-console-shared = { path = "../../lib/crowdb-console-shared" } -crowdb-kv = { path = "../../lib/crowdb-kv", features = ["test-util"] } +crowdb-paxos-kv = { path = "../../lib/crowdb-paxos-kv", features = ["test-util"] } crowdb-chunkdb-client = { path = "../../lib/crowdb-chunkdb-client" } crowdb-rpc-ffi = { path = "../../lib/crowdb-rpc/ffi", features = ["test-util"] } crowdb-test-harness = { path = "../../lib/crowdb-test-harness" } @@ -64,3 +64,8 @@ reqwest = { version = "0.12", features = ["json"] } serde_json = "1" crowdb-diskdb = { path = "../crowdb-diskdb" } crowdb-chunkdb = { path = ".", features = ["test-util"] } + +crowdb-chunk-kv-server = { path = "../crowdb-chunk-kv-server", features = ["test-util"] } +crowdb-chunk-kv = { path = "../../lib/crowdb-chunk-kv", features = ["test-util"] } +crowdb-chunk-stream = { path = "../../lib/crowdb-chunk-stream", features = ["test-util"] } +crowdb-tree-ffi = { path = "../../lib/crowdb-tree/ffi", features = ["test-util", "chunk-rpc"] } diff --git a/app/crowdb-chunkdb/src/allocator/pool.rs b/app/crowdb-chunkdb/src/allocator/pool.rs index 4a8e522f3..6608f3a8f 100644 --- a/app/crowdb-chunkdb/src/allocator/pool.rs +++ b/app/crowdb-chunkdb/src/allocator/pool.rs @@ -15,7 +15,7 @@ use std::time::{SystemTime, UNIX_EPOCH}; use arc_swap::ArcSwap; use crowdb_diskdb_client::{DiskdbClientError, DiskdbRpcTransport}; -use crowdb_kv_client::ServiceRegistryClient; +use crowdb_paxos_kv_client::ServiceRegistryClient; use crowdb_protocol::common::{ChunkId, DiskId}; use crowdb_protocol::diskdb::rpc::{ AllocateBlocksRequest, AllocateResponse, ExecuteRelocationRequest, ExecuteRelocationResponse, diff --git a/app/crowdb-chunkdb/src/chunkdb_config.rs b/app/crowdb-chunkdb/src/chunkdb_config.rs index e00cf64fa..609b3663d 100644 --- a/app/crowdb-chunkdb/src/chunkdb_config.rs +++ b/app/crowdb-chunkdb/src/chunkdb_config.rs @@ -6,7 +6,7 @@ use std::net::SocketAddr; use crowdb_common::config::BaseConfig; -use crowdb_protocol::{CHUNKDB_HTTP_BASE, CHUNKDB_RPC_BASE, KV_SERVER_MGMT_BASE}; +use crowdb_protocol::{CHUNKDB_HTTP_BASE, CHUNKDB_RPC_BASE, PAXOS_KV_SERVER_MGMT_BASE}; use serde::{Deserialize, Serialize}; use crate::selector::FailureDomainPriority; @@ -459,7 +459,7 @@ impl Default for ServerConfig { http_listen_addr: format!("0.0.0.0:{CHUNKDB_HTTP_BASE}"), rpc_listen_addr: default_rpc_listen_addr(), instance_id: None, - kv_server_mgmt_seeds: vec![format!("http://127.0.0.1:{KV_SERVER_MGMT_BASE}")], + kv_server_mgmt_seeds: vec![format!("http://127.0.0.1:{PAXOS_KV_SERVER_MGMT_BASE}")], kv_pool_size: default_client_pool_size(), kv_rpc_workers: default_client_rpc_workers(), diskdb_pool_size: default_client_pool_size(), diff --git a/app/crowdb-chunkdb/src/conversion/io.rs b/app/crowdb-chunkdb/src/conversion/io.rs index 90f356e6a..27c90a090 100644 --- a/app/crowdb-chunkdb/src/conversion/io.rs +++ b/app/crowdb-chunkdb/src/conversion/io.rs @@ -8,7 +8,7 @@ use std::sync::Arc; use arc_swap::ArcSwapOption; use bytes::Bytes; use crowdb_diskio_client::{DiskId, DiskioClient, DiskioClientConfig, Durability, SegmentTarget}; -use crowdb_kv_client::{HardwareClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{HardwareClient, ServiceRegistryClient}; use crowdb_protocol::diskdb::rpc::Segment; use crate::chunkdb_config::ConversionIoConfig; diff --git a/app/crowdb-chunkdb/src/lifecycle/handler.rs b/app/crowdb-chunkdb/src/lifecycle/handler.rs index 057bb9955..57bb861e4 100644 --- a/app/crowdb-chunkdb/src/lifecycle/handler.rs +++ b/app/crowdb-chunkdb/src/lifecycle/handler.rs @@ -39,6 +39,7 @@ use crate::task::TaskStore; use crate::topology::TopologyCache; mod allocation; +mod deletion; use super::replacement::validate_replacement_geometry; use super::state::{ChunkState, StateTransitionError}; @@ -603,112 +604,6 @@ impl LifecycleHandler { }); } - /// Delete a chunk — marks deleted and frees segments. - /// Repeated deletion returns the existing tombstone so retries are idempotent. - pub async fn delete_chunk(&self, chunk_id: &ChunkId) -> Result<Chunk, LifecycleError> { - self.check_range(chunk_id)?; - - let mut guard = if let Some(locks) = &self.locks { - Some( - locks - .acquire(chunk_id, &self.store, &LockPolicy::default(), CacheHint::Cache) - .await?, - ) - } else { - None - }; - - let mut chunk = match &guard { - Some(g) => g - .chunk() - .unwrap_or_else(|| unreachable!("acquire guarantees chunk on Ok")) - .clone(), - None => self.store.get_chunk(chunk_id).await?, - }; - let current_state = ChunkState::from_proto(chunk.state); - - // A Deleted record that still owns strips is a durable cleanup intent. - if current_state == ChunkState::Deleted { - if chunk.strips.is_empty() { - return Ok(chunk); - } - let segments: Vec<_> = chunk.strips.iter().flat_map(extract_segments).collect(); - self.allocator - .pool() - .free_blocks(segments) - .await - .map_err(LifecycleError::Cleanup)?; - chunk.strips.clear(); - chunk.capacity = 0; - chunk.modify_ts = chunk.modify_ts.saturating_add(1); - self.store.put_chunk(&chunk).await?; - if let Some(ref mut g) = guard { - g.refresh(chunk.clone()); - } - return Ok(chunk); - } - - current_state.check_can_delete()?; - - let reservation_groups = self.store.list_reservation_groups(chunk_id).await?; - if reservation_groups - .iter() - .any(|group| group.states.contains(&(StripReservationState::Consumed as i32))) - { - return Err(LifecycleError::StateConflict); - } - let mut reserved_strips = Vec::new(); - let mut reserved_parity = Vec::new(); - let mut reserved_parity_groups = Vec::new(); - for mut group in reservation_groups { - reservation::validate_group_shape(&group)?; - reserved_parity.extend_from_slice(&group.parity_segments); - for (index, state) in group.states.iter_mut().enumerate() { - if *state == StripReservationState::Reserved as i32 { - *state = StripReservationState::Cancelled as i32; - reserved_strips.push(group.strips[index].clone()); - } else if *state == StripReservationState::Cancelled as i32 { - reserved_strips.push(group.strips[index].clone()); - } - } - self.store.put_reservation_group(&group).await?; - reserved_parity_groups.push(group); - } - self.rollback_reserved_resources(&reserved_strips, reserved_parity, &reserved_parity_groups) - .await?; - for group in self.store.list_reservation_groups(chunk_id).await? { - if let Some(group_id) = group.group_id { - self.store.delete_reservation_group(chunk_id, &group_id).await?; - } - } - - // Publish Deleted before making any referenced block reusable. - let all_segments: Vec<_> = chunk.strips.iter().flat_map(extract_segments).collect(); - chunk.state = ProtoChunkState::Deleted as i32; - chunk.modify_ts = chunk.modify_ts.saturating_add(1); - self.store.put_chunk(&chunk).await?; - - if let Some(ref mut g) = guard { - g.refresh(chunk.clone()); - } - if !all_segments.is_empty() { - self.allocator - .pool() - .free_blocks(all_segments) - .await - .map_err(LifecycleError::Cleanup)?; - } - chunk.strips.clear(); - chunk.capacity = 0; - chunk.modify_ts = chunk.modify_ts.saturating_add(1); - self.store.put_chunk(&chunk).await?; - if let Some(ref mut g) = guard { - g.refresh(chunk.clone()); - } - info!(chunk_id = ?chunk_id, "chunk deleted"); - Ok(chunk) - } - /// Atomically replace a capacity-compatible strip range under a revision /// fence. Newly introduced segments are committed before publication; /// removed segments stay allocated until the reader-layout grace expires. diff --git a/app/crowdb-chunkdb/src/lifecycle/handler/allocation.rs b/app/crowdb-chunkdb/src/lifecycle/handler/allocation.rs index 66e312f5e..a2c4a3305 100644 --- a/app/crowdb-chunkdb/src/lifecycle/handler/allocation.rs +++ b/app/crowdb-chunkdb/src/lifecycle/handler/allocation.rs @@ -205,13 +205,11 @@ impl LifecycleHandler { let constraints = self.allocation_constraints(strip_type, &snap); // Convert write_granularity (KB) to unit_count using the unit - // size from the topology snapshot. Fall back to treating KB as - // units if unit_size_bytes is unavailable (0). + // size from the topology snapshot, rounding up to cover the complete + // requested strip. Fall back to treating KB as units if the unit size + // is unavailable (0). let unit_size_kb = snap.unit_size_bytes() / 1024; - let unit_count = write_granularity_kb - .checked_div(unit_size_kb) - .unwrap_or(write_granularity_kb) - .max(1); + let unit_count = write_granularity_kb.div_ceil(unit_size_kb.max(1)).max(1); let mut strips = self .allocator diff --git a/app/crowdb-chunkdb/src/lifecycle/handler/deletion.rs b/app/crowdb-chunkdb/src/lifecycle/handler/deletion.rs new file mode 100644 index 000000000..d1b9e09c4 --- /dev/null +++ b/app/crowdb-chunkdb/src/lifecycle/handler/deletion.rs @@ -0,0 +1,169 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Physical deletion and fixed allocation identity fencing. +use super::{ + extract_segments, info, reservation, unix_time_ms, CacheHint, Chunk, ChunkGuard, ChunkId, ChunkState, + LifecycleError, LifecycleHandler, LockPolicy, ProtoChunkState, StoreError, StripReservationState, +}; + +impl LifecycleHandler { + /// Delete a chunk — marks deleted and frees segments. + /// Repeated deletion returns the existing tombstone so retries are idempotent. + pub async fn delete_chunk(&self, chunk_id: &ChunkId) -> Result<Chunk, LifecycleError> { + self.delete_chunk_fenced(chunk_id, false).await + } + + /// Fence a fixed identity even when its allocation request has not arrived. + pub async fn delete_chunk_fenced( + &self, + chunk_id: &ChunkId, + fence_missing: bool, + ) -> Result<Chunk, LifecycleError> { + self.check_range(chunk_id)?; + if fence_missing && !matches!(chunk_id.high >> 56, 2 | 3) { + return Err(LifecycleError::InvalidRequest( + "allocation fencing requires a tree chunk identity".into(), + )); + } + let mut guard = if let Some(locks) = &self.locks { + Some(if fence_missing { + locks + .acquire_for_create(chunk_id, &LockPolicy::default(), CacheHint::Cache) + .await? + } else { + locks + .acquire(chunk_id, &self.store, &LockPolicy::default(), CacheHint::Cache) + .await? + }) + } else { + None + }; + + let chunk = if fence_missing { + match self.store.get_chunk(chunk_id).await { + Ok(chunk) => chunk, + Err(StoreError::ChunkNotFound) => { + let chunk = Chunk { + id: Some(*chunk_id), + modify_ts: 1, + state: ProtoChunkState::Deleted as i32, + create_ts_ms: unix_time_ms(), + chunk_type: if chunk_id.high >> 56 == 2 { 2 } else { 3 }, + ..Default::default() + }; + self.store.put_chunk(&chunk).await?; + if let Some(ref mut guard) = guard { + guard.refresh(chunk.clone()); + } + return Ok(chunk); + } + Err(error) => return Err(error.into()), + } + } else { + match &guard { + Some(guard) => guard + .chunk() + .unwrap_or_else(|| unreachable!("acquire guarantees a chunk")) + .clone(), + None => self.store.get_chunk(chunk_id).await?, + } + }; + self.finish_deletion(chunk_id, chunk, guard).await + } + + async fn finish_deletion( + &self, + chunk_id: &ChunkId, + mut chunk: Chunk, + mut guard: Option<ChunkGuard>, + ) -> Result<Chunk, LifecycleError> { + let current_state = ChunkState::from_proto(chunk.state); + + // A Deleted record that still owns strips is a durable cleanup intent. + if current_state == ChunkState::Deleted { + if chunk.strips.is_empty() { + return Ok(chunk); + } + let segments: Vec<_> = chunk.strips.iter().flat_map(extract_segments).collect(); + self.allocator + .pool() + .free_blocks(segments) + .await + .map_err(LifecycleError::Cleanup)?; + chunk.strips.clear(); + chunk.capacity = 0; + chunk.modify_ts = chunk.modify_ts.saturating_add(1); + self.store.put_chunk(&chunk).await?; + if let Some(ref mut g) = guard { + g.refresh(chunk.clone()); + } + return Ok(chunk); + } + + current_state.check_can_delete()?; + + self.retire_reservation_groups(chunk_id).await?; + + // Publish Deleted before making any referenced block reusable. + let all_segments: Vec<_> = chunk.strips.iter().flat_map(extract_segments).collect(); + chunk.state = ProtoChunkState::Deleted as i32; + chunk.modify_ts = chunk.modify_ts.saturating_add(1); + self.store.put_chunk(&chunk).await?; + + if let Some(ref mut g) = guard { + g.refresh(chunk.clone()); + } + if !all_segments.is_empty() { + self.allocator + .pool() + .free_blocks(all_segments) + .await + .map_err(LifecycleError::Cleanup)?; + } + chunk.strips.clear(); + chunk.capacity = 0; + chunk.modify_ts = chunk.modify_ts.saturating_add(1); + self.store.put_chunk(&chunk).await?; + if let Some(ref mut g) = guard { + g.refresh(chunk.clone()); + } + info!(chunk_id = ?chunk_id, "chunk deleted"); + Ok(chunk) + } + async fn retire_reservation_groups(&self, chunk_id: &ChunkId) -> Result<(), LifecycleError> { + let reservation_groups = self.store.list_reservation_groups(chunk_id).await?; + if reservation_groups + .iter() + .any(|group| group.states.contains(&(StripReservationState::Consumed as i32))) + { + return Err(LifecycleError::StateConflict); + } + let mut reserved_strips = Vec::new(); + let mut reserved_parity = Vec::new(); + let mut reserved_parity_groups = Vec::new(); + for mut group in reservation_groups { + reservation::validate_group_shape(&group)?; + reserved_parity.extend_from_slice(&group.parity_segments); + for (index, state) in group.states.iter_mut().enumerate() { + if *state == StripReservationState::Reserved as i32 { + *state = StripReservationState::Cancelled as i32; + reserved_strips.push(group.strips[index].clone()); + } else if *state == StripReservationState::Cancelled as i32 { + reserved_strips.push(group.strips[index].clone()); + } + } + self.store.put_reservation_group(&group).await?; + reserved_parity_groups.push(group); + } + self.rollback_reserved_resources(&reserved_strips, reserved_parity, &reserved_parity_groups) + .await?; + for group in self.store.list_reservation_groups(chunk_id).await? { + if let Some(group_id) = group.group_id { + self.store.delete_reservation_group(chunk_id, &group_id).await?; + } + } + + Ok(()) + } +} diff --git a/app/crowdb-chunkdb/src/main.rs b/app/crowdb-chunkdb/src/main.rs index 8a8741091..4742b4413 100644 --- a/app/crowdb-chunkdb/src/main.rs +++ b/app/crowdb-chunkdb/src/main.rs @@ -23,8 +23,8 @@ use crowdb_chunkdb::topology::{ build_snapshot, notify::NotifyHandler, refresh::run_refresh_loop, TopologyCache, }; use crowdb_common::metrics::{MetricsRegistry, MetricsRunner}; -use crowdb_kv_client::{ - ClientConfig, CrowdbKvClient, DomainMonitorClient, HardwareClient, KVClusterMetaClient, +use crowdb_paxos_kv_client::{ + ClientConfig, DomainMonitorClient, HardwareClient, PaxosKvClient, PaxosKvClusterMetaClient, ServiceRegistryClient, WatchNotifyClient, }; use tracing::{error, info, warn}; @@ -169,7 +169,7 @@ async fn main() { let mut kv_config = ClientConfig::new(config.server.kv_server_mgmt_seeds.clone()); kv_config.pool_size_per_endpoint = config.server.kv_pool_size; kv_config.rpc_workers = config.server.kv_rpc_workers; - let kv = Arc::new(CrowdbKvClient::new_with_isolated_point_reads(kv_config)); + let kv = Arc::new(PaxosKvClient::new_with_isolated_point_reads(kv_config)); let hw = HardwareClient::from_shared(Arc::clone(&kv)); let refresh_hw = HardwareClient::from_shared(Arc::clone(&kv)); let watch = WatchNotifyClient::from_shared(Arc::clone(&kv)); @@ -256,7 +256,7 @@ async fn main() { // Binding cache + chunk store. let bindings = BindingCache::new(); - let slot_maps = crowdb_kv_client::ChunkSlotMapClient::new(Arc::clone(&kv)); + let slot_maps = crowdb_paxos_kv_client::ChunkSlotMapClient::new(Arc::clone(&kv)); if let Some(bootstrap) = &config.slot_bootstrap { if let Err(error) = slot_maps.initialize_layout(bootstrap).await { error!(%error, "chunk slot initialization failed; refusing startup"); @@ -695,14 +695,14 @@ fn load_config(args: &Cli) -> ChunkdbConfig { config } -async fn validate_voting_topology(kv: &Arc<CrowdbKvClient>, mode: DeploymentMode) -> Result<(), String> { +async fn validate_voting_topology(kv: &Arc<PaxosKvClient>, mode: DeploymentMode) -> Result<(), String> { use std::collections::{BTreeMap, BTreeSet}; if mode == DeploymentMode::TestUnsafePlacement { return Ok(()); } - let metadata = KVClusterMetaClient::from_shared(Arc::clone(kv)); + let metadata = PaxosKvClusterMetaClient::from_shared(Arc::clone(kv)); let groups = metadata .list_all_groups() .await diff --git a/app/crowdb-chunkdb/src/range_guard.rs b/app/crowdb-chunkdb/src/range_guard.rs index dbe28929f..c23b21089 100644 --- a/app/crowdb-chunkdb/src/range_guard.rs +++ b/app/crowdb-chunkdb/src/range_guard.rs @@ -4,7 +4,7 @@ //! Immutable service-slot admission snapshots, independent of storage destinations. use arc_swap::ArcSwapOption; -use crowdb_kv_client::{ChunkSlotMapClient, CrowdbKvClient}; +use crowdb_paxos_kv_client::{ChunkSlotMapClient, PaxosKvClient}; use crowdb_protocol::chunk_slot::{ ChunkSlot, ChunkSlotAuthority, ChunkSlotBitmap, ChunkSlotMap, CHUNK_SLOT_COUNT, }; @@ -98,7 +98,7 @@ impl RangeGuard { &self, map: &ChunkSlotMap<ChunkSlotAuthority>, instance_id: u64, - ) -> crowdb_kv_client::Result<()> { + ) -> crowdb_paxos_kv_client::Result<()> { let slots = ChunkSlot::all() .filter(|slot| map.owner(*slot).instance_id() == instance_id) .collect(); @@ -178,7 +178,7 @@ impl RangeGuard { /// # Errors /// Rejects unsupported changes to initialized authority. New instances /// start with no slots until an explicit ownership migration assigns work. - pub fn install(&self, map: &ChunkSlotMap<u64>, instance_id: u64) -> crowdb_kv_client::Result<()> { + pub fn install(&self, map: &ChunkSlotMap<u64>, instance_id: u64) -> crowdb_paxos_kv_client::Result<()> { let binding = map .bindings() .iter() @@ -214,9 +214,9 @@ impl RangeGuard { /// Returns read, validation or unsupported ownership-change errors. pub async fn load_from_group0( &self, - kv: &Arc<CrowdbKvClient>, + kv: &Arc<PaxosKvClient>, instance_id: u64, - ) -> crowdb_kv_client::Result<()> { + ) -> crowdb_paxos_kv_client::Result<()> { let maps = ChunkSlotMapClient::new(Arc::clone(kv)); if let Some(snapshot) = maps.read_dynamic_service_snapshot().await? { self.install_dynamic(snapshot.authority(), instance_id) @@ -293,8 +293,8 @@ pub fn chunk_bucket(chunk_id: &ChunkId) -> u16 { ChunkSlot::for_chunk(chunk_id).value() } -fn invalid(reason: &str) -> crowdb_kv_client::Error { - crowdb_kv_client::Error::SysdataDecode { +fn invalid(reason: &str) -> crowdb_paxos_kv_client::Error { + crowdb_paxos_kv_client::Error::SysdataDecode { key: "/chunkdb/slot_service/".into(), reason: reason.into(), } diff --git a/app/crowdb-chunkdb/src/readiness.rs b/app/crowdb-chunkdb/src/readiness.rs index fbe4320e3..82da80c07 100644 --- a/app/crowdb-chunkdb/src/readiness.rs +++ b/app/crowdb-chunkdb/src/readiness.rs @@ -6,7 +6,7 @@ use crate::range_guard::RangeGuard; use arc_swap::ArcSwapOption; use axum::{http::StatusCode, routing::get, Router}; -use crowdb_kv_client::CrowdbKvClient; +use crowdb_paxos_kv_client::PaxosKvClient; use std::{future::Future, sync::Arc, time::Duration}; use tokio::{ sync::watch, @@ -71,7 +71,7 @@ impl RangeReadiness { /// Own one bounded refresh at a time; callers never initiate extra reads. pub fn start( self: Arc<Self>, - kv: Arc<CrowdbKvClient>, + kv: Arc<PaxosKvClient>, instance_id: u64, stop: watch::Receiver<bool>, ) -> JoinHandle<()> { diff --git a/app/crowdb-chunkdb/src/runtime.rs b/app/crowdb-chunkdb/src/runtime.rs index 85322d6b5..ded31ec9f 100644 --- a/app/crowdb-chunkdb/src/runtime.rs +++ b/app/crowdb-chunkdb/src/runtime.rs @@ -20,7 +20,7 @@ use crate::service::ChunkdbRpcService; use crate::storage::ChunkStore; use crate::task::{TaskManager, TaskStore}; use crate::topology::TopologyCache; -use crowdb_kv_client::CrowdbKvClient; +use crowdb_paxos_kv_client::PaxosKvClient; use crowdb_protocol::chunk_domain::ChunkDomain; mod background; @@ -30,7 +30,7 @@ mod recovery; #[derive(Clone)] pub struct RuntimeContext { pub config: ChunkdbConfig, - pub kv: Arc<CrowdbKvClient>, + pub kv: Arc<PaxosKvClient>, pub bindings: BindingCache, pub range_guard: Arc<RangeGuard>, pub allocator: Arc<ChunkAllocator>, diff --git a/app/crowdb-chunkdb/src/runtime/executor.rs b/app/crowdb-chunkdb/src/runtime/executor.rs index 823ce087f..78ae3b6e7 100644 --- a/app/crowdb-chunkdb/src/runtime/executor.rs +++ b/app/crowdb-chunkdb/src/runtime/executor.rs @@ -14,7 +14,7 @@ use crate::repair::{RepairCoordinator, RepairStripTaskHandler}; use crate::task::{ RelocateSegmentTaskHandler, TaskExecutor, TaskHandler, TaskManager, TaskScanner, TaskStore, }; -use crowdb_kv_client::{HardwareClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{HardwareClient, ServiceRegistryClient}; use std::sync::Arc; use std::time::Duration; use tracing::warn; diff --git a/app/crowdb-chunkdb/src/service/chunkdb_rpc_service/mutations.rs b/app/crowdb-chunkdb/src/service/chunkdb_rpc_service/mutations.rs index f3fee7ee9..fd14605a4 100644 --- a/app/crowdb-chunkdb/src/service/chunkdb_rpc_service/mutations.rs +++ b/app/crowdb-chunkdb/src/service/chunkdb_rpc_service/mutations.rs @@ -977,7 +977,9 @@ impl ChunkdbRpcService { return; }; - let result = handler.delete_chunk(&chunk_id).await; + let result = handler + .delete_chunk_fenced(&chunk_id, fb_req.fence_missing()) + .await; if result.is_ok() { request.mark_success(); } diff --git a/app/crowdb-chunkdb/src/storage.rs b/app/crowdb-chunkdb/src/storage.rs index 80f1f17a7..8641c54c8 100644 --- a/app/crowdb-chunkdb/src/storage.rs +++ b/app/crowdb-chunkdb/src/storage.rs @@ -17,7 +17,7 @@ use std::sync::Arc; use tracing::warn; use bytes::Bytes; -use crowdb_kv_client::{BatchOp, CrowdbKvClient, GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{BatchOp, GetOutcome, PaxosKvClient, ReadMode}; use crowdb_protocol::chunk_task::ChunkTaskValue; use crowdb_protocol::chunkdb::rpc::Chunk; use crowdb_protocol::common::ChunkId; @@ -57,14 +57,14 @@ pub type Result<T> = std::result::Result<T, StoreError>; /// Chunk metadata store — persists `Chunk` records to CROWDB KV. pub struct ChunkStore { - pub(super) kv: Arc<CrowdbKvClient>, + pub(super) kv: Arc<PaxosKvClient>, pub(super) bindings: BindingCache, scope: Option<(Arc<RangeGuard>, ChunkDomain)>, } impl ChunkStore { #[must_use] - pub fn new(kv: Arc<CrowdbKvClient>, bindings: BindingCache) -> Self { + pub fn new(kv: Arc<PaxosKvClient>, bindings: BindingCache) -> Self { Self { kv, bindings, @@ -116,12 +116,12 @@ impl ChunkStore { .await { Ok(_) => Ok(()), - Err(crowdb_kv_client::Error::OutcomeUnknown) => match self.get_chunk(id).await { + Err(crowdb_paxos_kv_client::Error::OutcomeUnknown) => match self.get_chunk(id).await { Ok(stored) if stored == *chunk => Ok(()), Ok(_) | Err(StoreError::ChunkNotFound) => Err(StoreError::Conflict), Err(error) => Err(StoreError::OutcomeUnknown(error.to_string())), }, - Err(crowdb_kv_client::Error::CasFailed { .. } | crowdb_kv_client::Error::CasBusy) => { + Err(crowdb_paxos_kv_client::Error::CasFailed { .. } | crowdb_paxos_kv_client::Error::CasBusy) => { Err(StoreError::ChunkAlreadyExists) } Err(error) => Err(StoreError::Kv(error.to_string())), @@ -181,9 +181,9 @@ impl ChunkStore { { Ok(_) => Ok(()), Err( - crowdb_kv_client::Error::CasFailed { .. } - | crowdb_kv_client::Error::CasBusy - | crowdb_kv_client::Error::OutcomeUnknown, + crowdb_paxos_kv_client::Error::CasFailed { .. } + | crowdb_paxos_kv_client::Error::CasBusy + | crowdb_paxos_kv_client::Error::OutcomeUnknown, ) => { match self .kv diff --git a/app/crowdb-chunkdb/src/storage/range_retirement.rs b/app/crowdb-chunkdb/src/storage/range_retirement.rs index 510dba10a..01650ae59 100644 --- a/app/crowdb-chunkdb/src/storage/range_retirement.rs +++ b/app/crowdb-chunkdb/src/storage/range_retirement.rs @@ -8,7 +8,7 @@ pub use record::{ChunkRangeRetirement, RetiredByteRange}; use super::{chunk_key, decode_chunk, encode_chunk, ChunkStore, Result, StoreError}; use bytes::Bytes; -use crowdb_kv_client::{BatchOp, GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{BatchOp, GetOutcome, ReadMode}; use crowdb_protocol::chunk_domain::ChunkDomain; use crowdb_protocol::chunkdb::rpc::{ChunkState, DeleteChunkRangeRequest, DeleteChunkRangeResponse}; use crowdb_protocol::common::ChunkId; @@ -106,7 +106,7 @@ impl ChunkStore { .await { Ok(_) => Ok(DeleteChunkRangeResponse { reclaimed: false }), - Err(crowdb_kv_client::Error::CasFailed { .. } | crowdb_kv_client::Error::CasBusy) => { + Err(crowdb_paxos_kv_client::Error::CasFailed { .. } | crowdb_paxos_kv_client::Error::CasBusy) => { Err(StoreError::Conflict) } Err(error) => Err(StoreError::Kv(error.to_string())), diff --git a/app/crowdb-chunkdb/src/storage/reservation.rs b/app/crowdb-chunkdb/src/storage/reservation.rs index 9fe718177..64ebe0d47 100644 --- a/app/crowdb-chunkdb/src/storage/reservation.rs +++ b/app/crowdb-chunkdb/src/storage/reservation.rs @@ -4,7 +4,7 @@ //! Durable strip-reservation values co-located with their owning chunk. use bytes::Bytes; -use crowdb_kv_client::{BatchOp, GetOutcome, ReadMode, ScanOutcome}; +use crowdb_paxos_kv_client::{BatchOp, GetOutcome, ReadMode, ScanOutcome}; use crowdb_protocol::chunkdb::rpc::{Chunk, StripReservationGroup}; use crowdb_protocol::common::ChunkId; @@ -217,9 +217,9 @@ impl ChunkStore { { Ok(_) => Ok(()), Err( - crowdb_kv_client::Error::CasFailed { .. } - | crowdb_kv_client::Error::CasBusy - | crowdb_kv_client::Error::OutcomeUnknown, + crowdb_paxos_kv_client::Error::CasFailed { .. } + | crowdb_paxos_kv_client::Error::CasBusy + | crowdb_paxos_kv_client::Error::OutcomeUnknown, ) => match self .kv .get(store_id, group_id, &key, ReadMode::Linearizable, None) diff --git a/app/crowdb-chunkdb/src/task/store.rs b/app/crowdb-chunkdb/src/task/store.rs index 58f8791b9..5b9e6e351 100644 --- a/app/crowdb-chunkdb/src/task/store.rs +++ b/app/crowdb-chunkdb/src/task/store.rs @@ -8,7 +8,7 @@ use std::sync::Arc; use crate::range_guard::RangeGuard; use bytes::Bytes; -use crowdb_kv_client::{BatchOp, CrowdbKvClient, Error as KvError, GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{BatchOp, Error as KvError, GetOutcome, PaxosKvClient, ReadMode}; use crowdb_protocol::chunk_domain::ChunkDomain; use crowdb_protocol::chunk_task::{ChunkTaskState, ChunkTaskValue, TASK_KIND_FINALIZE_CHUNK}; use crowdb_protocol::common::ChunkId; @@ -44,7 +44,7 @@ pub enum TaskStoreError { /// Persistent task storage. It is lock-free and relies on the caller's /// existing resource lifecycle guard for same-process transition ordering. pub struct TaskStore { - kv: Arc<CrowdbKvClient>, + kv: Arc<PaxosKvClient>, bindings: BindingCache, scope: Option<(Arc<RangeGuard>, ChunkDomain)>, } @@ -68,7 +68,7 @@ impl TaskStore { Ok(()) } #[must_use] - pub fn new(kv: Arc<CrowdbKvClient>, bindings: BindingCache) -> Self { + pub fn new(kv: Arc<PaxosKvClient>, bindings: BindingCache) -> Self { Self { kv, bindings, diff --git a/app/crowdb-chunkdb/src/topology.rs b/app/crowdb-chunkdb/src/topology.rs index bb11af798..5a5e12909 100644 --- a/app/crowdb-chunkdb/src/topology.rs +++ b/app/crowdb-chunkdb/src/topology.rs @@ -334,7 +334,7 @@ impl TopologySnapshot { } } - fn set_allocation_geometry(&mut self, disks: &[crowdb_kv_client::DiskRecord]) { + fn set_allocation_geometry(&mut self, disks: &[crowdb_paxos_kv_client::DiskRecord]) { match crowdb_protocol::chunk_allocation_geometry::uniform_allocation_unit( disks .iter() @@ -472,7 +472,7 @@ impl Default for TopologyCache { /// Fetches racks, nodes, and disk-groups from group-0 and assembles /// them into a snapshot. If the fetch fails or returns empty, returns /// `None` (caller should keep the previous snapshot). -pub async fn build_snapshot(hw: &crowdb_kv_client::HardwareClient) -> Option<TopologySnapshot> { +pub async fn build_snapshot(hw: &crowdb_paxos_kv_client::HardwareClient) -> Option<TopologySnapshot> { let racks = match hw.list_racks().await { Ok(r) => r, Err(e) => { @@ -505,7 +505,7 @@ pub async fn build_snapshot(hw: &crowdb_kv_client::HardwareClient) -> Option<Top } }; - let mut usages = match crowdb_kv_client::SpaceUsageClient::from_shared(hw.clone()) + let mut usages = match crowdb_paxos_kv_client::SpaceUsageClient::from_shared(hw.clone()) .list_disk_group_usages() .await { @@ -515,7 +515,7 @@ pub async fn build_snapshot(hw: &crowdb_kv_client::HardwareClient) -> Option<Top Vec::new() } }; - let service = crowdb_kv_client::ServiceRegistryClient::from_shared(hw.shared_kv()); + let service = crowdb_paxos_kv_client::ServiceRegistryClient::from_shared(hw.shared_kv()); let diskio = availability::read(&service).await?; if let Ok(instances) = service.read_all_diskdb_instances().await { for (_, instance) in instances { diff --git a/app/crowdb-chunkdb/src/topology/availability.rs b/app/crowdb-chunkdb/src/topology/availability.rs index 83418b31f..1d11b119d 100644 --- a/app/crowdb-chunkdb/src/topology/availability.rs +++ b/app/crowdb-chunkdb/src/topology/availability.rs @@ -10,7 +10,7 @@ use std::{ }; pub(super) async fn read( - service: &crowdb_kv_client::ServiceRegistryClient, + service: &crowdb_paxos_kv_client::ServiceRegistryClient, ) -> Option<Vec<(u64, InstanceValue)>> { match service.read_all_instance_observations("diskio").await { Ok(instances) => Some(instances), @@ -53,7 +53,7 @@ pub(super) fn deadlines( let deadline = if extra.owned_dg_ids.contains(&group.dg_id) { instance .last_heartbeat_ms - .saturating_add(crowdb_kv_client::SERVICE_INSTANCE_TTL_MS) + .saturating_add(crowdb_paxos_kv_client::SERVICE_INSTANCE_TTL_MS) } else { 0 }; diff --git a/app/crowdb-chunkdb/src/topology/notify.rs b/app/crowdb-chunkdb/src/topology/notify.rs index 0fed5f5fb..23f4b6a25 100644 --- a/app/crowdb-chunkdb/src/topology/notify.rs +++ b/app/crowdb-chunkdb/src/topology/notify.rs @@ -12,7 +12,7 @@ use tokio::sync::mpsc; use tracing::{info, warn}; use super::{build_snapshot, TopologyCache, CHUNKDB_WATCH_PREFIXES, G0_GROUP, G0_STORE}; -use crowdb_kv_client::{HardwareClient, WatchNotifyClient}; +use crowdb_paxos_kv_client::{HardwareClient, WatchNotifyClient}; /// chunkdb-side watch/notify handler for topology updates. pub struct NotifyHandler { diff --git a/app/crowdb-chunkdb/src/topology/refresh.rs b/app/crowdb-chunkdb/src/topology/refresh.rs index 67f230525..97b1b4d0e 100644 --- a/app/crowdb-chunkdb/src/topology/refresh.rs +++ b/app/crowdb-chunkdb/src/topology/refresh.rs @@ -11,7 +11,7 @@ use std::time::Duration; use tracing::{info, warn}; -use crowdb_kv_client::HardwareClient; +use crowdb_paxos_kv_client::HardwareClient; use super::{build_snapshot, TopologyCache}; diff --git a/app/crowdb-chunkdb/tests/allocation_fence_test.rs b/app/crowdb-chunkdb/tests/allocation_fence_test.rs new file mode 100644 index 000000000..4d9931399 --- /dev/null +++ b/app/crowdb-chunkdb/tests/allocation_fence_test.rs @@ -0,0 +1,111 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +//! A durable tombstone fences a fixed identity before a delayed allocation. +#[allow(dead_code)] +mod common; + +use common::cluster::{seed_hardware, ChunkdbHarness, DiskdbServer, KvCluster}; +use crowdb_chunkdb::lifecycle::LifecycleError; +use crowdb_protocol::chunkdb::rpc::{ChunkState, ChunkType, StripType}; +use crowdb_protocol::common::ChunkId; +use std::sync::Arc; + +#[tokio::test] +async fn delete_before_allocate_fences_the_identity_across_handler_restart() { + let cluster = KvCluster::start_pinned().await; + seed_hardware(&cluster.make_hardware_client()).await; + let _diskdb = DiskdbServer::start(&cluster).await; + let harness = ChunkdbHarness::start(&cluster).await; + let id = ChunkId { + high: 2 << 56, + low: 1007, + }; + let (send, receive) = tokio::sync::oneshot::channel(); + let handler = Arc::clone(&harness.handler); + let delayed = tokio::spawn(async move { + receive.await.unwrap(); + handler + .allocate_chunk( + Some(id), + 1, + 1, + StripType::Mirror, + 0, + 0, + 3, + ChunkType::BtreePage, + 17, + 30_000, + ) + .await + }); + let deleted = harness.handler.delete_chunk_fenced(&id, true).await.unwrap(); + assert_eq!(deleted.state, ChunkState::Deleted as i32); + assert!(deleted.strips.is_empty()); + assert_eq!(deleted.capacity, 0); + send.send(()).unwrap(); + assert!(matches!( + delayed.await.unwrap(), + Err(LifecycleError::ChunkAlreadyExists) + )); + let restarted = ChunkdbHarness::start(&cluster).await; + assert_eq!(restarted.store.get_chunk(&id).await.unwrap(), deleted); + assert_eq!( + restarted.handler.delete_chunk_fenced(&id, true).await.unwrap(), + deleted + ); + assert!(restarted + .handler + .allocate_chunk( + Some(id), + 1, + 1, + StripType::Mirror, + 0, + 0, + 3, + ChunkType::BtreePage, + 18, + 30_000 + ) + .await + .is_err()); +} + +#[tokio::test] +async fn allocated_tree_chunk_delete_releases_strips_before_success() { + let cluster = KvCluster::start_pinned().await; + seed_hardware(&cluster.make_hardware_client()).await; + let _diskdb = DiskdbServer::start(&cluster).await; + let harness = ChunkdbHarness::start(&cluster).await; + let id = ChunkId { + high: 3 << 56, + low: 1011, + }; + let active = harness + .handler + .allocate_chunk( + Some(id), + 1, + 1, + StripType::Mirror, + 0, + 0, + 3, + ChunkType::PageIndex, + 17, + 30_000, + ) + .await + .unwrap(); + assert!(!active.strips.is_empty()); + let deleted = harness.handler.delete_chunk_fenced(&id, true).await.unwrap(); + assert_eq!(deleted.state, ChunkState::Deleted as i32); + assert!(deleted.strips.is_empty()); + assert_eq!(deleted.capacity, 0); + assert_eq!(harness.store.get_chunk(&id).await.unwrap(), deleted); + assert_eq!( + harness.handler.delete_chunk_fenced(&id, true).await.unwrap(), + deleted + ); +} diff --git a/app/crowdb-chunkdb/tests/allocation_retry_test.rs b/app/crowdb-chunkdb/tests/allocation_retry_test.rs index 9eb726321..05134ff80 100644 --- a/app/crowdb-chunkdb/tests/allocation_retry_test.rs +++ b/app/crowdb-chunkdb/tests/allocation_retry_test.rs @@ -6,7 +6,7 @@ mod slot_groups; use crowdb_chunkdb_client::{ChunkdbClient, ChunkdbClientError, ChunkdbRpcTransport}; -use crowdb_kv_client::{ChunkSlotMapClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{ChunkSlotMapClient, ServiceRegistryClient}; use crowdb_protocol::{ chunk_slot::{ChunkSlotBootstrap, ChunkStorageGroup}, chunkdb::rpc::{AllocateChunkRequest, ChunkType}, diff --git a/app/crowdb-chunkdb/tests/allocation_rounding_test.rs b/app/crowdb-chunkdb/tests/allocation_rounding_test.rs new file mode 100644 index 000000000..e28d8e31e --- /dev/null +++ b/app/crowdb-chunkdb/tests/allocation_rounding_test.rs @@ -0,0 +1,55 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +mod common; + +use common::cluster::{seed_hardware, ChunkdbHarness, DiskdbServer, KvCluster, UNIT_SIZE_BYTES}; +use crowdb_protocol::chunkdb::rpc::{ChunkType, Strip, StripType}; + +#[tokio::test] +async fn requested_strip_capacity_rounds_up_to_complete_disk_units() { + let cluster = KvCluster::start_pinned().await; + seed_hardware(&cluster.make_hardware_client()).await; + let _diskdb = DiskdbServer::start(&cluster).await; + let harness = ChunkdbHarness::start(&cluster).await; + let unit_kb = UNIT_SIZE_BYTES / 1024; + for requested in [unit_kb - 1, unit_kb, unit_kb + 1] { + let chunk = harness + .handler + .allocate_chunk( + None, + requested, + 2, + StripType::Mirror, + 0, + 0, + 3, + ChunkType::BtreePage, + 17, + 30_000, + ) + .await + .unwrap(); + let expected_units = requested.div_ceil(unit_kb); + assert_eq!(chunk.capacity, 2 * expected_units * unit_kb); + assert_eq!(chunk.strips.len(), 2); + for (index, strip) in chunk.strips.iter().enumerate() { + assert!( + strip.capacity >= requested, + "requested {requested} KiB, allocated {} KiB", + strip.capacity + ); + assert_eq!(strip.capacity, expected_units * unit_kb); + assert_eq!(strip.chunk_offset, u32::try_from(index).unwrap() * strip.capacity); + let Some(Strip::MirrorStrip(mirror)) = &strip.strip else { + panic!("expected mirror layout") + }; + assert_eq!(mirror.segments.len(), 3); + assert!(mirror + .segments + .iter() + .all(|segment| segment.unit_count == expected_units)); + } + assert_eq!(harness.store.get_chunk(&chunk.id.unwrap()).await.unwrap(), chunk); + } +} diff --git a/app/crowdb-chunkdb/tests/allocator_owner_change_test.rs b/app/crowdb-chunkdb/tests/allocator_owner_change_test.rs index e29eeab16..b4252ce47 100644 --- a/app/crowdb-chunkdb/tests/allocator_owner_change_test.rs +++ b/app/crowdb-chunkdb/tests/allocator_owner_change_test.rs @@ -7,7 +7,7 @@ mod slot_groups; use crowdb_chunkdb::allocator::DiskdbClientPool; use crowdb_diskdb_client::DiskdbClientError; -use crowdb_kv_client::ServiceRegistryClient; +use crowdb_paxos_kv_client::ServiceRegistryClient; use crowdb_protocol::{ common::ChunkId, diskdb_fb::{FBAllocateResponse, FBAllocateResponseArgs, FBDiskdbRetCode}, diff --git a/app/crowdb-chunkdb/tests/allocator_pool_test.rs b/app/crowdb-chunkdb/tests/allocator_pool_test.rs index e61858e0d..a686c49c2 100644 --- a/app/crowdb-chunkdb/tests/allocator_pool_test.rs +++ b/app/crowdb-chunkdb/tests/allocator_pool_test.rs @@ -4,10 +4,10 @@ use std::sync::Arc; use crowdb_chunkdb::allocator::DiskdbClientPool; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient, ServiceRegistryClient}; fn pool() -> DiskdbClientPool { - let service = ServiceRegistryClient::new(CrowdbKvClient::new(ClientConfig::new(Vec::new()))); + let service = ServiceRegistryClient::new(PaxosKvClient::new(ClientConfig::new(Vec::new()))); DiskdbClientPool::new(service) } diff --git a/app/crowdb-chunkdb/tests/chunk_tree_runtime_test.rs b/app/crowdb-chunkdb/tests/chunk_tree_runtime_test.rs new file mode 100644 index 000000000..61c4a41ce --- /dev/null +++ b/app/crowdb-chunkdb/tests/chunk_tree_runtime_test.rs @@ -0,0 +1,430 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Real KV, `ChunkDB` and `DiskDB` handlers with the existing volatile `DiskIO` wire +//! fixture. This proves the host factory/catalog/Tree path, not disk durability. + +mod common; + +use common::cluster::{seed_hardware, ChunkdbHarness, DiskdbServer, KvCluster}; +use common::diskio_stub::TestDiskio; +use crowdb_chunk_client::{ChunkIoClient, ChunkIoClientConfig, SmallWritePolicy}; +use crowdb_chunk_kv::{ + MutationOperation, Partition, PartitionConfig, PartitionId, PartitionRange, RequestId, +}; +use crowdb_chunk_kv_server::{BootstrapPartitionConfig, ChunkKvStorage}; +use crowdb_chunk_stream::{StreamName, StreamPurpose, StreamRecord, StreamRecordState, StreamRegistry}; +use crowdb_chunkdb::{metrics::ChunkdbMetrics, service::ChunkdbRpcService}; +use crowdb_common::metrics::{MetricsRegistry, MetricsSampler}; +use crowdb_paxos_kv_client::{StreamProcessAuthority, StreamProcessLeaseClient, StreamProcessLeaseTiming}; +use crowdb_protocol::chunk_kv::Id128; +use crowdb_protocol::chunkdb::rpc::{ChunkState, ChunkType}; +use crowdb_tree_ffi::{ChunkPageStoreOptions, Config}; +use std::{sync::Arc, time::Instant}; + +async fn storage( + cluster: &KvCluster, + io: &ChunkIoClient, + process: &Arc<StreamProcessAuthority>, +) -> ChunkKvStorage { + let storage = ChunkKvStorage::from_parts_parallel_for_tests( + cluster.make_crowdb_client(), + io.clone(), + 0, + 300_000, + 2, + ) + .await + .unwrap(); + storage.bind_owner_process(process.clone()).unwrap(); + storage +} + +struct TestRuntimeStack { + cluster: KvCluster, + _diskdb: DiskdbServer, + _disks: Vec<TestDiskio>, + harness: ChunkdbHarness, + rpc: Arc<crowdb_rpc_ffi::RpcServer>, + io: ChunkIoClient, + process: Arc<StreamProcessAuthority>, +} +impl TestRuntimeStack { + async fn start() -> Self { + let cluster = KvCluster::start_pinned().await; + seed_hardware(&cluster.make_hardware_client()).await; + let diskdb = DiskdbServer::start(&cluster).await; + let mut disks = Vec::new(); + for index in 0..4_u64 { + disks.push( + TestDiskio::start( + &cluster.make_service_registry_client(), + 2000 + index, + 100 + index, + 10 + index, + 1000 + index, + ) + .await, + ); + } + let harness = ChunkdbHarness::start(&cluster).await; + let rpc = Arc::new(crowdb_rpc_ffi::RpcServer::new(None)); + rpc.listen("127.0.0.1", 0).unwrap(); + let mut metrics = MetricsRegistry::new(); + let service = Arc::new(ChunkdbRpcService::new( + harness.handler.clone(), + Arc::new(ChunkdbMetrics::register(&mut metrics)), + tokio::runtime::Handle::current(), + )); + service.register_handlers(&rpc); + rpc.start(); + cluster + .make_service_registry_client() + .heartbeat_chunkdb(100, &format!("http://127.0.0.1:{}", rpc.port())) + .await + .unwrap(); + let slots = crowdb_paxos_kv_client::ChunkSlotMapClient::new(cluster.make_crowdb_client()); + slots + .initialize_layout(&crowdb_protocol::chunk_slot::ChunkSlotBootstrap { + service_instances: vec![100], + storage_groups: vec![crowdb_protocol::chunk_slot::ChunkStorageGroup { + store_id: 0, + group_id: 1, + }], + }) + .await + .unwrap(); + slots.initialize_service_epochs().await.unwrap(); + let io = ChunkIoClient::connect_with_kv( + ChunkIoClientConfig { + management_seeds: vec![], + diskio_connections_per_endpoint: 1, + diskio_rpc_workers: 1, + small_write: SmallWritePolicy::new(ChunkType::Wal), + }, + cluster.make_crowdb_client(), + ) + .await + .unwrap(); + let now = StreamRecord::creation_time_ms(); + let lease = StreamProcessLeaseClient::new(cluster.make_crowdb_client()) + .claim(StreamPurpose::Wal, 71, [7; 16], None, now, 300_000) + .await + .unwrap(); + let process = Arc::new( + StreamProcessAuthority::new(lease, StreamProcessLeaseTiming::default(), now, Instant::now()) + .unwrap(), + ); + Self { + cluster, + _diskdb: diskdb, + _disks: disks, + harness, + rpc, + io, + process, + } + } +} +impl Drop for TestRuntimeStack { + fn drop(&mut self) { + self.rpc.stop(); + } +} + +async fn checkpoint_values( + host: &ChunkKvStorage, + bootstrap: &BootstrapPartitionConfig, +) -> ( + Partition, + Arc<crowdb_tree_ffi::PageStore>, + crowdb_chunk_kv::Checkpoint, +) { + host.streams() + .registry() + .create(StreamRecord { + purpose: StreamPurpose::Wal, + stream_name: bootstrap.stream_name, + metadata_group_id: 1, + record_version: 1, + state: StreamRecordState::Active, + consumer_kind: Some("chunk-kv-partition".into()), + consumer_id: Some(format!( + "{:016x}{:016x}", + bootstrap.partition_id.high, bootstrap.partition_id.low + )), + created_at_ms: StreamRecord::creation_time_ms(), + }) + .await + .unwrap(); + let stream = host + .create_registered_stream(bootstrap.stream_name, 1) + .await + .unwrap(); + let pages = host + .open_durable_tree_page_store( + ChunkPageStoreOptions { + tree_id: 42, + owner_epoch: 1, + open_generation: 0, + pack_bytes: 0, + iu_size: 0, + max_concurrent_packs: 0, + materialization_bytes_per_pass: 0, + max_chunk_bytes: 0, + mirror_copies: 0, + }, + 1, + ) + .await + .unwrap(); + let owned_pages = pages.clone(); + let config = Config { + runtime: Some(host.tree_runtime().clone()), + ..Default::default() + }; + let partition = tokio::task::spawn_blocking(move || { + Partition::open_native( + PartitionId { high: 17, low: 18 }, + PartitionRange { + start: Some(vec![]), + end: None, + }, + 1, + PartitionConfig { + retained_results: 1, + ..Default::default() + }, + 42, + config, + owned_pages, + stream, + ) + }) + .await + .unwrap() + .unwrap(); + let mut checkpoint = None; + for sequence in 1..=4_u64 { + partition + .mutate( + 1, + RequestId { + client_high: 99, + client_low: 1, + client_sequence: sequence, + }, + MutationOperation::Put { + key: b"key".to_vec(), + value: vec![u8::try_from(sequence).unwrap(); 20_000], + }, + ) + .await + .unwrap(); + checkpoint = Some(partition.checkpoint(1).await.unwrap()); + } + let checkpoint = checkpoint.unwrap(); + (partition, pages, checkpoint) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn runtime_host_recovers_root_wal_pair_and_collects_real_allocations() { + let stack = TestRuntimeStack::start().await; + let host = Arc::new(storage(&stack.cluster, &stack.io, &stack.process).await); + let bootstrap = BootstrapPartitionConfig { + partition_id: Id128 { high: 17, low: 18 }, + tree_id: 42, + stream_name: StreamName { high: 19, low: 20 }, + owner_epoch: 1, + metadata_group_id: 1, + }; + let (partition, pages, checkpoint) = checkpoint_values(&host, &bootstrap).await; + let service = Arc::new(crowdb_chunk_kv_server::ChunkKvService::new(stack.process.clone(), 8).unwrap()); + service.install_partition(&partition).unwrap(); + let mut sampler = + crowdb_chunk_kv_server::metrics::TreeResources::new(host.tree_runtime().clone(), &service) + .with_storage(&host); + sampler.sample(); + let mut resource_log = Vec::new(); + sampler.flush(&mut resource_log, "test").unwrap(); + let resource_log = String::from_utf8(resource_log).unwrap(); + let pins = resource_log + .lines() + .find(|line| line.contains("host.tree.resource.chunk_pin_release_slots ")) + .unwrap(); + assert!(pins.contains("limit=1048576"), "{pins}"); + let registered: u64 = pins + .split_whitespace() + .find_map(|field| field.strip_prefix("used=")) + .unwrap() + .parse() + .unwrap(); + assert!(registered > 0, "{pins}"); + assert!(pins.contains("allocated_sample_max="), "{pins}"); + drop((sampler, service)); + assert!(checkpoint.replay_offset > 0); + assert_eq!(pages.wal_replay_offset().unwrap(), checkpoint.replay_offset); + assert!(pages.reclaim_chunk_generations_before(42, checkpoint.root_manifest_generation) > 0); + let before = stack.harness.store.list_chunks(None, 100).await.unwrap(); + assert!(before + .iter() + .any(|chunk| chunk.chunk_type == ChunkType::BtreePage as i32)); + partition.suspend_for_shutdown(1).await.unwrap(); + tokio::task::spawn_blocking(move || drop((partition, pages))) + .await + .unwrap(); + host.drain_pin_releases().await; + drop(host); + + let reopened_host = storage(&stack.cluster, &stack.io, &stack.process).await; + let recovered = reopened_host.bootstrap_partition(&bootstrap).await.unwrap(); + recovered.activate_recovered(1).unwrap(); + assert_eq!( + recovered.get(1, b"key", None).await.unwrap().unwrap().value, + vec![4; 20_000] + ); + let stable = recovered.checkpoint(1).await.unwrap(); + assert_eq!(stable.replay_offset, checkpoint.replay_offset); + let frontier = recovered.snapshot(); + for _ in 0..64 { + if !recovered.maintain_serving_metadata(1).await.unwrap() { + break; + } + } + let after = stack.harness.store.list_chunks(None, 100).await.unwrap(); + assert!(after + .iter() + .any(|chunk| chunk.chunk_type == ChunkType::BtreePage as i32 + && chunk.state == ChunkState::Deleted as i32 + && chunk.strips.is_empty())); + assert_eq!(recovered.snapshot().applied_seq, frontier.applied_seq); + assert_eq!( + recovered.snapshot().journal_durable_offset, + frontier.journal_durable_offset + ); + recovered.suspend_for_shutdown(1).await.unwrap(); + tokio::task::spawn_blocking(move || drop(recovered)) + .await + .unwrap(); + reopened_host.drain_pin_releases().await; +} + +async fn runtime_tree_config(host: &ChunkKvStorage, tree_id: u64) -> Config { + let pages = host + .open_durable_tree_page_store( + ChunkPageStoreOptions { + tree_id, + owner_epoch: 1, + open_generation: 0, + pack_bytes: 0, + iu_size: 0, + max_concurrent_packs: 0, + materialization_bytes_per_pass: 0, + max_chunk_bytes: 0, + mirror_copies: 0, + }, + 1, + ) + .await + .unwrap(); + Config { + page_store: Some(pages), + runtime: Some(host.tree_runtime().clone()), + ..Config::default() + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn chunk_partition_snapshot_rebuilds_after_writer_close_without_reopening_source() { + use crowdb_chunk_kv::{ChunkKvError, CrowdbPartitionTree, PartitionTree}; + let stack = TestRuntimeStack::start().await; + let host = storage(&stack.cluster, &stack.io, &stack.process).await; + let config = runtime_tree_config(&host, 42).await; + let source = tokio::task::spawn_blocking(move || CrowdbPartitionTree::open(42, &config)) + .await + .unwrap() + .unwrap(); + source + .apply( + 1, + &MutationOperation::Put { + key: b"a".to_vec(), + value: b"left".to_vec(), + }, + ) + .await + .unwrap(); + source + .apply( + 2, + &MutationOperation::Put { + key: b"z".to_vec(), + value: vec![b'v'; 9000], + }, + ) + .await + .unwrap(); + let (_, coverage, snapshot) = source.checkpoint_snapshot(0).await.unwrap(); + source + .apply( + 3, + &MutationOperation::Put { + key: b"z".to_vec(), + value: b"new".to_vec(), + }, + ) + .await + .unwrap(); + tokio::task::spawn_blocking(move || drop(source)).await.unwrap(); + assert_eq!(snapshot.get(b"z").await.unwrap().unwrap().value, vec![b'v'; 9000]); + assert!(matches!( + snapshot.advance_noop(4).await, + Err(ChunkKvError::InvalidRequest(_)) + )); + let config = runtime_tree_config(&host, 43).await; + let (child, stats) = snapshot + .rebuild_range( + 43, + &PartitionRange { + start: Some(b"m".to_vec()), + end: None, + }, + config, + ) + .await + .unwrap(); + assert_eq!(stats.entries_emitted, 1); + assert_eq!(child.last_applied_seq(), coverage); + assert_eq!(child.get(b"z").await.unwrap().unwrap().value, vec![b'v'; 9000]); + tokio::task::spawn_blocking(move || drop((child, snapshot))) + .await + .unwrap(); + host.drain_pin_releases().await; +} + +#[test] +fn diskio_fixture_only_acknowledges_valid_fixed_read_ages() { + use common::diskio_stub::checked_read_age; + use crowdb_protocol::diskio_fb::FBDiskIoRetCode; + use crowdb_protocol::timing::{MAX_WRITE_CLOCK_SKEW_MS, MAX_WRITE_REQUEST_AGE_MS}; + let now = 1_000_000; + assert_eq!(checked_read_age(now, true, 0, now), Ok(true)); + assert_eq!(checked_read_age(now, false, 16, now), Ok(true)); + assert_eq!(checked_read_age(0, false, 16, now), Ok(false)); + for (created, probe, size) in [ + (0, true, 0), + (now, true, 1), + ( + now - MAX_WRITE_REQUEST_AGE_MS - MAX_WRITE_CLOCK_SKEW_MS, + false, + 16, + ), + (now + MAX_WRITE_CLOCK_SKEW_MS + 1, false, 16), + (u64::MAX, false, 16), + ] { + assert_eq!( + checked_read_age(created, probe, size, now), + Err(FBDiskIoRetCode::OldRequest) + ); + } +} diff --git a/app/crowdb-chunkdb/tests/client_owner_refresh_test.rs b/app/crowdb-chunkdb/tests/client_owner_refresh_test.rs index a7fa3f8d5..bd55000fc 100644 --- a/app/crowdb-chunkdb/tests/client_owner_refresh_test.rs +++ b/app/crowdb-chunkdb/tests/client_owner_refresh_test.rs @@ -6,7 +6,7 @@ mod slot_groups; use crowdb_chunkdb_client::{ChunkdbClient, ChunkdbRpcTransport}; -use crowdb_kv_client::{ChunkSlotMapClient, RangeBindingClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{ChunkSlotMapClient, RangeBindingClient, ServiceRegistryClient}; use crowdb_protocol::chunk_slot::{ChunkSlot, ChunkSlotBootstrap, ChunkStorageGroup}; use crowdb_protocol::chunkdb::rpc::AllocateChunkRequest; use crowdb_protocol::fb::FBMsgType; diff --git a/app/crowdb-chunkdb/tests/common/cluster.rs b/app/crowdb-chunkdb/tests/common/cluster.rs index 3d9aa5fa1..a6207e851 100644 --- a/app/crowdb-chunkdb/tests/common/cluster.rs +++ b/app/crowdb-chunkdb/tests/common/cluster.rs @@ -33,22 +33,22 @@ use crowdb_diskdb::recovery::ZoneLoader; use crowdb_diskdb::scanner::ScanState; use crowdb_diskdb::service::DiskdbRpcService; use crowdb_diskdb_client::DiskdbRpcTransport; -use crowdb_kv::cluster::KvServer; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, HardwareClient, ServiceRegistryClient}; +use crowdb_paxos_kv::cluster::PaxosKvServer; +use crowdb_paxos_kv_client::{ClientConfig, HardwareClient, PaxosKvClient, ServiceRegistryClient}; use crowdb_protocol::chunkdb::rpc::{RelocateSegmentHandoffRequest, RelocationHandoffDisposition}; use crowdb_protocol::common::{DiskId, HwStatus, NodeValue, RackValue}; use crowdb_protocol::diskdb::rpc::{DiskGroupValue, DiskType, DiskValue, Segment}; pub struct KvCluster { - memory_store: Arc<crowdb_kv::cluster::PxKvStore>, + memory_store: Arc<crowdb_paxos_kv::cluster::PxKvStore>, endpoint: String, } impl KvCluster { /// Pinned memory dependency for component handler tests. pub async fn start_pinned() -> Self { - use crowdb_kv::cluster::group::PxGroup; - use crowdb_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; + use crowdb_paxos_kv::cluster::group::PxGroup; + use crowdb_paxos_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; let store = Arc::new(PxKvStore::new(0, "127.0.0.1:0".parse().unwrap())); for id in [0, 1] { store.add_group(PxGroup::new( @@ -64,8 +64,8 @@ impl KvCluster { } } - pub fn make_crowdb_client(&self) -> Arc<CrowdbKvClient> { - let kv = CrowdbKvClient::new(ClientConfig::new(vec![])); + pub fn make_crowdb_client(&self) -> Arc<PaxosKvClient> { + let kv = PaxosKvClient::new(ClientConfig::new(vec![])); kv.seed_leader(0, 0, self.endpoint.clone()); kv.seed_leader(0, 1, self.endpoint.clone()); Arc::new(kv) diff --git a/app/crowdb-chunkdb/tests/common/diskio_stub.rs b/app/crowdb-chunkdb/tests/common/diskio_stub.rs index 0f1d5ac4e..8d5a9fce8 100644 --- a/app/crowdb-chunkdb/tests/common/diskio_stub.rs +++ b/app/crowdb-chunkdb/tests/common/diskio_stub.rs @@ -4,10 +4,11 @@ //! Volatile protocol stub for pinned placement/task proofs. Packaged Diskio //! discovery, bytes and durability are verified by the container acceptance. -use crowdb_kv_client::ServiceRegistryClient; +use crowdb_paxos_kv_client::ServiceRegistryClient; use crowdb_protocol::diskio_fb::{ - FBDiskFsyncRequest, FBDiskFsyncResponse, FBDiskFsyncResponseArgs, FBDiskReadRequest, FBDiskReadResponse, - FBDiskReadResponseArgs, FBDiskWriteRequest, FBDiskWriteResponse, FBDiskWriteResponseArgs, + FBDiskFsyncRequest, FBDiskFsyncResponse, FBDiskFsyncResponseArgs, FBDiskIoRetCode, FBDiskReadRequest, + FBDiskReadResponse, FBDiskReadResponseArgs, FBDiskWriteRequest, FBDiskWriteResponse, + FBDiskWriteResponseArgs, }; use crowdb_protocol::fb::{ ConnectionPingRequest, ConnectionPingResponse, ConnectionPingResponseArgs, FBMsgType, @@ -155,11 +156,34 @@ fn response( } else if kind == FBMsgType::EDiskReadRequest.0 as u16 { control.read_calls.fetch_add(1, Ordering::Relaxed); let request = flatbuffers::root::<FBDiskReadRequest>(ctrl).unwrap(); - payload = read_blocks(blocks, &request); + let now_ms = u64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_millis(), + ) + .unwrap(); + let age = checked_read_age( + request.read_create_time_ms(), + request.fence_probe(), + request.size(), + now_ms, + ); + let (ret_code, request_age_checked) = match age { + Ok(checked) => { + if !request.fence_probe() { + payload = read_blocks(blocks, &request); + } + (FBDiskIoRetCode::Success, checked) + } + Err(error) => (error, false), + }; let reply = FBDiskReadResponse::create( &mut builder, &FBDiskReadResponseArgs { id: request.id(), + ret_code, + request_age_checked, ..Default::default() }, ); @@ -208,3 +232,26 @@ fn read_blocks(blocks: &Blocks, request: &FBDiskReadRequest<'_>) -> Vec<u8> { } bytes } + +// Mirror the production DiskIO admission age ceiling for synchronous fixture reads. +pub fn checked_read_age( + created_ms: u64, + probe: bool, + size: u32, + now_ms: u64, +) -> Result<bool, FBDiskIoRetCode> { + use crowdb_protocol::timing::{MAX_WRITE_CLOCK_SKEW_MS, MAX_WRITE_REQUEST_AGE_MS}; + if probe && (created_ms == 0 || size != 0) { + return Err(FBDiskIoRetCode::OldRequest); + } + if created_ms == 0 { + return Ok(false); + } + let deadline = created_ms.checked_add(MAX_WRITE_REQUEST_AGE_MS + MAX_WRITE_CLOCK_SKEW_MS); + if created_ms.saturating_sub(now_ms) > MAX_WRITE_CLOCK_SKEW_MS + || deadline.map_or(true, |deadline| deadline <= now_ms) + { + return Err(FBDiskIoRetCode::OldRequest); + } + Ok(true) +} diff --git a/app/crowdb-chunkdb/tests/common/dynamic_runtime.rs b/app/crowdb-chunkdb/tests/common/dynamic_runtime.rs index 6c856f06e..aa3269558 100644 --- a/app/crowdb-chunkdb/tests/common/dynamic_runtime.rs +++ b/app/crowdb-chunkdb/tests/common/dynamic_runtime.rs @@ -6,7 +6,7 @@ use crowdb_chunkdb::{ range_guard::RangeGuard, routing::{BindingCache, BindingTable}, }; -use crowdb_kv_client::CrowdbKvClient; +use crowdb_paxos_kv_client::PaxosKvClient; use crowdb_protocol::{ chunk_domain::ChunkDomain, chunk_slot::{ChunkSlotAuthority, ChunkSlotBootstrap, ChunkSlotMap, ChunkStorageGroup}, @@ -22,7 +22,7 @@ pub struct TestDynamicRuntime { workers: Vec<tokio::task::JoinHandle<()>>, } impl TestDynamicRuntime { - pub async fn start(kv: Arc<CrowdbKvClient>, map: &ChunkSlotMap<ChunkSlotAuthority>, owner: u64) -> Self { + pub async fn start(kv: Arc<PaxosKvClient>, map: &ChunkSlotMap<ChunkSlotAuthority>, owner: u64) -> Self { let routes = BindingCache::new(); routes .replace(BindingTable::new( diff --git a/app/crowdb-chunkdb/tests/common/readiness_failures.rs b/app/crowdb-chunkdb/tests/common/readiness_failures.rs index c50dc1fd9..95513f5e7 100644 --- a/app/crowdb-chunkdb/tests/common/readiness_failures.rs +++ b/app/crowdb-chunkdb/tests/common/readiness_failures.rs @@ -3,15 +3,15 @@ use crowdb_chunkdb::range_guard::RangeGuard; use crowdb_chunkdb_client::{ChunkdbClient, ChunkdbClientError}; -use crowdb_kv_client::CrowdbKvClient; +use crowdb_paxos_kv_client::PaxosKvClient; use crowdb_protocol::chunk_id::{ChunkIdParts, CHUNK_TYPE_S3}; use crowdb_rpc_ffi::RpcServer; use std::time::Duration; pub async fn assert_storage_and_owner_failures( client: &ChunkdbClient, - data: &CrowdbKvClient, - control: &CrowdbKvClient, + data: &PaxosKvClient, + control: &PaxosKvClient, guard: &RangeGuard, server: &RpcServer, ) { diff --git a/app/crowdb-chunkdb/tests/common/readiness_registry.rs b/app/crowdb-chunkdb/tests/common/readiness_registry.rs index adf4896fc..676baad56 100644 --- a/app/crowdb-chunkdb/tests/common/readiness_registry.rs +++ b/app/crowdb-chunkdb/tests/common/readiness_registry.rs @@ -2,14 +2,14 @@ // Licensed under the Apache License, Version 2.0. use crowdb_chunkdb_client::ChunkdbClient; -use crowdb_kv_client::CrowdbKvClient; +use crowdb_paxos_kv_client::PaxosKvClient; use crowdb_rpc_ffi::RpcServer; use std::time::Duration; pub async fn assert_warm_registry_independence( client: &ChunkdbClient, - control: &CrowdbKvClient, - data: &CrowdbKvClient, + control: &PaxosKvClient, + data: &PaxosKvClient, server: &RpcServer, endpoint: String, ) { diff --git a/app/crowdb-chunkdb/tests/common/repair_targets.rs b/app/crowdb-chunkdb/tests/common/repair_targets.rs index cf9e15e4b..e6d81d203 100644 --- a/app/crowdb-chunkdb/tests/common/repair_targets.rs +++ b/app/crowdb-chunkdb/tests/common/repair_targets.rs @@ -14,7 +14,7 @@ use crowdb_chunkdb::routing::{default_binding_table, BindingCache}; use crowdb_chunkdb::task::{TaskExecutor, TaskManager, TaskStore}; use crowdb_chunkdb::topology::build_snapshot; use crowdb_common::metrics::MetricsRegistry; -use crowdb_kv_client::{GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{GetOutcome, ReadMode}; use crowdb_protocol::chunk_task::{ChunkTaskValue, RepairTargetCheckpoint, RepairTargetPhase}; use crowdb_protocol::chunkdb::rpc::{Chunk, ChunkType, Strip, StripType}; use crowdb_protocol::common::{ChunkId, InstanceValue}; diff --git a/app/crowdb-chunkdb/tests/common/runtime_context.rs b/app/crowdb-chunkdb/tests/common/runtime_context.rs index b3f9fa351..2570f7681 100644 --- a/app/crowdb-chunkdb/tests/common/runtime_context.rs +++ b/app/crowdb-chunkdb/tests/common/runtime_context.rs @@ -10,12 +10,12 @@ use crowdb_chunkdb::routing::BindingCache; use crowdb_chunkdb::runtime::RuntimeContext; use crowdb_chunkdb::topology::TopologyCache; use crowdb_common::metrics::MetricsRegistry; -use crowdb_kv_client::{CrowdbKvClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{PaxosKvClient, ServiceRegistryClient}; use std::sync::Arc; use std::time::Duration; pub fn runtime_context( - kv: Arc<CrowdbKvClient>, + kv: Arc<PaxosKvClient>, bindings: BindingCache, guard: Arc<RangeGuard>, ) -> (RuntimeContext, tokio::sync::watch::Sender<bool>) { diff --git a/app/crowdb-chunkdb/tests/common/slot_groups.rs b/app/crowdb-chunkdb/tests/common/slot_groups.rs index 1954464b4..69c28c50b 100644 --- a/app/crowdb-chunkdb/tests/common/slot_groups.rs +++ b/app/crowdb-chunkdb/tests/common/slot_groups.rs @@ -1,10 +1,10 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -use crowdb_kv::cluster::group::PxGroup; -use crowdb_kv::cluster::kv_server::KvServer; -use crowdb_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient}; +use crowdb_paxos_kv::cluster::group::PxGroup; +use crowdb_paxos_kv::cluster::paxos_kv_server::PaxosKvServer; +use crowdb_paxos_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient}; use crowdb_protocol::chunk_task::{ ChunkTaskState, ChunkTaskValue, CHUNK_TASK_SCHEMA_VERSION, TASK_KIND_FINALIZE_CHUNK, }; @@ -13,7 +13,7 @@ use std::sync::Arc; pub struct TestGroups { pub server: Arc<PxKvStore>, - pub kv: Arc<CrowdbKvClient>, + pub kv: Arc<PaxosKvClient>, } impl TestGroups { @@ -26,7 +26,7 @@ impl TestGroups { )); } server.start().await.unwrap(); - let kv = Arc::new(CrowdbKvClient::new(ClientConfig::new(Vec::new()))); + let kv = Arc::new(PaxosKvClient::new(ClientConfig::new(Vec::new()))); for group in 0..=3 { kv.seed_leader(0, group, server.listen_addr().unwrap().to_string()); } diff --git a/app/crowdb-chunkdb/tests/common/topology_availability.rs b/app/crowdb-chunkdb/tests/common/topology_availability.rs index 6efba4a20..23a1f6cf0 100644 --- a/app/crowdb-chunkdb/tests/common/topology_availability.rs +++ b/app/crowdb-chunkdb/tests/common/topology_availability.rs @@ -1,7 +1,7 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -use crowdb_kv_client::ServiceRegistryClient; +use crowdb_paxos_kv_client::ServiceRegistryClient; use crowdb_protocol::{common::DiskioHardwareScope, sysdata::DiskGroupEntry}; pub async fn heartbeat(registry: &ServiceRegistryClient, group: &DiskGroupEntry) { diff --git a/app/crowdb-chunkdb/tests/dynamic_retry_test.rs b/app/crowdb-chunkdb/tests/dynamic_retry_test.rs index 05ad8c568..313d4b0af 100644 --- a/app/crowdb-chunkdb/tests/dynamic_retry_test.rs +++ b/app/crowdb-chunkdb/tests/dynamic_retry_test.rs @@ -13,7 +13,7 @@ use crowdb_chunkdb::{ storage::ChunkStore, }; use crowdb_chunkdb_client::{ChunkdbClient, ChunkdbRpcTransport}; -use crowdb_kv_client::{ChunkSlotMapClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{ChunkSlotMapClient, ServiceRegistryClient}; use crowdb_protocol::chunk_slot::{ChunkSlot, ChunkSlotBootstrap, ChunkStorageGroup}; use crowdb_protocol::chunkdb::rpc::{Chunk, ChunkState, DeleteChunkRequest}; use crowdb_protocol::common::ChunkId; diff --git a/app/crowdb-chunkdb/tests/e2e_test.rs b/app/crowdb-chunkdb/tests/e2e_test.rs index cc06b8e39..ed287249f 100644 --- a/app/crowdb-chunkdb/tests/e2e_test.rs +++ b/app/crowdb-chunkdb/tests/e2e_test.rs @@ -20,7 +20,7 @@ use crowdb_chunkdb::service::ChunkdbRpcService; use crowdb_chunkdb::storage::ChunkStore; use crowdb_chunkdb::topology::TopologyCache; use crowdb_common::metrics::MetricsRegistry; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient, ServiceRegistryClient}; use crowdb_protocol::common::{ChunkId, HwStatus}; use crowdb_protocol::diskdb::rpc::DiskGroupValue; use crowdb_protocol::sysdata::DiskGroupEntry; @@ -49,7 +49,7 @@ fn build_test_topology() -> TopologyCache { /// Build a fully wired LifecycleHandler with a dummy KV client /// (calls will fail, but construction verifies all wiring). fn build_handler() -> LifecycleHandler { - let kv = Arc::new(CrowdbKvClient::new(ClientConfig::new(vec![ + let kv = Arc::new(PaxosKvClient::new(ClientConfig::new(vec![ "http://127.0.0.1:1".into() ]))); let svc = ServiceRegistryClient::from_shared(Arc::clone(&kv)); diff --git a/app/crowdb-chunkdb/tests/http_readiness_test.rs b/app/crowdb-chunkdb/tests/http_readiness_test.rs index ebaa33153..972e8e1bc 100644 --- a/app/crowdb-chunkdb/tests/http_readiness_test.rs +++ b/app/crowdb-chunkdb/tests/http_readiness_test.rs @@ -8,7 +8,7 @@ mod http_readiness; mod slot_groups; use crowdb_chunkdb::{range_guard::RangeGuard, readiness::RangeReadiness}; -use crowdb_kv_client::ChunkSlotMapClient; +use crowdb_paxos_kv_client::ChunkSlotMapClient; use http_readiness::TestReadiness; use std::{ sync::{atomic::Ordering, Arc}, diff --git a/app/crowdb-chunkdb/tests/journal_handoff_test.rs b/app/crowdb-chunkdb/tests/journal_handoff_test.rs index 7618a3a11..75d6a1929 100644 --- a/app/crowdb-chunkdb/tests/journal_handoff_test.rs +++ b/app/crowdb-chunkdb/tests/journal_handoff_test.rs @@ -6,11 +6,11 @@ use std::time::Duration; use bytes::Bytes; use crowdb_chunkdb::storage::{decode_chunk_for_tests, encode_chunk_for_tests}; -use crowdb_kv::cluster::group::{ProposeResult, PxGroup}; -use crowdb_kv::cluster::{PxLocalReplica, PxLocalReplicaRole}; -use crowdb_kv::kv::{CrowdbTreeConfig, CrowdbTreeEngine, KVEngine}; -use crowdb_kv::paxos::learner::PxLearner; -use crowdb_kv::paxos::roles::{PxBallot, PxLogEntry}; +use crowdb_paxos_kv::cluster::group::{ProposeResult, PxGroup}; +use crowdb_paxos_kv::cluster::{PxLocalReplica, PxLocalReplicaRole}; +use crowdb_paxos_kv::kv::{CrowdbTreeConfig, CrowdbTreeEngine, KVEngine}; +use crowdb_paxos_kv::paxos::learner::PxLearner; +use crowdb_paxos_kv::paxos::roles::{PxBallot, PxLogEntry}; use crowdb_protocol::chunkdb::rpc::{Chunk, ChunkType}; use crowdb_protocol::common::ChunkId; @@ -97,13 +97,30 @@ async fn journal_cas_retains_future_revision_through_flush_and_reopen() { handle.flush().unwrap(); assert_eq!(handle.snapshot().unwrap(), 851); - learn(&group, 853, encode_puts(&[(&chunk_key(), &metadata(60, 24040))])).await; + let pending_group = Arc::clone(&group); + let pending = tokio::spawn(async move { + learn( + &pending_group, + 853, + encode_puts(&[(&chunk_key(), &metadata(60, 24040))]), + ) + .await; + }); + tokio::time::timeout(Duration::from_secs(5), async { + while group.local_replica().last_chosen_slot() < 853 { + tokio::task::yield_now().await; + } + }) + .await + .expect("slot 853 must be chosen while its contiguous apply waits for 852"); let engine = group.local_replica().learner.engine(); - // Keep slot 852 absent: every maintenance pass must retain the future value. + // The chosen task owns the future value until the gap is repaired. Flush + // must preserve that work while reads still observe the contiguous prefix. for _ in 0..3 { handle.flush().unwrap(); assert_eq!(handle.snapshot().unwrap(), 851); - assert_metadata(engine, 853, 60, 24040).await; + assert_metadata(engine, 843, 58, 23039).await; + assert!(!pending.is_finished(), "future apply must wait for the gap"); } group.set_next_slot(854); let journal: Vec<u8> = (0..24643).map(|i| u8::try_from(i % 251).unwrap()).collect(); @@ -116,10 +133,14 @@ async fn journal_cas_retains_future_revision_through_flush_and_reopen() { tokio::task::yield_now().await; handle.flush().unwrap(); assert_eq!(handle.snapshot().unwrap(), 851); - assert_metadata(engine, 853, 60, 24040).await; + assert_metadata(engine, 843, 58, 23039).await; assert!(!cas.is_finished(), "the apply fence must not cross the gap"); } learn(&group, 852, Vec::new()).await; + tokio::time::timeout(Duration::from_secs(5), pending) + .await + .expect("gap repair must release the chosen apply") + .unwrap(); let result = tokio::time::timeout(Duration::from_secs(5), cas) .await .unwrap() diff --git a/app/crowdb-chunkdb/tests/mixed_geometry_test.rs b/app/crowdb-chunkdb/tests/mixed_geometry_test.rs index a6e2c2f04..693a2478a 100644 --- a/app/crowdb-chunkdb/tests/mixed_geometry_test.rs +++ b/app/crowdb-chunkdb/tests/mixed_geometry_test.rs @@ -6,7 +6,7 @@ mod common; use crowdb_chunkdb::allocator::{ChunkAllocator, DiskdbClientPool, StripAllocType}; use crowdb_chunkdb::selector::PlacementConstraints; use crowdb_chunkdb::topology::build_snapshot; -use crowdb_kv_client::ServiceRegistryClient; +use crowdb_paxos_kv_client::ServiceRegistryClient; use std::sync::Arc; #[tokio::test] diff --git a/app/crowdb-chunkdb/tests/readiness_probe_test.rs b/app/crowdb-chunkdb/tests/readiness_probe_test.rs index cc34224e1..a08abb9d6 100644 --- a/app/crowdb-chunkdb/tests/readiness_probe_test.rs +++ b/app/crowdb-chunkdb/tests/readiness_probe_test.rs @@ -9,8 +9,8 @@ use crowdb_chunkdb::{ storage::ChunkStore, }; use crowdb_chunkdb_client::{ChunkdbClient, ChunkdbRpcTransport}; -use crowdb_kv::cluster::kv_server::KvServer; -use crowdb_kv_client::{ChunkSlotMapClient, ClientConfig, CrowdbKvClient, ServiceRegistryClient}; +use crowdb_paxos_kv::cluster::paxos_kv_server::PaxosKvServer; +use crowdb_paxos_kv_client::{ChunkSlotMapClient, ClientConfig, PaxosKvClient, ServiceRegistryClient}; use crowdb_protocol::{ chunk_domain::ChunkDomain, chunk_id::CHUNK_TYPE_S3, @@ -66,7 +66,7 @@ async fn readiness_reads_both_domains_without_scanning_or_mutating_the_catalog() .unwrap() .unwrap(); } - let control = Arc::new(CrowdbKvClient::new(ClientConfig::new(Vec::new()))); + let control = Arc::new(PaxosKvClient::new(ClientConfig::new(Vec::new()))); control.seed_leader(0, 0, test.server.listen_addr().unwrap().to_string()); let registry = ServiceRegistryClient::from_shared(Arc::clone(&control)); registry @@ -75,7 +75,7 @@ async fn readiness_reads_both_domains_without_scanning_or_mutating_the_catalog() .unwrap(); let client = ChunkdbClient::new(registry, Arc::new(ChunkdbRpcTransport::new())); Box::pin(client.observe_readiness()).await.unwrap(); - let reference = crowdb_kv_client::RangeBindingClient::from_shared(Arc::clone(&control)); + let reference = crowdb_paxos_kv_client::RangeBindingClient::from_shared(Arc::clone(&control)); let _ = control.drain_window(); reference.refresh().await.unwrap(); let binding_scans = control.drain_window().scan.len(); diff --git a/app/crowdb-chunkdb/tests/repair_target_recovery_test.rs b/app/crowdb-chunkdb/tests/repair_target_recovery_test.rs index f559c1926..e4ffddbe0 100644 --- a/app/crowdb-chunkdb/tests/repair_target_recovery_test.rs +++ b/app/crowdb-chunkdb/tests/repair_target_recovery_test.rs @@ -7,7 +7,7 @@ mod repair_targets; use crowdb_chunkdb::repair::decode_payload; use crowdb_chunkdb::task::SegmentOwnerResolver; -use crowdb_kv_client::{GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{GetOutcome, ReadMode}; use crowdb_protocol::chunk_task::{ChunkTaskState, RepairTargetPhase}; use crowdb_protocol::chunkdb::rpc::{SegmentOwnerDisposition, Strip}; use crowdb_protocol::key::{BinaryKey, BusyBlockKey}; diff --git a/app/crowdb-chunkdb/tests/runtime_scope_test.rs b/app/crowdb-chunkdb/tests/runtime_scope_test.rs index 3a780b6e6..7683d752d 100644 --- a/app/crowdb-chunkdb/tests/runtime_scope_test.rs +++ b/app/crowdb-chunkdb/tests/runtime_scope_test.rs @@ -110,7 +110,7 @@ async fn production_runtime_dispatches_all_purposes_and_lists_only_owned_chunks( #[tokio::test] async fn native_snapshots_route_by_service_slot_and_refresh_endpoints_without_storage_map() { use crowdb_chunkdb_client::ChunkdbClient; - use crowdb_kv_client::{ChunkSlotMapClient, ServiceRegistryClient}; + use crowdb_paxos_kv_client::{ChunkSlotMapClient, ServiceRegistryClient}; use crowdb_protocol::chunk_slot::ChunkSlot; use crowdb_protocol::common::ChunkId; diff --git a/app/crowdb-chunkdb/tests/slot_storage_test.rs b/app/crowdb-chunkdb/tests/slot_storage_test.rs index d2c9e9205..bd06fd0f7 100644 --- a/app/crowdb-chunkdb/tests/slot_storage_test.rs +++ b/app/crowdb-chunkdb/tests/slot_storage_test.rs @@ -6,7 +6,7 @@ use std::sync::Arc; use crowdb_chunkdb::routing::{BindingCache, BindingTable}; use crowdb_chunkdb::storage::ChunkStore; use crowdb_chunkdb::task::TaskStore; -use crowdb_kv_client::{ChunkSlotMapClient, GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{ChunkSlotMapClient, GetOutcome, ReadMode}; use crowdb_protocol::chunk_slot::{ChunkSlot, ChunkSlotBootstrap, ChunkStorageGroup}; use crowdb_protocol::chunkdb::rpc::{Chunk, ChunkState, ChunkType}; use crowdb_protocol::common::ChunkId; diff --git a/app/crowdb-chunkdb/tests/submission_epoch_test.rs b/app/crowdb-chunkdb/tests/submission_epoch_test.rs index 683b91932..f5f9f828d 100644 --- a/app/crowdb-chunkdb/tests/submission_epoch_test.rs +++ b/app/crowdb-chunkdb/tests/submission_epoch_test.rs @@ -158,7 +158,7 @@ async fn stale_execution_cannot_write_metadata_reservations_or_tasks_and_fresh_e #[tokio::test(flavor = "multi_thread", worker_threads = 4)] async fn epoch_publication_does_not_drain_an_accepted_data_write_and_the_final_check_race_is_allowed() { - use crowdb_kv_client::{ChunkSlotMapClient, GetOutcome, ReadMode}; + use crowdb_paxos_kv_client::{ChunkSlotMapClient, GetOutcome, ReadMode}; use crowdb_protocol::chunk_slot::{ChunkSlotBootstrap, ChunkStorageGroup}; use std::{sync::Arc, time::Duration}; let test = slot_groups::TestGroups::start().await; diff --git a/app/crowdb-chunkdb/tests/topology_availability_test.rs b/app/crowdb-chunkdb/tests/topology_availability_test.rs index 8bf884342..1d585ea2e 100644 --- a/app/crowdb-chunkdb/tests/topology_availability_test.rs +++ b/app/crowdb-chunkdb/tests/topology_availability_test.rs @@ -9,7 +9,7 @@ use crowdb_chunkdb::{ selector::{MirrorPlacement, PlacementConstraints}, topology::build_snapshot, }; -use crowdb_kv_client::ServiceRegistryClient; +use crowdb_paxos_kv_client::ServiceRegistryClient; use crowdb_protocol::{ common::{HwStatus, InstanceValue}, key::InstanceKey, @@ -35,7 +35,7 @@ async fn expired_physical_io_is_excluded_without_changing_hardware_and_returns_a .as_millis(), ) .unwrap() - + crowdb_kv_client::SERVICE_INSTANCE_TTL_MS + + crowdb_paxos_kv_client::SERVICE_INSTANCE_TTL_MS + 1; assert!(before.healthy_disk_groups_at_for_tests(expired).is_empty()); let stopped = &groups[0]; @@ -50,12 +50,12 @@ async fn expired_physical_io_is_excluded_without_changing_hardware_and_returns_a 0, 0, key.as_bytes(), - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ) .await .unwrap(); - let crowdb_kv_client::GetOutcome::Found { value, .. } = value else { + let crowdb_paxos_kv_client::GetOutcome::Found { value, .. } = value else { panic!("missing IO registration") }; let mut observation: InstanceValue = serde_json::from_slice(&value).unwrap(); diff --git a/app/crowdb-cli/Cargo.toml b/app/crowdb-cli/Cargo.toml index 6fe0b758a..25d8bc84a 100644 --- a/app/crowdb-cli/Cargo.toml +++ b/app/crowdb-cli/Cargo.toml @@ -18,7 +18,7 @@ path = "src/main.rs" futures = "0.3" uuid = { version = "1", features = ["v4"] } crowdb-console-shared = { path = "../../lib/crowdb-console-shared" } -crowdb-kv-client = { path = "../../lib/crowdb-kv-client" } +crowdb-paxos-kv-client = { path = "../../lib/crowdb-paxos-kv-client" } crowdb-common = { path = "../../lib/crowdb-common/rust" } crowdb-protocol = { path = "../../lib/crowdb-protocol" } crowdb-rpc-ffi = { path = "../../lib/crowdb-rpc/ffi" } @@ -27,7 +27,7 @@ crowdb-chunkdb-client = { path = "../../lib/crowdb-chunkdb-client" } crowdb-chunk-client = { path = "../../lib/crowdb-chunk-client" } flatbuffers = { workspace = true } clap = { version = "4", features = ["derive", "env"] } -tokio = { workspace = true, features = ["rt-multi-thread", "macros", "time", "sync", "net"] } +tokio = { workspace = true, features = ["rt-multi-thread", "macros", "time", "sync", "net", "process"] } serde = { version = "1", features = ["derive"] } serde_json = "1" hex = "0.4" @@ -84,3 +84,7 @@ path = "tests/cluster_lifecycle_test.rs" [[test]] name = "cluster_workloads_test" path = "tests/cluster_workloads_test.rs" + +[[test]] +name = "kv_bench_test" +path = "tests/kv_bench_test.rs" diff --git a/app/crowdb-cli/src/commands.rs b/app/crowdb-cli/src/commands.rs index 0dfcc48aa..15ed6088a 100644 --- a/app/crowdb-cli/src/commands.rs +++ b/app/crowdb-cli/src/commands.rs @@ -9,15 +9,15 @@ pub(crate) mod bench; pub(crate) mod cluster; -pub(crate) mod kv; +pub(crate) mod paxos_kv; pub(crate) mod port_alloc; pub(crate) mod s3; pub(crate) use bench::{run_bench_verb, BenchVerb}; pub(crate) use cluster::{run_cluster_verb, ClusterVerb}; -pub(crate) use kv::{ - run_group_verb, run_kv_data_verb, run_replica_verb, run_store_verb, GroupVerb, KvDataVerb, ReplicaVerb, - StoreVerb, +pub(crate) use paxos_kv::{ + run_group_verb, run_paxos_kv_data_verb, run_replica_verb, run_store_verb, GroupVerb, PaxosKvDataVerb, + ReplicaVerb, StoreVerb, }; pub(crate) use port_alloc::{run as run_port_alloc, PortAllocArgs}; pub(crate) use s3::{run_s3_verb, S3Verb}; @@ -47,7 +47,7 @@ pub(crate) async fn group0_connection(cli: &Cli) -> Result<(Vec<String>, String) eprintln!("error: cluster native transport: {error}"); ExitCode::from(2) })?; - let native = crowdb_kv_client::CrowdbKvClient::new(crowdb_kv_client::ClientConfig::new( + let native = crowdb_paxos_kv_client::PaxosKvClient::new(crowdb_paxos_kv_client::ClientConfig::new( target.identity.group0_management_endpoints.clone(), )); native.seed_leader(0, 0, target.identity.group0_rpc_endpoints[0].clone()); diff --git a/app/crowdb-cli/src/commands/bench.rs b/app/crowdb-cli/src/commands/bench.rs index 2919f81da..6db01d143 100644 --- a/app/crowdb-cli/src/commands/bench.rs +++ b/app/crowdb-cli/src/commands/bench.rs @@ -9,13 +9,14 @@ //! under `bench/`. pub mod chunk; +mod client; pub mod disk; pub mod io; -pub mod kv; pub mod loader; pub mod metrics; -pub mod result; +pub mod paxos_kv; pub mod s3; pub mod verb; +pub mod workload; pub use verb::{run_bench_verb, BenchVerb}; diff --git a/app/crowdb-cli/src/commands/bench/chunk.rs b/app/crowdb-cli/src/commands/bench/chunk.rs index a562fcc31..a31fab99a 100644 --- a/app/crowdb-cli/src/commands/bench/chunk.rs +++ b/app/crowdb-cli/src/commands/bench/chunk.rs @@ -10,7 +10,7 @@ use std::time::{Duration, Instant}; use crowdb_chunkdb_client::{ChunkdbClient, ChunkdbRpcTransport, RangeBindingClient, RetryConfig}; use crowdb_diskdb_client::{DiskdbClient, DiskdbClientError, DiskdbRpcTransport}; -use crowdb_kv_client::{ReadEndpointPolicy, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{ReadEndpointPolicy, ServiceRegistryClient}; use crowdb_protocol::chunkdb::rpc::{ AllocateChunkRequest, AppendChunkRequest, Chunk, ChunkState, ChunkType, DeleteChunkRequest, QueryChunkRequest, SealChunkRequest, Strip, StripType, @@ -21,7 +21,7 @@ use crowdb_protocol::generate_chunk_id; use rand::rngs::SmallRng; use rand::{Rng, SeedableRng}; -use super::kv::{build_kv_client, KvClientTunables}; +use super::client::{build_kv_client, KvClientTunables}; use super::metrics::BenchMetrics; use super::verb::{ChunkdbArgs, ChunkdbBenchVerb, ChunkdbStripMode}; use crate::Cli; diff --git a/app/crowdb-cli/src/commands/bench/client.rs b/app/crowdb-cli/src/commands/bench/client.rs new file mode 100644 index 000000000..a6102c0dc --- /dev/null +++ b/app/crowdb-cli/src/commands/bench/client.rs @@ -0,0 +1,75 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Shared helper for building a [`PaxosKvClient`] from the CLI endpoint +//! with a bench-specific [`ReadEndpointPolicy`]. The standard `op_context` +//! helper always uses the default `Leader` policy; bench read/scan +//! commands need `AnyReplica` for distributed `MinSlot` reads. + +use std::process::ExitCode; + +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient, ReadEndpointPolicy}; + +use crate::Cli; + +/// Client-side tunables for bench KV commands. Mirrors the server-side +/// `KvDeployTunables` — the client transport must match the server's +/// `event_write` and `rpc_workers` for optimal performance. +#[derive(Debug, Clone)] +pub(crate) struct KvClientTunables { + pub event_write: bool, + pub rpc_workers: u32, + pub enable_nagle: bool, + pub quickack: bool, + pub send_queue_capacity: u32, + pub pool_size: usize, +} + +impl Default for KvClientTunables { + fn default() -> Self { + Self { + event_write: false, + rpc_workers: 2, + enable_nagle: false, + quickack: false, + send_queue_capacity: 4096, + pool_size: 1, + } + } +} + +/// Build a `PaxosKvClient` from the resolved cluster target with the given +/// `read_endpoint_policy`. +/// +/// # Errors +/// Returns `ExitCode::from(2)` if the management endpoint is unavailable. +pub(crate) async fn build_kv_client( + cli: &Cli, + read_endpoint_policy: ReadEndpointPolicy, + tunables: &KvClientTunables, +) -> Result<PaxosKvClient, ExitCode> { + let (seeds, effective_g0) = crate::commands::group0_connection(cli).await?; + + let mut client_config = ClientConfig::new(seeds); + client_config.read_endpoint_policy = read_endpoint_policy; + client_config.event_write = tunables.event_write; + client_config.rpc_workers = tunables.rpc_workers; + client_config.enable_nagle = tunables.enable_nagle; + client_config.quickack = tunables.quickack; + client_config.send_queue_capacity = tunables.send_queue_capacity; + client_config.pool_size_per_endpoint = tunables.pool_size; + + eprintln!( + "bench paxos-kv client: read_endpoint_policy={:?} event_write={} rpc_workers={} nagle={} quickack={} send_queue={}", + client_config.read_endpoint_policy, + client_config.event_write, + client_config.rpc_workers, + client_config.enable_nagle, + client_config.quickack, + client_config.send_queue_capacity, + ); + + let client = PaxosKvClient::new(client_config); + client.seed_leader(0, 0, effective_g0); + Ok(client) +} diff --git a/app/crowdb-cli/src/commands/bench/disk/db.rs b/app/crowdb-cli/src/commands/bench/disk/db.rs index da0ca92aa..d95cff466 100644 --- a/app/crowdb-cli/src/commands/bench/disk/db.rs +++ b/app/crowdb-cli/src/commands/bench/disk/db.rs @@ -10,14 +10,14 @@ use std::sync::Arc; use std::time::{Duration, Instant}; use crowdb_diskdb_client::{DiskdbClient, DiskdbClientError, DiskdbRpcTransport}; -use crowdb_kv_client::{ReadEndpointPolicy, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{ReadEndpointPolicy, ServiceRegistryClient}; use crowdb_protocol::common::ChunkId; use crowdb_protocol::diskdb::rpc::{AllocateBlocksRequest, CompactZoneRequest, FreeBlocksRequest, Segment}; use rand::rngs::SmallRng; use rand::SeedableRng; use super::workload::select_free; -use crate::commands::bench::kv::{build_kv_client, KvClientTunables}; +use crate::commands::bench::client::{build_kv_client, KvClientTunables}; use crate::commands::bench::loader::BenchRecorder; use crate::commands::bench::metrics::BenchMetrics; use crate::commands::bench::verb::{DiskdbArgs, DiskdbBenchVerb}; @@ -136,7 +136,14 @@ pub async fn run(cli: &Cli, verb: DiskdbBenchVerb) -> ExitCode { let elapsed = started.elapsed(); metrics.stop().await; let live: Vec<_> = receiver.try_iter().collect(); - if mixed && !compact_for_verification(&client, &groups).await { + if mixed + && !compact_for_verification( + &client, + &groups, + baseline + live_bytes(&live, args.unit_size_bytes), + ) + .await + { total.errors += 1; } let final_busy = busy_bytes(&client, &groups).await.unwrap_or(u64::MAX); @@ -181,14 +188,28 @@ fn valid_args(args: &DiskdbArgs) -> bool { valid } -async fn compact_for_verification(client: &Arc<DiskdbClient>, groups: &[u64]) -> bool { - for _ in 0..3 { +async fn compact_for_verification(client: &Arc<DiskdbClient>, groups: &[u64], expected_busy: u64) -> bool { + let deadline = Instant::now() + Duration::from_secs(30); + loop { if let Err(error) = compact_all(client, groups).await { eprintln!("diskdb compaction failed: {error}"); return false; } + match busy_bytes(client, groups).await { + Ok(actual) if actual == expected_busy => return true, + Ok(actual) if Instant::now() >= deadline => { + eprintln!( + "diskdb compaction did not drain pending frees: busy={actual} expected={expected_busy}" + ); + return false; + } + Ok(_) => tokio::time::sleep(Duration::from_millis(20)).await, + Err(error) => { + eprintln!("diskdb compaction capacity query failed: {error}"); + return false; + } + } } - true } fn report(total: &mut TaskResult, input: &ReportInput<'_>) -> ExitCode { diff --git a/app/crowdb-cli/src/commands/bench/kv.rs b/app/crowdb-cli/src/commands/bench/kv.rs deleted file mode 100644 index 2f6fff9eb..000000000 --- a/app/crowdb-cli/src/commands/bench/kv.rs +++ /dev/null @@ -1,14 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -//! KV-layer bench workloads (prepare / read / write / scan) against a -//! crowdb-kv store. The shared client builder is re-exported for the -//! `disk` and `chunk` sub-modules, which also drive a KV client. - -pub mod client; -pub mod prepare; -pub mod read; -pub mod scan; -pub mod write; - -pub(crate) use client::{build_kv_client, KvClientTunables}; diff --git a/app/crowdb-cli/src/commands/bench/kv/client.rs b/app/crowdb-cli/src/commands/bench/kv/client.rs deleted file mode 100644 index d8c5031e2..000000000 --- a/app/crowdb-cli/src/commands/bench/kv/client.rs +++ /dev/null @@ -1,75 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -//! Shared helper for building a [`CrowdbKvClient`] from the CLI endpoint -//! with a bench-specific [`ReadEndpointPolicy`]. The standard `op_context` -//! helper always uses the default `Leader` policy; bench read/scan -//! commands need `AnyReplica` for distributed `MinSlot` reads. - -use std::process::ExitCode; - -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, ReadEndpointPolicy}; - -use crate::Cli; - -/// Client-side tunables for bench KV commands. Mirrors the server-side -/// `KvDeployTunables` — the client transport must match the server's -/// `event_write` and `rpc_workers` for optimal performance. -#[derive(Debug, Clone)] -pub(crate) struct KvClientTunables { - pub event_write: bool, - pub rpc_workers: u32, - pub enable_nagle: bool, - pub quickack: bool, - pub send_queue_capacity: u32, - pub pool_size: usize, -} - -impl Default for KvClientTunables { - fn default() -> Self { - Self { - event_write: false, - rpc_workers: 2, - enable_nagle: false, - quickack: false, - send_queue_capacity: 4096, - pool_size: 1, - } - } -} - -/// Build a `CrowdbKvClient` from the resolved cluster target with the given -/// `read_endpoint_policy`. -/// -/// # Errors -/// Returns `ExitCode::from(2)` if the management endpoint is unavailable. -pub(crate) async fn build_kv_client( - cli: &Cli, - read_endpoint_policy: ReadEndpointPolicy, - tunables: &KvClientTunables, -) -> Result<CrowdbKvClient, ExitCode> { - let (seeds, effective_g0) = crate::commands::group0_connection(cli).await?; - - let mut client_config = ClientConfig::new(seeds); - client_config.read_endpoint_policy = read_endpoint_policy; - client_config.event_write = tunables.event_write; - client_config.rpc_workers = tunables.rpc_workers; - client_config.enable_nagle = tunables.enable_nagle; - client_config.quickack = tunables.quickack; - client_config.send_queue_capacity = tunables.send_queue_capacity; - client_config.pool_size_per_endpoint = tunables.pool_size; - - eprintln!( - "bench kv client: read_endpoint_policy={:?} event_write={} rpc_workers={} nagle={} quickack={} send_queue={}", - client_config.read_endpoint_policy, - client_config.event_write, - client_config.rpc_workers, - client_config.enable_nagle, - client_config.quickack, - client_config.send_queue_capacity, - ); - - let client = CrowdbKvClient::new(client_config); - client.seed_leader(0, 0, effective_g0); - Ok(client) -} diff --git a/app/crowdb-cli/src/commands/bench/kv/prepare.rs b/app/crowdb-cli/src/commands/bench/kv/prepare.rs deleted file mode 100644 index b0ddecfed..000000000 --- a/app/crowdb-cli/src/commands/bench/kv/prepare.rs +++ /dev/null @@ -1,115 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -//! `bench kv prepare` writes only the requested dataset into the selected group. -//! -//! Concurrent bulk-put loop with `--concurrency` tasks. No latency -//! histogram — this is a setup step, not a measurement. Reports total -//! keys written, elapsed time, and errors to stderr. - -use std::process::ExitCode; -use std::sync::atomic::{AtomicU64, Ordering}; -use std::sync::Arc; -use std::time::{Duration, Instant}; - -use super::client::{build_kv_client, KvClientTunables}; -use crate::commands::bench::verb::PrepareArgs; -use crate::Cli; - -pub async fn run(cli: &Cli, args: PrepareArgs) -> ExitCode { - let store_id = args.store; - let group_id = args.group; - let client = match build_kv_client( - cli, - crowdb_kv_client::ReadEndpointPolicy::Leader, - &KvClientTunables::default(), - ) - .await - { - Ok(c) => Arc::new(c), - Err(c) => return c, - }; - - let value_size = args.value_size.max(1); - - let next_id = Arc::new(AtomicU64::new(0)); - let ok_count = Arc::new(AtomicU64::new(0)); - let err_count = Arc::new(AtomicU64::new(0)); - let total = args.keys; - - let concurrency = args.concurrency.max(1); - let start = Instant::now(); - - let mut handles = Vec::with_capacity(concurrency); - for _ in 0..concurrency { - let client = Arc::clone(&client); - let next_id = Arc::clone(&next_id); - let ok_count = Arc::clone(&ok_count); - let err_count = Arc::clone(&err_count); - handles.push(tokio::spawn(async move { - loop { - let id = next_id.fetch_add(1, Ordering::Relaxed); - if id >= total { - break; - } - let key = format!("k{id:020}"); - let value = build_value(id, value_size); - // The client's put() already retries 4 times, but the - // leader may still be stabilizing. Add one extra retry - // with a short backoff for the first few keys. - let mut succeeded = false; - let mut last_err = None; - for attempt in 0..3u32 { - match client.put(store_id, group_id, key.as_bytes(), &value, None).await { - Ok(_) => { - ok_count.fetch_add(1, Ordering::Relaxed); - succeeded = true; - break; - } - Err(e) => { - last_err = Some(e); - if attempt < 2 { - tokio::time::sleep(Duration::from_millis(50 * u64::from(attempt + 1))).await; - } - } - } - } - if !succeeded { - err_count.fetch_add(1, Ordering::Relaxed); - if err_count.load(Ordering::Relaxed) <= 3 { - if let Some(e) = &last_err { - eprintln!("prepare: put key={key} failed: {e}"); - } - } - } - } - })); - } - - for h in handles { - let _ = h.await; - } - - let elapsed = start.elapsed(); - let ok = ok_count.load(Ordering::Relaxed); - let err = err_count.load(Ordering::Relaxed); - eprintln!( - "bench kv prepare: {ok} keys written, {err} errors, {:.1}s", - elapsed.as_secs_f64() - ); - - if err > 0 { - ExitCode::from(2) - } else { - ExitCode::SUCCESS - } -} - -/// Build a deterministic value for key `id`: byte `i` = `(id + i) % 256`. -/// This pattern is reused by `bench kv read --verify-bytes` for -/// correctness verification. -fn build_value(id: u64, size: usize) -> Vec<u8> { - (0..size) - .map(|i| u8::try_from((id + i as u64) % 256).unwrap_or(0)) - .collect() -} diff --git a/app/crowdb-cli/src/commands/bench/kv/read.rs b/app/crowdb-cli/src/commands/bench/kv/read.rs deleted file mode 100644 index 169987c36..000000000 --- a/app/crowdb-cli/src/commands/bench/kv/read.rs +++ /dev/null @@ -1,164 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -//! `bench kv read` — point-get workload against store 0 / group 0. -//! -//! Spawns `--loader-num` tokio tasks, each looping random `get` calls -//! for `--duration-secs`. Supports `--read-mode` (linearizable|minslot), -//! `--min-slot` (auto|zero), `--read-endpoint-policy` (leader|any-replica), -//! and `--verify-bytes` correctness checking. - -use std::process::ExitCode; -use std::sync::Arc; -use std::time::{Duration, Instant}; - -use crowdb_kv_client::{CrowdbKvClient, GetOutcome, ReadEndpointPolicy, ReadMode}; -use rand::rngs::SmallRng; -use rand::{Rng, SeedableRng}; - -use super::client::{build_kv_client, KvClientTunables}; -use crate::commands::bench::loader::{run_workload, BenchRecorder}; -use crate::commands::bench::metrics::BenchMetrics; -use crate::commands::bench::result::{BenchOps, BenchResult}; -use crate::commands::bench::verb::{BenchMinSlot, BenchReadEndpoint, BenchReadMode, ReadArgs}; -use crate::Cli; - -pub async fn run(cli: &Cli, args: ReadArgs) -> ExitCode { - let store_id = args.store; - let group_id = args.group; - let read_mode = match args.read_mode { - BenchReadMode::Linearizable => ReadMode::Linearizable, - BenchReadMode::Minslot => ReadMode::MinSlot, - }; - let endpoint_policy = match args.read_endpoint_policy { - BenchReadEndpoint::Leader => ReadEndpointPolicy::Leader, - BenchReadEndpoint::AnyReplica => ReadEndpointPolicy::AnyReplica, - }; - - let client = match build_kv_client( - cli, - endpoint_policy, - &KvClientTunables { - pool_size: args.connections, - ..Default::default() - }, - ) - .await - { - Ok(c) => Arc::new(c), - Err(c) => return c, - }; - - let key_space = args.key_space.max(1); - let verify_bytes = args.verify_bytes; - let value_size = args.value_size; - let min_slot_arg = args.min_slot; - - let mut metrics = BenchMetrics::new(&cli.log_dir, args.metrics_interval); - metrics.start(); - - let duration = Duration::from_secs(args.duration_secs); - let start = Instant::now(); - let recorder = Arc::clone(&metrics.recorder); - let run = run_workload( - recorder, - args.loader_num, - duration, - make_read_workload( - client, - key_space, - verify_bytes, - value_size, - read_mode, - min_slot_arg, - store_id, - group_id, - ), - || {}, - ) - .await; - - let duration_ms: u64 = start.elapsed().as_millis().try_into().unwrap_or(u64::MAX); - metrics.stop().await; - - let snap = run.recorder.hist_snapshot(); - eprintln!( - "bench kv read: {} ops, {} errors, {} corr errors, {}ms", - run.recorder.ops(), - run.recorder.errors(), - run.recorder.correctness_errors(), - duration_ms, - ); - - let result = BenchResult { - total_ops: run.recorder.ops(), - duration_ms, - total_errors: run.recorder.errors(), - correctness_errors: Some(run.recorder.correctness_errors()), - by_op: BenchOps::default().read(snap), - client_transport_stats: None, - server_metrics: None, - }; - let json = serde_json::to_value(&result).unwrap_or_default(); - tracing::info!(report = %json, "bench_report"); - result.finish(&cli.log_dir) -} - -#[allow(clippy::too_many_arguments)] -fn make_read_workload( - client: Arc<CrowdbKvClient>, - key_space: u64, - verify_bytes: usize, - value_size: usize, - read_mode: ReadMode, - min_slot_arg: BenchMinSlot, - store_id: u64, - group_id: u64, -) -> impl Fn(Arc<BenchRecorder>) -> std::pin::Pin<Box<dyn std::future::Future<Output = ()> + Send>> { - move |rec: Arc<BenchRecorder>| { - let client = Arc::clone(&client); - Box::pin(async move { - let mut rng = SmallRng::from_entropy(); - let id = rng.gen_range(0..key_space); - let key = format!("k{id:020}"); - let min_slot = match min_slot_arg { - BenchMinSlot::Auto => None, - BenchMinSlot::Zero => Some(0), - }; - let t0 = Instant::now(); - match client - .get(store_id, group_id, key.as_bytes(), read_mode, min_slot) - .await - { - Ok(GetOutcome::Found { value, .. }) => { - rec.record_ok( - t0.elapsed().as_micros().try_into().unwrap_or(u64::MAX), - value.len() as u64, - ); - if verify_bytes > 0 { - let expected = expected_value(id, value_size, verify_bytes); - if value.len() < verify_bytes || value[..verify_bytes] != expected[..] { - rec.record_correctness_err(); - } - } - } - Ok(GetOutcome::NotFound) => { - rec.record_err(); - if verify_bytes > 0 { - rec.record_correctness_err(); - } - } - Err(_) => rec.record_err(), - } - }) - } -} - -/// Build the expected first `verify` bytes for key `id`: byte `i` = -/// `(id + i) % 256` (same pattern as `super::prepare::build_value`). -fn expected_value(id: u64, size: usize, verify: usize) -> Vec<u8> { - let len = verify.min(size); - (0..len) - .map(|i| u8::try_from((id + i as u64) % 256).unwrap_or(0)) - .collect() -} diff --git a/app/crowdb-cli/src/commands/bench/kv/scan.rs b/app/crowdb-cli/src/commands/bench/kv/scan.rs deleted file mode 100644 index 3c43826f9..000000000 --- a/app/crowdb-cli/src/commands/bench/kv/scan.rs +++ /dev/null @@ -1,183 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -//! `bench kv scan` — list/range workload against store 0 / group 0. -//! -//! Spawns `--loader-num` tokio tasks, each looping scan calls for -//! `--duration-secs`. Supports `--scan-limit`, `--scan-prefix`, -//! `--scan-start-after`, `--read-mode`, `--min-slot`, -//! `--read-endpoint-policy`, and `--value-size-mix`. - -use std::process::ExitCode; -use std::sync::Arc; -use std::time::{Duration, Instant}; - -use crowdb_kv_client::{ReadEndpointPolicy, ReadMode}; - -use super::client::{build_kv_client, KvClientTunables}; -use crate::commands::bench::loader::{run_workload, BenchRecorder}; -use crate::commands::bench::metrics::BenchMetrics; -use crate::commands::bench::result::{BenchOps, BenchResult}; -use crate::commands::bench::verb::{ - BenchMinSlot, BenchReadEndpoint, BenchReadMode, BenchScanDirection, ScanArgs, -}; -use crate::Cli; - -#[allow(clippy::too_many_lines)] -pub async fn run(cli: &Cli, args: ScanArgs) -> ExitCode { - let store_id = args.store; - let group_id = args.group; - let read_mode = match args.read_mode { - BenchReadMode::Linearizable => ReadMode::Linearizable, - BenchReadMode::Minslot => ReadMode::MinSlot, - }; - let endpoint_policy = match args.read_endpoint_policy { - BenchReadEndpoint::Leader => ReadEndpointPolicy::Leader, - BenchReadEndpoint::AnyReplica => ReadEndpointPolicy::AnyReplica, - }; - - let client = match build_kv_client( - cli, - endpoint_policy, - &KvClientTunables { - pool_size: args.connections, - ..Default::default() - }, - ) - .await - { - Ok(c) => Arc::new(c), - Err(c) => return c, - }; - - let prefix = args.scan_prefix.as_bytes().to_vec(); - let start_after = args.scan_start_after.as_bytes().to_vec(); - let limit = args.scan_limit; - let min_slot_arg = args.min_slot; - let value_size = args.value_size; - let value_size_mix = parse_value_size_mix(args.value_size_mix.as_deref()); - let direction = args.scan_direction; - - let mut metrics = BenchMetrics::new(&cli.log_dir, args.metrics_interval); - metrics.start(); - - let duration = Duration::from_secs(args.duration_secs); - let start = Instant::now(); - let recorder = Arc::clone(&metrics.recorder); - let run = run_workload( - recorder, - args.loader_num, - duration, - move |rec: Arc<BenchRecorder>| { - let client = Arc::clone(&client); - let prefix = prefix.clone(); - let start_after = start_after.clone(); - let limit = limit; - let read_mode = read_mode; - let min_slot_arg = min_slot_arg; - let value_size = value_size; - let value_size_mix = value_size_mix.clone(); - let direction = direction; - async move { - let min_slot = match min_slot_arg { - BenchMinSlot::Auto => None, - BenchMinSlot::Zero => Some(0), - }; - let t0 = Instant::now(); - let scan = match direction { - BenchScanDirection::Forward => { - client - .scan( - store_id, - group_id, - &prefix, - &start_after, - &[], - limit, - read_mode, - min_slot, - false, - None, - ) - .await - } - BenchScanDirection::Reverse => { - client - .scan_reverse( - store_id, - group_id, - &prefix, - &start_after, - &[], - limit, - read_mode, - min_slot, - false, - None, - ) - .await - } - }; - match scan { - Ok(outcome) => { - // Touch value bytes to prevent lazy-load optimization - // skewing latency. Both --value-size and --value-size-mix - // paths touch the same bytes. - let total_bytes: u64 = outcome.items.iter().map(|(_, v)| v.len() as u64).sum(); - rec.record_ok( - t0.elapsed().as_micros().try_into().unwrap_or(u64::MAX), - total_bytes, - ); - let _ = outcome.items.len(); - if value_size > 0 || !value_size_mix.is_empty() { - for (_, v) in &outcome.items { - let _ = v.len(); - } - } - } - Err(_) => rec.record_err(), - } - } - }, - || {}, - ) - .await; - - let duration_ms: u64 = start.elapsed().as_millis().try_into().unwrap_or(u64::MAX); - metrics.stop().await; - - let snap = run.recorder.hist_snapshot(); - eprintln!( - "bench kv scan: {} ops, {} errors, {}ms", - run.recorder.ops(), - run.recorder.errors(), - duration_ms, - ); - - let result = BenchResult { - total_ops: run.recorder.ops(), - duration_ms, - total_errors: run.recorder.errors(), - correctness_errors: Some(run.recorder.correctness_errors()), - by_op: BenchOps::default().list(snap), - client_transport_stats: None, - server_metrics: None, - }; - let json = serde_json::to_value(&result).unwrap_or_default(); - tracing::info!(report = %json, "bench_report"); - result.finish(&cli.log_dir) -} - -/// Parse a `--value-size-mix` string like `64:70,1024:20,16384:10` -/// into a list of `(size, percent)` pairs. Empty/None → empty vec. -fn parse_value_size_mix(s: Option<&str>) -> Vec<(usize, u8)> { - let Some(s) = s else { return Vec::new() }; - s.split(',') - .filter_map(|pair| { - let mut parts = pair.split(':'); - let size = parts.next()?.trim().parse::<usize>().ok()?; - let pct = parts.next()?.trim().parse::<u8>().ok()?; - Some((size, pct)) - }) - .collect() -} diff --git a/app/crowdb-cli/src/commands/bench/kv/write.rs b/app/crowdb-cli/src/commands/bench/kv/write.rs deleted file mode 100644 index 58ff06a39..000000000 --- a/app/crowdb-cli/src/commands/bench/kv/write.rs +++ /dev/null @@ -1,460 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -//! `bench kv write` — put workload against store 0 / group 0. -//! -//! Spawns `--loader-num` tokio tasks, each looping random `put` calls -//! for `--duration-secs`. After the workload, fetches server-side -//! metrics from every node's `/metrics` endpoint and aggregates them -//! into the `server_metrics` JSON section. - -use std::process::ExitCode; -use std::sync::Arc; -use std::time::{Duration, Instant}; - -use crowdb_console_shared::clients::http::ServerClient; -use crowdb_console_shared::snapshot::MetricFieldView; -use rand::rngs::SmallRng; -use rand::{Rng, SeedableRng}; - -use super::client::{build_kv_client, KvClientTunables}; -use crate::commands::authority_context; -use crate::commands::bench::loader::{run_workload, BenchRecorder}; -use crate::commands::bench::metrics::BenchMetrics; -use crate::commands::bench::result::{ - BenchOps, BenchResult, ReplicaStats, ServerMetrics, ServerRpcLatency, SnapshotStats, TransportStats, -}; -use crate::commands::bench::verb::WriteArgs; -use crate::Cli; - -#[allow(clippy::too_many_lines)] -pub async fn run(cli: &Cli, args: WriteArgs) -> ExitCode { - let store_id = args.store; - let group_id = args.group; - let client = match build_kv_client( - cli, - crowdb_kv_client::ReadEndpointPolicy::Leader, - &KvClientTunables { - event_write: args.event_write, - rpc_workers: args.rpc_workers, - send_queue_capacity: if args.send_queue_capacity > 0 { - args.send_queue_capacity - } else { - 4096 - }, - pool_size: args.connections, - ..Default::default() - }, - ) - .await - { - Ok(c) => Arc::new(c), - Err(c) => return c, - }; - - let key_space = args.key_space.max(1); - let value_size = args.value_size; - - let mut metrics = BenchMetrics::new(&cli.log_dir, args.metrics_interval); - metrics.start(); - - let server_metrics_baseline = fetch_server_metrics(cli, store_id, group_id).await; - - let duration = Duration::from_secs(args.duration_secs); - tracing::info!( - duration_secs = duration.as_secs(), - loaders = args.loader_num, - connections = args.connections, - key_space, - value_size, - event_write = args.event_write, - rpc_workers = args.rpc_workers, - send_queue_capacity = if args.send_queue_capacity > 0 { - args.send_queue_capacity - } else { - 4096 - }, - metrics_interval_secs = args.metrics_interval, - "bench kv write: starting" - ); - let start = Instant::now(); - let recorder = Arc::clone(&metrics.recorder); - let stats_client = Arc::clone(&client); - let dump_client = Arc::clone(&client); - let run = run_workload( - recorder, - args.loader_num, - duration, - move |rec: Arc<BenchRecorder>| { - let client = Arc::clone(&client); - let key_space = key_space; - let value_size = value_size; - async move { - let mut rng = SmallRng::from_entropy(); - let id = rng.gen_range(0..key_space); - let key = format!("k{id:020}"); - let value = build_value(id, value_size); - let t0 = Instant::now(); - match client.put(store_id, group_id, key.as_bytes(), &value, None).await { - Ok(_) => rec.record_ok( - t0.elapsed().as_micros().try_into().unwrap_or(u64::MAX), - value.len() as u64, - ), - Err(_) => rec.record_err(), - } - } - }, - move || { - let next_id = dump_client.next_req_id(); - tracing::info!( - next_req_id = next_id, - "bench: dumping pending RPC requests at stop time" - ); - dump_client.dump_pending_requests(); - }, - ) - .await; - - let duration_ms: u64 = start.elapsed().as_millis().try_into().unwrap_or(u64::MAX); - tracing::info!( - duration_ms, - ops = run.recorder.ops(), - errors = run.recorder.errors(), - "bench kv write: workload complete" - ); - - metrics.stop().await; - - // Fetch server-side metrics from every node. - tracing::info!("bench kv write: fetching server metrics"); - let server_metrics = fetch_server_metrics(cli, store_id, group_id) - .await - .map(|current| match server_metrics_baseline.as_ref() { - Some(baseline) => current.delta_since(baseline), - None => current, - }); - tracing::info!( - elapsed_ms = start.elapsed().as_millis(), - "bench kv write: server metrics fetched" - ); - - let snap = run.recorder.hist_snapshot(); - eprintln!( - "bench kv write: {} ops, {} errors, {}ms", - run.recorder.ops(), - run.recorder.errors(), - duration_ms, - ); - - // Client-side crowdb-rpc transport stats (submit_to_writev latency). - let client_transport_stats = stats_client.transport_stats().map(|s| TransportStats { - submit_to_writev_avg_us: (s.submit_to_writev.sum_ns / 1000) - .checked_div(s.submit_to_writev.count) - .unwrap_or(0), - ..Default::default() - }); - - let result = BenchResult { - total_ops: run.recorder.ops(), - duration_ms, - total_errors: run.recorder.errors(), - correctness_errors: Some(run.recorder.correctness_errors()), - by_op: BenchOps::default().write(snap), - client_transport_stats, - server_metrics, - }; - metrics.write_report(&result); - result.finish(&cli.log_dir) -} - -/// Build a deterministic value for key `id`: byte `i` = `(id + i) % 256`. -fn build_value(id: u64, size: usize) -> Vec<u8> { - (0..size) - .map(|i| u8::try_from((id + i as u64) % 256).unwrap_or(0)) - .collect() -} - -/// Fetch `/metrics` from confirmed replica hosts and aggregate into -/// `ServerMetrics`. Metrics are summed across nodes except for averages -/// (which are averaged across nodes that report them). -#[allow(clippy::too_many_lines)] -async fn fetch_server_metrics(cli: &Cli, store_id: u64, group_id: u64) -> Option<ServerMetrics> { - let ctx = authority_context(cli).await.ok()?; - let replicas = ctx - .sysmd() - .list_replicas_in_group(store_id, group_id) - .await - .ok()?; - let group_prefix = format!("s.{store_id}.g.{group_id}."); - let store_prefix = format!("s.{store_id}.rpc."); - - let mut mgmt_urls = Vec::with_capacity(replicas.len()); - for replica in replicas { - mgmt_urls.push(ctx.live_node_mgmt_url(replica.node_id).await.ok()?); - } - mgmt_urls.sort(); - mgmt_urls.dedup(); - if mgmt_urls.is_empty() { - return None; - } - - let mut wal_append_count = 0u64; - let mut inflight_enqueued = 0u64; - let mut inflight_wait_sum = 0u64; - let mut inflight_wait_count = 0u64; - let mut election_count = 0u64; - let mut snapshot_completed = 0u64; - let mut snapshot_failed = 0u64; - let mut snapshot_max_latency_us = 0u64; - let mut rpc_s2w_sum = 0u64; - let mut rpc_s2w_count = 0u64; - // Server-side per-op RPC latency accumulators (averaged across nodes). - let mut rpc_lat = RpcLatencyAcc::default(); - // Replica RPC: collect all `rpc.l@<peer>` summaries across nodes. - // The leader sends accept rounds to peers; followers only send - // fetchgap/chosen_notice. We aggregate by peer id and report the - // two peers with the highest `total` (round-trips) as r2/r3. - let mut peer_totals: Vec<(u64, u64, u64)> = Vec::new(); // (peer_id, avg_us, total) - - for url in &mgmt_urls { - let Ok(sc) = ServerClient::new(url.clone()) else { - continue; - }; - // Group-level metrics: WAL, inflight, replica RPC. - if let Ok(resp) = sc.metrics(&group_prefix).await { - for point in &resp.metrics { - let fields = field_map(&point.fields); - match point.name.as_str() { - // WAL append summary: `total` = cumulative count. - n if n.ends_with(".wal.mem.append.l") - || n.ends_with(".wal.file.append.l") - || n.ends_with(".wal.uring.append.l") => - { - wal_append_count += get_u64(&fields, "total"); - } - n if n.ends_with(".paxos.elections.c") => { - election_count += get_u64(&fields, "total"); - } - n if n.ends_with(".maintenance.snapshot.success.c") => { - snapshot_completed += get_u64(&fields, "total"); - } - n if n.ends_with(".maintenance.snapshot.failure.c") => { - snapshot_failed += get_u64(&fields, "total"); - } - n if n.ends_with(".maintenance.snapshot.max_us.g") => { - snapshot_max_latency_us = snapshot_max_latency_us.max(get_u64(&fields, "value")); - } - n if n.ends_with(".write.inflight_enqueued.c") => { - inflight_enqueued += get_u64(&fields, "total"); - } - n if n.ends_with(".write.inflight_wait.l") => { - if get_u64(&fields, "count") > 0 { - inflight_wait_sum += get_u64(&fields, "avg_ns") / 1000; - inflight_wait_count += 1; - } - } - // Replica RPC latency: s.0.g.0.rpc.l@<peer_id> - n if n.contains(".rpc.l@") => { - let peer_id = n - .rsplit('@') - .next() - .and_then(|s| s.parse::<u64>().ok()) - .unwrap_or(0); - let avg_us = get_u64(&fields, "avg_ns") / 1000; - let total = get_u64(&fields, "total"); - peer_totals.push((peer_id, avg_us, total)); - } - _ => {} - } - } - } - // Store-level RPC transport stats. - if let Ok(resp) = sc.metrics(&store_prefix).await { - for point in &resp.metrics { - let fields = field_map(&point.fields); - match point.name.as_str() { - n if n.ends_with(".rpc.submit_to_writev.avg_us.g") => { - rpc_s2w_sum += get_u64(&fields, "value"); - rpc_s2w_count += 1; - } - n if n.ends_with(".rpc.put.lh") => { - rpc_lat.put.add(&fields); - } - n if n.ends_with(".rpc.get.lh") => { - rpc_lat.get.add(&fields); - } - n if n.ends_with(".rpc.scan.lh") => { - rpc_lat.scan.add(&fields); - } - n if n.ends_with(".rpc.delete.lh") => { - rpc_lat.delete.add(&fields); - } - _ => {} - } - } - } - } - - Some(build_server_metrics( - wal_append_count, - inflight_enqueued, - inflight_wait_sum, - inflight_wait_count, - rpc_s2w_sum, - rpc_s2w_count, - &rpc_lat, - peer_totals, - election_count, - SnapshotStats { - completed: snapshot_completed, - failed: snapshot_failed, - max_latency_us: snapshot_max_latency_us, - }, - )) -} - -#[allow(clippy::too_many_arguments)] -fn build_server_metrics( - wal_append_count: u64, - inflight_enqueued: u64, - inflight_wait_sum: u64, - inflight_wait_count: u64, - rpc_s2w_sum: u64, - rpc_s2w_count: u64, - rpc_lat: &RpcLatencyAcc, - mut peer_totals: Vec<(u64, u64, u64)>, - election_count: u64, - snapshot: SnapshotStats, -) -> ServerMetrics { - // Aggregate peer totals by peer_id (sum across nodes), then pick - // the top 2 by total round-trips as r2/r3. - peer_totals.sort_by_key(|&(_, _, total)| std::cmp::Reverse(total)); - let (r2_avg, r2_tps) = peer_totals - .first() - .map_or((0, 0), |(_, avg, total)| (*avg, *total)); - let (r3_avg, r3_tps) = peer_totals - .get(1) - .map_or((0, 0), |(_, avg, total)| (*avg, *total)); - - let inflight_wait_avg_us = inflight_wait_sum.checked_div(inflight_wait_count).unwrap_or(0); - let submit_to_writev_avg_us = rpc_s2w_sum.checked_div(rpc_s2w_count).unwrap_or(0); - - // RPC transport stats (writev_calls, frames_sent, etc.) are not - // exposed via the metrics registry — they live in the crowdb-rpc - // FFI transport and are not yet surfaced to the management API. - // Reported as 0; the regression script tolerates this via jq `// 0`. - ServerMetrics { - wal_append_count, - rpc: TransportStats { - writev_calls: 0, - frames_sent: 0, - read_calls: 0, - frames_parsed: 0, - submit_to_writev_avg_us, - }, - replica: ReplicaStats { - r2: r2_avg, - r2_tps, - r3: r3_avg, - r3_tps, - }, - inflight_enqueued, - inflight_wait_avg_us, - election_count, - snapshot, - rpc_latency: rpc_lat.finalize(), - } -} - -fn field_map(fields: &[MetricFieldView]) -> std::collections::HashMap<&str, f64> { - fields.iter().map(|f| (f.key.as_str(), f.value)).collect() -} - -fn get_u64(map: &std::collections::HashMap<&str, f64>, key: &str) -> u64 { - // Metric values are non-negative counts/latencies; the filter - // guards against NaN/negative before the truncating cast. - map.get(key) - .filter(|v| v.is_finite() && **v >= 0.0) - .map_or(0, |v| { - #[allow(clippy::cast_possible_truncation, clippy::cast_sign_loss)] - { - *v as u64 - } - }) -} - -fn get_f64(map: &std::collections::HashMap<&str, f64>, key: &str) -> f64 { - // avg_ns is now a true f64; preserve fractional precision. The - // filter guards against NaN/negative. - map.get(key) - .filter(|v| v.is_finite() && **v >= 0.0) - .map_or(0.0, |v| *v) -} - -/// Accumulator for one op-type's RPC latency across nodes. Histograms -/// report `avg_ns`/`p50_ns`/`p99_ns`; we sum and divide by node count. -#[derive(Default)] -struct RpcLatencyAccOp { - avg_ns: f64, - p50_ns: u64, - p99_ns: u64, - count: u64, -} - -impl RpcLatencyAccOp { - fn add(&mut self, fields: &std::collections::HashMap<&str, f64>) { - if get_u64(fields, "count") == 0 { - return; - } - self.avg_ns += get_f64(fields, "avg_ns"); - self.p50_ns += get_u64(fields, "p50_ns"); - self.p99_ns += get_u64(fields, "p99_ns"); - self.count += 1; - } - fn finalize_us(&self) -> (f64, u64, u64) { - let n = self.count.max(1); - ( - #[allow(clippy::cast_precision_loss)] - { - self.avg_ns / n as f64 / 1000.0 - }, - self.p50_ns / n / 1000, - self.p99_ns / n / 1000, - ) - } -} - -/// Accumulator for all op-type RPC latencies across nodes. -#[derive(Default)] -struct RpcLatencyAcc { - put: RpcLatencyAccOp, - get: RpcLatencyAccOp, - scan: RpcLatencyAccOp, - delete: RpcLatencyAccOp, -} - -impl RpcLatencyAcc { - fn finalize(&self) -> Option<ServerRpcLatency> { - if self.put.count == 0 && self.get.count == 0 && self.scan.count == 0 && self.delete.count == 0 { - return None; - } - let (put_avg, put_p50, put_p99) = self.put.finalize_us(); - let (get_avg, get_p50, get_p99) = self.get.finalize_us(); - let (scan_avg, scan_p50, scan_p99) = self.scan.finalize_us(); - let (delete_avg, delete_p50, delete_p99) = self.delete.finalize_us(); - Some(ServerRpcLatency { - put_avg_us: put_avg, - put_p50_us: put_p50, - put_p99_us: put_p99, - get_avg_us: get_avg, - get_p50_us: get_p50, - get_p99_us: get_p99, - scan_avg_us: scan_avg, - scan_p50_us: scan_p50, - scan_p99_us: scan_p99, - delete_avg_us: delete_avg, - delete_p50_us: delete_p50, - delete_p99_us: delete_p99, - }) - } -} diff --git a/app/crowdb-cli/src/commands/bench/loader.rs b/app/crowdb-cli/src/commands/bench/loader.rs index 1ad30f7db..442d7d878 100644 --- a/app/crowdb-cli/src/commands/bench/loader.rs +++ b/app/crowdb-cli/src/commands/bench/loader.rs @@ -1,30 +1,11 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -//! Shared loader-loop helper for tokio-based bench workloads. -//! -//! `run_workload` spawns `num_loaders` async tasks, each looping the -//! supplied `op` closure until a deadline, plus one timer task that -//! flips a shared `running` flag. The closure receives the shared -//! [`BenchRecorder`] and records each op's latency / errors itself. -//! The coroutine RPC path does not use this (the C++ `co_spawn` manages -//! its own loop); it records into a `BenchRecorder` directly from the -//! `on_response` callback. -//! -//! `BenchRecorder` wraps the project's `Counter` + `LatencyHistogram` + -//! `Bandwidth` handles from `crowdb_common::metrics`, all registered on -//! the `MetricsRunner`'s registry. The `LatencyHistogram` tracks both -//! window state (flushed/reset each interval for the periodic metrics -//! log) and cumulative state (never reset, for the final JSON report) -//! in a single `observe()` call — no double recording. Its `count` is -//! the success-ops count, so a separate success counter is not needed. +//! Metrics recorder retained by the chunk and disk workload adapters. -use std::future::Future; -use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::Arc; -use std::time::{Duration, Instant}; -use crowdb_common::metrics::{Bandwidth, Counter, HistogramSnapshot, LatencyHistogram, MetricsRegistry}; +use crowdb_common::metrics::{Bandwidth, Counter, LatencyHistogram, MetricsRegistry}; /// Shared error counters + latency histogram + bandwidth + running flag /// for one bench run. All fields are lock-free atomics — safe to update @@ -38,7 +19,6 @@ pub struct BenchRecorder { latency: Arc<LatencyHistogram>, /// Payload bandwidth (bytes/s, registered → flushed each interval). bytes: Arc<Bandwidth>, - running: AtomicBool, } impl BenchRecorder { @@ -51,7 +31,6 @@ impl BenchRecorder { correctness_errors: reg.register_counter("bench.ops.correctness_error.c"), latency: reg.register_histogram("bench.latency.lh"), bytes: reg.register_bandwidth("bench.bytes.bw"), - running: AtomicBool::new(true), } } @@ -76,119 +55,4 @@ impl BenchRecorder { pub fn record_correctness_err(&self) { self.correctness_errors.inc(); } - - /// Whether loader tasks should keep running. - pub fn is_running(&self) -> bool { - self.running.load(Ordering::Relaxed) - } - - /// Stop all loader tasks (set by the timer task at the deadline). - pub fn stop(&self) { - self.running.store(false, Ordering::Relaxed); - } - - /// Total successful ops (cumulative count from the latency histogram). - #[must_use] - pub fn ops(&self) -> u64 { - self.latency.snapshot_total().total_count - } - #[must_use] - pub fn errors(&self) -> u64 { - self.errors.snapshot().total - } - #[must_use] - #[allow(dead_code)] - pub fn correctness_errors(&self) -> u64 { - self.correctness_errors.snapshot().total - } - /// Cumulative latency snapshot (avg/p50/p99/max in ns) across all - /// observations — for the final JSON report. Window state may have - /// been reset by periodic flushes; cumulative state is preserved. - #[must_use] - pub fn hist_snapshot(&self) -> HistogramSnapshot { - self.latency.snapshot_total() - } -} - -/// The recorder + measured wall-clock duration of a workload run. -pub struct WorkloadRun { - pub recorder: Arc<BenchRecorder>, - #[allow(dead_code)] - pub duration_ms: u64, -} - -/// Spawn `num_loaders` tasks each looping `op` until `duration` elapses. -/// `op` performs exactly one operation and records its outcome into the -/// shared recorder. Returns the recorder + the measured run duration. -/// -/// `on_stop` is called once, immediately after the timer fires and -/// `running` is set to false, but before the loader tasks have exited — -/// so any in-flight requests are still pending. Use it to dump diagnostics. -/// -/// `num_loaders` is clamped to >= 1. -pub async fn run_workload<F, Fut, S>( - recorder: Arc<BenchRecorder>, - num_loaders: usize, - duration: Duration, - op: F, - on_stop: S, -) -> WorkloadRun -where - F: Fn(Arc<BenchRecorder>) -> Fut + Send + Sync + 'static, - Fut: Future<Output = ()> + Send + 'static, - S: FnOnce() + Send + 'static, -{ - let loaders = num_loaders.max(1); - let start = Instant::now(); - let op = Arc::new(op); - - let mut tasks = Vec::with_capacity(loaders); - for _ in 0..loaders { - let rec = Arc::clone(&recorder); - let op = Arc::clone(&op); - tasks.push(tokio::spawn(async move { - while rec.is_running() { - (*op)(Arc::clone(&rec)).await; - } - })); - } - - // Timer task: flip running=false at the deadline, then call on_stop - // while in-flight requests are still pending (before loaders exit). - let timer_rec = Arc::clone(&recorder); - let timer = tokio::spawn(async move { - tokio::time::sleep(duration).await; - let ops_at_deadline = timer_rec.ops(); - timer_rec.stop(); - tracing::info!( - deadline_secs = duration.as_secs(), - ops_at_deadline, - "bench: stop signaled — loaders draining in-flight ops" - ); - on_stop(); - }); - - for (i, t) in tasks.into_iter().enumerate() { - let _ = t.await; - if i == 0 { - tracing::info!( - task = i, - elapsed_ms = start.elapsed().as_millis(), - "bench: first loader exited" - ); - } - } - let elapsed_ms = start.elapsed().as_millis(); - tracing::info!( - elapsed_ms, - ops = recorder.ops(), - errors = recorder.errors(), - "bench: all loaders joined" - ); - let _ = timer.await; - - WorkloadRun { - recorder, - duration_ms: elapsed_ms.try_into().unwrap_or(u64::MAX), - } } diff --git a/app/crowdb-cli/src/commands/bench/metrics.rs b/app/crowdb-cli/src/commands/bench/metrics.rs index 03793ec3e..1fb1b3d2d 100644 --- a/app/crowdb-cli/src/commands/bench/metrics.rs +++ b/app/crowdb-cli/src/commands/bench/metrics.rs @@ -10,7 +10,7 @@ //! counters (e.g. `rpc.client.*`) are flushed into the same file via a //! `set_cpp_flush` callback that calls //! `crowdb_rpc_ffi::flush_cpp_global_metrics` — the same pattern -//! `crowdb-kv-server` uses. After the workload, the final JSON report +//! `crowdb-paxos-kv-server` uses. After the workload, the final JSON report //! is appended to the metrics log via `write_raw`. use std::path::Path; @@ -78,7 +78,7 @@ impl BenchMetrics { let mut runner = MetricsRunner::new(file, metrics_interval); // Merge C++ crowdb-rpc process-level metrics (rpc.client.*) into - // the same log file. Mirrors crowdb-kv-server's engine_collector. + // the same log file. Mirrors crowdb-paxos-kv-server's engine_collector. runner.set_cpp_flush(|writer, window_secs, timestamp, rust_width, count_w, tps_w| { let cpp_max = crowdb_rpc_ffi::cpp_global_metrics_max_name_len(); let shared_width = rust_width.max(cpp_max); @@ -119,15 +119,4 @@ impl BenchMetrics { r.stop().await; } } - - /// Write the final report to the metrics log file as readable text. - /// Falls back to `eprintln!` if the runner is disabled (no metrics log). - pub fn write_report(&self, result: &super::result::BenchResult) { - let text = result.to_string(); - if let Some(ref r) = self.runner { - r.write_raw(&text); - } else { - eprint!("{text}"); - } - } } diff --git a/app/crowdb-cli/src/commands/bench/paxos_kv.rs b/app/crowdb-cli/src/commands/bench/paxos_kv.rs new file mode 100644 index 000000000..a3442a64f --- /dev/null +++ b/app/crowdb-cli/src/commands/bench/paxos_kv.rs @@ -0,0 +1,13 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Read-only deployment discovery followed by host-generated KV load. +pub mod args; +mod client; +mod collect; +mod counters; +mod dataset; +mod deployment; +mod run; +mod verification; +pub use run::run; diff --git a/app/crowdb-cli/src/commands/bench/paxos_kv/args.rs b/app/crowdb-cli/src/commands/bench/paxos_kv/args.rs new file mode 100644 index 000000000..c6b8cdbbd --- /dev/null +++ b/app/crowdb-cli/src/commands/bench/paxos_kv/args.rs @@ -0,0 +1,127 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use clap::{Args, Subcommand, ValueEnum}; +use serde::Serialize; + +use super::super::workload::LoadOptions; + +#[derive(Args, Debug, Serialize)] +#[allow(clippy::struct_excessive_bools)] +pub struct PaxosKvArgs { + #[arg(long, default_value = "kv", global = true)] + pub label: String, + /// Management origins of all three running nodes; repeat three times. + #[arg(long = "endpoint", global = true, value_parser = endpoint)] + pub endpoints: Vec<String>, + /// Local running containers in the same order as endpoints (host network). + #[arg(long = "container", global = true, conflicts_with = "server_pids")] + pub containers: Vec<String>, + /// Native diagnostic: three existing server PIDs in separate cgroups. + #[arg(long = "server-pid", global = true, conflicts_with = "containers")] + pub server_pids: Vec<u32>, + /// Snapshot success > 0, failure = 0, elections = 0 over the complete load. + #[arg(long, global = true)] + pub require_snapshot: bool, + #[arg(long, global = true)] + pub expect_window: Option<u64>, + #[arg(long, global = true)] + pub expect_coalesce: Option<u64>, + /// 70% 64B, 20% 1KiB, 10% 16KiB, assigned deterministically by key ID. + #[arg(long, global = true)] + pub mixed_values: bool, + #[arg(long, default_value_t = 0, global = true)] + pub store: u64, + /// Existing workload group. Group 0 is reserved for control-plane data. + #[arg(long, default_value_t = 1, global = true)] + pub group: u64, + #[arg(long, default_value_t = 1_000_000, global = true)] + pub key_space: u64, + #[arg(long, default_value_t = 64, global = true)] + pub value_size: usize, + #[arg(long, default_value_t = 2 * 1024 * 1024 * 1024, global = true)] + pub max_dataset_bytes: u64, + /// Stop injecting requests at this percentage of any node's memory limit. + #[arg(long, default_value_t = 90, global = true, value_parser = clap::value_parser!(u8).range(1..=99))] + pub memory_guard_percent: u8, + /// Upper bound on concurrent materialized scan responses in the host CLI. + #[arg(long, default_value_t = 512 * 1024 * 1024, global = true)] + pub max_inflight_bytes: u64, + #[arg(long, default_value_t = 4, global = true)] + pub connections: usize, + #[arg(long, default_value_t = 2, global = true)] + pub rpc_workers: u32, + #[arg(long, default_value_t = false, global = true)] + pub event_write: bool, + #[arg(long, default_value_t = 4096, global = true)] + pub send_queue_capacity: u32, + #[arg(long, default_value_t = 1, global = true)] + pub seed: u64, + #[arg(long, value_enum, default_value_t = ReadMode::Linearizable, global = true)] + pub read_mode: ReadMode, + #[arg(long, value_enum, default_value_t = ReadEndpoint::Leader, global = true)] + pub read_endpoint_policy: ReadEndpoint, + #[arg(long, default_value_t = 100, global = true)] + pub scan_limit: u32, + #[arg(long, default_value = "", global = true)] + pub scan_start_after: String, + #[arg(long, default_value = "", global = true)] + pub scan_prefix: String, + #[arg(long, default_value_t = false, global = true)] + pub reverse: bool, + #[command(flatten)] + pub load: LoadOptions, + #[command(subcommand)] + pub operation: Operation, +} + +#[derive(Subcommand, Debug, Clone, Copy, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum Operation { + /// Populate and verify the bounded dataset; no performance verdict. + Prepare, + Read, + Write, + Scan, +} + +#[derive(ValueEnum, Debug, Clone, Copy, Serialize)] +pub enum ReadMode { + Linearizable, + Minslot, +} + +#[derive(ValueEnum, Debug, Clone, Copy, Serialize)] +pub enum ReadEndpoint { + Leader, + AnyReplica, +} + +fn endpoint(value: &str) -> Result<String, String> { + let url = reqwest::Url::parse(value).map_err(|error| error.to_string())?; + if !matches!(url.scheme(), "http" | "https") + || url.host_str().is_none() + || url.port() == Some(0) + || url.path() != "/" + || url.query().is_some() + || url.fragment().is_some() + || !url.username().is_empty() + || url.password().is_some() + { + return Err("endpoint must be an HTTP(S) management origin without credentials or path".into()); + } + Ok(value.trim_end_matches('/').to_owned()) +} + +impl PaxosKvArgs { + pub(super) fn dataset_bytes(&self) -> Option<u64> { + let values = if self.mixed_values { + let complete = (self.key_space / 10).checked_mul(7 * 64 + 2 * 1024 + 16384)?; + let rest = self.key_space % 10; + complete.checked_add(rest.min(7) * 64 + rest.saturating_sub(7) * 1024)? + } else { + self.key_space.checked_mul(self.value_size as u64)? + }; + values.checked_add(self.key_space.checked_mul(21)?) + } +} diff --git a/app/crowdb-cli/src/commands/bench/paxos_kv/client.rs b/app/crowdb-cli/src/commands/bench/paxos_kv/client.rs new file mode 100644 index 000000000..fbd580215 --- /dev/null +++ b/app/crowdb-cli/src/commands/bench/paxos_kv/client.rs @@ -0,0 +1,99 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use super::args::{PaxosKvArgs, ReadEndpoint}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient, ReadEndpointPolicy}; +use crowdb_protocol::mgmt::TopologyResponse; +use serde_json::Value; +use std::{ + collections::{BTreeMap, BTreeSet}, + time::Duration, +}; + +pub(super) async fn fetch(endpoint: &str, path: &str) -> Result<Value, String> { + let response = reqwest::Client::builder() + .no_proxy() + .timeout(Duration::from_secs(5)) + .build() + .map_err(|e| e.to_string())? + .get(format!("{endpoint}{path}")) + .send() + .await + .map_err(|e| format!("{endpoint}{path}: {e}"))? + .error_for_status() + .map_err(|e| e.to_string())?; + serde_json::from_slice(&response.bytes().await.map_err(|e| e.to_string())?).map_err(|e| e.to_string()) +} + +pub(super) async fn connect(args: &PaxosKvArgs) -> Result<(PaxosKvClient, Vec<Value>), String> { + if args.endpoints.len() != 3 || args.endpoints.iter().collect::<BTreeSet<_>>().len() != 3 { + return Err("provide exactly three distinct --endpoint management origins".into()); + } + let mut topologies = Vec::new(); + let mut replicas = BTreeMap::new(); + let mut memberships = Vec::new(); + let mut leaders = BTreeSet::new(); + for endpoint in &args.endpoints { + let raw = fetch(endpoint, "/topology").await?; + let topology: TopologyResponse = serde_json::from_value(raw.clone()).map_err(|e| e.to_string())?; + let store = topology + .stores + .iter() + .find(|s| s.store_id == args.store) + .ok_or("store absent on a node")?; + let group = store + .groups + .iter() + .find(|g| g.group_id == args.group) + .ok_or("group absent on a node")?; + let address = store.listen_addr.clone().ok_or("RPC listener absent")?; + let mut socket: std::net::SocketAddr = + address.parse().map_err(|e| format!("invalid RPC address: {e}"))?; + if socket.ip().is_unspecified() { + let origin = reqwest::Url::parse(endpoint).map_err(|e| e.to_string())?; + let ip = origin + .host_str() + .ok_or("missing host")? + .parse() + .map_err(|_| "use a numeric endpoint IP")?; + socket.set_ip(ip); + } + let address = socket.to_string(); + tokio::time::timeout(Duration::from_secs(3), tokio::net::TcpStream::connect(socket)) + .await + .map_err(|_| format!("RPC endpoint unreachable: {socket}"))? + .map_err(|e| e.to_string())?; + if replicas.insert(group.local_replica_id, address.clone()).is_some() { + return Err("duplicate replica identity".into()); + } + let mut membership = BTreeMap::from([(group.local_replica_id, address)]); + for remote in &group.remotes { + if !remote.voting || membership.insert(remote.id, remote.endpoint.clone()).is_some() { + return Err("expected three distinct voting replicas".into()); + } + } + memberships.push(membership); + leaders.insert(group.leader_id); + topologies.push(raw); + } + if memberships.iter().any(|members| members != &replicas) || leaders.len() != 1 { + return Err("nodes disagree on replica membership or leader; wait for deployment readiness".into()); + } + let leader = leaders + .first() + .and_then(|id| replicas.get(id)) + .ok_or("no elected leader")?; + let mut config = ClientConfig::new(args.endpoints.clone()); + config.pool_size_per_endpoint = args.connections; + config.rpc_workers = args.rpc_workers; + config.event_write = args.event_write; + config.send_queue_capacity = args.send_queue_capacity; + config.read_endpoint_policy = match args.read_endpoint_policy { + ReadEndpoint::Leader => ReadEndpointPolicy::Leader, + ReadEndpoint::AnyReplica => ReadEndpointPolicy::AnyReplica, + }; + let client = PaxosKvClient::new(config); + client.refresh_topology().await.map_err(|e| e.to_string())?; + client.seed_leader(args.store, args.group, leader.clone()); + Ok((client, topologies)) +} diff --git a/app/crowdb-cli/src/commands/bench/paxos_kv/collect.rs b/app/crowdb-cli/src/commands/bench/paxos_kv/collect.rs new file mode 100644 index 000000000..6e009a74a --- /dev/null +++ b/app/crowdb-cli/src/commands/bench/paxos_kv/collect.rs @@ -0,0 +1,17 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use super::{args::PaxosKvArgs, client::fetch, deployment::ServerProcess}; +use serde_json::{json, Value}; + +pub(super) async fn sample(args: &PaxosKvArgs, nodes: &[ServerProcess]) -> Result<Value, String> { + let start = chrono::Utc::now(); + let metrics = futures::future::try_join_all(args.endpoints.iter().map(|e| fetch(e, "/metrics"))).await?; + let resources: Vec<_> = nodes + .iter() + .map(ServerProcess::sample) + .collect::<Result<_, _>>()?; + Ok(json!({"started_at":start,"finished_at":chrono::Utc::now(),"metrics":metrics,"resources":resources})) +} + +pub(super) use super::counters::{deltas, sentinel}; diff --git a/app/crowdb-cli/src/commands/bench/paxos_kv/counters.rs b/app/crowdb-cli/src/commands/bench/paxos_kv/counters.rs new file mode 100644 index 000000000..b122030c8 --- /dev/null +++ b/app/crowdb-cli/src/commands/bench/paxos_kv/counters.rs @@ -0,0 +1,105 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +use serde_json::{json, Value}; + +pub fn counter(node: &Value, name: &str) -> Result<f64, String> { + node["metrics"] + .as_array() + .ok_or("metrics response has no points")? + .iter() + .find(|v| v["name"] == name) + .and_then(|v| v["fields"].as_array()) + .and_then(|f| f.iter().find(|v| v["key"] == "total")) + .and_then(|v| v["value"].as_f64()) + .ok_or_else(|| format!("missing required cumulative counter {name}")) +} + +pub fn sentinel(store: u64, group: u64, before: &Value, after: &Value) -> Result<Value, String> { + let mut sums = [0.0; 3]; + for (i, suffix) in [ + "paxos.elections.c", + "maintenance.snapshot.success.c", + "maintenance.snapshot.failure.c", + ] + .iter() + .enumerate() + { + let name = format!("s.{store}.g.{group}.{suffix}"); + for node in 0..3 { + let delta = counter(&after["metrics"][node], &name)? - counter(&before["metrics"][node], &name)?; + if delta < 0.0 { + return Err(format!("counter reset: {name}")); + } + sums[i] += delta; + } + } + Ok( + json!({"elections":sums[0],"snapshot_completed":sums[1],"snapshot_failed":sums[2], + "passed":sums[0]==0.0 && sums[1]>0.0 && sums[2]==0.0, + "window":"entire load including warmup, tail and drain; preparation excluded"}), + ) +} + +fn stat(sample: &Value, file: &str, key: &str) -> Result<u64, String> { + sample[file] + .as_str() + .and_then(|s| { + s.lines().find_map(|line| { + let mut fields = line.split_whitespace(); + (fields.next() == Some(key)) + .then(|| fields.next()?.parse().ok()) + .flatten() + }) + }) + .ok_or_else(|| format!("missing cgroup {file}/{key}")) +} + +#[allow(clippy::cast_precision_loss)] +pub fn deltas(before: &Value, after: &Value) -> Result<Value, String> { + let mut nodes = Vec::new(); + for index in 0..3 { + let mut counters = serde_json::Map::new(); + let points = before["metrics"][index]["metrics"] + .as_array() + .ok_or("metrics missing")?; + for point in points { + let Some(name) = point["name"] + .as_str() + .filter(|s| s.rsplit('.').next() == Some("c")) + else { + continue; + }; + let delta = counter(&after["metrics"][index], name)? - counter(&before["metrics"][index], name)?; + if delta < 0.0 { + return Err(format!("counter reset: {name}")); + } + counters.insert(name.into(), json!(delta)); + } + let b = &before["resources"][index]; + let a = &after["resources"][index]; + let elapsed = (chrono::DateTime::parse_from_rfc3339(a["at"].as_str().ok_or("sample timestamp")?) + .map_err(|e| e.to_string())? + - chrono::DateTime::parse_from_rfc3339(b["at"].as_str().ok_or("sample timestamp")?) + .map_err(|e| e.to_string())?) + .num_microseconds() + .ok_or("invalid sample interval")?; + if elapsed <= 0 { + return Err("nonpositive resource interval".into()); + } + let delta = |file, key| { + stat(a, file, key)? + .checked_sub(stat(b, file, key)?) + .ok_or_else(|| format!("cgroup counter reset {file}/{key}")) + }; + let usage = delta("cpu_stat", "usage_usec")?; + let oom = delta("memory_events", "oom")?; + let oom_kill = delta("memory_events", "oom_kill")?; + if oom > 0 || oom_kill > 0 { + return Err("container memory exhaustion during load".into()); + } + nodes.push(json!({"counters":counters,"cpu_usage_usec":usage,"cpu_percent":usage as f64/elapsed as f64*100.0, + "cpu_throttled_usec":delta("cpu_stat","throttled_usec")?,"memory_current":a["memory_current"], + "memory_peak":a["memory_peak"],"memory_peak_scope":"container lifetime", "sample_elapsed_us":elapsed})); + } + Ok(json!(nodes)) +} diff --git a/app/crowdb-cli/src/commands/bench/paxos_kv/dataset.rs b/app/crowdb-cli/src/commands/bench/paxos_kv/dataset.rs new file mode 100644 index 000000000..2a928fb92 --- /dev/null +++ b/app/crowdb-cli/src/commands/bench/paxos_kv/dataset.rs @@ -0,0 +1,204 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use super::args::{Operation, PaxosKvArgs, ReadMode as Mode}; +use crowdb_paxos_kv_client::{GetOutcome, PaxosKvClient, ReadMode}; +use futures::{stream, StreamExt, TryStreamExt}; +use std::sync::Arc; + +pub(super) fn key(id: u64) -> String { + format!("k{id:020}") +} +pub(super) fn size(args: &PaxosKvArgs, id: u64) -> usize { + if args.mixed_values { + match id % 10 { + 0..=6 => 64, + 7..=8 => 1024, + _ => 16384, + } + } else { + args.value_size + } +} +fn value(args: &PaxosKvArgs, id: u64) -> Vec<u8> { + (0..size(args, id)) + .map(|offset| id.wrapping_add(offset as u64).to_le_bytes()[0]) + .collect() +} +fn verify_value(args: &PaxosKvArgs, id: u64, actual: &[u8]) -> Result<(), String> { + if actual.len() != size(args, id) + || actual + .iter() + .enumerate() + .any(|(offset, b)| *b != id.wrapping_add(offset as u64).to_le_bytes()[0]) + { + return Err(format!("incorrect bytes for key {}", key(id))); + } + Ok(()) +} + +pub(super) async fn prepare(client: Arc<PaxosKvClient>, args: Arc<PaxosKvArgs>) -> Result<(), String> { + stream::iter(0..args.key_space) + .map(|id| { + let client = client.clone(); + let args = args.clone(); + async move { + client + .put( + args.store, + args.group, + key(id).as_bytes(), + &value(&args, id), + None, + ) + .await + .map(|_| ()) + .map_err(|e| e.to_string()) + } + }) + .buffer_unordered(args.load.loader_num) + .try_collect::<Vec<_>>() + .await?; + verify(client, args).await +} +pub(super) async fn verify(client: Arc<PaxosKvClient>, args: Arc<PaxosKvArgs>) -> Result<(), String> { + stream::iter(0..args.key_space) + .map(|id| { + let client = client.clone(); + let args = args.clone(); + async move { + match client + .get( + args.store, + args.group, + key(id).as_bytes(), + ReadMode::Linearizable, + None, + ) + .await + .map_err(|e| format!("verification read {} failed: {e}", key(id)))? + { + GetOutcome::Found { value, .. } => verify_value(&args, id, &value), + GetOutcome::NotFound => Err(format!("missing key {}", key(id))), + } + } + }) + .buffer_unordered(args.load.loader_num.max(16)) + .try_collect::<Vec<_>>() + .await?; + Ok(()) +} + +pub(super) async fn operation(client: &PaxosKvClient, args: &PaxosKvArgs, id: u64) -> Result<u64, String> { + let mode = match args.read_mode { + Mode::Linearizable => ReadMode::Linearizable, + Mode::Minslot => ReadMode::MinSlot, + }; + match args.operation { + Operation::Write => { + client + .put(args.store, args.group, key(id).as_bytes(), &value(args, id), None) + .await + .map_err(|e| e.to_string())?; + Ok(size(args, id) as u64) + } + Operation::Read => match client + .get(args.store, args.group, key(id).as_bytes(), mode, Some(0)) + .await + .map_err(|e| e.to_string())? + { + GetOutcome::Found { value, .. } if value.len() == size(args, id) => Ok(value.len() as u64), + GetOutcome::Found { .. } => Err("incorrect value length".into()), + GetOutcome::NotFound => Err(format!("missing key {}", key(id))), + }, + Operation::Scan => scan(client, args, mode, false).await, + Operation::Prepare => Err("prepare cannot be timed".into()), + } +} + +pub(super) async fn scan( + client: &PaxosKvClient, + args: &PaxosKvArgs, + mode: ReadMode, + verify: bool, +) -> Result<u64, String> { + // Keep scans inside this bounded dataset even after a larger earlier run. + let upper = key(args.key_space); + let prefix = if args.scan_prefix.is_empty() { + "k" + } else { + &args.scan_prefix + }; + let start = if args.reverse && (args.scan_start_after.is_empty() || args.scan_start_after > upper) { + &upper + } else { + &args.scan_start_after + }; + let end = if args.reverse { "" } else { &upper }; + let result = if args.reverse { + client + .scan_reverse( + args.store, + args.group, + prefix.as_bytes(), + start.as_bytes(), + end.as_bytes(), + args.scan_limit, + mode, + Some(0), + false, + None, + ) + .await + } else { + client + .scan( + args.store, + args.group, + prefix.as_bytes(), + start.as_bytes(), + end.as_bytes(), + args.scan_limit, + mode, + Some(0), + false, + None, + ) + .await + } + .map_err(|e| e.to_string())?; + if result.items.is_empty() { + return Err("scan returned no items; choose a nonempty dataset range".into()); + } + if result.timed_out { + return Err("partial timed-out scan".into()); + } + if verify { + let mut ids: Vec<_> = (0..args.key_space) + .filter(|id| { + let k = key(*id); + k.starts_with(&args.scan_prefix) + && (args.scan_start_after.is_empty() + || if args.reverse { + k < args.scan_start_after + } else { + k > args.scan_start_after + }) + }) + .collect(); + if args.reverse { + ids.reverse(); + } + let expected_len = ids.len().min(args.scan_limit as usize); + if result.items.len() != expected_len || result.truncated != (ids.len() > expected_len) { + return Err("scan count/truncated mismatch".into()); + } + for ((k, v), id) in result.items.iter().zip(ids) { + if k.as_ref() != key(id).as_bytes() { + return Err("scan key ordering/bound mismatch".into()); + } + verify_value(args, id, v)?; + } + } + Ok(result.items.iter().map(|(k, v)| (k.len() + v.len()) as u64).sum()) +} diff --git a/app/crowdb-cli/src/commands/bench/paxos_kv/deployment.rs b/app/crowdb-cli/src/commands/bench/paxos_kv/deployment.rs new file mode 100644 index 000000000..eb202c5fd --- /dev/null +++ b/app/crowdb-cli/src/commands/bench/paxos_kv/deployment.rs @@ -0,0 +1,156 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Read-only deployment evidence and cgroup v2 resource accounting. +mod memory; +mod native; +use super::args::PaxosKvArgs; +use serde_json::{json, Value}; +use std::{collections::BTreeSet, path::PathBuf, time::Duration}; + +pub(super) struct ServerProcess { + pub evidence: Value, + cgroup: PathBuf, +} +impl ServerProcess { + pub fn sample(&self) -> Result<Value, String> { + let read = + |name| std::fs::read_to_string(self.cgroup.join(name)).map_err(|e| format!("cgroup {name}: {e}")); + Ok(json!({"at":chrono::Utc::now(),"cpu_stat":read("cpu.stat")?, + "memory_current":read("memory.current")?,"memory_peak":read("memory.peak").ok(), + "memory_events":read("memory.events")?,"cpu_pressure":read("cpu.pressure").ok(), + "memory_pressure":read("memory.pressure").ok()})) + } +} + +fn memory(node: &ServerProcess) -> Result<(Option<u64>, u64), String> { + let read = |file| -> Result<String, String> { + Ok(std::fs::read_to_string(node.cgroup.join(file)) + .map_err(|e| format!("cgroup {file}: {e}"))? + .trim() + .to_owned()) + }; + Ok(( + memory::parse_limit(&read("memory.max")?)?, + read("memory.current")? + .parse() + .map_err(|_| "invalid cgroup memory.current")?, + )) +} + +pub(super) fn validate_budget(args: &PaxosKvArgs, nodes: &[ServerProcess]) -> Result<(), String> { + let payload = args.dataset_bytes().ok_or("dataset size overflow")?; + for node in nodes { + let (limit, current) = memory(node)?; + let Some(limit) = limit else { continue }; + // Startup estimate for replicas' WAL, Tree and working buffers; + // the runtime guard remains necessary because write amplification is variable. + if (u128::from(current) + u128::from(payload) * 2) * 100 + > u128::from(limit) * u128::from(args.memory_guard_percent) + { + return Err(format!("{}: dataset {payload} bytes exceeds memory budget: current={current}, limit={limit}; reduce --key-space or --value-size", node.evidence["name"])); + } + } + Ok(()) +} + +pub(super) async fn memory_guard(nodes: &[ServerProcess], percent: u8) -> String { + loop { + for node in nodes { + match memory(node) { + Ok((None, _)) => {} + Ok((Some(limit), current)) + if u128::from(current) * 100 < u128::from(limit) * u128::from(percent) => {} + Ok((Some(limit), current)) => { + return format!( + "{}: memory guard stopped load at {current}/{limit} bytes ({percent}% limit)", + node.evidence["name"] + ) + } + Err(error) => return error, + } + } + tokio::time::sleep(Duration::from_millis(200)).await; + } +} + +pub(super) async fn inspect(args: &PaxosKvArgs) -> Result<Vec<ServerProcess>, String> { + if !args.server_pids.is_empty() { + return native::inspect(args); + } + if args.containers.len() != 3 || args.containers.iter().collect::<BTreeSet<_>>().len() != 3 { + return Err("provide three distinct --container names in endpoint order".into()); + } + let mut result = Vec::new(); + for (name, endpoint) in args.containers.iter().zip(&args.endpoints) { + if name.starts_with('-') { + return Err("invalid container name".into()); + } + let output = tokio::time::timeout( + Duration::from_secs(10), + tokio::process::Command::new("docker") + // Instrument the load client/server, not the Docker Go process. + .env_remove("LD_PRELOAD") + .env_remove("ASAN_OPTIONS") + .args(["inspect", name]) + .kill_on_drop(true) + .output(), + ) + .await + .map_err(|_| "docker inspect timeout")? + .map_err(|e| e.to_string())?; + if !output.status.success() { + return Err(String::from_utf8_lossy(&output.stderr).into_owned()); + } + let data: Vec<Value> = serde_json::from_slice(&output.stdout).map_err(|e| e.to_string())?; + let node = data.first().ok_or("container absent")?; + if node["State"]["Running"] != true || node["HostConfig"]["NetworkMode"] != "host" { + return Err(format!( + "{name}: benchmark requires a running host-network container" + )); + } + let cmd = node["Config"]["Cmd"] + .as_array() + .ok_or("container has no command")?; + let flag = |name: &str| cmd.windows(2).find(|v| v[0] == name).and_then(|v| v[1].as_str()); + if flag("--kv-backend") != Some("mem-block") || flag("--wal-backend") != Some("mem-block") { + return Err("benchmark requires explicit mem-block KV and WAL backends".into()); + } + for (expected, name) in [ + (args.expect_window, "--max-inflight"), + (args.expect_coalesce, "--coalesce-max-keys"), + ] { + if let Some(expected) = expected { + if flag(name).and_then(|s| s.parse::<u64>().ok()) != Some(expected) { + return Err(format!("container {name} does not match expected {expected}")); + } + } + } + let origin = reqwest::Url::parse(endpoint).map_err(|e| e.to_string())?; + if !matches!(origin.host_str(), Some("127.0.0.1" | "localhost" | "[::1]")) { + return Err("local host-network benchmark endpoints must use loopback".into()); + } + if flag("--management-port").and_then(|v| v.parse::<u16>().ok()) != origin.port_or_known_default() { + return Err(format!( + "{name}: endpoint port does not match the container command" + )); + } + let pid = node["State"]["Pid"].as_u64().ok_or("missing container PID")?; + let groups = std::fs::read_to_string(format!("/proc/{pid}/cgroup")).map_err(|e| e.to_string())?; + let path = groups + .lines() + .find_map(|l| l.strip_prefix("0::/")) + .ok_or("cgroup v2 is required")?; + let cgroup = PathBuf::from("/sys/fs/cgroup").join(path); + result.push(ServerProcess { + cgroup, + evidence: json!({"kind":"container","name":name,"id":node["Id"],"image":node["Image"], + "image_labels":node["Config"]["Labels"],"command":cmd, + "pid":pid,"started_at":node["State"]["StartedAt"],"endpoint":endpoint, + "network":"host","memory_limit":node["HostConfig"]["Memory"], + "nano_cpus":node["HostConfig"]["NanoCpus"],"cpu_quota":node["HostConfig"]["CpuQuota"], + "cpu_period":node["HostConfig"]["CpuPeriod"],"cpuset":node["HostConfig"]["CpusetCpus"]}), + }); + } + Ok(result) +} diff --git a/app/crowdb-cli/src/commands/bench/paxos_kv/deployment/memory.rs b/app/crowdb-cli/src/commands/bench/paxos_kv/deployment/memory.rs new file mode 100644 index 000000000..0ca67b62b --- /dev/null +++ b/app/crowdb-cli/src/commands/bench/paxos_kv/deployment/memory.rs @@ -0,0 +1,12 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +pub(super) fn parse_limit(raw: &str) -> Result<Option<u64>, String> { + if raw.trim() == "max" { + return Ok(None); + } + raw.trim() + .parse() + .map(Some) + .map_err(|_| "invalid cgroup memory.max".into()) +} diff --git a/app/crowdb-cli/src/commands/bench/paxos_kv/deployment/native.rs b/app/crowdb-cli/src/commands/bench/paxos_kv/deployment/native.rs new file mode 100644 index 000000000..af7ecc792 --- /dev/null +++ b/app/crowdb-cli/src/commands/bench/paxos_kv/deployment/native.rs @@ -0,0 +1,106 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use super::{json, BTreeSet, PathBuf, PaxosKvArgs, ServerProcess}; + +pub(super) fn inspect(args: &PaxosKvArgs) -> Result<Vec<ServerProcess>, String> { + if !args.containers.is_empty() + || args.server_pids.len() != 3 + || args.endpoints.len() != 3 + || args.server_pids.iter().collect::<BTreeSet<_>>().len() != 3 + { + return Err("native diagnostic requires three distinct --server-pid values and three endpoints, without --container".into()); + } + let mut nodes = Vec::new(); + let mut cgroups = BTreeSet::new(); + for (&pid, endpoint) in args.server_pids.iter().zip(&args.endpoints) { + let node = inspect_process(args, pid, endpoint)?; + if !cgroups.insert(node.cgroup.clone()) { + return Err("native server PIDs must have separate cgroups".into()); + } + nodes.push(node); + } + Ok(nodes) +} + +fn inspect_process(args: &PaxosKvArgs, pid: u32, endpoint: &str) -> Result<ServerProcess, String> { + let proc = PathBuf::from(format!("/proc/{pid}")); + let read = |name| std::fs::read_to_string(proc.join(name)).map_err(|e| format!("PID {pid}/{name}: {e}")); + let stat = read("stat")?; + let fields: Vec<_> = stat + .rsplit_once(')') + .ok_or("invalid process stat")? + .1 + .split_whitespace() + .collect(); + if matches!(fields.first(), Some(&"Z" | &"X")) { + return Err(format!("PID {pid} has exited")); + } + let started = fields.get(19).ok_or("missing process start time")?; + let exe = std::fs::read_link(proc.join("exe")).map_err(|e| e.to_string())?; + if exe.file_name().and_then(|n| n.to_str()) != Some("crowdb-paxos-kv-server") { + return Err(format!("PID {pid} is not crowdb-paxos-kv-server")); + } + let command: Vec<String> = read("cmdline")? + .split('\0') + .filter(|s| !s.is_empty()) + .map(str::to_owned) + .collect(); + validate_command(args, &command, endpoint)?; + let group = read("cgroup")?; + let relative = group + .lines() + .find_map(|l| l.strip_prefix("0::/")) + .ok_or("cgroup v2 required")?; + if relative.is_empty() { + return Err("native server requires a dedicated cgroup".into()); + } + let cgroup = PathBuf::from("/sys/fs/cgroup").join(relative); + let policy = |file| std::fs::read_to_string(cgroup.join(file)).map_err(|e| format!("cgroup {file}: {e}")); + let pids = policy("cgroup.procs")?; + if pids.split_whitespace().collect::<Vec<_>>() != [pid.to_string()] { + return Err(format!("PID {pid}: cgroup must contain only this server process")); + } + let boot = std::fs::read_to_string("/proc/sys/kernel/random/boot_id").map_err(|e| e.to_string())?; + // User services may not have the cpuset controller delegated. The process + // status still reports the effective CPU affinity inherited from ancestors. + let status = read("status")?; + let affinity = status + .lines() + .find_map(|line| line.strip_prefix("Cpus_allowed_list:")) + .ok_or("missing process CPU affinity")? + .trim(); + let evidence = json!({"kind":"native-process","name":format!("pid-{pid}"), + "id":format!("{}:{pid}:{started}",boot.trim()),"started_at":started, + "pid":pid,"exe":exe,"command":command,"endpoint":endpoint,"network":"host", + "cgroup":cgroup,"memory_limit":policy("memory.max")?,"cpu_max":policy("cpu.max")?, + "cpuset":std::fs::read_to_string(cgroup.join("cpuset.cpus.effective")).ok(), + "cpu_affinity_list":affinity}); + Ok(ServerProcess { evidence, cgroup }) +} + +fn validate_command(args: &PaxosKvArgs, command: &[String], endpoint: &str) -> Result<(), String> { + let flag = |name: &str| command.windows(2).find(|v| v[0] == name).map(|v| v[1].as_str()); + for name in ["--kv-backend", "--wal-backend"] { + if flag(name) != Some("mem-block") { + return Err("native benchmark requires explicit mem-block KV and WAL backends".into()); + } + } + for (expected, name) in [ + (args.expect_window, "--max-inflight"), + (args.expect_coalesce, "--coalesce-max-keys"), + ] { + if let Some(expected) = expected { + if flag(name).and_then(|s| s.parse::<u64>().ok()) != Some(expected) { + return Err(format!("native server {name} does not match expected {expected}")); + } + } + } + let origin = reqwest::Url::parse(endpoint).map_err(|e| e.to_string())?; + if !matches!(origin.host_str(), Some("127.0.0.1" | "localhost" | "[::1]")) + || flag("--management-port").and_then(|p| p.parse::<u16>().ok()) != origin.port_or_known_default() + { + return Err("native endpoint must match the server's loopback management port".into()); + } + Ok(()) +} diff --git a/app/crowdb-cli/src/commands/bench/paxos_kv/run.rs b/app/crowdb-cli/src/commands/bench/paxos_kv/run.rs new file mode 100644 index 000000000..798b481b9 --- /dev/null +++ b/app/crowdb-cli/src/commands/bench/paxos_kv/run.rs @@ -0,0 +1,272 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use super::super::workload::{self, Window}; +use super::{ + args::{Operation, PaxosKvArgs}, + client, collect, dataset, deployment, +}; +use crate::Cli; +use crowdb_common::metrics::system::SystemCollector; +use serde_json::{json, Value}; +use std::{process::ExitCode, sync::Arc, time::Duration}; + +pub async fn run(_cli: &Cli, args: PaxosKvArgs) -> ExitCode { + let output = args.load.output.clone(); + let result = execute(Arc::new(args)).await; + let (report, success) = match result { + Ok(r) => { + let ok = r["passed"] == true; + (r, ok) + } + Err(e) => { + eprintln!("bench paxos-kv failed: {e}"); + (json!({"passed":false,"error":e}), false) + } + }; + let mut report = report; + if report.get("load").is_some() { + report["history_line"] = json!(format!( + "# {} {} {} ops/s={} latency_ns={} passed={}", + chrono::Utc::now().format("%Y-%m-%d"), + report["config"]["label"].as_str().unwrap_or("kv"), + report["deployment_kind"].as_str().unwrap_or("host-container"), + report["load"]["ops_per_sec"], + report["load"]["latency_ns"], + success + )); + } + let text = serde_json::to_string_pretty(&report).expect("JSON value"); + if let Some(path) = output { + if let Err(error) = std::fs::write(path, &text) { + eprintln!("write benchmark report: {error}"); + return ExitCode::from(2); + } + } + println!("{text}"); + if success { + ExitCode::SUCCESS + } else { + ExitCode::from(2) + } +} + +fn validate(args: &PaxosKvArgs) -> Result<(), String> { + args.load.validate()?; + let size = if args.mixed_values { 16384 } else { args.value_size }; + if args.group == 0 + || args.key_space == 0 + || args.key_space > 10_000_000 + || size == 0 + || size > 1024 * 1024 + || args.dataset_bytes().map_or(true, |n| n > args.max_dataset_bytes) + || args.connections == 0 + || args.rpc_workers == 0 + || args.send_queue_capacity == 0 + || args.scan_limit == 0 + { + return Err("invalid group, dataset budget, value size or client capacity".into()); + } + if matches!(args.operation, Operation::Scan) + && args + .key_space + .min(u64::from(args.scan_limit)) + .checked_mul(size as u64 + 21) + .and_then(|n| n.checked_mul(args.load.loader_num as u64)) + .map_or(true, |n| n > args.max_inflight_bytes) + { + return Err("scan responses exceed --max-inflight-bytes".into()); + } + if args.require_snapshot && (!matches!(args.operation, Operation::Write) || args.value_size < 16384) { + return Err("snapshot sentinel requires a write workload with values >=16KiB".into()); + } + Ok(()) +} + +async fn execute(args: Arc<PaxosKvArgs>) -> Result<Value, String> { + validate(&args)?; + let containers = deployment::inspect(&args).await?; + deployment::validate_budget(&args, &containers)?; + let (client, topology) = client::connect(&args).await?; + let client = Arc::new(client); + eprintln!("bench paxos-kv: checking deterministic dataset before timing"); + prepare_dataset(client.clone(), args.clone(), &containers).await?; + if matches!(args.operation, Operation::Prepare) { + return Ok(json!({"passed":true,"operation":"prepare","keys":args.key_space})); + } + if matches!(args.operation, Operation::Scan) { + dataset::scan( + &client, + &args, + crowdb_paxos_kv_client::ReadMode::Linearizable, + true, + ) + .await?; + } + let before = collect::sample(&args, &containers).await?; + let window = Window::new(&args.load); + let operation_args = args.clone(); + let operation_client = client.clone(); + let load = workload::run_until( + &args.load, + window, + move |worker, sequence| { + let args = operation_args.clone(); + let client = operation_client.clone(); + async move { + // Stateless SplitMix64 avoids shared RNG contention and repeating adjacent keys. + let mut n = args + .seed + .wrapping_add((worker as u64) << 32) + .wrapping_add(sequence) + .wrapping_add(0x9e37_79b9_7f4a_7c15); + n = (n ^ (n >> 30)).wrapping_mul(0xbf58_476d_1ce4_e5b9); + n = (n ^ (n >> 27)).wrapping_mul(0x94d0_49bb_1331_11eb); + let id = (n ^ (n >> 31)) % args.key_space; + dataset::operation(&client, &args, id).await + } + }, + deployment::memory_guard(&containers, args.memory_guard_percent), + ); + let measurement = async { + tokio::time::sleep_until(window.measure).await; + let mut system = SystemCollector::new(); + let start = collect::sample(&args, &containers).await?; + tokio::time::sleep_until(window.tail).await; + let cli_system = system.collect(); + let end = collect::sample(&args, &containers).await?; + Ok::<_, String>( + json!({"delta":collect::deltas(&start,&end)?,"before":start,"after":end,"cli_system":cli_system, + "note":"resource snapshots carry actual sampling timestamps; latency uses exact monotonic window; host TCP/DRAM are shared host metrics"}), + ) + }; + eprintln!( + "bench paxos-kv: warmup={}s measurement={}s tail={}s", + args.load.warmup_secs, args.load.duration_secs, args.load.tail_secs + ); + let (mut result, measurement) = tokio::join!(load, measurement); + let metrics = match measurement { + Ok(v) => v, + Err(e) => { + result.errors.push(e); + Value::Null + } + }; + let (after, sentinel) = finish_collection(&args, &containers, &before, &mut result).await; + let verified = + super::verification::after_load(&mut result.errors, verify_dataset(client, args.clone())).await; + let verification = if verified { + if result.errors.is_empty() { + "passed" + } else { + "failed" + } + } else { + eprintln!("bench paxos-kv: skipping dataset verification after failed load or service validation"); + "skipped_after_failure" + }; + let evidence: Vec<_> = containers.iter().map(|c| &c.evidence).collect(); + Ok( + json!({"schema_version":1,"cli_version":env!("CARGO_PKG_VERSION"),"debug_build":cfg!(debug_assertions),"at":chrono::Utc::now(),"passed":result.errors.is_empty(),"errors":result.errors, + "kind":if args.load.duration_secs>=20 {"benchmark"}else{"smoke"},"config":&*args,"load":result.report, + "deployment_kind":if args.server_pids.is_empty() {"host-container"} else {"native-process"}, + "containers":if args.server_pids.is_empty() {json!(evidence)} else {json!([])}, + "processes":if args.server_pids.is_empty() {json!([])} else {json!(evidence)},"initial_topology":topology,"measurement_resources":metrics, + "load_before":before,"load_after":after,"sentinel":sentinel,"verification":verification, + "dataset_policy":"bounded deterministic overwrite dataset; preparation and full verification outside timing", + "host_cpu":std::fs::read_to_string("/proc/cpuinfo").ok().and_then(|s|s.lines().find(|l|l.starts_with("model name")).map(str::to_owned))}), + ) +} + +async fn verify_dataset( + client: Arc<crowdb_paxos_kv_client::PaxosKvClient>, + args: Arc<PaxosKvArgs>, +) -> Result<(), String> { + eprintln!("bench paxos-kv: verifying all dataset bytes outside the timing window"); + tokio::time::timeout( + Duration::from_secs(300), + dataset::verify(client.clone(), args.clone()), + ) + .await + .map_err(|_| "verification timeout".to_owned())??; + if matches!(args.operation, Operation::Scan) { + dataset::scan( + &client, + &args, + crowdb_paxos_kv_client::ReadMode::Linearizable, + true, + ) + .await?; + } + Ok(()) +} + +async fn prepare_dataset( + client: Arc<crowdb_paxos_kv_client::PaxosKvClient>, + args: Arc<PaxosKvArgs>, + containers: &[deployment::ServerProcess], +) -> Result<(), String> { + let prepare = matches!(args.operation, Operation::Prepare | Operation::Write); + let preparation = async { + if prepare { + dataset::prepare(client.clone(), args.clone()).await + } else { + dataset::verify(client.clone(), args.clone()).await + } + }; + tokio::select! { + result = preparation => result?, + error = deployment::memory_guard(containers, args.memory_guard_percent) => return Err(error), + } + Ok(()) +} + +async fn finish_collection( + args: &PaxosKvArgs, + containers: &[deployment::ServerProcess], + before: &Value, + result: &mut workload::LoadResult, +) -> (Option<Value>, Value) { + let after = collect::sample(args, containers).await; + if let Ok(after) = after.as_ref() { + if let Err(error) = collect::deltas(before, after) { + result.errors.push(error); + } + } + match deployment::inspect(args).await { + Ok(current) + if current.iter().zip(containers).all(|(a, b)| { + a.evidence["id"] == b.evidence["id"] && a.evidence["started_at"] == b.evidence["started_at"] + }) => {} + Ok(_) => result + .errors + .push("server process incarnation changed during load".into()), + Err(error) => result.errors.push(error), + } + if result.report["measured_operations"] == 0 { + result + .errors + .push("no successful operations in measurement window".into()); + } + let sentinel = match after.as_ref() { + Ok(after) => match collect::sentinel(args.store, args.group, before, after) { + Ok(v) => { + if args.require_snapshot && v["passed"] != true { + result.errors.push("snapshot sentinel failed".into()); + } + v + } + Err(e) => { + if args.require_snapshot { + result.errors.push(e.clone()); + } + json!({"unsupported":e}) + } + }, + Err(e) => { + result.errors.push(e.clone()); + Value::Null + } + }; + (after.ok(), sentinel) +} diff --git a/app/crowdb-cli/src/commands/bench/paxos_kv/verification.rs b/app/crowdb-cli/src/commands/bench/paxos_kv/verification.rs new file mode 100644 index 000000000..369f74d43 --- /dev/null +++ b/app/crowdb-cli/src/commands/bench/paxos_kv/verification.rs @@ -0,0 +1,18 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use std::future::Future; + +// A failed load must not inject a second full dataset's worth of reads. +pub(super) async fn after_load<F>(errors: &mut Vec<String>, verification: F) -> bool +where + F: Future<Output = Result<(), String>>, +{ + if !errors.is_empty() { + return false; + } + if let Err(error) = verification.await { + errors.push(error); + } + true +} diff --git a/app/crowdb-cli/src/commands/bench/result.rs b/app/crowdb-cli/src/commands/bench/result.rs deleted file mode 100644 index ecea040fd..000000000 --- a/app/crowdb-cli/src/commands/bench/result.rs +++ /dev/null @@ -1,330 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -//! `BenchResult` — the structured result shared by `bench` subcommands. -//! -//! The CLI prints its human-readable [`fmt::Display`] representation and -//! retains the serializable shape for metrics reports. Optional sections -//! (`correctness_errors`, `client_transport_stats`, `server_metrics`) -//! are present only on the subcommands that produce them; `jq`'s `// 0` -//! fallback tolerates their absence. - -use std::fmt; - -use serde::Serialize; - -use crowdb_common::metrics::HistogramSnapshot; - -/// Top-level bench result. `by_op` carries one entry per op kind -/// actually exercised (`read` / `write` / `list`). -#[derive(Debug, Serialize)] -pub struct BenchResult { - pub total_ops: u64, - pub duration_ms: u64, - pub total_errors: u64, - #[serde(skip_serializing_if = "Option::is_none")] - pub correctness_errors: Option<u64>, - pub by_op: BenchOps, - #[serde(skip_serializing_if = "Option::is_none")] - pub client_transport_stats: Option<TransportStats>, - #[serde(skip_serializing_if = "Option::is_none")] - pub server_metrics: Option<ServerMetrics>, -} - -impl BenchResult { - /// Keep the console report and a structured latest-run artifact for measurement tools. - #[must_use] - pub fn finish(&self, directory: &std::path::Path) -> std::process::ExitCode { - println!("{self}"); - let path = directory.join("report.json"); - let result = (|| -> Result<(), Box<dyn std::error::Error>> { - let file = std::fs::File::create(&path)?; - serde_json::to_writer_pretty(file, self)?; - Ok(()) - })(); - if let Err(error) = result { - eprintln!("error: retain benchmark report {}: {error}", path.display()); - return std::process::ExitCode::from(2); - } - std::process::ExitCode::SUCCESS - } -} - -#[derive(Debug, Default, Serialize)] -pub struct BenchOps { - #[serde(skip_serializing_if = "Option::is_none")] - pub read: Option<OpStats>, - #[serde(skip_serializing_if = "Option::is_none")] - pub write: Option<OpStats>, - #[serde(skip_serializing_if = "Option::is_none")] - pub list: Option<OpStats>, -} - -impl BenchOps { - /// Attach a read op section. - #[allow(dead_code)] - pub fn read(mut self, s: HistogramSnapshot) -> Self { - self.read = Some(OpStats::from(s)); - self - } - /// Attach a write op section. - pub fn write(mut self, s: HistogramSnapshot) -> Self { - self.write = Some(OpStats::from(s)); - self - } - /// Attach a list (scan) op section. - #[allow(dead_code)] - pub fn list(mut self, s: HistogramSnapshot) -> Self { - self.list = Some(OpStats::from(s)); - self - } -} - -#[derive(Debug, Serialize)] -pub struct OpStats { - pub latency_us: LatencyUs, - pub latency_ns: LatencyNs, -} - -impl From<HistogramSnapshot> for OpStats { - fn from(s: HistogramSnapshot) -> Self { - // LatencyHistogram records in ns; convert to µs for the JSON report. - Self { - latency_us: LatencyUs { - avg: s.avg / 1000.0, - p50: s.p50 / 1000, - p99: s.p99 / 1000, - }, - latency_ns: LatencyNs { - count: s.count, - avg: s.avg, - min: s.min, - max: s.max, - p50: s.p50, - p90: s.p90, - p95: s.p95, - p99: s.p99, - p999: s.p999, - zero_count: s.zero_count, - overflow_count: s.overflow_count, - }, - } - } -} - -/// Nanosecond values preserve sub-microsecond precision in machine reports. -#[derive(Debug, Serialize)] -pub struct LatencyNs { - pub count: u64, - pub avg: f64, - pub min: u64, - pub max: u64, - pub p50: u64, - pub p90: u64, - pub p95: u64, - pub p99: u64, - pub p999: u64, - pub zero_count: u64, - pub overflow_count: u64, -} - -impl fmt::Display for LatencyNs { - #[allow(clippy::cast_precision_loss)] - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - write!( - f, - "avg={:.3}us p50={:.3}us p90={:.3}us p95={:.3}us p99={:.3}us p99.9={:.3}us max={:.3}us overflow={}", - self.avg / 1000.0, - self.p50 as f64 / 1000.0, - self.p90 as f64 / 1000.0, - self.p95 as f64 / 1000.0, - self.p99 as f64 / 1000.0, - self.p999 as f64 / 1000.0, - self.max as f64 / 1000.0, - self.overflow_count, - ) - } -} - -/// Per-op latency percentiles. Field names are serialized as -/// `*_us` (the contract the regression scripts parse); the Rust names -/// drop the shared `_us` postfix to satisfy `clippy::struct_field_names`. -#[derive(Debug, Serialize)] -pub struct LatencyUs { - #[serde(rename = "avg_us")] - pub avg: f64, - #[serde(rename = "p50_us")] - pub p50: u64, - #[serde(rename = "p99_us")] - pub p99: u64, -} - -/// crowdb-rpc transport stats (client-side or server-side). Fields not -/// exposed by the FFI for a given side are reported as 0. -#[derive(Debug, Default, Serialize)] -pub struct TransportStats { - pub writev_calls: u64, - pub frames_sent: u64, - pub read_calls: u64, - pub frames_parsed: u64, - pub submit_to_writev_avg_us: u64, -} - -/// Server-side metrics fetched from the KV server management API after -/// a write run. Aggregated across the cluster where noted. -#[derive(Debug, Default, Serialize)] -pub struct ServerMetrics { - /// WAL append count summed across nodes. - pub wal_append_count: u64, - pub rpc: TransportStats, - pub replica: ReplicaStats, - pub inflight_enqueued: u64, - pub inflight_wait_avg_us: u64, - /// Election counter delta for the measured interval. - pub election_count: u64, - /// Maintenance snapshot evidence for the measured interval. - pub snapshot: SnapshotStats, - /// Server-side per-op RPC latency (averaged across nodes). - pub rpc_latency: Option<ServerRpcLatency>, -} - -#[derive(Debug, Default, Serialize)] -pub struct SnapshotStats { - pub completed: u64, - pub failed: u64, - pub max_latency_us: u64, -} - -impl ServerMetrics { - /// Convert cumulative counters into values for one measured interval. - #[must_use] - pub fn delta_since(mut self, baseline: &Self) -> Self { - self.wal_append_count = self.wal_append_count.saturating_sub(baseline.wal_append_count); - self.inflight_enqueued = self.inflight_enqueued.saturating_sub(baseline.inflight_enqueued); - self.election_count = self.election_count.saturating_sub(baseline.election_count); - self.snapshot.completed = self - .snapshot - .completed - .saturating_sub(baseline.snapshot.completed); - self.snapshot.failed = self.snapshot.failed.saturating_sub(baseline.snapshot.failed); - self.replica.r2_tps = self.replica.r2_tps.saturating_sub(baseline.replica.r2_tps); - self.replica.r3_tps = self.replica.r3_tps.saturating_sub(baseline.replica.r3_tps); - self - } -} - -/// Server-side per-request-type RPC latency, averaged across nodes. -#[derive(Debug, Default, Serialize)] -#[allow(clippy::struct_field_names)] -pub struct ServerRpcLatency { - pub put_avg_us: f64, - pub put_p50_us: u64, - pub put_p99_us: u64, - pub get_avg_us: f64, - pub get_p50_us: u64, - pub get_p99_us: u64, - pub scan_avg_us: f64, - pub scan_p50_us: u64, - pub scan_p99_us: u64, - pub delete_avg_us: f64, - pub delete_p50_us: u64, - pub delete_p99_us: u64, -} - -#[derive(Debug, Default, Serialize)] -pub struct ReplicaStats { - /// Round-trip latency (µs) to replica 2. - pub r2: u64, - /// Round-trips/s to replica 2. - pub r2_tps: u64, - /// Round-trip latency (µs) to replica 3. - pub r3: u64, - /// Round-trips/s to replica 3. - pub r3_tps: u64, -} - -impl fmt::Display for BenchResult { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - let ops_s = (self.total_ops * 1000).checked_div(self.duration_ms).unwrap_or(0); - writeln!(f, "=== bench report ===")?; - writeln!( - f, - " ops: {} duration: {}ms ops/s: {} errors: {}", - self.total_ops, self.duration_ms, ops_s, self.total_errors, - )?; - if let Some(ce) = self.correctness_errors { - writeln!(f, " correctness_errors: {ce}")?; - } - if let Some(s) = &self.by_op.read { - writeln!(f, " read: {}", s.latency_ns)?; - } - if let Some(s) = &self.by_op.write { - writeln!(f, " write: {}", s.latency_ns)?; - } - if let Some(s) = &self.by_op.list { - writeln!(f, " list: {}", s.latency_ns)?; - } - if let Some(cts) = &self.client_transport_stats { - writeln!( - f, - " client_transport: writev={} frames_sent={} read={} frames_parsed={} s2w_avg={}us", - cts.writev_calls, - cts.frames_sent, - cts.read_calls, - cts.frames_parsed, - cts.submit_to_writev_avg_us, - )?; - } - if let Some(sm) = &self.server_metrics { - let wal_per_node = sm.wal_append_count / 3; - writeln!( - f, - " server: wal_append={} ({} /node) inflight_enq={} inflight_wait_avg={}us", - sm.wal_append_count, wal_per_node, sm.inflight_enqueued, sm.inflight_wait_avg_us, - )?; - writeln!( - f, - " server_rpc: writev={} frames_sent={} read={} frames_parsed={} s2w_avg={}us", - sm.rpc.writev_calls, - sm.rpc.frames_sent, - sm.rpc.read_calls, - sm.rpc.frames_parsed, - sm.rpc.submit_to_writev_avg_us, - )?; - writeln!( - f, - " replica: r2={}us/{}tps r3={}us/{}tps", - sm.replica.r2, sm.replica.r2_tps, sm.replica.r3, sm.replica.r3_tps, - )?; - writeln!( - f, - " maintenance: snapshots={} failed={} max={}us elections={}", - sm.snapshot.completed, sm.snapshot.failed, sm.snapshot.max_latency_us, sm.election_count, - )?; - if let Some(rl) = &sm.rpc_latency { - writeln!(f, " server_rpc_latency (avg/p50/p99 us):")?; - writeln!( - f, - " put: {}/{}/{}", - rl.put_avg_us, rl.put_p50_us, rl.put_p99_us, - )?; - writeln!( - f, - " get: {}/{}/{}", - rl.get_avg_us, rl.get_p50_us, rl.get_p99_us, - )?; - writeln!( - f, - " scan: {}/{}/{}", - rl.scan_avg_us, rl.scan_p50_us, rl.scan_p99_us, - )?; - writeln!( - f, - " delete: {}/{}/{}", - rl.delete_avg_us, rl.delete_p50_us, rl.delete_p99_us, - )?; - } - } - write!(f, "=== end report ===") - } -} diff --git a/app/crowdb-cli/src/commands/bench/verb.rs b/app/crowdb-cli/src/commands/bench/verb.rs index 6c193de46..b04ba4226 100644 --- a/app/crowdb-cli/src/commands/bench/verb.rs +++ b/app/crowdb-cli/src/commands/bench/verb.rs @@ -13,9 +13,8 @@ use crate::Cli; /// Workloads attached to a selected application cluster. #[derive(Subcommand, Debug)] pub enum BenchVerb { - /// KV-layer workload (prepare / read / write / scan). - #[command(subcommand)] - Kv(KvBenchVerb), + /// Load an already running three-container Paxos-KV deployment. + PaxosKv(super::paxos_kv::args::PaxosKvArgs), /// Distributed disk-block allocation benchmark. #[command(subcommand)] Diskdb(DiskdbBenchVerb), @@ -290,191 +289,12 @@ pub struct DiskdbArgs { pub metrics_interval: u64, } -#[derive(Subcommand, Debug)] -pub enum KvBenchVerb { - /// Pre-populate `--keys` keys into store 0 / group 0. - Prepare(PrepareArgs), - /// Point-get workload. - Read(ReadArgs), - /// Put workload. - Write(WriteArgs), - /// Scan (list) workload. - Scan(ScanArgs), -} - -#[derive(clap::Args, Debug)] -pub struct PrepareArgs { - /// Number of keys to write into the target store/group. - #[arg(long, default_value_t = 100_000)] - pub keys: u64, - /// Value size in bytes. - #[arg(long, default_value_t = 64)] - pub value_size: usize, - /// Concurrent put tasks. - #[arg(long, default_value_t = 16)] - pub concurrency: usize, - /// Target store ID (default 0 = system store). - #[arg(long, default_value_t = 0)] - pub store: u64, - /// Target group ID (default 0 = system group; use 1+ for bench groups). - #[arg(long, default_value_t = 0)] - pub group: u64, -} - -#[derive(clap::Args, Debug)] -pub struct ReadArgs { - #[arg(long, default_value_t = 10)] - pub duration_secs: u64, - #[arg(long, default_value_t = 1)] - pub loader_num: usize, - #[arg(long, default_value_t = 1)] - pub connections: usize, - /// Read consistency: `linearizable` (lease barrier) or `minslot`. - #[arg(long, value_enum, default_value_t = BenchReadMode::Linearizable)] - pub read_mode: BenchReadMode, - /// `MinSlot` lower bound: `auto` (client high-watermark) or `zero`. - #[arg(long, value_enum, default_value_t = BenchMinSlot::Auto)] - pub min_slot: BenchMinSlot, - /// `MinSlot` endpoint selection: `leader` or `any-replica`. - #[arg(long, value_enum, default_value_t = BenchReadEndpoint::Leader)] - pub read_endpoint_policy: BenchReadEndpoint, - /// Keyspace size (keys are `k{id:020}`, id in `[0, key_space)`). - #[arg(long, default_value_t = 100_000)] - pub key_space: u64, - /// Value size in bytes (for correctness verification). - #[arg(long, default_value_t = 64)] - pub value_size: usize, - /// Verify the first N value bytes match the expected pattern - /// (`0..N` mod 256). 0 disables verification. - #[arg(long, default_value_t = 0)] - pub verify_bytes: usize, - /// Metrics flush interval in seconds. 0 disables the metrics log. - #[arg(long, default_value_t = 1)] - pub metrics_interval: u64, - /// Target store ID (default 0 = system store). - #[arg(long, default_value_t = 0)] - pub store: u64, - /// Target group ID (default 0 = system group; use 1+ for bench groups). - #[arg(long, default_value_t = 0)] - pub group: u64, -} - -#[derive(clap::Args, Debug)] -pub struct WriteArgs { - #[arg(long, default_value_t = 10)] - pub duration_secs: u64, - #[arg(long, default_value_t = 1)] - pub loader_num: usize, - #[arg(long, default_value_t = 1)] - pub connections: usize, - #[arg(long, default_value_t = 1_000_000)] - pub key_space: u64, - #[arg(long, default_value_t = 512)] - pub value_size: usize, - #[arg(long, default_value_t = 0)] - pub verify_bytes: usize, - /// Client transport: event-write mode (coalesce frames via I/O worker). - #[arg(long, default_value_t = false)] - pub event_write: bool, - /// Client transport: I/O worker threads (match server --rpc-workers). - #[arg(long, default_value_t = 2)] - pub rpc_workers: u32, - /// Client transport: per-connection send queue capacity. 0 = default (4096). - #[arg(long, default_value_t = 0)] - pub send_queue_capacity: u32, - /// Metrics flush interval in seconds. 0 disables the metrics log. - #[arg(long, default_value_t = 1)] - pub metrics_interval: u64, - /// Target store ID (default 0 = system store). - #[arg(long, default_value_t = 0)] - pub store: u64, - /// Target group ID (default 0 = system group; use 1+ for bench groups). - #[arg(long, default_value_t = 0)] - pub group: u64, -} - -#[derive(clap::Args, Debug)] -pub struct ScanArgs { - #[arg(long, default_value_t = 10)] - pub duration_secs: u64, - #[arg(long, default_value_t = 1)] - pub loader_num: usize, - #[arg(long, default_value_t = 1)] - pub connections: usize, - #[arg(long, value_enum, default_value_t = BenchReadMode::Linearizable)] - pub read_mode: BenchReadMode, - #[arg(long, value_enum, default_value_t = BenchMinSlot::Auto)] - pub min_slot: BenchMinSlot, - #[arg(long, value_enum, default_value_t = BenchReadEndpoint::Leader)] - pub read_endpoint_policy: BenchReadEndpoint, - /// Maximum items per scan request. - #[arg(long, default_value_t = 1000)] - pub scan_limit: u32, - /// Prefix filter (empty = no prefix). - #[arg(long, default_value = "")] - pub scan_prefix: String, - /// Exclusive lower bound (empty = start from beginning). - #[arg(long, default_value = "")] - pub scan_start_after: String, - /// Scan traversal direction. - #[arg(long, value_enum, default_value_t = BenchScanDirection::Forward)] - pub scan_direction: BenchScanDirection, - #[arg(long, default_value_t = 64)] - pub value_size: usize, - #[arg(long, default_value_t = 100_000)] - pub key_space: u64, - #[arg(long, default_value_t = 0)] - pub verify_bytes: usize, - /// Mixed value-size distribution, e.g. `64:70,1024:20,16384:10` - /// (size:percent). When set, overrides `--value-size`. - #[arg(long)] - pub value_size_mix: Option<String>, - /// Metrics flush interval in seconds. 0 disables the metrics log. - #[arg(long, default_value_t = 1)] - pub metrics_interval: u64, - /// Target store ID (default 0 = system store). - #[arg(long, default_value_t = 0)] - pub store: u64, - /// Target group ID (default 0 = system group; use 1+ for bench groups). - #[arg(long, default_value_t = 0)] - pub group: u64, -} - -#[derive(Clone, Copy, Debug, PartialEq, Eq, ValueEnum)] -pub enum BenchReadMode { - Linearizable, - Minslot, -} - -#[derive(Clone, Copy, Debug, PartialEq, Eq, ValueEnum)] -pub enum BenchMinSlot { - Auto, - Zero, -} - -#[derive(Clone, Copy, Debug, PartialEq, Eq, ValueEnum)] -pub enum BenchReadEndpoint { - Leader, - AnyReplica, -} - -#[derive(Clone, Copy, Debug, PartialEq, Eq, ValueEnum)] -pub enum BenchScanDirection { - Forward, - Reverse, -} - pub async fn run_bench_verb(cli: &Cli, verb: BenchVerb) -> ExitCode { match verb { BenchVerb::Diskdb(verb) => super::disk::db::run(cli, verb).await, BenchVerb::Chunkdb(verb) => super::chunk::run(cli, verb).await, BenchVerb::Chunkio(verb) => super::io::run(cli, verb).await, BenchVerb::S3(verb) => super::s3::run(cli, verb).await, - BenchVerb::Kv(kv) => match kv { - KvBenchVerb::Prepare(args) => super::kv::prepare::run(cli, args).await, - KvBenchVerb::Read(args) => super::kv::read::run(cli, args).await, - KvBenchVerb::Write(args) => super::kv::write::run(cli, args).await, - KvBenchVerb::Scan(args) => super::kv::scan::run(cli, args).await, - }, + BenchVerb::PaxosKv(args) => super::paxos_kv::run(cli, args).await, } } diff --git a/app/crowdb-cli/src/commands/bench/workload.rs b/app/crowdb-cli/src/commands/bench/workload.rs new file mode 100644 index 000000000..2c5738167 --- /dev/null +++ b/app/crowdb-cli/src/commands/bench/workload.rs @@ -0,0 +1,252 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Continuous closed-loop load with one shared monotonic measurement window. +use clap::Args; +use crowdb_common::metrics::histogram::{HistogramData, LatencyHistogram}; +use serde::Serialize; +use serde_json::{json, Value}; +use std::{ + future::Future, + path::PathBuf, + sync::{ + atomic::{AtomicU64, Ordering}, + Arc, + }, + time::Duration, +}; +use tokio::{ + task::JoinSet, + time::{sleep_until, Instant}, +}; + +#[derive(Args, Debug, Clone, Serialize)] +pub struct LoadOptions { + #[arg(long, default_value_t = 5, global = true)] + pub warmup_secs: u64, + #[arg(long, default_value_t = 20, global = true)] + pub duration_secs: u64, + #[arg(long, default_value_t = 5, global = true)] + pub tail_secs: u64, + #[arg(long, default_value_t = 30, global = true)] + pub drain_timeout_secs: u64, + #[arg(long, default_value_t = 1, global = true)] + pub loader_num: usize, + #[arg(long, global = true)] + pub output: Option<PathBuf>, +} + +impl LoadOptions { + pub fn validate(&self) -> Result<(), String> { + if self.duration_secs == 0 + || self.duration_secs > 86400 + || self.warmup_secs > 3600 + || self.tail_secs > 3600 + || self.drain_timeout_secs == 0 + || self.drain_timeout_secs > 3600 + || self.loader_num == 0 + || self.loader_num > 4096 + { + return Err( + "invalid load window or concurrency (measurement 1..86400s, concurrency 1..4096)".into(), + ); + } + Ok(()) + } +} + +#[derive(Clone, Copy)] +pub struct Window { + pub start: Instant, + pub measure: Instant, + pub tail: Instant, + pub stop: Instant, +} +impl Window { + pub fn new(options: &LoadOptions) -> Self { + let start = Instant::now(); + let measure = start + Duration::from_secs(options.warmup_secs); + let tail = measure + Duration::from_secs(options.duration_secs); + Self { + start, + measure, + tail, + stop: tail + Duration::from_secs(options.tail_secs), + } + } + pub fn phase(self, time: Instant) -> usize { + if time < self.measure { + 0 + } else if time < self.tail { + 1 + } else { + 2 + } + } +} + +struct Counts { + admitted: AtomicU64, + completed: AtomicU64, + failed: AtomicU64, +} +impl Counts { + fn new() -> Self { + Self { + admitted: AtomicU64::new(0), + completed: AtomicU64::new(0), + failed: AtomicU64::new(0), + } + } + fn json(&self) -> Value { + let admitted = self.admitted.load(Ordering::Relaxed); + let completed = self.completed.load(Ordering::Relaxed); + let failed = self.failed.load(Ordering::Relaxed); + json!({"admitted": admitted, "completed": completed, "failed": failed, + "incomplete": admitted - completed - failed}) + } +} +struct Worker { + phases: [Counts; 3], + measured: LatencyHistogram, + bytes: AtomicU64, + crossing: AtomicU64, +} +impl Worker { + fn new() -> Self { + Self { + phases: std::array::from_fn(|_| Counts::new()), + measured: LatencyHistogram::new("bench.latency".into()), + bytes: AtomicU64::new(0), + crossing: AtomicU64::new(0), + } + } +} + +pub struct LoadResult { + pub report: Value, + pub errors: Vec<String>, +} + +/// Stop and join all workers if a resource guard reports a failure. +pub async fn run_until<F, Fut, Stop>( + options: &LoadOptions, + window: Window, + operation: F, + stop: Stop, +) -> LoadResult +where + F: Fn(usize, u64) -> Fut + Send + Sync + 'static, + Fut: Future<Output = Result<u64, String>> + Send + 'static, + Stop: Future<Output = String>, +{ + tokio::pin!(stop); + let operation = Arc::new(operation); + let workers: Vec<_> = (0..options.loader_num).map(|_| Arc::new(Worker::new())).collect(); + let mut tasks = JoinSet::new(); + for (index, worker) in workers.iter().enumerate() { + let worker = worker.clone(); + let operation = operation.clone(); + tasks.spawn(async move { + let mut sequence = 0; + let mut first_error = None; + loop { + let admitted = Instant::now(); + if admitted >= window.stop { + break; + } + let phase = window.phase(admitted); + worker.phases[phase].admitted.fetch_add(1, Ordering::Relaxed); + let outcome = operation(index, sequence).await; + let completed = Instant::now(); + if window.phase(completed) != phase || completed >= window.stop { + worker.crossing.fetch_add(1, Ordering::Relaxed); + } + match outcome { + Ok(bytes) => { + worker.phases[phase].completed.fetch_add(1, Ordering::Relaxed); + if phase == 1 && completed < window.tail { + worker.measured.observe( + u64::try_from((completed - admitted).as_nanos()).unwrap_or(u64::MAX), + ); + worker.bytes.fetch_add(bytes, Ordering::Relaxed); + } + } + Err(error) => { + worker.phases[phase].failed.fetch_add(1, Ordering::Relaxed); + if first_error.is_none() { + first_error = Some(error); + } + // Yield even for immediately rejected operations. + tokio::task::yield_now().await; + } + } + sequence += 1; + } + first_error + }); + } + let mut errors = Vec::new(); + let deadline = window.stop + Duration::from_secs(options.drain_timeout_secs); + loop { + tokio::select! { + reason = &mut stop => { + errors.push(reason); + tasks.abort_all(); + while tasks.join_next().await.is_some() {} + break; + }, + result = tasks.join_next() => match result { + None => break, + Some(Ok(Some(error))) => errors.push(error), + Some(Err(error)) => errors.push(format!("load worker: {error}")), + Some(Ok(None)) => {}, + }, + () = sleep_until(deadline) => { + errors.push("drain timeout; unfinished operations have unknown outcomes".into()); + tasks.abort_all(); + while tasks.join_next().await.is_some() {} + break; + } + } + } + let report = summarize(options, window, &workers); + if report["failed"].as_u64().unwrap_or(0) > 0 && errors.is_empty() { + errors.push("operation failures in an aborted worker".into()); + } + LoadResult { report, errors } +} + +#[allow(clippy::cast_precision_loss)] +fn summarize(options: &LoadOptions, window: Window, workers: &[Arc<Worker>]) -> Value { + let mut histogram = HistogramData::default(); + let mut phases = [[0_u64; 4]; 3]; + let mut bytes = 0; + let mut crossing = 0; + for worker in workers { + histogram.merge(&worker.measured.capture_total()); + bytes += worker.bytes.load(Ordering::Relaxed); + crossing += worker.crossing.load(Ordering::Relaxed); + for (sum, counts) in phases.iter_mut().zip(&worker.phases) { + let counts = counts.json(); + for (i, field) in ["admitted", "completed", "failed", "incomplete"] + .iter() + .enumerate() + { + sum[i] += counts[field].as_u64().unwrap_or(0); + } + } + } + let h = histogram.snapshot(0); + let phases_json: Vec<_> = phases.iter().zip(["warmup", "measurement", "tail"]).map(|(v, name)| + json!({"phase":name,"admitted":v[0],"completed":v[1],"failed":v[2],"incomplete":v[3]})).collect(); + json!({"phases":phases_json,"measured_operations":h.count,"measured_bytes":bytes, + "ops_per_sec":h.count as f64 / options.duration_secs as f64, + "bytes_per_sec":bytes as f64 / options.duration_secs as f64, + "failed":phases.iter().map(|p|p[2]).sum::<u64>(), + "incomplete":phases.iter().map(|p|p[3]).sum::<u64>(), + "boundary_crossings":crossing,"duration_secs":options.duration_secs, + "elapsed_secs_including_drain":window.start.elapsed().as_secs_f64(), + "selection":"successful requests admitted and completed within measurement; crossings excluded", + "latency_ns":{"count":h.count,"avg":h.avg,"min":h.min,"max":h.max,"p50":h.p50,"p99":h.p99,"p999":h.p999}}) +} diff --git a/app/crowdb-cli/src/commands/cluster/local.rs b/app/crowdb-cli/src/commands/cluster/local.rs index b8e87764c..dc899a3d4 100644 --- a/app/crowdb-cli/src/commands/cluster/local.rs +++ b/app/crowdb-cli/src/commands/cluster/local.rs @@ -29,7 +29,7 @@ pub struct StartArgs { /// Already loaded CROWDB node image; start never builds or pulls. #[arg(long)] image: Option<String>, - /// First localhost console port; each node reserves four ports, ten apart. + /// First localhost console port; Access uses the default relative offsets, nodes add 0/1/2. #[arg(long)] port_base: Option<u16>, #[arg(long)] diff --git a/app/crowdb-cli/src/commands/cluster/local/host.rs b/app/crowdb-cli/src/commands/cluster/local/host.rs index b1025a621..115f3d18e 100644 --- a/app/crowdb-cli/src/commands/cluster/local/host.rs +++ b/app/crowdb-cli/src/commands/cluster/local/host.rs @@ -113,7 +113,7 @@ pub(super) fn selected(name: &str, args: &StartArgs) -> Result<bool> { } if args.port_base.is_some() { return Err(invalid( - "host networking uses port 9090; --port-base is for a local bridge", + "host networking uses the default Console Web port; --port-base is for a local bridge", )); } return Ok(true); diff --git a/app/crowdb-cli/src/commands/cluster/local/host/args.rs b/app/crowdb-cli/src/commands/cluster/local/host/args.rs index e369022ab..a564708ca 100644 --- a/app/crowdb-cli/src/commands/cluster/local/host/args.rs +++ b/app/crowdb-cli/src/commands/cluster/local/host/args.rs @@ -37,7 +37,7 @@ pub(crate) struct HostArgs { physical_host_id: Option<String>, #[arg(long)] interface: Option<String>, - /// This host's management IP on the selected interface; port 9090. + /// This host's management IP on the selected interface; uses the default Console Web port. #[arg(long)] management_address: Option<IpAddr>, #[arg(long)] diff --git a/app/crowdb-cli/src/commands/cluster/local/launch.rs b/app/crowdb-cli/src/commands/cluster/local/launch.rs index 3582497c8..a6b6cf939 100644 --- a/app/crowdb-cli/src/commands/cluster/local/launch.rs +++ b/app/crowdb-cli/src/commands/cluster/local/launch.rs @@ -9,6 +9,7 @@ use super::{ StartArgs, }; use crowdb_console_shared::error::Result; +use crowdb_protocol::WEB_BASE; use serde_json::{json, Value}; use std::time::Duration; @@ -41,12 +42,12 @@ pub(super) async fn accepted(docker: &Docker, cluster: &str, args: &StartArgs) - return Err(invalid("local start requires the Linux amd64 CROWDB node image")); } let record = Launch { - version: 2, + version: 3, cluster: cluster.into(), owner: uuid::Uuid::new_v4().to_string(), runtime: docker.identity().await?, nodes: args.nodes.unwrap_or(1), - port_base: args.port_base.unwrap_or(9090), + port_base: args.port_base.unwrap_or(WEB_BASE), image: image.into(), image_id: metadata["Id"] .as_str() @@ -110,11 +111,16 @@ fn arguments(record: &Launch, index: u8) -> Vec<String> { "--mount", &format!("type=volume,source={volume},target=/opt/crowdb/data"), ])); - let base = record.port_base + u16::from(index - 1) * 10; + let base = record.port_base + u16::from(index - 1) * if record.version < 3 { 10 } else { 1 }; for ((offset, protocol), internal) in [(0_u16, ""), (1, "S3"), (2, "ICEBERG"), (3, "DATASET")] .into_iter() .zip(record.container_ports()) { + let offset = if record.version < 3 { + offset + } else { + internal - WEB_BASE + }; args.extend(strings(&[ "-p", &format!("127.0.0.1:{}:{internal}", base + offset), @@ -138,9 +144,14 @@ fn arguments(record: &Launch, index: u8) -> Vec<String> { pub(super) fn verify(item: &Value, record: &Launch, index: u8) -> Result<()> { require_owned(item, record, "container")?; - let base = record.port_base + u16::from(index - 1) * 10; + let base = record.port_base + u16::from(index - 1) * if record.version < 3 { 10 } else { 1 }; let mut ports = serde_json::Map::new(); for (offset, internal) in (0_u16..).zip(record.container_ports()) { + let offset = if record.version < 3 { + offset + } else { + internal - WEB_BASE + }; ports.insert( format!("{internal}/tcp"), json!([{"HostIp":"127.0.0.1", "HostPort":(base+offset).to_string()}]), @@ -176,7 +187,15 @@ pub(super) fn verify(item: &Value, record: &Launch, index: u8) -> Result<()> { format!("CROWDB_CLUSTER={}", record.cluster), format!("CROWDB_INITIAL_VOTERS={}", record.nodes), ]; - for (offset, protocol) in [(1, "S3"), (2, "ICEBERG"), (3, "DATASET")] { + for ((offset, protocol), internal) in [(1_u16, "S3"), (2, "ICEBERG"), (3, "DATASET")] + .into_iter() + .zip(record.container_ports().into_iter().skip(1)) + { + let offset = if record.version < 3 { + offset + } else { + internal - WEB_BASE + }; expected.push(format!( "CROWDB_{protocol}_PUBLIC_URI=http://127.0.0.1:{}", base + offset diff --git a/app/crowdb-cli/src/commands/cluster/local/state.rs b/app/crowdb-cli/src/commands/cluster/local/state.rs index 20769bb0c..3aa4432ca 100644 --- a/app/crowdb-cli/src/commands/cluster/local/state.rs +++ b/app/crowdb-cli/src/commands/cluster/local/state.rs @@ -152,12 +152,18 @@ impl Launch { pub fn validate(&self) -> Result<()> { let owner = uuid::Uuid::parse_str(&self.owner).map_err(|_| invalid("invalid cluster ownership token"))?; - if !matches!(self.version, 1 | 2) + if !matches!(self.version, 1..=3) || !name_valid(&self.cluster) || owner.is_nil() || owner.to_string() != self.owner || !matches!(self.nodes, 1 | 3) - || !(1..=65512).contains(&self.port_base) + || !(1..=u16::MAX + - if self.version < 3 { + 23 + } else { + crowdb_protocol::ACCESS_SERVER_DATASET_HTTP_BASE - crowdb_protocol::WEB_BASE + 2 + }) + .contains(&self.port_base) || !self.cpus.is_finite() || self.cpus <= 0.0 || self.cpus > f64::from(u32::MAX) @@ -179,21 +185,27 @@ impl Launch { } pub fn origins(&self) -> Vec<String> { (0..self.nodes) - .map(|index| format!("http://127.0.0.1:{}", self.port_base + u16::from(index) * 10)) + .map(|index| { + format!( + "http://127.0.0.1:{}", + self.port_base + u16::from(index) * if self.version < 3 { 10 } else { 1 } + ) + }) .collect() } pub fn container_ports(&self) -> [u16; 4] { - if self.version == 1 { - [9090, 9091, 9092, 9093] - } else { - [ - 9090, + match self.version { + 1 => [9090, 9091, 9092, 9093], + 2 => [9090, 16000, 17000, 18000], + _ => [ + crowdb_protocol::WEB_BASE, crowdb_protocol::ACCESS_SERVER_HTTP_BASE, crowdb_protocol::ACCESS_SERVER_ICEBERG_HTTP_BASE, crowdb_protocol::ACCESS_SERVER_DATASET_HTTP_BASE, - ] + ], } } + pub fn resource_name(&self, kind: &str, index: u8) -> String { let base = format!("crowdb-{}", self.cluster); if kind == "network" { diff --git a/app/crowdb-cli/src/commands/cluster/target.rs b/app/crowdb-cli/src/commands/cluster/target.rs index 1c9b05229..f3c5f5cb0 100644 --- a/app/crowdb-cli/src/commands/cluster/target.rs +++ b/app/crowdb-cli/src/commands/cluster/target.rs @@ -10,6 +10,19 @@ use crowdb_console_shared::deployment::target::{self, ClusterTargetBinding, Reso use crate::Cli; pub(crate) async fn resolve(cli: &Cli) -> Result<Option<ResolvedClusterTarget>, ExitCode> { + if matches!( + cli.command, + crate::Domain::Bench { + verb: crate::commands::BenchVerb::PaxosKv(_) + } + ) { + if cli.cluster.is_some() || !cli.management_endpoints.is_empty() { + return Err(failure( + "bench paxos-kv uses container --endpoint addresses, not a Console cluster target", + )); + } + return Ok(None); + } if matches!( cli.command, crate::Domain::Cluster { @@ -49,7 +62,7 @@ fn validate_command(command: &crate::Domain) -> Result<(), ExitCode> { fn supports_attachment(command: &crate::Domain) -> bool { use crate::{ commands::{BenchVerb, ClusterVerb}, - Domain, KvVerb, + Domain, PaxosKvVerb, }; matches!( command, @@ -61,15 +74,14 @@ fn supports_attachment(command: &crate::Domain) -> bool { | ClusterVerb::Node { .. } | ClusterVerb::DiskGroup { .. } | ClusterVerb::Disk { .. } - } | Domain::Kv { - verb: KvVerb::Store(_) | KvVerb::Group(_) | KvVerb::Replica(_) | KvVerb::Data(_) + } | Domain::PaxosKv { + verb: PaxosKvVerb::Store(_) + | PaxosKvVerb::Group(_) + | PaxosKvVerb::Replica(_) + | PaxosKvVerb::Data(_) } | Domain::S3 { .. } | Domain::Bench { - verb: BenchVerb::Kv(_) - | BenchVerb::Diskdb(_) - | BenchVerb::Chunkdb(_) - | BenchVerb::Chunkio(_) - | BenchVerb::S3(_) + verb: BenchVerb::Diskdb(_) | BenchVerb::Chunkdb(_) | BenchVerb::Chunkio(_) | BenchVerb::S3(_) } ) } diff --git a/app/crowdb-cli/src/commands/kv.rs b/app/crowdb-cli/src/commands/kv.rs deleted file mode 100644 index 0b41af25c..000000000 --- a/app/crowdb-cli/src/commands/kv.rs +++ /dev/null @@ -1,14 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -//! `kv` domain — logical concepts (store/group/replica), -//! and data-plane operations. Each sub-module owns one slice of the -//! domain. - -pub mod data; -pub mod logical; - -pub(crate) use data::{run_kv_data_verb, KvDataVerb}; -pub(crate) use logical::{ - run_group_verb, run_replica_verb, run_store_verb, GroupVerb, ReplicaVerb, StoreVerb, -}; diff --git a/app/crowdb-cli/src/commands/paxos_kv.rs b/app/crowdb-cli/src/commands/paxos_kv.rs new file mode 100644 index 000000000..6cbdcdd9d --- /dev/null +++ b/app/crowdb-cli/src/commands/paxos_kv.rs @@ -0,0 +1,14 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! `paxos-kv` domain — logical concepts (store/group/replica), +//! and data-plane operations. Each sub-module owns one slice of the +//! domain. + +pub mod data; +pub mod logical; + +pub(crate) use data::{run_paxos_kv_data_verb, PaxosKvDataVerb}; +pub(crate) use logical::{ + run_group_verb, run_replica_verb, run_store_verb, GroupVerb, ReplicaVerb, StoreVerb, +}; diff --git a/app/crowdb-cli/src/commands/kv/data.rs b/app/crowdb-cli/src/commands/paxos_kv/data.rs similarity index 94% rename from app/crowdb-cli/src/commands/kv/data.rs rename to app/crowdb-cli/src/commands/paxos_kv/data.rs index 4e22f9b1e..9cf478863 100644 --- a/app/crowdb-cli/src/commands/kv/data.rs +++ b/app/crowdb-cli/src/commands/paxos_kv/data.rs @@ -1,18 +1,18 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -//! `kv put/get/delete/scan/snapshot` command handlers — data-plane. +//! `paxos-kv data put/get/delete/scan/snapshot` command handlers — data-plane. use std::process::ExitCode; use clap::Subcommand; -use crowdb_kv_client::GetOutcome; +use crowdb_paxos_kv_client::GetOutcome; use crate::commands::authority_context; use crate::Cli; #[derive(Subcommand, Debug)] -pub enum KvDataVerb { +pub enum PaxosKvDataVerb { Put { #[arg(short = 's', long)] store: String, @@ -78,9 +78,9 @@ pub enum SnapshotVerb { } #[allow(clippy::too_many_lines)] -pub async fn run_kv_data_verb(cli: &Cli, verb: KvDataVerb) -> ExitCode { +pub async fn run_paxos_kv_data_verb(cli: &Cli, verb: PaxosKvDataVerb) -> ExitCode { match verb { - KvDataVerb::Put { + PaxosKvDataVerb::Put { store, group, key, @@ -114,7 +114,7 @@ pub async fn run_kv_data_verb(cli: &Cli, verb: KvDataVerb) -> ExitCode { } } } - KvDataVerb::Get { store, group, key } => { + PaxosKvDataVerb::Get { store, group, key } => { let (store_id, group_id) = match parse_store_group(&store, &group) { Ok(ids) => ids, Err(c) => return c, @@ -142,7 +142,7 @@ pub async fn run_kv_data_verb(cli: &Cli, verb: KvDataVerb) -> ExitCode { } } } - KvDataVerb::Delete { store, group, key } => { + PaxosKvDataVerb::Delete { store, group, key } => { let (store_id, group_id) = match parse_store_group(&store, &group) { Ok(ids) => ids, Err(c) => return c, @@ -164,7 +164,7 @@ pub async fn run_kv_data_verb(cli: &Cli, verb: KvDataVerb) -> ExitCode { } } } - KvDataVerb::Scan { + PaxosKvDataVerb::Scan { store, group, prefix, @@ -200,7 +200,7 @@ pub async fn run_kv_data_verb(cli: &Cli, verb: KvDataVerb) -> ExitCode { } } } - KvDataVerb::Snapshot(sv) => run_snapshot_verb(cli, sv).await, + PaxosKvDataVerb::Snapshot(sv) => run_snapshot_verb(cli, sv).await, } } diff --git a/app/crowdb-cli/src/commands/kv/logical.rs b/app/crowdb-cli/src/commands/paxos_kv/logical.rs similarity index 100% rename from app/crowdb-cli/src/commands/kv/logical.rs rename to app/crowdb-cli/src/commands/paxos_kv/logical.rs diff --git a/app/crowdb-cli/src/commands/port_alloc.rs b/app/crowdb-cli/src/commands/port_alloc.rs index 2785e8352..cc1e212a5 100644 --- a/app/crowdb-cli/src/commands/port_alloc.rs +++ b/app/crowdb-cli/src/commands/port_alloc.rs @@ -18,16 +18,12 @@ use crowdb_protocol::ServicePort; pub enum ServiceArg { #[value(name = "kv-mgmt")] KvMgmt, - #[value(name = "kv-listen")] - KvListen, - #[value(name = "diskdb-listen")] - DiskdbListen, + #[value(name = "kv-rpc")] + KvRpc, #[value(name = "diskdb-http")] DiskdbHttp, #[value(name = "diskdb-rpc")] DiskdbRpc, - #[value(name = "chunkdb-listen")] - ChunkdbListen, #[value(name = "chunkdb-http")] ChunkdbHttp, #[value(name = "chunkdb-rpc")] @@ -49,12 +45,10 @@ pub enum ServiceArg { impl From<ServiceArg> for ServicePort { fn from(arg: ServiceArg) -> Self { match arg { - ServiceArg::KvMgmt => Self::KvServerMgmt, - ServiceArg::KvListen => Self::KvServerListen, - ServiceArg::DiskdbListen => Self::DiskdbListen, + ServiceArg::KvMgmt => Self::PaxosKvServerMgmt, + ServiceArg::KvRpc => Self::PaxosKvServerRpc, ServiceArg::DiskdbHttp => Self::DiskdbHttp, ServiceArg::DiskdbRpc => Self::DiskdbRpc, - ServiceArg::ChunkdbListen => Self::ChunkdbListen, ServiceArg::ChunkdbHttp => Self::ChunkdbHttp, ServiceArg::ChunkdbRpc => Self::ChunkdbRpc, ServiceArg::DiskioRpc => Self::DiskioRpc, @@ -79,7 +73,7 @@ pub struct PortAllocArgs { #[arg(long, default_value_t = 0)] pub offset: u16, - /// Service type (e.g. "kv-mgmt", "kv-listen", "diskdb-rpc"). + /// Service type (e.g. "kv-mgmt", "kv-rpc", "diskdb-rpc"). #[arg(long)] pub service: Option<ServiceArg>, diff --git a/app/crowdb-cli/src/main.rs b/app/crowdb-cli/src/main.rs index 7e777130d..f4d5792fb 100644 --- a/app/crowdb-cli/src/main.rs +++ b/app/crowdb-cli/src/main.rs @@ -33,9 +33,9 @@ use std::process::ExitCode; use clap::{Parser, Subcommand}; use commands::{ - run_bench_verb, run_cluster_verb, run_group_verb, run_kv_data_verb, run_port_alloc, run_replica_verb, - run_s3_verb, run_store_verb, BenchVerb, ClusterVerb, GroupVerb, KvDataVerb, PortAllocArgs, ReplicaVerb, - S3Verb, StoreVerb, + run_bench_verb, run_cluster_verb, run_group_verb, run_paxos_kv_data_verb, run_port_alloc, + run_replica_verb, run_s3_verb, run_store_verb, BenchVerb, ClusterVerb, GroupVerb, PaxosKvDataVerb, + PortAllocArgs, ReplicaVerb, S3Verb, StoreVerb, }; #[derive(Parser, Debug)] @@ -66,7 +66,7 @@ impl Cli { fn benchmark_slug(&self) -> Option<&'static str> { match &self.command { Domain::Bench { verb } => Some(match verb { - BenchVerb::Kv(_) => "bench-kv", + BenchVerb::PaxosKv(_) => "bench-paxos-kv", BenchVerb::Diskdb(_) => "bench-diskdb", BenchVerb::Chunkdb(_) => "bench-chunkdb", BenchVerb::Chunkio(_) => "bench-chunkio", @@ -85,10 +85,10 @@ enum Domain { #[command(subcommand)] verb: ClusterVerb, }, - /// Logical KV topology and data operations. - Kv { + /// Logical Paxos-KV topology and data operations. + PaxosKv { #[command(subcommand)] - verb: KvVerb, + verb: PaxosKvVerb, }, /// Load injection only. Bench { @@ -110,7 +110,7 @@ enum Domain { } #[derive(Subcommand, Debug)] -enum KvVerb { +enum PaxosKvVerb { #[command(subcommand)] Store(StoreVerb), #[command(subcommand)] @@ -118,7 +118,7 @@ enum KvVerb { #[command(subcommand)] Replica(ReplicaVerb), #[command(subcommand)] - Data(KvDataVerb), + Data(PaxosKvDataVerb), } fn main() -> ExitCode { @@ -161,8 +161,8 @@ fn main() -> ExitCode { "crowdb-cli", 30, 1, - "warn,crowdb_cli=info,crowdb_console_shared=info,crowdb_kv_client=info", - "warn,crowdb_cli=info,crowdb_console_shared=info,crowdb_kv_client=info", + "warn,crowdb_cli=info,crowdb_console_shared=info,crowdb_paxos_kv_client=info", + "warn,crowdb_cli=info,crowdb_console_shared=info,crowdb_paxos_kv_client=info", ) .ok(); crowdb_rpc_ffi::init_logging( @@ -238,11 +238,11 @@ async fn dispatch(mut cli: Cli) -> ExitCode { ); match command { Domain::Cluster { verb } => run_cluster_verb(&cli, verb).await, - Domain::Kv { verb } => match verb { - KvVerb::Store(sv) => run_store_verb(&cli, sv).await, - KvVerb::Group(gv) => run_group_verb(&cli, gv).await, - KvVerb::Replica(rv) => run_replica_verb(&cli, rv).await, - KvVerb::Data(dv) => run_kv_data_verb(&cli, dv).await, + Domain::PaxosKv { verb } => match verb { + PaxosKvVerb::Store(sv) => run_store_verb(&cli, sv).await, + PaxosKvVerb::Group(gv) => run_group_verb(&cli, gv).await, + PaxosKvVerb::Replica(rv) => run_replica_verb(&cli, rv).await, + PaxosKvVerb::Data(dv) => run_paxos_kv_data_verb(&cli, dv).await, }, Domain::Bench { verb } => run_bench_verb(&cli, verb).await, Domain::S3 { verb } => run_s3_verb(&cli, verb).await, diff --git a/app/crowdb-cli/tests/cluster_cli_test.rs b/app/crowdb-cli/tests/cluster_cli_test.rs index e0cac2801..4fae1790f 100644 --- a/app/crowdb-cli/tests/cluster_cli_test.rs +++ b/app/crowdb-cli/tests/cluster_cli_test.rs @@ -2,7 +2,7 @@ //! CLI e2e for cluster observation: `cluster status` and `cluster //! topology` through a system-group endpoint against a real -//! `crowdb-kv-server` with the system group initialized. +//! `crowdb-paxos-kv-server` with the system group initialized. mod common; @@ -68,7 +68,7 @@ fn retired_storage_process_commands_are_absent_and_measurements_remain() { assert_eq!(output.status.code(), Some(2), "{arguments:?}: {output:?}"); assert!(String::from_utf8_lossy(&output.stderr).contains("unrecognized subcommand")); } - for family in ["kv", "diskdb", "chunkdb", "chunkio", "s3"] { + for family in ["paxos-kv", "diskdb", "chunkdb", "chunkio", "s3"] { let output = Command::new(&cli) .args(["bench", family, "--help"]) .output() diff --git a/app/crowdb-cli/tests/cluster_lifecycle_test.rs b/app/crowdb-cli/tests/cluster_lifecycle_test.rs index c871e83ab..0f72e947b 100644 --- a/app/crowdb-cli/tests/cluster_lifecycle_test.rs +++ b/app/crowdb-cli/tests/cluster_lifecycle_test.rs @@ -188,8 +188,14 @@ async fn default_single_node_start_seals_identity_and_resumes_without_duplicate_ ); let original = fs::read(test.receipt("local")).unwrap(); let accepted: Value = serde_json::from_slice(&original).unwrap(); - assert_eq!(accepted["version"], 2); - for (internal, offset) in [(9090, 0), (16000, 1), (17000, 2), (18000, 3)] { + assert_eq!(accepted["version"], 3); + for internal in [ + crowdb_protocol::port::ports::WEB_BASE, + crowdb_protocol::port::ports::ACCESS_SERVER_HTTP_BASE, + crowdb_protocol::port::ports::ACCESS_SERVER_ICEBERG_HTTP_BASE, + crowdb_protocol::port::ports::ACCESS_SERVER_DATASET_HTTP_BASE, + ] { + let offset = internal - crowdb_protocol::port::ports::WEB_BASE; assert_eq!( test.state()["container"]["crowdb-local-1"]["HostConfig"]["PortBindings"] [format!("{internal}/tcp")][0]["HostPort"], @@ -225,43 +231,59 @@ async fn default_single_node_start_seals_identity_and_resumes_without_duplicate_ #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn legacy_receipts_retain_protocol_ports_on_resume_and_recreation() { - let test = TestDocker::new(); - let (port, server) = serve("local", 1, true).await; - test.run( - None, - &["start", "--port-base", &port.to_string(), "--wait", "0"], - 0, - ); - let path = test.receipt("local"); - let mut legacy: Value = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); - legacy["version"] = json!(1); - let accepted = serde_json::to_vec(&legacy).unwrap(); - fs::write(&path, &accepted).unwrap(); - let ports: serde_json::Map<_, _> = (0_u16..4) - .map(|offset| { - ( - format!("{}/tcp", 9090 + offset), - json!([{"HostIp":"127.0.0.1","HostPort":(port+offset).to_string()}]), - ) - }) - .collect(); - let mut state = test.state(); - state["container"]["crowdb-local-1"]["HostConfig"]["PortBindings"] = json!(ports); - test.save(&state); - test.run(None, &["stop"], 0); - test.run(None, &["start", "--wait", "0"], 0); - assert_eq!(fs::read(&path).unwrap(), accepted); - let mut state = test.state(); - state["container"] = json!({}); - test.save(&state); - test.run(None, &["start", "--wait", "0"], 0); - assert_eq!(fs::read(&path).unwrap(), accepted); - assert_eq!( - test.state()["container"]["crowdb-local-1"]["HostConfig"]["PortBindings"], - json!(ports) - ); - assert_eq!(test.state()["volume"].as_object().unwrap().len(), 1); - server.abort(); + for version in [1, 2] { + let test = TestDocker::new(); + let (port, server) = serve("local", 1, true).await; + test.run( + None, + &["start", "--port-base", &port.to_string(), "--wait", "0"], + 0, + ); + let path = test.receipt("local"); + let mut legacy: Value = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); + legacy["version"] = json!(version); + let accepted = serde_json::to_vec(&legacy).unwrap(); + fs::write(&path, &accepted).unwrap(); + let internal = if version == 1 { + [9090, 9091, 9092, 9093] + } else { + [9090, 16000, 17000, 18000] + }; + let ports: serde_json::Map<_, _> = internal + .into_iter() + .zip(0_u16..) + .map(|(internal, offset)| { + ( + format!("{internal}/tcp"), + json!([{"HostIp":"127.0.0.1","HostPort":(port+offset).to_string()}]), + ) + }) + .collect(); + let mut state = test.state(); + state["container"]["crowdb-local-1"]["HostConfig"]["PortBindings"] = json!(ports); + state["container"]["crowdb-local-1"]["Config"]["Env"] = json!([ + "CROWDB_CLUSTER=local", + "CROWDB_INITIAL_VOTERS=1", + format!("CROWDB_S3_PUBLIC_URI=http://127.0.0.1:{}", port + 1), + format!("CROWDB_ICEBERG_PUBLIC_URI=http://127.0.0.1:{}", port + 2), + format!("CROWDB_DATASET_PUBLIC_URI=http://127.0.0.1:{}", port + 3), + ]); + test.save(&state); + test.run(None, &["stop"], 0); + test.run(None, &["start", "--wait", "0"], 0); + assert_eq!(fs::read(&path).unwrap(), accepted); + let mut state = test.state(); + state["container"] = json!({}); + test.save(&state); + test.run(None, &["start", "--wait", "0"], 0); + assert_eq!(fs::read(&path).unwrap(), accepted); + assert_eq!( + test.state()["container"]["crowdb-local-1"]["HostConfig"]["PortBindings"], + json!(ports) + ); + assert_eq!(test.state()["volume"].as_object().unwrap().len(), 1); + server.abort(); + } } #[tokio::test(flavor = "multi_thread", worker_threads = 2)] diff --git a/app/crowdb-cli/tests/cluster_target_test.rs b/app/crowdb-cli/tests/cluster_target_test.rs index 049d2f0db..db8ff14bb 100644 --- a/app/crowdb-cli/tests/cluster_target_test.rs +++ b/app/crowdb-cli/tests/cluster_target_test.rs @@ -20,7 +20,7 @@ fn run(root: &Path, selected: &[&str]) -> String { .env_remove("CROWDB_SYSTEM_PORT") .args(selected) .args([ - "kv", + "paxos-kv", "data", "put", "--store", @@ -117,7 +117,6 @@ fn cli_rejects_removed_system_address_flags() { #[test] fn retained_workloads_require_a_target_before_any_runtime_or_native_effect() { for args in [ - vec!["bench", "kv", "prepare", "--keys", "1"], vec!["bench", "diskdb", "allocate"], vec!["bench", "chunkdb", "allocate"], vec!["bench", "chunkio", "write"], diff --git a/app/crowdb-cli/tests/cluster_workloads_test.rs b/app/crowdb-cli/tests/cluster_workloads_test.rs index 999e66937..956f1fe75 100644 --- a/app/crowdb-cli/tests/cluster_workloads_test.rs +++ b/app/crowdb-cli/tests/cluster_workloads_test.rs @@ -7,7 +7,7 @@ use crowdb_console_shared::deployment::target; use crowdb_diskdb_client::{DiskdbClient, DiskdbRpcTransport}; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient, ServiceRegistryClient}; use crowdb_protocol::{ common::ChunkId, diskdb::rpc::{AllocateBlocksRequest, FreeBlocksRequest}, @@ -23,7 +23,7 @@ async fn mixed_disk_workload_normalizes_frees_predating_its_baseline() { .await .unwrap(); selected.require_native_transport().await.unwrap(); - let kv = Arc::new(CrowdbKvClient::new(ClientConfig::new( + let kv = Arc::new(PaxosKvClient::new(ClientConfig::new( selected.identity.group0_management_endpoints.clone(), ))); kv.seed_leader(0, 0, selected.identity.group0_rpc_endpoints[0].clone()); diff --git a/app/crowdb-cli/tests/common/target_commands.rs b/app/crowdb-cli/tests/common/target_commands.rs index c1efeaa40..dd4433e01 100644 --- a/app/crowdb-cli/tests/common/target_commands.rs +++ b/app/crowdb-cli/tests/common/target_commands.rs @@ -10,13 +10,13 @@ pub fn test_operations() -> Vec<Vec<&'static str>> { vec!["cluster", "disk-group", "list"], vec!["cluster", "disk", "list"], vec!["cluster", "topology", "--node", "1"], - vec!["kv", "store", "list"], - vec!["kv", "group", "list", "--store", "0"], + vec!["paxos-kv", "store", "list"], + vec!["paxos-kv", "group", "list", "--store", "0"], vec![ - "kv", "replica", "add", "--store", "0", "--group", "1", "--node", "2", + "paxos-kv", "replica", "add", "--store", "0", "--group", "1", "--node", "2", ], vec![ - "kv", + "paxos-kv", "data", "put", "--store", @@ -29,7 +29,6 @@ pub fn test_operations() -> Vec<Vec<&'static str>> { "untouched", ], vec!["s3", "bucket", "put", "target-marker"], - vec!["bench", "kv", "prepare", "--keys", "1"], vec!["bench", "diskdb", "allocate"], vec!["bench", "chunkdb", "allocate"], vec!["bench", "chunkio", "write"], diff --git a/app/crowdb-cli/tests/direct_cli_test.rs b/app/crowdb-cli/tests/direct_cli_test.rs index b7559e462..e2846bba4 100644 --- a/app/crowdb-cli/tests/direct_cli_test.rs +++ b/app/crowdb-cli/tests/direct_cli_test.rs @@ -8,7 +8,7 @@ fn retired_process_commands_and_selectors_fail_before_runtime_access() { let root = crowdb_test_harness::test_dirs::tempdir_in_test_data("cli-removed-paths"); for args in [ vec!["launch", "list"], - vec!["kv", "server", "list"], + vec!["paxos-kv", "server", "list"], vec!["cluster", "init"], vec!["cluster", "local-deploy"], vec!["cluster", "destroy"], diff --git a/app/crowdb-cli/tests/kv_bench_test.rs b/app/crowdb-cli/tests/kv_bench_test.rs new file mode 100644 index 000000000..51263b3d8 --- /dev/null +++ b/app/crowdb-cli/tests/kv_bench_test.rs @@ -0,0 +1,237 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +#[path = "../src/commands/bench/paxos_kv/counters.rs"] +mod counters; +#[path = "../src/commands/bench/paxos_kv/deployment/memory.rs"] +mod memory; +#[path = "../src/commands/bench/paxos_kv/verification.rs"] +mod verification; +#[path = "../src/commands/bench/workload.rs"] +mod workload; +use serde_json::{json, Value}; +use std::time::Duration; +use workload::{LoadOptions, Window}; + +#[test] +fn cgroup_memory_limit_accepts_unlimited_without_hiding_invalid_values() { + assert_eq!(memory::parse_limit("max\n").unwrap(), None); + assert_eq!( + memory::parse_limit("21474836480\n").unwrap(), + Some(21_474_836_480) + ); + assert!(memory::parse_limit("").is_err()); + assert!(memory::parse_limit("unknown").is_err()); + assert!(memory::parse_limit("-1").is_err()); +} + +fn options() -> LoadOptions { + LoadOptions { + warmup_secs: 1, + duration_secs: 1, + tail_secs: 1, + drain_timeout_secs: 1, + loader_num: 2, + output: None, + } +} + +#[tokio::test] +async fn continuous_load_separates_boundaries_and_drains() { + let options = options(); + options.validate().unwrap(); + let result = workload::run_until( + &options, + Window::new(&options), + |_, _| async { + tokio::time::sleep(Duration::from_millis(120)).await; + Ok(64) + }, + std::future::pending(), + ) + .await; + assert!(result.errors.is_empty(), "{:?}", result.errors); + assert_eq!(result.report["incomplete"], 0); + assert!(result.report["boundary_crossings"].as_u64().unwrap() >= 4); + let middle = &result.report["phases"][1]; + assert!(result.report["measured_operations"].as_u64().unwrap() < middle["completed"].as_u64().unwrap()); + for phase in result.report["phases"].as_array().unwrap() { + assert_eq!(phase["admitted"], phase["completed"]); + } +} + +#[tokio::test] +async fn failures_in_warmup_and_tail_invalidate_run() { + let options = options(); + let window = Window::new(&options); + let result = workload::run_until( + &options, + window, + move |_, _| async move { + tokio::time::sleep(Duration::from_millis(50)).await; + if window.phase(tokio::time::Instant::now()) == 1 { + Ok(1) + } else { + Err("injected phase error".into()) + } + }, + std::future::pending(), + ) + .await; + assert!(!result.errors.is_empty()); + assert!(result.report["phases"][0]["failed"].as_u64().unwrap() > 0); + assert!(result.report["phases"][2]["failed"].as_u64().unwrap() > 0); +} + +#[tokio::test] +async fn drain_timeout_preserves_incomplete_and_joins_workers() { + let mut options = options(); + options.warmup_secs = 0; + options.tail_secs = 0; + let result = workload::run_until( + &options, + Window::new(&options), + |_, _| async { std::future::pending::<Result<u64, String>>().await }, + std::future::pending(), + ) + .await; + assert_eq!(result.report["incomplete"], 2); + assert!(result.errors.iter().any(|e| e.contains("drain timeout"))); +} + +#[tokio::test] +async fn worker_panics_are_failures() { + let options = options(); + let result = workload::run_until( + &options, + Window::new(&options), + |_, _| async { + panic!("injected worker panic"); + #[allow(unreachable_code)] + Ok(0) + }, + std::future::pending(), + ) + .await; + assert_eq!(result.errors.len(), 2); + assert_eq!(result.report["incomplete"], 2); +} + +fn snapshot(elections: u64, completed: u64, failed: u64) -> Value { + let points:Vec<_>=[("paxos.elections.c",elections),("maintenance.snapshot.success.c",completed),("maintenance.snapshot.failure.c",failed)] + .into_iter().map(|(suffix,value)|json!({"name":format!("s.0.g.1.{suffix}"),"fields":[{"key":"total","value":value}]})).collect(); + json!({"metrics":[{"metrics":points},{"metrics":points},{"metrics":points}]}) +} +#[test] +fn snapshot_sentinel_uses_deltas_and_rejects_missing_or_reset_counters() { + let before = snapshot(2, 10, 1); + assert_eq!( + counters::sentinel(0, 1, &before, &snapshot(2, 11, 1)).unwrap()["passed"], + true + ); + for after in [snapshot(2, 10, 1), snapshot(3, 11, 1), snapshot(2, 11, 2)] { + assert_eq!( + counters::sentinel(0, 1, &before, &after).unwrap()["passed"], + false + ); + } + assert!(counters::sentinel(0, 1, &before, &snapshot(1, 11, 1)).is_err()); + assert!(counters::sentinel(0, 1, &before, &json!({})).is_err()); + assert!(counters::deltas(&json!({}), &json!({})).is_err()); +} + +#[test] +fn container_bench_rejects_missing_deployment_and_console_target() { + for (extra, message) in [ + (vec![], "three distinct --container"), + ( + vec!["--cluster", "missing"], + "bench paxos-kv uses container --endpoint", + ), + ] { + let dir = crowdb_test_harness::test_dirs::tempdir_in_test_data("kv-bench-contract"); + let output = std::process::Command::new(env!("CARGO_BIN_EXE_crowdb-cli")) + .env_remove("CROWDB_CLUSTER") + .env("CROWDB_RUNTIME_ROOT", dir.path()) + .args(extra) + .args(["bench", "paxos-kv", "read"]) + .output() + .unwrap(); + assert_eq!(output.status.code(), Some(2)); + assert!( + String::from_utf8_lossy(&output.stderr).contains(message), + "{output:?}" + ); + } +} + +#[tokio::test] +async fn resource_guard_cancels_inflight_work_and_fails_the_run() { + let options = options(); + let result = workload::run_until( + &options, + Window::new(&options), + |_, _| async { std::future::pending::<Result<u64, String>>().await }, + async { + tokio::time::sleep(Duration::from_millis(30)).await; + "memory guard stopped load".to_owned() + }, + ) + .await; + assert_eq!(result.errors, ["memory guard stopped load"]); + assert_eq!(result.report["incomplete"], 2); + assert_eq!(result.report["measured_operations"], 0); + assert!(result.report["elapsed_secs_including_drain"].as_f64().unwrap() < 1.0); +} + +#[tokio::test] +async fn failed_load_does_not_poll_the_dataset_verification() { + let mut errors = vec!["memory guard stopped load".to_owned()]; + let mut requests = 0; + let started = verification::after_load(&mut errors, async { + requests += 1; + Ok(()) + }) + .await; + assert!(!started); + assert_eq!(requests, 0); + assert_eq!(errors, ["memory guard stopped load"]); + + errors.clear(); + assert!( + verification::after_load(&mut errors, async { + requests += 1; + Err("incorrect dataset bytes".to_owned()) + }) + .await + ); + assert_eq!(requests, 1); + assert_eq!(errors, ["incorrect dataset bytes"]); +} + +#[test] +fn native_bench_rejects_duplicate_pids_and_mixed_deployments() { + for (extra, message) in [ + ( + vec!["--server-pid", "1", "--server-pid", "1", "--server-pid", "1"], + "three distinct --server-pid", + ), + ( + vec!["--server-pid", "1", "--container", "node1"], + "cannot be used with", + ), + ] { + let dir = crowdb_test_harness::test_dirs::tempdir_in_test_data("kv-bench-native-contract"); + let output = std::process::Command::new(env!("CARGO_BIN_EXE_crowdb-cli")) + .env_remove("CROWDB_CLUSTER") + .env("CROWDB_RUNTIME_ROOT", dir.path()) + .args(["bench", "paxos-kv", "read"]) + .args(extra) + .output() + .unwrap(); + assert_eq!(output.status.code(), Some(2)); + assert!( + String::from_utf8_lossy(&output.stderr).contains(message), + "{output:?}" + ); + } +} diff --git a/app/crowdb-diskdb/Cargo.toml b/app/crowdb-diskdb/Cargo.toml index 546028912..e42fa6938 100644 --- a/app/crowdb-diskdb/Cargo.toml +++ b/app/crowdb-diskdb/Cargo.toml @@ -28,7 +28,7 @@ workspace = true [dependencies] crowdb-protocol = { path = "../../lib/crowdb-protocol" } -crowdb-kv-client = { path = "../../lib/crowdb-kv-client" } +crowdb-paxos-kv-client = { path = "../../lib/crowdb-paxos-kv-client" } crowdb-common = { path = "../../lib/crowdb-common/rust" } crowdb-rpc-ffi = { path = "../../lib/crowdb-rpc/ffi" } crowdb-chunkdb-client = { path = "../../lib/crowdb-chunkdb-client" } @@ -55,9 +55,10 @@ rand = "0.9.5" futures = "0.3" [dev-dependencies] -crowdb-kv = { path = "../../lib/crowdb-kv" } +crowdb-paxos-kv = { path = "../../lib/crowdb-paxos-kv" } crowdb-rpc-ffi = { path = "../../lib/crowdb-rpc/ffi", features = ["test-util"] } crowdb-test-harness = { path = "../../lib/crowdb-test-harness" } +crowdb-tree-ffi = { path = "../../lib/crowdb-tree/ffi" } serde_json = "1" tower = { version = "0.5", features = ["util"] } # Self dev-dependency: turns on the `test-util` feature for this diff --git a/app/crowdb-diskdb/conf/crowdb_diskdb_config.toml b/app/crowdb-diskdb/conf/crowdb_diskdb_config.toml index 5caf605fe..c2b9b4c64 100644 --- a/app/crowdb-diskdb/conf/crowdb_diskdb_config.toml +++ b/app/crowdb-diskdb/conf/crowdb_diskdb_config.toml @@ -1,3 +1,4 @@ +# Generated by tools/ports/generate.py from ports.rs; edit the .in template. # Canonical diskdb server config template. Tracked so the # `tracked_config_file_loads_and_validates` test guards against # stale/mismatched template edits. All sections are explicit; omitting @@ -5,10 +6,9 @@ [server] rpc_workers = 2 -listen_addr = "0.0.0.0:11000" -http_listen_addr = "0.0.0.0:11100" -rpc_listen_addr = "0.0.0.0:11200" -kv_server_mgmt_seeds = ["http://127.0.0.1:10000"] +http_listen_addr = "0.0.0.0:13135" +rpc_listen_addr = "0.0.0.0:14135" +kv_server_mgmt_seeds = ["http://127.0.0.1:10135"] [storage] zone_size_bytes = 17179869184 diff --git a/app/crowdb-diskdb/conf/crowdb_diskdb_config.toml.ports.in b/app/crowdb-diskdb/conf/crowdb_diskdb_config.toml.ports.in new file mode 100644 index 000000000..e0eeddaf5 --- /dev/null +++ b/app/crowdb-diskdb/conf/crowdb_diskdb_config.toml.ports.in @@ -0,0 +1,71 @@ +# Canonical diskdb server config template. Tracked so the +# `tracked_config_file_loads_and_validates` test guards against +# stale/mismatched template edits. All sections are explicit; omitting +# any non-`[server]` section is also valid (serde fills from `Default`). + +[server] +rpc_workers = 2 +http_listen_addr = "0.0.0.0:@DISKDB_HTTP_BASE@" +rpc_listen_addr = "0.0.0.0:@DISKDB_RPC_BASE@" +kv_server_mgmt_seeds = ["http://127.0.0.1:@PAXOS_KV_SERVER_MGMT_BASE@"] + +[storage] +zone_size_bytes = 17179869184 +block_size_bytes = 1048576 +allocate_granularity = 1048576 +zone_rotate_count = 4 +cas_retry_limit = 100 + +[heartbeat] +interval_secs = 10 +miss_threshold = 3 +temp_failure_timeout_secs = 900 + +[persistence] +# Coalesce concurrent frees without delaying the first request. +free_batch_enabled = false +# Maximum records per KV proposal; one oversized RPC remains atomic. +free_flush_max_batch = 256 +compaction_cadence_secs = 300 +snapshot_compaction_threshold = 4096 +load_concurrency = 16 + +[scanner] +scan_interval_secs = 600 +# Reconcile tentative ownership independently and at a deliberately low rate. +tentative_owner_scan_interval_secs = 600 +# Traverse disk-group, disk, and zone serially, pausing after every zone. +tentative_owner_zone_delay_secs = 3 +# Require a durable Absent observation to age for 24 hours before reclaiming. +tentative_owner_grace_secs = 86400 +reverify_delay_ms = 1000 + +[scanner.ghost] +detect = true +auto_correct = false + +[scanner.integrity] +verify = true +detect_owner_mismatch = false + +[rebalance] +enabled = true +plan_interval_secs = 300 +imbalance_threshold_pct = 20 +max_jobs_per_cycle = 1 +zone_delay_secs = 3 + +[allocator] +load_aware = true +load_aware_weight = "free_bytes" + +[sync] +group0_store_id = 0 +group0_group_id = 0 +sync_interval_secs = 10 + +[reporting] +interval_secs = 10 + +[notify] +notify_enabled = false diff --git a/app/crowdb-diskdb/src/ddb_config.rs b/app/crowdb-diskdb/src/ddb_config.rs index 9bc85a74d..3e0891111 100644 --- a/app/crowdb-diskdb/src/ddb_config.rs +++ b/app/crowdb-diskdb/src/ddb_config.rs @@ -7,7 +7,7 @@ use std::net::SocketAddr; use std::time::Duration; use crowdb_common::config::BaseConfig; -use crowdb_protocol::{DISKDB_HTTP_BASE, DISKDB_LISTEN_BASE, DISKDB_RPC_BASE, KV_SERVER_MGMT_BASE}; +use crowdb_protocol::{DISKDB_HTTP_BASE, DISKDB_RPC_BASE, PAXOS_KV_SERVER_MGMT_BASE}; use serde::{Deserialize, Serialize}; /// Top-level configuration for a diskdb instance. @@ -40,24 +40,21 @@ impl BaseConfig for DdbConfig { } } -/// main listener + HTTP + crowdb-rpc listen addresses. +/// HTTP management and crowdb-rpc listen addresses. #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(default)] pub struct ServerConfig { /// static: crowdb-rpc I/O workers serving this process. #[serde(default = "default_rpc_workers")] pub rpc_workers: u32, - /// static: main listen address. - pub listen_addr: String, /// static: HTTP management listen address. pub http_listen_addr: String, - /// static: crowdb-rpc listen address (R115 migration — runs alongside - /// the main listener during the mixed-rollout window). + /// static: crowdb-rpc listen address. pub rpc_listen_addr: String, /// static: unique instance ID (auto-generated UUID if absent). pub instance_id: Option<String>, /// static: HTTP management-API seed endpoints (`http://host:port`) - /// of the crowdb-kv-server(s) used to discover the system group + /// of the crowdb-paxos-kv-server(s) used to discover the system group /// (store 0, group 0) leader and, via `/topology`, the data-group /// leaders. The client refreshes this lazily on first use, so the /// leader does not need to be pre-seeded. At least one must be @@ -87,11 +84,10 @@ impl Default for ServerConfig { fn default() -> Self { Self { rpc_workers: default_rpc_workers(), - listen_addr: format!("0.0.0.0:{DISKDB_LISTEN_BASE}"), http_listen_addr: format!("0.0.0.0:{DISKDB_HTTP_BASE}"), rpc_listen_addr: format!("0.0.0.0:{DISKDB_RPC_BASE}"), instance_id: None, - kv_server_mgmt_seeds: vec![format!("http://127.0.0.1:{KV_SERVER_MGMT_BASE}")], + kv_server_mgmt_seeds: vec![format!("http://127.0.0.1:{PAXOS_KV_SERVER_MGMT_BASE}")], kv_pool_size: default_kv_pool_size(), kv_rpc_workers: default_kv_rpc_workers(), } @@ -462,12 +458,6 @@ pub fn validate(config: &DdbConfig) -> Result<(), String> { if config.persistence.load_concurrency == 0 { return Err("load_concurrency must be > 0".to_string()); } - if config.server.listen_addr.parse::<SocketAddr>().is_err() { - return Err(format!( - "listen_addr {:?} is not a valid SocketAddr", - config.server.listen_addr, - )); - } if config.server.http_listen_addr.parse::<SocketAddr>().is_err() { return Err(format!( "http_listen_addr {:?} is not a valid SocketAddr", diff --git a/app/crowdb-diskdb/src/ddb_kv_client.rs b/app/crowdb-diskdb/src/ddb_kv_client.rs index 5a0e7d06d..2904a7b16 100644 --- a/app/crowdb-diskdb/src/ddb_kv_client.rs +++ b/app/crowdb-diskdb/src/ddb_kv_client.rs @@ -1,7 +1,7 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -//! [`DdbKvClient`] — wraps [`CrowdbKvClient`] for put/delete/scan on +//! [`DdbKvClient`] — wraps [`PaxosKvClient`] for put/delete/scan on //! the disk-group's bound paxos data group. //! //! Uses `(store_id, group_id)` from `DdbDiskGroup.bind` (set by the @@ -13,7 +13,7 @@ use std::sync::Arc; use bytes::Bytes; -use crowdb_kv_client::{BatchOp, CrowdbKvClient, GetOutcome, JournalOp, ReadMode, Result}; +use crowdb_paxos_kv_client::{BatchOp, GetOutcome, JournalOp, PaxosKvClient, ReadMode, Result}; use crowdb_protocol::common::DiskId; use crowdb_protocol::diskdb::rpc::{ BusyBlockValue, FreeBlockValue, RecoveryScanProgressValue, RelocationJournalValue, ZoneValue, @@ -39,11 +39,11 @@ pub type Bind = (u64, u64); /// Client for the disk-group's bound paxos data group. /// /// All methods take `(store_id, group_id)` from `DdbDiskGroup.bind`. The -/// wrapped `CrowdbKvClient` must have its topology seeded with the +/// wrapped `PaxosKvClient` must have its topology seeded with the /// data-group leader endpoint. #[derive(Clone)] pub struct DdbKvClient { - kv: Arc<CrowdbKvClient>, + kv: Arc<PaxosKvClient>, metrics: Option<Arc<crate::metrics::DiskdbMetrics>>, require_fence: bool, disks: Option<Arc<Vec<Arc<crate::model::disk::DdbDisk>>>>, @@ -113,7 +113,7 @@ impl DdbKvClient { let mut journals = Vec::with_capacity(outcome.items.len()); for (raw_key, raw_value) in outcome.items { let key = RelocationJournalKey::from_bytes(&raw_key).map_err(|error| { - crowdb_kv_client::Error::SysdataDecode { + crowdb_paxos_kv_client::Error::SysdataDecode { key: format!("{raw_key:02x?}"), reason: error.to_string(), } @@ -137,7 +137,7 @@ impl DdbKvClient { .await? { GetOutcome::Found { value, .. } => bincode::deserialize(&value).map(Some).map_err(|error| { - crowdb_kv_client::Error::SysdataDecode { + crowdb_paxos_kv_client::Error::SysdataDecode { key: format!("{:02x?}", key.to_bytes()), reason: error.to_string(), } @@ -179,9 +179,9 @@ impl DdbKvClient { .map(|_| ()) } - /// Wrap a `CrowdbKvClient` for data-group access. + /// Wrap a `PaxosKvClient` for data-group access. #[must_use] - pub fn new(kv: CrowdbKvClient) -> Self { + pub fn new(kv: PaxosKvClient) -> Self { Self { kv: Arc::new(kv), metrics: None, @@ -191,9 +191,9 @@ impl DdbKvClient { } } - /// Wrap an already-shared `CrowdbKvClient`. + /// Wrap an already-shared `PaxosKvClient`. #[must_use] - pub fn from_shared(kv: Arc<CrowdbKvClient>) -> Self { + pub fn from_shared(kv: Arc<PaxosKvClient>) -> Self { Self { kv, metrics: None, @@ -210,9 +210,9 @@ impl DdbKvClient { self } - /// Access the underlying `CrowdbKvClient`. + /// Access the underlying `PaxosKvClient`. #[must_use] - pub fn kv(&self) -> &CrowdbKvClient { + pub fn kv(&self) -> &PaxosKvClient { &self.kv } @@ -238,11 +238,12 @@ impl DdbKvClient { .await?; match outcome { GetOutcome::Found { value, revision } => { - let bv = - bincode::deserialize(&value).map_err(|e| crowdb_kv_client::Error::SysdataDecode { + let bv = bincode::deserialize(&value).map_err(|e| { + crowdb_paxos_kv_client::Error::SysdataDecode { key: format!("{:02x?}", key.to_bytes()), reason: e.to_string(), - })?; + } + })?; Ok(Some((bv, revision))) } GetOutcome::NotFound => Ok(None), @@ -435,13 +436,12 @@ impl DdbKvClient { .get(store_id, group_id, &zone_bytes, ReadMode::Linearizable, None) .await?; if let GetOutcome::Found { value, .. } = zone_get { - records.zone_value = - Some( - ZoneValue::from_bytes(&value).map_err(|e| crowdb_kv_client::Error::SysdataDecode { - key: format!("{zone_bytes:02x?}"), - reason: e, - })?, - ); + records.zone_value = Some(ZoneValue::from_bytes(&value).map_err(|e| { + crowdb_paxos_kv_client::Error::SysdataDecode { + key: format!("{zone_bytes:02x?}"), + reason: e, + } + })?); } // 2. BusyBlock records @@ -521,19 +521,20 @@ impl DdbKvClient { .await?; let cutoff = free_scan.scan_cutoff; if free_scan.items.len() != free_scan.commit_slots.len() { - return Err(crowdb_kv_client::Error::SysdataDecode { + return Err(crowdb_paxos_kv_client::Error::SysdataDecode { key: format!("{free_prefix:02x?}"), reason: "bounded free scan item/commit-slot count mismatch".to_string(), }); } for ((key, value), commit_slot) in free_scan.items.iter().zip(&free_scan.commit_slots) { - let decoded_key = - FreeBlockKey::from_bytes(key).map_err(|error| crowdb_kv_client::Error::SysdataDecode { + let decoded_key = FreeBlockKey::from_bytes(key).map_err(|error| { + crowdb_paxos_kv_client::Error::SysdataDecode { key: format!("{key:02x?}"), reason: error.to_string(), - })?; + } + })?; let decoded_value = bincode::deserialize::<FreeBlockValue>(value).map_err(|error| { - crowdb_kv_client::Error::SysdataDecode { + crowdb_paxos_kv_client::Error::SysdataDecode { key: format!("{key:02x?}"), reason: error.to_string(), } @@ -551,19 +552,20 @@ impl DdbKvClient { .scan_bounded_at(store_id, group_id, &busy_prefix, &[], &[], 0, false, None, cutoff) .await?; if busy_scan.items.len() != busy_scan.commit_slots.len() { - return Err(crowdb_kv_client::Error::SysdataDecode { + return Err(crowdb_paxos_kv_client::Error::SysdataDecode { key: format!("{busy_prefix:02x?}"), reason: "bounded busy scan item/commit-slot count mismatch".to_string(), }); } for ((key, value), commit_slot) in busy_scan.items.iter().zip(&busy_scan.commit_slots) { - let decoded_key = - BusyBlockKey::from_bytes(key).map_err(|error| crowdb_kv_client::Error::SysdataDecode { + let decoded_key = BusyBlockKey::from_bytes(key).map_err(|error| { + crowdb_paxos_kv_client::Error::SysdataDecode { key: format!("{key:02x?}"), reason: error.to_string(), - })?; + } + })?; let decoded_value = bincode::deserialize::<BusyBlockValue>(value).map_err(|error| { - crowdb_kv_client::Error::SysdataDecode { + crowdb_paxos_kv_client::Error::SysdataDecode { key: format!("{key:02x?}"), reason: error.to_string(), } @@ -646,11 +648,12 @@ impl DdbKvClient { .await?; match outcome { GetOutcome::Found { value, .. } => { - let zv = - ZoneValue::from_bytes(&value).map_err(|e| crowdb_kv_client::Error::SysdataDecode { + let zv = ZoneValue::from_bytes(&value).map_err(|e| { + crowdb_paxos_kv_client::Error::SysdataDecode { key: format!("{:02x?}", key.to_bytes()), reason: e, - })?; + } + })?; Ok(Some(zv)) } GetOutcome::NotFound => Ok(None), @@ -684,7 +687,7 @@ impl DdbKvClient { /// Slot-ordered journal scan over `BusyBlockKey` ops for one zone /// (R73 recovery strategy 2 step c, scan 1). Wraps - /// `CrowdbKvClient::journal_scan` with `BusyBlockKey::prefix_for_zone`. + /// `PaxosKvClient::journal_scan` with `BusyBlockKey::prefix_for_zone`. /// Returns ops in slot order; the caller replays them to rebuild /// the bitmap. pub async fn journal_scan_busy( @@ -716,7 +719,7 @@ impl DdbKvClient { /// Slot-ordered journal scan over `FreeBlockKey` ops for one zone /// (R73 recovery strategy 2 step c, scan 2). Wraps - /// `CrowdbKvClient::journal_scan` with `FreeBlockKey::prefix_for_zone`. + /// `PaxosKvClient::journal_scan` with `FreeBlockKey::prefix_for_zone`. pub async fn journal_scan_free( &self, bind: Bind, @@ -785,7 +788,7 @@ impl DdbKvClient { match outcome { GetOutcome::Found { value, .. } => { let v = RecoveryScanProgressValue::from_bytes(&value).map_err(|e| { - crowdb_kv_client::Error::SysdataDecode { + crowdb_paxos_kv_client::Error::SysdataDecode { key: format!("{:02x?}", key.to_bytes()), reason: e, } @@ -814,7 +817,7 @@ impl DdbKvClient { } fn decode_relocation_journal(key: &RelocationJournalKey, bytes: &[u8]) -> Result<RelocationJournalValue> { - bincode::deserialize(bytes).map_err(|error| crowdb_kv_client::Error::SysdataDecode { + bincode::deserialize(bytes).map_err(|error| crowdb_paxos_kv_client::Error::SysdataDecode { key: format!("{:02x?}", key.to_bytes()), reason: error.to_string(), }) diff --git a/app/crowdb-diskdb/src/ddb_kv_client/disk_writes.rs b/app/crowdb-diskdb/src/ddb_kv_client/disk_writes.rs index e7fb192ad..c0624f9cd 100644 --- a/app/crowdb-diskdb/src/ddb_kv_client/disk_writes.rs +++ b/app/crowdb-diskdb/src/ddb_kv_client/disk_writes.rs @@ -42,7 +42,7 @@ impl DdbKvClient { .filter_map(disk_id) .any(|id| !disks.iter().any(|disk| disk.disk_id == id)) { - return Err(crowdb_kv_client::Error::Server( + return Err(crowdb_paxos_kv_client::Error::Server( "disk is absent from the captured membership".into(), )); } @@ -58,7 +58,7 @@ impl DdbKvClient { for permit in permits { permit.complete(); } - return Err(crowdb_kv_client::Error::Server( + return Err(crowdb_paxos_kv_client::Error::Server( "disk is administratively excluded; metadata write rejected".into(), )); }; diff --git a/app/crowdb-diskdb/src/ddb_kv_client/ownership.rs b/app/crowdb-diskdb/src/ddb_kv_client/ownership.rs index b078228f7..02ebb3944 100644 --- a/app/crowdb-diskdb/src/ddb_kv_client/ownership.rs +++ b/app/crowdb-diskdb/src/ddb_kv_client/ownership.rs @@ -3,7 +3,7 @@ //! Data-group fencing for `DiskGroup` management handovers. -use crowdb_kv_client::{BatchOp, Error, GetOutcome, ReadMode, Result}; +use crowdb_paxos_kv_client::{BatchOp, Error, GetOutcome, ReadMode, Result}; use crowdb_protocol::{ common::{DiskId, HwStatus}, key::DiskKey, @@ -44,7 +44,7 @@ impl DdbKvClient { group: &crate::model::disk_group::DdbDiskGroup, disk: &DiskId, ) -> Result<()> { - let hardware = crowdb_kv_client::HardwareClient::from_shared(self.kv.clone()); + let hardware = crowdb_paxos_kv_client::HardwareClient::from_shared(self.kv.clone()); let Some(fence) = &self.fence else { if self.require_fence { return Err(fenced()); diff --git a/app/crowdb-diskdb/src/ddb_kv_client/retained.rs b/app/crowdb-diskdb/src/ddb_kv_client/retained.rs index 26018bed2..996b86744 100644 --- a/app/crowdb-diskdb/src/ddb_kv_client/retained.rs +++ b/app/crowdb-diskdb/src/ddb_kv_client/retained.rs @@ -5,7 +5,7 @@ use super::{BinaryKey, Bind, BusyBlockKey, DdbKvClient, DiskId, FreeBlockKey, ZoneKey}; use crate::model::records::{BusyRecord, FreeRecord, ZoneRecords}; use bincode::Options; -use crowdb_kv_client::{Error, Result}; +use crowdb_paxos_kv_client::{Error, Result}; use crowdb_protocol::diskdb::rpc::ZoneValue; use crowdb_protocol::ZoneValueExt; use serde::de::DeserializeOwned; @@ -36,7 +36,7 @@ impl DdbKvClient { &self, key: &crowdb_protocol::key::DiskKey, ) -> Result<bool> { - let hardware = crowdb_kv_client::HardwareClient::from_shared(self.kv.clone()); + let hardware = crowdb_paxos_kv_client::HardwareClient::from_shared(self.kv.clone()); Ok(hardware.disk_rejoin_receipt(key).await?.is_some() || hardware.disk_online_operation(key).await?.is_some()) } diff --git a/app/crowdb-diskdb/src/ddb_kv_client/zone_init.rs b/app/crowdb-diskdb/src/ddb_kv_client/zone_init.rs index 6b5ed431b..858cc4d1b 100644 --- a/app/crowdb-diskdb/src/ddb_kv_client/zone_init.rs +++ b/app/crowdb-diskdb/src/ddb_kv_client/zone_init.rs @@ -8,7 +8,8 @@ use super::{ ZoneKey, ZoneValue, ZoneValueExt, }; -pub const MAX_ZONE_INIT_BATCH: usize = 128; +// Each proposal must also fit the native tree's reserved gap-repair window. +pub const MAX_ZONE_INIT_BATCH: usize = 64; impl DdbKvClient { /// Any snapshot, live allocation or free fact requires ordinary recovery. @@ -49,9 +50,9 @@ impl DdbKvClient { /// Rejects an empty or oversized batch and propagates ownership or write failures. pub async fn put_zone_batch(&self, bind: Bind, disk: &DiskId, values: &[(u32, ZoneValue)]) -> Result<()> { if values.is_empty() || values.len() > MAX_ZONE_INIT_BATCH { - return Err(crowdb_kv_client::Error::Server( - "zone initialization batch must contain 1..=128 records".into(), - )); + return Err(crowdb_paxos_kv_client::Error::Server(format!( + "zone initialization batch must contain 1..={MAX_ZONE_INIT_BATCH} records" + ))); } let ops: Vec<_> = values .iter() diff --git a/app/crowdb-diskdb/src/liveness/keepalive/observation.rs b/app/crowdb-diskdb/src/liveness/keepalive/observation.rs index ba36ef506..418cd4813 100644 --- a/app/crowdb-diskdb/src/liveness/keepalive/observation.rs +++ b/app/crowdb-diskdb/src/liveness/keepalive/observation.rs @@ -130,7 +130,7 @@ impl KeepAlive { &self, owner: &crowdb_protocol::DiskdbOwnerEntry, bind: (u64, u64), - ) -> crowdb_kv_client::Result<std::sync::Arc<crate::ddb_kv_client::OwnershipFence>> { + ) -> crowdb_paxos_kv_client::Result<std::sync::Arc<crate::ddb_kv_client::OwnershipFence>> { use crowdb_protocol::key::OwnerMapKey; let epoch = self.ensure_process_session().await?; let generation = self diff --git a/app/crowdb-diskdb/src/liveness/keepalive/online.rs b/app/crowdb-diskdb/src/liveness/keepalive/online.rs index 6543a0b38..8abd9a802 100644 --- a/app/crowdb-diskdb/src/liveness/keepalive/online.rs +++ b/app/crowdb-diskdb/src/liveness/keepalive/online.rs @@ -132,7 +132,7 @@ impl KeepAlive { // published before its strict background loader can expose allocations. group.remove_disk_from_memory(&old.disk_id); self.disk_add_init(group, old.disk_id, &disk); - Ok::<(), crowdb_kv_client::Error>(()) + Ok::<(), crowdb_paxos_kv_client::Error>(()) } .await; if let Err(error) = result { diff --git a/app/crowdb-diskdb/src/liveness/keepalive/scheduler.rs b/app/crowdb-diskdb/src/liveness/keepalive/scheduler.rs index 34e29e249..3016f28cc 100644 --- a/app/crowdb-diskdb/src/liveness/keepalive/scheduler.rs +++ b/app/crowdb-diskdb/src/liveness/keepalive/scheduler.rs @@ -12,7 +12,7 @@ use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, AtomicUsize, Ordering} use std::sync::{Arc, RwLock}; use crowdb_common::metrics::Counter; -use crowdb_kv_client::{HardwareClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{HardwareClient, ServiceRegistryClient}; use crowdb_protocol::common::{DiskGroupUsageSummary, DiskId, HwStatus}; use crowdb_protocol::diskdb::rpc::DiskValue; use crowdb_protocol::{DiskGroupId, NodeId, RackId}; diff --git a/app/crowdb-diskdb/src/liveness/keepalive/session.rs b/app/crowdb-diskdb/src/liveness/keepalive/session.rs index 1c6d341e5..15f47b988 100644 --- a/app/crowdb-diskdb/src/liveness/keepalive/session.rs +++ b/app/crowdb-diskdb/src/liveness/keepalive/session.rs @@ -3,7 +3,7 @@ //! Freeze one predecessor before claiming a new per-process epoch. use super::{KeepAlive, Ordering}; -use crowdb_kv_client::{Error, Result}; +use crowdb_paxos_kv_client::{Error, Result}; use std::sync::Arc; pub(super) struct Predecessor { diff --git a/app/crowdb-diskdb/src/liveness/notify.rs b/app/crowdb-diskdb/src/liveness/notify.rs index 61980d442..b1d862a54 100644 --- a/app/crowdb-diskdb/src/liveness/notify.rs +++ b/app/crowdb-diskdb/src/liveness/notify.rs @@ -10,7 +10,7 @@ use tokio::sync::mpsc; use tracing::{info, warn}; use crate::bg_task::StopHandle; -use crowdb_kv_client::WatchNotifyClient; +use crowdb_paxos_kv_client::WatchNotifyClient; use crowdb_protocol::DISKDB_WATCH_PREFIXES; /// Group-0 store + group ids (system group). diff --git a/app/crowdb-diskdb/src/main.rs b/app/crowdb-diskdb/src/main.rs index e1e76bfcb..2863448f4 100644 --- a/app/crowdb-diskdb/src/main.rs +++ b/app/crowdb-diskdb/src/main.rs @@ -24,8 +24,8 @@ use crowdb_diskdb::recovery::ZoneLoader; use crowdb_diskdb::scanner::{BusyBlockOwnerScanner, ScanState, ScannerTask, SegmentOwnerQuery}; use crowdb_diskdb::service::DiskdbRpcService; use crowdb_diskdb_client::{DiskdbClient, DiskdbRpcTransport}; -use crowdb_kv_client::{ - ClientConfig, CrowdbKvClient, DomainMonitorClient, HardwareClient, +use crowdb_paxos_kv_client::{ + ClientConfig, DomainMonitorClient, HardwareClient, PaxosKvClient, RangeBindingClient as KvRangeBindingClient, ServiceRegistryClient, WatchNotifyClient, }; use crowdb_protocol::chunk_kv::{EnsureDomainMonitorOutcome, EnsureDomainMonitorRequest}; @@ -39,10 +39,6 @@ struct Cli { #[arg(long)] config: String, - /// crowdb-rpc listen address (overrides config). - #[arg(long)] - listen_addr: Option<String>, - /// HTTP management listen address (overrides config). #[arg(long)] http_addr: Option<String>, @@ -51,10 +47,6 @@ struct Cli { #[arg(long)] rpc_listen_addr: Option<String>, - /// Main listener port (overrides the port in config `listen_addr`). - #[arg(long, value_parser = clap::value_parser!(u16).range(1..))] - listen_port: Option<u16>, - /// HTTP management port (overrides the port in config `http_listen_addr`). #[arg(long, value_parser = clap::value_parser!(u16).range(1..))] http_port: Option<u16>, @@ -198,7 +190,7 @@ async fn main() { // leader and data-group leaders are discovered lazily via the // kv-server HTTP management API seeds from config — no pre-seeding. // One client is shared across all service classes (hardware, - // service-registry, data-group) since `CrowdbKvClient` is fully + // service-registry, data-group) since `PaxosKvClient` is fully // interior-mutable; each service class takes it via `from_shared`. let kv_config = { let loaded = config.load(); @@ -207,7 +199,7 @@ async fn main() { client.rpc_workers = loaded.server.kv_rpc_workers; client }; - let kv_client = Arc::new(CrowdbKvClient::new(kv_config)); + let kv_client = Arc::new(PaxosKvClient::new(kv_config)); let (sys_store, sys_group) = kv_client.system_group(); info!( store_id = sys_store, @@ -323,14 +315,6 @@ async fn main() { let relocation_worker = Arc::new( RelocationWorker::new(owner_client.clone(), relocation_io).with_source_free(relocation_source_free), ); - let listen_addr: SocketAddr = config - .load() - .server - .listen_addr - .parse() - .expect("valid listen_addr"); - info!(%listen_addr, "RPC server listening (zone load pending)"); - // Build the crowdb-rpc server. The RpcServer listens on the RPC // port and dispatches to DiskdbRpcService handlers. let rpc_listen_addr: SocketAddr = config @@ -616,18 +600,12 @@ fn load_config(args: &Cli) -> DdbConfig { panic!("failed to load config file {config_path}: {e}"); }); - if let Some(addr) = &args.listen_addr { - config.server.listen_addr.clone_from(addr); - } if let Some(addr) = &args.http_addr { config.server.http_listen_addr.clone_from(addr); } if let Some(addr) = &args.rpc_listen_addr { config.server.rpc_listen_addr.clone_from(addr); } - if let Some(port) = args.listen_port { - config.server.listen_addr = replace_port(&config.server.listen_addr, port); - } if let Some(port) = args.http_port { config.server.http_listen_addr = replace_port(&config.server.http_listen_addr, port); } @@ -693,7 +671,7 @@ fn resolve_instance_id(config: &arc_swap::ArcSwap<DdbConfig>) -> u64 { } // 3. Generate new and persist. - let id = crowdb_kv_client::new_client_id(); + let id = crowdb_paxos_kv_client::new_client_id(); if let Err(e) = std::fs::create_dir_all(state_dir) { warn!(error = %e, "failed to create state dir; instance_id will not persist across restarts"); } else if let Err(e) = std::fs::write(&state_file, id.to_string()) { diff --git a/app/crowdb-diskdb/src/model/alloc.rs b/app/crowdb-diskdb/src/model/alloc.rs index 5b9187716..92868352a 100644 --- a/app/crowdb-diskdb/src/model/alloc.rs +++ b/app/crowdb-diskdb/src/model/alloc.rs @@ -35,7 +35,7 @@ fn matching_tentative(dg: &DdbDiskGroup, segment: &Segment, disk_id: DiskId) -> #[derive(Clone, Debug)] pub enum FreeError { /// KV client error during lookup or persist. - Kv(Arc<crowdb_kv_client::Error>), + Kv(Arc<crowdb_paxos_kv_client::Error>), /// Block is not busy (no `BusyBlockKey` exists) — double-free or /// never allocated. NotBusy { @@ -71,10 +71,10 @@ impl std::fmt::Display for FreeError { impl std::error::Error for FreeError {} -impl From<crowdb_kv_client::Error> for FreeError { - fn from(e: crowdb_kv_client::Error) -> Self { +impl From<crowdb_paxos_kv_client::Error> for FreeError { + fn from(e: crowdb_paxos_kv_client::Error) -> Self { match e { - crowdb_kv_client::Error::OutcomeUnknown => Self::OutcomeUnknown, + crowdb_paxos_kv_client::Error::OutcomeUnknown => Self::OutcomeUnknown, error => Self::Kv(Arc::new(error)), } } @@ -389,7 +389,7 @@ pub async fn free_block( let scoped_kv = kv.for_group(dg); let kv = &scoped_kv; let disk_id = segment.disk_id.ok_or_else(|| { - FreeError::Kv(Arc::new(crowdb_kv_client::Error::SysdataDecode { + FreeError::Kv(Arc::new(crowdb_paxos_kv_client::Error::SysdataDecode { key: "segment.disk_id".to_string(), reason: "missing disk_id in Segment".to_string(), })) @@ -529,7 +529,7 @@ pub fn prepare_free( let mut records = Vec::with_capacity(unique.len()); for segment in &unique { let disk_id = segment.disk_id.ok_or_else(|| { - FreeError::Kv(Arc::new(crowdb_kv_client::Error::SysdataDecode { + FreeError::Kv(Arc::new(crowdb_paxos_kv_client::Error::SysdataDecode { key: "segment.disk_id".to_string(), reason: "missing disk_id in Segment".to_string(), })) @@ -630,7 +630,7 @@ pub async fn commit_blocks( let mut unique = Vec::with_capacity(segments.len()); for seg in segments { let disk_id = seg.disk_id.ok_or_else(|| { - FreeError::Kv(Arc::new(crowdb_kv_client::Error::SysdataDecode { + FreeError::Kv(Arc::new(crowdb_paxos_kv_client::Error::SysdataDecode { key: "segment.disk_id".to_string(), reason: "missing disk_id in Segment".to_string(), })) @@ -699,7 +699,7 @@ pub async fn mark_blocks_corrupt( let mut marked = 0u32; for segment in segments { let disk_id = segment.disk_id.ok_or_else(|| { - FreeError::Kv(Arc::new(crowdb_kv_client::Error::SysdataDecode { + FreeError::Kv(Arc::new(crowdb_paxos_kv_client::Error::SysdataDecode { key: "segment.disk_id".into(), reason: "missing disk ID".into(), })) diff --git a/app/crowdb-diskdb/src/rebalance/relocation.rs b/app/crowdb-diskdb/src/rebalance/relocation.rs index a36b393e3..616399922 100644 --- a/app/crowdb-diskdb/src/rebalance/relocation.rs +++ b/app/crowdb-diskdb/src/rebalance/relocation.rs @@ -12,7 +12,7 @@ use bytes::Bytes; use crowdb_chunkdb_client::ChunkdbClient; use crowdb_diskdb_client::DiskdbClient; use crowdb_diskio_client::{DiskId as IoDiskId, DiskioClient, DiskioClientConfig, Durability, SegmentTarget}; -use crowdb_kv_client::{HardwareClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{HardwareClient, ServiceRegistryClient}; use crowdb_protocol::chunk_task::relocation_operation_id; use crowdb_protocol::chunkdb::rpc::{RelocateSegmentHandoffRequest, RelocationHandoffDisposition}; use crowdb_protocol::common::DiskId; diff --git a/app/crowdb-diskdb/src/recovery.rs b/app/crowdb-diskdb/src/recovery.rs index a17bb93fc..49aa9b54f 100644 --- a/app/crowdb-diskdb/src/recovery.rs +++ b/app/crowdb-diskdb/src/recovery.rs @@ -45,7 +45,7 @@ pub use full_scan::rebuild_zone_bitmap_full_scan; #[derive(Debug)] pub enum ZoneLoadError { /// KV client error. - Kv(crowdb_kv_client::Error), + Kv(crowdb_paxos_kv_client::Error), /// A `JournalScan` returned `KV_ERROR_JOURNAL_SCAN_GC_GAP` — slots /// already GC'd below the WAL trim point. The caller should fall /// back to strategy 1 (full scan) for this zone. @@ -70,8 +70,8 @@ impl std::fmt::Display for ZoneLoadError { impl std::error::Error for ZoneLoadError {} -impl From<crowdb_kv_client::Error> for ZoneLoadError { - fn from(e: crowdb_kv_client::Error) -> Self { +impl From<crowdb_paxos_kv_client::Error> for ZoneLoadError { + fn from(e: crowdb_paxos_kv_client::Error) -> Self { Self::Kv(e) } } diff --git a/app/crowdb-diskdb/src/recovery/disk_recovery.rs b/app/crowdb-diskdb/src/recovery/disk_recovery.rs index 8174a2eed..17856edbb 100644 --- a/app/crowdb-diskdb/src/recovery/disk_recovery.rs +++ b/app/crowdb-diskdb/src/recovery/disk_recovery.rs @@ -291,7 +291,7 @@ impl RecoveryScanTask { /// Scan one zone: read zone records, extract live busy blocks, /// call the placeholder recovery function. Returns the count of /// impacted blocks in this zone. - async fn scan_zone(&self, zone_index: u32) -> Result<u32, crowdb_kv_client::Error> { + async fn scan_zone(&self, zone_index: u32) -> Result<u32, crowdb_paxos_kv_client::Error> { let records = self .kv .read_zone_records(self.bind, &self.disk.disk_id, zone_index) diff --git a/app/crowdb-diskdb/src/recovery/journal_replay.rs b/app/crowdb-diskdb/src/recovery/journal_replay.rs index b49e3e6b4..ed0de7271 100644 --- a/app/crowdb-diskdb/src/recovery/journal_replay.rs +++ b/app/crowdb-diskdb/src/recovery/journal_replay.rs @@ -68,7 +68,7 @@ pub async fn load_zone_inner( let busy_ops = kv.journal_scan_busy(bind, min_slot, 0, &disk_id, zone_idx).await; let busy_ops = match busy_ops { Ok(ops) => ops, - Err(crowdb_kv_client::Error::JournalScanGcGap) => { + Err(crowdb_paxos_kv_client::Error::JournalScanGcGap) => { return Err(ZoneLoadError::JournalScanGcGap); } Err(e) => return Err(ZoneLoadError::Kv(e)), @@ -120,7 +120,7 @@ pub async fn load_zone_inner( let free_ops = kv.journal_scan_free(bind, min_slot, 0, &disk_id, zone_idx).await; let free_ops = match free_ops { Ok(ops) => ops, - Err(crowdb_kv_client::Error::JournalScanGcGap) => { + Err(crowdb_paxos_kv_client::Error::JournalScanGcGap) => { return Err(ZoneLoadError::JournalScanGcGap); } Err(e) => return Err(ZoneLoadError::Kv(e)), diff --git a/app/crowdb-diskdb/src/scanner/integrity.rs b/app/crowdb-diskdb/src/scanner/integrity.rs index d458c89fe..37d8eea07 100644 --- a/app/crowdb-diskdb/src/scanner/integrity.rs +++ b/app/crowdb-diskdb/src/scanner/integrity.rs @@ -164,13 +164,13 @@ async fn scan_zone_records( // ZoneValue. let zone_key = crowdb_protocol::key::ZoneKey { disk_id, zone_index }; let zone_bytes = zone_key.to_bytes(); - if let Ok(crowdb_kv_client::GetOutcome::Found { value, .. }) = kv + if let Ok(crowdb_paxos_kv_client::GetOutcome::Found { value, .. }) = kv .kv() .get( store_id, group_id, &zone_bytes, - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ) .await @@ -203,7 +203,7 @@ async fn scan_zone_records( &[], &[], 0, - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, false, None, @@ -230,7 +230,7 @@ async fn scan_zone_records( &[], &[], 0, - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, false, None, diff --git a/app/crowdb-diskdb/tests/bg_task_test.rs b/app/crowdb-diskdb/tests/bg_task_test.rs index b59b6616c..aa2395ea8 100644 --- a/app/crowdb-diskdb/tests/bg_task_test.rs +++ b/app/crowdb-diskdb/tests/bg_task_test.rs @@ -63,8 +63,8 @@ impl BackgroundTask for MockTask { fn make_ctx() -> Arc<BgCtx> { let container = Arc::new(DdbDiskGroupContainer::new(1)); - let kv = Arc::new(DdbKvClient::new(crowdb_kv_client::CrowdbKvClient::new( - crowdb_kv_client::ClientConfig::new(vec!["127.0.0.1:0".to_string()]), + let kv = Arc::new(DdbKvClient::new(crowdb_paxos_kv_client::PaxosKvClient::new( + crowdb_paxos_kv_client::ClientConfig::new(vec!["127.0.0.1:0".to_string()]), ))); let mut registry = crowdb_common::metrics::MetricsRegistry::new(); let metrics = DiskdbMetrics::register(&mut registry); diff --git a/app/crowdb-diskdb/tests/common/journal_recovery.rs b/app/crowdb-diskdb/tests/common/journal_recovery.rs index 08b481885..39ec98aff 100644 --- a/app/crowdb-diskdb/tests/common/journal_recovery.rs +++ b/app/crowdb-diskdb/tests/common/journal_recovery.rs @@ -3,14 +3,16 @@ use bytes::Bytes; use crowdb_diskdb::model::zone::DdbZone; -use crowdb_kv::cluster::{PxLocalReplica, PxLocalReplicaRole}; -use crowdb_kv::kv::{Batch, BatchOp, CrowdbTreeConfig, CrowdbTreeEngine, KVEngine, Op}; -use crowdb_kv::paxos::roles::{PxBallot, PxLogEntry}; -use crowdb_kv::wal::{index::SegmentIndex, record::WALRecord, replay::ReplayResult}; +use crowdb_paxos_kv::cluster::{PxLocalReplica, PxLocalReplicaRole}; +use crowdb_paxos_kv::kv::{Batch, BatchOp, CrowdbTreeConfig, CrowdbTreeEngine, KVEngine, Op}; +use crowdb_paxos_kv::paxos::roles::{PxBallot, PxLogEntry}; +use crowdb_paxos_kv::wal::{index::SegmentIndex, record::WALRecord, replay::ReplayResult}; use crowdb_protocol::common::{ChunkId, DiskId}; use crowdb_protocol::diskdb::rpc::BusyBlockValue; use crowdb_protocol::key::{BinaryKey, BusyBlockKey, ZoneKey}; use crowdb_protocol::ZoneValueExt; +use crowdb_tree_ffi::PageStore; +use std::sync::Arc; pub struct TestReclaimedJournal; @@ -38,7 +40,11 @@ impl TestReclaimedJournal { allocation_ts: 100, ..Default::default() }; - let engine = CrowdbTreeEngine::open(&CrowdbTreeConfig::default()).unwrap(); + let config = CrowdbTreeConfig { + page_store: Some(Arc::new(PageStore::open_mem(4096).unwrap())), + ..Default::default() + }; + let engine = CrowdbTreeEngine::open(&config).unwrap(); for (slot, key, value) in [ (1, zone_key, baseline.to_zone_value().to_bytes()), (2, busy_key, bincode::serialize(&busy).unwrap()), @@ -58,6 +64,13 @@ impl TestReclaimedJournal { } engine.noop(3); engine.handle().flush().unwrap(); + assert_eq!(engine.resume_from_slot(), 0); + let checkpoint = engine.persist_snapshot_async().unwrap().await.unwrap(); + assert!(checkpoint.status.is_ok(), "{:?}", checkpoint.status); + assert!(checkpoint.durable_committed); + assert_eq!(engine.resume_from_slot(), 3); + drop(engine); + let engine = CrowdbTreeEngine::open(&config).unwrap(); assert_eq!(engine.resume_from_slot(), 3); let replay = ReplayResult { records: vec![WALRecord::from_accepted( diff --git a/app/crowdb-diskdb/tests/common/metadata.rs b/app/crowdb-diskdb/tests/common/metadata.rs index 20a1668ef..06b98dd61 100644 --- a/app/crowdb-diskdb/tests/common/metadata.rs +++ b/app/crowdb-diskdb/tests/common/metadata.rs @@ -5,10 +5,10 @@ //! Election, server packaging and persisted KV recovery belong to container E2E. use crowdb_diskdb::ddb_kv_client::DdbKvClient; -use crowdb_kv::cluster::group::PxGroup; -use crowdb_kv::cluster::kv_server::KvServer; -use crowdb_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, HardwareClient, ServiceRegistryClient}; +use crowdb_paxos_kv::cluster::group::PxGroup; +use crowdb_paxos_kv::cluster::paxos_kv_server::PaxosKvServer; +use crowdb_paxos_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; +use crowdb_paxos_kv_client::{ClientConfig, HardwareClient, PaxosKvClient, ServiceRegistryClient}; use std::sync::Arc; use std::time::{Duration, Instant}; @@ -40,8 +40,8 @@ impl TestMetadata { } } - fn client(&self) -> CrowdbKvClient { - let client = CrowdbKvClient::new(ClientConfig::new(vec![])); + fn client(&self) -> PaxosKvClient { + let client = PaxosKvClient::new(ClientConfig::new(vec![])); client.seed_leader(0, 0, self.group0_leader_endpoint.clone()); client.seed_leader(0, 1, self.group1_leader_endpoint.clone()); client diff --git a/app/crowdb-diskdb/tests/common/online.rs b/app/crowdb-diskdb/tests/common/online.rs index d6cb4fa36..c3f031c13 100644 --- a/app/crowdb-diskdb/tests/common/online.rs +++ b/app/crowdb-diskdb/tests/common/online.rs @@ -7,7 +7,7 @@ use crowdb_diskdb::{ liveness::keepalive::KeepAlive, model::{disk::DdbDisk, disk_group_container::DdbDiskGroupContainer}, }; -use crowdb_kv_client::HardwareClient; +use crowdb_paxos_kv_client::HardwareClient; use crowdb_protocol::{ common::{DiskId, DiskioHardwareScope, HwStatus, NodeValue, RackValue}, disk_storage::{DiskExclusionRecord, DiskExclusionReporter, DiskOnlineState, DiskStorageRecord}, diff --git a/app/crowdb-diskdb/tests/config_reload_test.rs b/app/crowdb-diskdb/tests/config_reload_test.rs index 95d3abc3a..7411e9d15 100644 --- a/app/crowdb-diskdb/tests/config_reload_test.rs +++ b/app/crowdb-diskdb/tests/config_reload_test.rs @@ -17,7 +17,7 @@ use crowdb_diskdb::liveness::keepalive::KeepAlive; use crowdb_diskdb::metrics::DiskdbMetrics; use crowdb_diskdb::model::disk_group_container::DdbDiskGroupContainer; use crowdb_diskdb::recovery::compaction::CompactionEngine; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, HardwareClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{ClientConfig, HardwareClient, PaxosKvClient, ServiceRegistryClient}; fn make_config_handle(interval_secs: u32) -> Arc<ArcSwap<DdbConfig>> { let mut config = DdbConfig::default(); @@ -28,7 +28,7 @@ fn make_config_handle(interval_secs: u32) -> Arc<ArcSwap<DdbConfig>> { } fn make_keepalive(handle: &Arc<ArcSwap<DdbConfig>>) -> KeepAlive { - let kv_client = Arc::new(CrowdbKvClient::new(ClientConfig::new(vec![ + let kv_client = Arc::new(PaxosKvClient::new(ClientConfig::new(vec![ "127.0.0.1:0".to_string() ]))); let hw = HardwareClient::from_shared(Arc::clone(&kv_client)); @@ -66,7 +66,7 @@ fn keepalive_trigger_uses_timer_fn_when_config_handle_set() { #[test] fn keepalive_trigger_uses_fixed_timer_without_config_handle() { - let kv_client = Arc::new(CrowdbKvClient::new(ClientConfig::new(vec![ + let kv_client = Arc::new(PaxosKvClient::new(ClientConfig::new(vec![ "127.0.0.1:0".to_string() ]))); let hw = HardwareClient::from_shared(Arc::clone(&kv_client)); @@ -89,7 +89,7 @@ fn keepalive_trigger_uses_fixed_timer_without_config_handle() { #[test] fn compaction_engine_trigger_uses_timer_fn_when_config_handle_set() { let handle = make_config_handle(30); - let kv_client = Arc::new(CrowdbKvClient::new(ClientConfig::new(vec![ + let kv_client = Arc::new(PaxosKvClient::new(ClientConfig::new(vec![ "127.0.0.1:0".to_string() ]))); let ddb_kv = Arc::new(DdbKvClient::from_shared(kv_client)); @@ -113,7 +113,7 @@ fn compaction_engine_trigger_uses_timer_fn_when_config_handle_set() { #[test] fn compaction_engine_trigger_uses_fixed_timer_without_config_handle() { - let kv_client = Arc::new(CrowdbKvClient::new(ClientConfig::new(vec![ + let kv_client = Arc::new(PaxosKvClient::new(ClientConfig::new(vec![ "127.0.0.1:0".to_string() ]))); let ddb_kv = Arc::new(DdbKvClient::from_shared(kv_client)); @@ -136,7 +136,7 @@ async fn compaction_engine_run_cycle_reads_threshold_from_ctx_config() { config.persistence.snapshot_compaction_threshold = 7777; handle.store(Arc::new(config)); - let kv_client = Arc::new(CrowdbKvClient::new(ClientConfig::new(vec![ + let kv_client = Arc::new(PaxosKvClient::new(ClientConfig::new(vec![ "127.0.0.1:0".to_string() ]))); let ddb_kv = Arc::new(DdbKvClient::from_shared(kv_client)); diff --git a/app/crowdb-diskdb/tests/ddb_config_test.rs b/app/crowdb-diskdb/tests/ddb_config_test.rs index 1678c8521..3c90135f1 100644 --- a/app/crowdb-diskdb/tests/ddb_config_test.rs +++ b/app/crowdb-diskdb/tests/ddb_config_test.rs @@ -46,9 +46,9 @@ fn config_validate_rejects_granularity_not_equal_to_block() { } #[test] -fn config_validate_rejects_bad_listen_addr() { +fn config_validate_rejects_bad_rpc_listen_addr() { let mut config = DdbConfig::default(); - config.server.listen_addr = "not-an-addr".to_string(); + config.server.rpc_listen_addr = "not-an-addr".to_string(); assert!(validate(&config).is_err()); } @@ -93,14 +93,13 @@ fn config_defaults_match_design() { #[test] fn partial_persistence_config_keeps_defaults() { let config: DdbConfig = toml::from_str( - r#" + r" [server] - listen_addr = "127.0.0.1:1" [persistence] free_batch_enabled = true free_flush_max_batch = 32 - "#, + ", ) .unwrap(); assert!(config.persistence.free_batch_enabled); @@ -114,7 +113,6 @@ fn allocator_weight_policy_parses_and_keeps_other_defaults() { let config: DdbConfig = toml::from_str( r#" [server] - listen_addr = "127.0.0.1:1" [allocator] load_aware_weight = "inverse_used_pct" @@ -198,7 +196,6 @@ fn config_load_from_file_roundtrip() { std::fs::write(&tmp, &toml).expect("write temp"); let loaded = crowdb_common::config::load_from_file::<DdbConfig>(&tmp).expect("load"); assert_eq!(loaded.storage.block_size_bytes, config.storage.block_size_bytes); - assert_eq!(loaded.server.listen_addr, config.server.listen_addr); let _ = std::fs::remove_file(&tmp); } @@ -209,7 +206,6 @@ fn config_load_from_file_roundtrip() { #[test] fn minimal_server_only_config_uses_section_defaults() { let toml = "[server]\n\ - listen_addr = \"0.0.0.0:11000\"\n\ http_listen_addr = \"0.0.0.0:11100\"\n\ rpc_listen_addr = \"0.0.0.0:11200\"\n\ kv_server_mgmt_seeds = [\"http://127.0.0.1:10000\"]\n"; diff --git a/app/crowdb-diskdb/tests/disk_initialization_test.rs b/app/crowdb-diskdb/tests/disk_initialization_test.rs index 4dc6e9a13..6793cf923 100644 --- a/app/crowdb-diskdb/tests/disk_initialization_test.rs +++ b/app/crowdb-diskdb/tests/disk_initialization_test.rs @@ -149,7 +149,10 @@ async fn atomic_multi_disk_rejection_does_not_poison_unsubmitted_writes() { assert!(second.write_gate.quiescent()); } -async fn prepare_hardware(metadata: &TestMetadata, status: HwStatus) -> crowdb_kv_client::HardwareClient { +async fn prepare_hardware( + metadata: &TestMetadata, + status: HwStatus, +) -> crowdb_paxos_kv_client::HardwareClient { let hw = metadata.make_hardware_client(); let id = DiskId { high: 4, low: 5 }; hw.add_rack( diff --git a/app/crowdb-diskdb/tests/journal_gap_recovery_test.rs b/app/crowdb-diskdb/tests/journal_gap_recovery_test.rs index bf46047f1..954d501e7 100644 --- a/app/crowdb-diskdb/tests/journal_gap_recovery_test.rs +++ b/app/crowdb-diskdb/tests/journal_gap_recovery_test.rs @@ -2,10 +2,10 @@ // Licensed under the Apache License, Version 2.0. use crowdb_diskdb::{ddb_kv_client::DdbKvClient, recovery::ZoneLoader}; -use crowdb_kv::cluster::group::PxGroup; -use crowdb_kv::cluster::kv_server::KvServer; -use crowdb_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient}; +use crowdb_paxos_kv::cluster::group::PxGroup; +use crowdb_paxos_kv::cluster::paxos_kv_server::PaxosKvServer; +use crowdb_paxos_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient}; use crowdb_protocol::common::{DiskId, HwStatus}; use crowdb_protocol::diskdb::rpc::DiskValue; use std::sync::Arc; @@ -24,7 +24,7 @@ async fn restart_with_reclaimed_journal_rebuilds_busy_units_before_allocating() )); store.add_group(PxGroup::new(1, restored)); store.start().await.unwrap(); - let client = CrowdbKvClient::new(ClientConfig::new(vec![])); + let client = PaxosKvClient::new(ClientConfig::new(vec![])); let endpoint = store.listen_addr().unwrap().to_string(); for group in [0, 1] { client.seed_leader(0, group, endpoint.clone()); diff --git a/app/crowdb-diskdb/tests/keepalive_sync_test.rs b/app/crowdb-diskdb/tests/keepalive_sync_test.rs index c1793247c..fd1811fd3 100644 --- a/app/crowdb-diskdb/tests/keepalive_sync_test.rs +++ b/app/crowdb-diskdb/tests/keepalive_sync_test.rs @@ -32,7 +32,10 @@ async fn owner_without_bind_does_not_publish_disk_group() { .set_owner(RACK_ID, NODE_ID, DG_ID, INSTANCE_ID + 1, u64::MAX) .await; assert!( - matches!(replacement, Err(crowdb_kv_client::Error::OwnerConflict { .. })), + matches!( + replacement, + Err(crowdb_paxos_kv_client::Error::OwnerConflict { .. }) + ), "different owner must be rejected" ); diff --git a/app/crowdb-diskdb/tests/metadata_flow_test.rs b/app/crowdb-diskdb/tests/metadata_flow_test.rs index ad2bb5ae4..663ed3d8d 100644 --- a/app/crowdb-diskdb/tests/metadata_flow_test.rs +++ b/app/crowdb-diskdb/tests/metadata_flow_test.rs @@ -19,7 +19,7 @@ use crowdb_diskdb::ddb_kv_client::DdbKvClient; use crowdb_diskdb::liveness::keepalive::KeepAlive; use crowdb_diskdb::model::alloc; use crowdb_diskdb::model::disk_group_container::DdbDiskGroupContainer; -use crowdb_kv_client::{GetOutcome, HardwareClient}; +use crowdb_paxos_kv_client::{GetOutcome, HardwareClient}; use crowdb_protocol::common::{ChunkId, DiskId, HwStatus, NodeValue, RackValue}; use crowdb_protocol::diskdb::rpc::{BlockState, CommitState, DiskGroupValue, DiskType, DiskValue}; use crowdb_protocol::key::{BinaryKey, BusyBlockKey, FreeBlockKey}; @@ -56,7 +56,7 @@ async fn kv_get(kv: &DdbKvClient, key: &[u8]) -> Option<bytes::Bytes> { STORE_ID, DATA_GROUP_ID, key, - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ) .await diff --git a/app/crowdb-diskdb/tests/ownership_fence_test.rs b/app/crowdb-diskdb/tests/ownership_fence_test.rs index 52371fa63..eccc708da 100644 --- a/app/crowdb-diskdb/tests/ownership_fence_test.rs +++ b/app/crowdb-diskdb/tests/ownership_fence_test.rs @@ -15,7 +15,7 @@ use std::sync::Arc; #[tokio::test] async fn stale_zone_loader_cannot_mark_a_successor_disk_offline() { - use crowdb_kv_client::ReadMode; + use crowdb_paxos_kv_client::ReadMode; use crowdb_protocol::{ common::{HwStatus, OwnerMapValue}, key::{OwnerMapKey, TextKey}, @@ -62,7 +62,7 @@ async fn stale_zone_loader_cannot_mark_a_successor_disk_offline() { HwStatus::Offline as i32 ); hardware.add_disk(1, 1, 1, &disk, &online).await.unwrap(); - let crowdb_kv_client::GetOutcome::Found { revision, .. } = kv + let crowdb_paxos_kv_client::GetOutcome::Found { revision, .. } = kv .kv() .get(0, 0, owner_key.as_bytes(), ReadMode::Linearizable, None) .await @@ -174,11 +174,15 @@ async fn concurrent_disk_initialization_persists_every_zone_under_one_owner() { ) }) .collect(); - assert!(scoped.put_zone_batch((0, 1), &disk, &records).await.is_err()); + let limit = crowdb_diskdb::ddb_kv_client::MAX_ZONE_INIT_BATCH; + assert!(scoped + .put_zone_batch((0, 1), &disk, &records[..=limit]) + .await + .is_err()); for batch in records.chunks(crowdb_diskdb::ddb_kv_client::MAX_ZONE_INIT_BATCH) { scoped.put_zone_batch((0, 1), &disk, batch).await?; } - Ok::<_, crowdb_kv_client::Error>(disk) + Ok::<_, crowdb_paxos_kv_client::Error>(disk) } })) .await; @@ -195,12 +199,12 @@ async fn concurrent_disk_initialization_persists_every_zone_under_one_owner() { 0, 1, &key.to_bytes(), - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ) .await .unwrap(); - let crowdb_kv_client::GetOutcome::Found { value, .. } = record else { + let crowdb_paxos_kv_client::GetOutcome::Found { value, .. } = record else { panic!("missing zone baseline") }; let snapshot = crowdb_protocol::diskdb::rpc::ZoneValue::from_bytes(&value).unwrap(); @@ -213,7 +217,7 @@ async fn concurrent_disk_initialization_persists_every_zone_under_one_owner() { #[tokio::test] async fn ordinary_writes_cannot_bypass_ownership_handover() { use bytes::Bytes; - use crowdb_kv_client::{BatchOp, GetOutcome, ReadMode}; + use crowdb_paxos_kv_client::{BatchOp, GetOutcome, ReadMode}; let cluster = TestMetadata::start().await; let ddb = cluster.make_ddb_kv_client(); let kv = ddb.kv(); diff --git a/app/crowdb-diskdb/tests/query_handler_test.rs b/app/crowdb-diskdb/tests/query_handler_test.rs index 54ec58f49..4d9bec0b6 100644 --- a/app/crowdb-diskdb/tests/query_handler_test.rs +++ b/app/crowdb-diskdb/tests/query_handler_test.rs @@ -194,13 +194,13 @@ async fn disk_rpc_reports_effective_status() { use crowdb_diskdb::scanner::ScanState; use crowdb_diskdb::service::DiskdbRpcService; use crowdb_diskdb_client::rpc_transport::DiskdbRpcTransport; - use crowdb_kv_client::{ClientConfig, CrowdbKvClient}; + use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient}; let dg = make_dg(); let disk = dg.get_disk(disk_id(1)).unwrap(); let container = Arc::new(DdbDiskGroupContainer::new(1)); container.replace_disk_group(&dg); - let kv = Arc::new(DdbKvClient::new(CrowdbKvClient::new(ClientConfig::new(vec![])))); + let kv = Arc::new(DdbKvClient::new(PaxosKvClient::new(ClientConfig::new(vec![])))); let service = Arc::new(DiskdbRpcService::new( container.clone(), kv.clone(), diff --git a/app/crowdb-diskdb/tests/recovery_failure_test.rs b/app/crowdb-diskdb/tests/recovery_failure_test.rs index 92c7845d1..74d0dad92 100644 --- a/app/crowdb-diskdb/tests/recovery_failure_test.rs +++ b/app/crowdb-diskdb/tests/recovery_failure_test.rs @@ -6,7 +6,7 @@ use std::time::Duration; use crowdb_diskdb::ddb_kv_client::DdbKvClient; use crowdb_diskdb::recovery::ZoneLoader; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, RetryConfig}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient, RetryConfig}; use crowdb_protocol::common::DiskId; use crowdb_protocol::diskdb::rpc::{DiskType, DiskValue}; @@ -19,8 +19,8 @@ async fn failed_recovery_does_not_return_writable_disk_group() { backoff_base: Duration::from_millis(10), backoff_max: Duration::from_millis(10), }; - let kv = CrowdbKvClient::new(config); - let port = crowdb_protocol::port::alloc::alloc_test_port(crowdb_protocol::ServicePort::KvServerListen); + let kv = PaxosKvClient::new(config); + let port = crowdb_protocol::port::alloc::alloc_test_port(crowdb_protocol::ServicePort::PaxosKvServerRpc); kv.seed_leader(0, 999, format!("127.0.0.1:{port}")); let loader = ZoneLoader::new(Arc::new(DdbKvClient::new(kv)), 1); let disk = DiskValue { diff --git a/app/crowdb-diskdb/tests/recovery_test.rs b/app/crowdb-diskdb/tests/recovery_test.rs index 6017629dc..6c22ddea6 100644 --- a/app/crowdb-diskdb/tests/recovery_test.rs +++ b/app/crowdb-diskdb/tests/recovery_test.rs @@ -25,7 +25,7 @@ use crowdb_diskdb::model::disk_group_container::DdbDiskGroupContainer; use crowdb_diskdb::model::zone::DdbZone; use crowdb_diskdb::recovery::compaction::CompactionEngine; use crowdb_diskdb::recovery::ZoneLoader; -use crowdb_kv_client::{GetOutcome, HardwareClient}; +use crowdb_paxos_kv_client::{GetOutcome, HardwareClient}; use crowdb_protocol::common::{ChunkId, DiskId, HwStatus, NodeValue, RackValue}; use crowdb_protocol::diskdb::rpc::{DiskGroupValue, DiskType, DiskValue}; use crowdb_protocol::key::BinaryKey; @@ -639,7 +639,7 @@ async fn compaction_compact_zone_writes_snapshot_and_deletes_free_records() { STORE_ID, DATA_GROUP_ID, &free_key.to_bytes(), - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ) .await @@ -666,7 +666,7 @@ async fn compaction_compact_zone_writes_snapshot_and_deletes_free_records() { STORE_ID, DATA_GROUP_ID, &free_key.to_bytes(), - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ) .await @@ -781,7 +781,7 @@ async fn compaction_watermark_prevents_double_free_after_crashed_compaction() { STORE_ID, DATA_GROUP_ID, &free_key.to_bytes(), - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ) .await @@ -887,7 +887,7 @@ async fn compaction_watermark_prevents_double_free_after_crashed_compaction() { STORE_ID, DATA_GROUP_ID, &free_key.to_bytes(), - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ) .await diff --git a/app/crowdb-diskdb/tests/relocation_journal_test.rs b/app/crowdb-diskdb/tests/relocation_journal_test.rs index 87ae11cf9..073c95d3c 100644 --- a/app/crowdb-diskdb/tests/relocation_journal_test.rs +++ b/app/crowdb-diskdb/tests/relocation_journal_test.rs @@ -26,7 +26,7 @@ use crowdb_diskdb::recovery::ZoneLoader; use crowdb_diskdb::scanner::ScanState; use crowdb_diskdb::service::DiskdbRpcService; use crowdb_diskdb_client::DiskdbRpcTransport; -use crowdb_kv_client::HardwareClient; +use crowdb_paxos_kv_client::HardwareClient; use crowdb_protocol::chunkdb::rpc::{RelocateSegmentHandoffRequest, RelocationHandoffDisposition}; use crowdb_protocol::common::{ChunkId, DiskId, HwStatus, NodeValue, RackValue}; use crowdb_protocol::diskdb::rpc::{ diff --git a/app/crowdb-diskdb/tests/scanner_test.rs b/app/crowdb-diskdb/tests/scanner_test.rs index 1476935a1..f7479f954 100644 --- a/app/crowdb-diskdb/tests/scanner_test.rs +++ b/app/crowdb-diskdb/tests/scanner_test.rs @@ -31,7 +31,7 @@ use crowdb_diskdb::model::disk_group_container::DdbDiskGroupContainer; use crowdb_diskdb::scanner::ghost::diff_bitmaps; use crowdb_diskdb::scanner::integrity::{is_zero_chunk, is_zero_owner}; use crowdb_diskdb::scanner::{BusyBlockOwnerScanner, ScanState, ScanSummary, SegmentOwnerQuery, ZonePacer}; -use crowdb_kv_client::HardwareClient; +use crowdb_paxos_kv_client::HardwareClient; use crowdb_protocol::chunkdb::rpc::{QuerySegmentOwnerRequest, SegmentOwnerDisposition}; use crowdb_protocol::common::{ChunkId, DiskId, HwStatus, NodeValue, RackValue}; use crowdb_protocol::diskdb::rpc::{BusyBlockValue, CommitState, DiskGroupValue, DiskType, DiskValue}; diff --git a/app/crowdb-diskdb/tests/startup_sync_test.rs b/app/crowdb-diskdb/tests/startup_sync_test.rs index ac2ca9176..d0b14c10d 100644 --- a/app/crowdb-diskdb/tests/startup_sync_test.rs +++ b/app/crowdb-diskdb/tests/startup_sync_test.rs @@ -34,7 +34,7 @@ fn recovered_group_replaces_only_the_retained_owner_generation() { #[tokio::test] async fn startup_reconstruction_keeps_the_original_administrative_write_gate() { use crowdb_diskdb::{ddb_kv_client::DdbKvClient, model::disk::DdbDisk, recovery::ZoneLoader}; - use crowdb_kv_client::{ClientConfig, CrowdbKvClient}; + use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient}; use crowdb_protocol::{ common::{DiskId, HwStatus}, diskdb::rpc::DiskValue, @@ -51,7 +51,7 @@ async fn startup_reconstruction_keeps_the_original_administrative_write_gate() { let group = DdbDiskGroup::new(3, 2, 1); let disk = Arc::new(DdbDisk::new(id, 3, 2, 1, value.clone())); group.add_disk(disk.clone()); - let scoped = DdbKvClient::new(CrowdbKvClient::new(ClientConfig::new(vec![]))).for_group(&group); + let scoped = DdbKvClient::new(PaxosKvClient::new(ClientConfig::new(vec![]))).for_group(&group); let loader = ZoneLoader::new(Arc::new(scoped), 1); // The loader captured an Up value before exclusion, but it must neither // manufacture an empty baseline nor publish a new writable incarnation. diff --git a/app/crowdb-diskio/CMakeLists.txt b/app/crowdb-diskio/CMakeLists.txt index 534fc83b6..c94d17f1c 100644 --- a/app/crowdb-diskio/CMakeLists.txt +++ b/app/crowdb-diskio/CMakeLists.txt @@ -19,11 +19,11 @@ add_compile_options(-Wall -Wextra -Werror) # add crowdb-common separately here (avoids duplicate target collision). add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/../../lib/crowdb-rpc crowdb-rpc-build) -# ── crowdb-kv-client FFI (Rust cdylib with --features ffi) ────────── -# crowdb-kv-client produces both an rlib (for Rust dependents) and a +# ── crowdb-paxos-kv-client FFI (Rust cdylib with --features ffi) ────────── +# crowdb-paxos-kv-client produces both an rlib (for Rust dependents) and a # cdylib (for C++ consumers). The cdylib exports C ABI symbols only # when built with the `ffi` feature. -set(CROWDB_KV_CLIENT_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../../lib/crowdb-kv-client) +set(CROWDB_PAXOS_KV_CLIENT_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../../lib/crowdb-paxos-kv-client) set(CROWDB_ROOT_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../..) if(CMAKE_BUILD_TYPE STREQUAL "Release" OR CMAKE_BUILD_TYPE STREQUAL "RelWithDebInfo") set(CARGO_PROFILE release) @@ -31,16 +31,16 @@ else() set(CARGO_PROFILE debug) endif() if(APPLE) - set(CROWDB_KV_CLIENT_FFI_LIB ${CROWDB_ROOT_DIR}/target/${CARGO_PROFILE}/libcrowdb_kv_client.dylib) + set(CROWDB_PAXOS_KV_CLIENT_FFI_LIB ${CROWDB_ROOT_DIR}/target/${CARGO_PROFILE}/libcrowdb_paxos_kv_client.dylib) else() - set(CROWDB_KV_CLIENT_FFI_LIB ${CROWDB_ROOT_DIR}/target/${CARGO_PROFILE}/libcrowdb_kv_client.so) + set(CROWDB_PAXOS_KV_CLIENT_FFI_LIB ${CROWDB_ROOT_DIR}/target/${CARGO_PROFILE}/libcrowdb_paxos_kv_client.so) endif() -add_custom_target(crowdb-kv-client-ffi - BYPRODUCTS ${CROWDB_KV_CLIENT_FFI_LIB} - COMMAND pixi run cargo build -p crowdb-kv-client --features ffi +add_custom_target(crowdb-paxos-kv-client-ffi + BYPRODUCTS ${CROWDB_PAXOS_KV_CLIENT_FFI_LIB} + COMMAND pixi run cargo build -p crowdb-paxos-kv-client --features ffi $<$<STREQUAL:${CARGO_PROFILE},release>:--release> WORKING_DIRECTORY ${CROWDB_ROOT_DIR} - COMMENT "Building crowdb-kv-client (Rust cdylib with ffi feature)" + COMMENT "Building crowdb-paxos-kv-client (Rust cdylib with ffi feature)" VERBATIM ) @@ -53,7 +53,7 @@ target_include_directories(crowdb-diskio-lib PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/src ${CMAKE_CURRENT_SOURCE_DIR}/../../lib/crowdb-common/cpp/include ${CMAKE_CURRENT_SOURCE_DIR}/../../lib/crowdb-rpc/include - ${CROWDB_KV_CLIENT_DIR}/include + ${CROWDB_PAXOS_KV_CLIENT_DIR}/include PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/crowdb-rpc-build/generated ${CMAKE_CURRENT_SOURCE_DIR}/../../lib/crowdb-protocol/cpp/include ) @@ -67,8 +67,8 @@ target_include_directories(crowdb-diskio PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/src ${CMAKE_CURRENT_BINARY_DIR}/crowdb-rpc-build/generated ) -target_link_libraries(crowdb-diskio PRIVATE crowdb-diskio-lib ${CROWDB_KV_CLIENT_FFI_LIB} ${CMAKE_DL_LIBS}) -add_dependencies(crowdb-diskio crowdb-kv-client-ffi) +target_link_libraries(crowdb-diskio PRIVATE crowdb-diskio-lib ${CROWDB_PAXOS_KV_CLIENT_FFI_LIB} ${CMAKE_DL_LIBS}) +add_dependencies(crowdb-diskio crowdb-paxos-kv-client-ffi) set_target_properties(crowdb-diskio PROPERTIES BUILD_RPATH ${CROWDB_ROOT_DIR}/target/${CARGO_PROFILE} ) diff --git a/app/crowdb-diskio/src/dio_config.cpp b/app/crowdb-diskio/src/dio_config.cpp index d81196471..fe04c341c 100644 --- a/app/crowdb-diskio/src/dio_config.cpp +++ b/app/crowdb-diskio/src/dio_config.cpp @@ -282,7 +282,7 @@ bool DioConfig::parse_args(int argc, char *argv[], DioConfig &out, std::string & } else if (arg == "--kv-seeds" && i + 1 < argc) { // Comma-separated list of kv-server management endpoints. - // e.g. --kv-seeds http://127.0.0.1:10000,http://127.0.0.1:10001 + // e.g. --kv-seeds http://127.0.0.1:10135,http://127.0.0.1:10136 std::string seeds = argv[++i]; if (!cli_kv_seeds) { out.kv_seeds.clear(); diff --git a/app/crowdb-diskio/src/dio_config.h b/app/crowdb-diskio/src/dio_config.h index d582a9907..dad7ba995 100644 --- a/app/crowdb-diskio/src/dio_config.h +++ b/app/crowdb-diskio/src/dio_config.h @@ -6,6 +6,7 @@ #pragma once #include "crowdb-common/runtime_path.h" +#include "crowdb-protocol/ports.h" #include "crowdb-protocol/timing.h" #include "disk/disk_properties.h" #include "disk/types.h" @@ -35,10 +36,9 @@ struct DiskEntry // Server configuration. struct DioConfig { - // RPC listen. Default port 13000 matches DISKIO_RPC_BASE in - // lib/crowdb-protocol/src/ports.rs — keep in sync. + // RPC listen; defaults are generated from protocol ports.rs. std::string bind_address = "127.0.0.1"; - int listen_port = 13000; + int listen_port = protocol::DISKIO_RPC_BASE; uint32_t rpc_workers = 4; // Node identity (64-bit; each diskio service manages one node). diff --git a/app/crowdb-diskio/src/dio_main.cpp b/app/crowdb-diskio/src/dio_main.cpp index d86744bce..fa6841d3f 100644 --- a/app/crowdb-diskio/src/dio_main.cpp +++ b/app/crowdb-diskio/src/dio_main.cpp @@ -16,7 +16,7 @@ #include "crowdb-common/metrics/metrics.h" #include "crowdb-common/metrics/system_metrics.h" -#include "crowdb-kv-client/c_api.h" +#include "crowdb-paxos-kv-client/c_api.h" #include "crowdb-rpc/scheduled_executor.h" #include "crowdb-rpc/server/server.h" #include "crowdb-rpc/transport/socket_transport.h" diff --git a/app/crowdb-diskio/src/group0/group0_exclusion.cpp b/app/crowdb-diskio/src/group0/group0_exclusion.cpp index 5f9d60211..152fe2766 100644 --- a/app/crowdb-diskio/src/group0/group0_exclusion.cpp +++ b/app/crowdb-diskio/src/group0/group0_exclusion.cpp @@ -1,6 +1,6 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -#include "crowdb-kv-client/c_api.h" +#include "crowdb-paxos-kv-client/c_api.h" #include "group0/ffi_result.h" #include "group0/group0_sync.h" diff --git a/app/crowdb-diskio/src/group0/group0_sync.cpp b/app/crowdb-diskio/src/group0/group0_sync.cpp index 70c11509a..2d694d1cc 100644 --- a/app/crowdb-diskio/src/group0/group0_sync.cpp +++ b/app/crowdb-diskio/src/group0/group0_sync.cpp @@ -3,7 +3,7 @@ #include "group0/group0_sync.h" -#include "crowdb-kv-client/c_api.h" +#include "crowdb-paxos-kv-client/c_api.h" #include "disk/block_disk.h" #include "disk/mem_disk.h" #include "disk/null_disk.h" diff --git a/app/crowdb-diskio/src/group0/group0_sync.h b/app/crowdb-diskio/src/group0/group0_sync.h index c748d723d..68ffbc255 100644 --- a/app/crowdb-diskio/src/group0/group0_sync.h +++ b/app/crowdb-diskio/src/group0/group0_sync.h @@ -4,7 +4,7 @@ // Group0Sync: periodic group-0 synchronization for crowdb-diskio. // // When kv_seeds are configured, Group0Sync connects to group-0 via -// the crowdb-kv-client FFI C ABI, fetches the disk list for this node's +// the crowdb-paxos-kv-client FFI C ABI, fetches the disk list for this node's // disk-group, reconciles the DiskSet, and heartbeats to the service // registry. // @@ -81,7 +81,7 @@ class Group0Sync DiskExclusion exclusions_; bool registered_ = false; - // FFI handles (opaque pointers from crowdb-kv-client FFI). + // FFI handles (opaque pointers from crowdb-paxos-kv-client FFI). void *hw_client_ = nullptr; void *svc_client_ = nullptr; }; diff --git a/app/crowdb-diskio/src/rpc/dio_read.cpp b/app/crowdb-diskio/src/rpc/dio_read.cpp index 88f7698d5..2eb42bcfe 100644 --- a/app/crowdb-diskio/src/rpc/dio_read.cpp +++ b/app/crowdb-diskio/src/rpc/dio_read.cpp @@ -4,6 +4,7 @@ #include "crowdb-common/metrics/metrics.h" #include "rpc/dio_read_buffer.h" #include "rpc/dio_request.h" +#include "rpc/dio_request_deadline.h" #include "rpc/dio_server.h" #include <msg_type_generated.h> @@ -23,14 +24,30 @@ crowdb::rpc::OutFrame *DiskioServer::handle_read(crowdb::rpc::Frame *request, cr { uint64_t req_id = request->request_id; uint64_t create_nano = request->rpc_create_nano; - uint16_t msg_type = static_cast<uint16_t>(rproto::FBMsgType_EDiskReadResponse); + auto msg_type = static_cast<uint16_t>(rproto::FBMsgType_EDiskReadResponse); - auto *fb_req = decode_disk_request<dproto::FBDiskReadRequest>(*request); + const auto *fb_req = decode_disk_request<dproto::FBDiskReadRequest>(*request); if (fb_req == nullptr || (request->data_buf != nullptr && request->data_buf->len != 0)) { delete request; send_error_response(conn, req_id, create_nano, msg_type, static_cast<int16_t>(dproto::FBDiskIoRetCode_IoError)); return nullptr; } + const bool read_age_checked = fb_req->read_create_time_ms() != 0; + const auto deadline = read_age_checked ? request_deadline(fb_req->read_create_time_ms(), max_write_request_age_ms_, + max_clock_skew_ms_) + : std::nullopt; + if ((read_age_checked && !deadline) || (fb_req->fence_probe() && (!read_age_checked || fb_req->size() != 0))) { + delete request; + send_error_response(conn, req_id, create_nano, msg_type, + static_cast<int16_t>(dproto::FBDiskIoRetCode_OldRequest)); + return nullptr; + } + if (fb_req->fence_probe()) { + delete request; + send_response(transport_, conn->weak_from_this().lock(), req_id, create_nano, msg_type, + static_cast<int16_t>(dproto::FBDiskIoRetCode_Success), nullptr, false, true); + return nullptr; + } DiskId did = parse_disk_id(fb_req->disk_id()); uint32_t zone_index = fb_req->zone_index(); uint64_t zone_offset = fb_req->zone_offset(); @@ -67,7 +84,15 @@ crowdb::rpc::OutFrame *DiskioServer::handle_read(crowdb::rpc::Frame *request, cr root->ownership.load(std::memory_order_acquire) == crowdb::rpc::BufferOwnership::CallerOwned; // Control has been decoded, and READ carries no input payload. Retain its // existing receive data region until disk and response SEND both finish. - auto *read_buf = registered ? (receive_owned ? root->slice(0, size) : nullptr) : conn->pool()->alloc(size); + crowdb::rpc::Buffer *read_buf = nullptr; + if (registered) { + if (receive_owned) { + read_buf = root->slice(0, size); + } + } + else { + read_buf = conn->pool()->alloc(size); + } if (read_buf == nullptr) { delete request; send_error_response(conn, req_id, create_nano, msg_type, static_cast<int16_t>(dproto::FBDiskIoRetCode_IoError)); @@ -82,26 +107,36 @@ crowdb::rpc::OutFrame *DiskioServer::handle_read(crowdb::rpc::Frame *request, cr send_error_response(conn, req_id, create_nano, msg_type, static_cast<int16_t>(aligned)); return nullptr; } + if (deadline && std::chrono::steady_clock::now() >= *deadline) { + read_buf->release(); + send_error_response(conn, req_id, create_nano, msg_type, + static_cast<int16_t>(dproto::FBDiskIoRetCode_OldRequest)); + return nullptr; + } if (!io.begin()) { conn->close(); return nullptr; } - Disk *disk_ptr = disk.get(); - auto started = std::chrono::steady_clock::now(); - auto *io_data = io.data(); - disk_ptr->engine()->submit_read(disk_ptr, io.offset, io_data, io.io_size, test_pattern_offset, + Disk *disk_ptr = disk.get(); + auto started = std::chrono::steady_clock::now(); + auto *io_data = io.data(); + const auto io_offset = io.offset; + const auto io_size = io.io_size; + disk_ptr->engine()->submit_read(disk_ptr, io_offset, io_data, io_size, test_pattern_offset, [transport = transport_, connection = conn->weak_from_this().lock(), latency = read_latency_, req_id, create_nano, msg_type, io = std::move(io), - disk = std::move(disk), started](int result) mutable { + disk = std::move(disk), started, read_age_checked](int result) mutable { latency->observe(elapsed_nanos(started)); const auto completed = io.finish(result); if (!completed.valid_owner) { - if (connection != nullptr) + if (connection != nullptr) { connection->close(); + } return; } send_response(transport, connection, req_id, create_nano, msg_type, - static_cast<int16_t>(completed.code), completed.data); + static_cast<int16_t>(completed.code), completed.data, false, + read_age_checked); }); return nullptr; diff --git a/app/crowdb-diskio/src/rpc/dio_request_deadline.h b/app/crowdb-diskio/src/rpc/dio_request_deadline.h new file mode 100644 index 000000000..3123ac425 --- /dev/null +++ b/app/crowdb-diskio/src/rpc/dio_request_deadline.h @@ -0,0 +1,36 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +#pragma once + +#include <chrono> +#include <cstdint> +#include <optional> +#include <utility> + +namespace crowdb::diskio +{ +inline std::optional<std::chrono::steady_clock::time_point> request_deadline(uint64_t created_ms, uint64_t max_age_ms, + uint64_t skew_ms) +{ + const auto steady_now = std::chrono::steady_clock::now(); + const auto now_ms = static_cast<uint64_t>( + std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch()) + .count()); + if (created_ms == 0 || max_age_ms == 0 || (created_ms > now_ms && created_ms - now_ms > skew_ms) || + max_age_ms > UINT64_MAX - skew_ms) { + return std::nullopt; + } + const auto allowance = max_age_ms + skew_ms; + if (created_ms > UINT64_MAX - allowance || created_ms + allowance <= now_ms) { + return std::nullopt; + } + const auto remaining_ms = created_ms + allowance - now_ms; + const auto max_wait_ms = + std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::time_point::max() - steady_now) + .count(); + if (std::cmp_greater(remaining_ms, max_wait_ms)) { + return std::nullopt; + } + return steady_now + std::chrono::milliseconds(static_cast<int64_t>(remaining_ms)); +} +} // namespace crowdb::diskio diff --git a/app/crowdb-diskio/src/rpc/dio_server.cpp b/app/crowdb-diskio/src/rpc/dio_server.cpp index 2515f0171..f5fbae4dc 100644 --- a/app/crowdb-diskio/src/rpc/dio_server.cpp +++ b/app/crowdb-diskio/src/rpc/dio_server.cpp @@ -57,7 +57,6 @@ DiskioServer::DiskioServer(std::shared_ptr<DiskSet> disk_set, std::shared_ptr<cr uint64_t max_write_request_age_ms, uint64_t max_clock_skew_ms) : disk_set_(std::move(disk_set)), transport_(std::move(transport)), - aligned_writer_(), read_latency_(&engine_read_latency_metric()), write_latency_(&engine_write_latency_metric()), fsync_latency_(&engine_fsync_latency_metric()), @@ -69,7 +68,7 @@ DiskioServer::DiskioServer(std::shared_ptr<DiskSet> disk_set, std::shared_ptr<cr // Build a diskio response control buffer (flatbuffer). crowdb::rpc::Buffer *DiskioServer::build_response_ctrl(crowdb::rpc::BufferPool *pool, uint64_t request_id, uint64_t rpc_create_nano, int16_t ret_code, uint16_t msg_type, - bool sync_completed) + bool sync_completed, bool read_age_checked) { auto fb_ret = static_cast<dproto::FBDiskIoRetCode>(ret_code); flatbuffers::FlatBufferBuilder fbb(64); @@ -78,7 +77,7 @@ crowdb::rpc::Buffer *DiskioServer::build_response_ctrl(crowdb::rpc::BufferPool * fbb.Finish(off); } else if (msg_type == static_cast<uint16_t>(rproto::FBMsgType_EDiskReadResponse)) { - auto off = dproto::CreateFBDiskReadResponse(fbb, request_id, rpc_create_nano, fb_ret); + auto off = dproto::CreateFBDiskReadResponse(fbb, request_id, rpc_create_nano, fb_ret, read_age_checked); fbb.Finish(off); } else { // fsync response @@ -103,16 +102,19 @@ void DiskioServer::send_error_response(crowdb::rpc::Connection *conn, uint64_t r void DiskioServer::send_response(const std::shared_ptr<crowdb::rpc::Transport> &transport, const std::shared_ptr<crowdb::rpc::Connection> &connection, uint64_t request_id, uint64_t created, uint16_t type, int16_t code, crowdb::rpc::Buffer *data, - bool sync_completed) + bool sync_completed, bool read_age_checked) { const auto release = [](crowdb::rpc::Buffer *buffer) { - if (buffer != nullptr) + if (buffer != nullptr) { buffer->release(); + } }; std::unique_ptr<crowdb::rpc::Buffer, decltype(release)> payload(data, release); - if (connection == nullptr || !connection->is_open()) + if (connection == nullptr || !connection->is_open()) { return; - auto *control = build_response_ctrl(connection->pool(), request_id, created, code, type, sync_completed); + } + auto *control = + build_response_ctrl(connection->pool(), request_id, created, code, type, sync_completed, read_age_checked); if (control == nullptr) { connection->close(); return; diff --git a/app/crowdb-diskio/src/rpc/dio_server.h b/app/crowdb-diskio/src/rpc/dio_server.h index e235c031f..9d84245d0 100644 --- a/app/crowdb-diskio/src/rpc/dio_server.h +++ b/app/crowdb-diskio/src/rpc/dio_server.h @@ -69,7 +69,7 @@ class DiskioServer // pulling the generated header into this .h). static crowdb::rpc::Buffer *build_response_ctrl(crowdb::rpc::BufferPool *pool, uint64_t request_id, uint64_t rpc_create_nano, int16_t ret_code, uint16_t msg_type, - bool sync_completed = false); + bool sync_completed = false, bool read_age_checked = false); // Send an error response synchronously (disk not found, parse error). void send_error_response(crowdb::rpc::Connection *conn, uint64_t request_id, uint64_t rpc_create_nano, @@ -78,7 +78,7 @@ class DiskioServer static void send_response(const std::shared_ptr<crowdb::rpc::Transport> &transport, const std::shared_ptr<crowdb::rpc::Connection> &connection, uint64_t request_id, uint64_t created, uint16_t type, int16_t code, crowdb::rpc::Buffer *data = nullptr, - bool sync_completed = false); + bool sync_completed = false, bool read_age_checked = false); }; } // namespace crowdb::diskio diff --git a/app/crowdb-diskio/src/rpc/dio_server_write.cpp b/app/crowdb-diskio/src/rpc/dio_server_write.cpp index 8362b90ef..74033ddbd 100644 --- a/app/crowdb-diskio/src/rpc/dio_server_write.cpp +++ b/app/crowdb-diskio/src/rpc/dio_server_write.cpp @@ -4,6 +4,7 @@ #include "crowdb-common/metrics/metrics.h" #include "crowdb-rpc/server/message.h" #include "disk/disk.h" +#include "rpc/dio_request_deadline.h" #include "rpc/dio_server.h" #include <diskio_generated.h> @@ -23,41 +24,12 @@ namespace rproto = crowdb::rpc::proto; namespace { -uint64_t unix_time_ms() -{ - return static_cast<uint64_t>( - std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch()) - .count()); -} - uint64_t elapsed_nanos(std::chrono::steady_clock::time_point started) { return static_cast<uint64_t>( std::chrono::duration_cast<std::chrono::nanoseconds>(std::chrono::steady_clock::now() - started).count()); } -std::optional<std::chrono::steady_clock::time_point> write_deadline(uint64_t created_ms, uint64_t max_age_ms, - uint64_t skew_ms) -{ - const auto steady_now = std::chrono::steady_clock::now(); - const auto now_ms = unix_time_ms(); - if (created_ms == 0 || max_age_ms == 0 || (created_ms > now_ms && created_ms - now_ms > skew_ms) || - max_age_ms > UINT64_MAX - skew_ms) { - return std::nullopt; - } - const auto allowance = max_age_ms + skew_ms; - if (created_ms > UINT64_MAX - allowance || created_ms + allowance <= now_ms) { - return std::nullopt; - } - const auto remaining_ms = created_ms + allowance - now_ms; - const auto max_wait_ms = - std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::time_point::max() - steady_now) - .count(); - if (std::cmp_greater(remaining_ms, max_wait_ms)) { - return std::nullopt; - } - return steady_now + std::chrono::milliseconds(static_cast<int64_t>(remaining_ms)); -} } // namespace crowdb::rpc::OutFrame *DiskioServer::handle_write(crowdb::rpc::Frame *request, crowdb::rpc::Connection *conn) @@ -82,7 +54,7 @@ crowdb::rpc::OutFrame *DiskioServer::handle_write(crowdb::rpc::Frame *request, c uint64_t write_create_time_ms = fb_req->write_create_time_ms(); const bool sync = fb_req->sync(); - const auto deadline = write_deadline(write_create_time_ms, max_write_request_age_ms_, max_clock_skew_ms_); + const auto deadline = request_deadline(write_create_time_ms, max_write_request_age_ms_, max_clock_skew_ms_); if (!deadline) { delete request; send_error_response(conn, req_id, create_nano, msg_type, diff --git a/app/crowdb-diskio/tests/dio_lifetime_test.cpp b/app/crowdb-diskio/tests/dio_lifetime_test.cpp index 86f55a4a2..f4e64dbfd 100644 --- a/app/crowdb-diskio/tests/dio_lifetime_test.cpp +++ b/app/crowdb-diskio/tests/dio_lifetime_test.cpp @@ -32,19 +32,21 @@ class TestDeferredEngine final : public IoEngine std::vector<Pending> pending; - void submit_write(Disk *, off_t, const uint8_t *data, size_t size, std::function<void(int)> done) override + void submit_write(Disk * /*unused*/, off_t /*unused*/, const uint8_t *data, size_t size, + std::function<void(int)> done) override { - pending.push_back({data, nullptr, size, std::move(done)}); + pending.push_back({.data = data, .output = nullptr, .size = size, .done = std::move(done)}); } - void submit_read(Disk *, off_t, uint8_t *data, size_t size, uint64_t, std::function<void(int)> done) override + void submit_read(Disk * /*unused*/, off_t /*unused*/, uint8_t *data, size_t size, uint64_t /*unused*/, + std::function<void(int)> done) override { - pending.push_back({nullptr, data, size, std::move(done)}); + pending.push_back({.data = nullptr, .output = data, .size = size, .done = std::move(done)}); } - void submit_fsync(Disk *, std::function<void(int)> done) override + void submit_fsync(Disk * /*unused*/, std::function<void(int)> done) override { - pending.push_back({nullptr, nullptr, 0, std::move(done)}); + pending.push_back({.data = nullptr, .output = nullptr, .size = 0, .done = std::move(done)}); } void complete(int result) @@ -52,8 +54,9 @@ class TestDeferredEngine final : public IoEngine ASSERT_FALSE(pending.empty()); auto io = std::move(pending.front()); pending.erase(pending.begin()); - if (io.output != nullptr && result > 0) + if (io.output != nullptr && result > 0) { std::memset(io.output, 0x5a, std::min(io.size, static_cast<size_t>(result))); + } io.done(result); } }; @@ -66,22 +69,22 @@ class TestDeferredDisk final : public Disk zones_.push_back({.zone_index = 0, .base_offset = 0, .capacity = 8 << 20}); } - DiskType type() const override + [[nodiscard]] DiskType type() const override { return DiskType::Mem; } - int fd() const override + [[nodiscard]] int fd() const override { return 1; } - bool is_o_direct() const override + [[nodiscard]] bool is_o_direct() const override { return false; } - size_t block_size() const override + [[nodiscard]] size_t block_size() const override { return 1; } @@ -91,14 +94,14 @@ class TestDeferredDisk final : public Disk return engine_; } - DiskId id() const override + [[nodiscard]] DiskId id() const override { - return {7, 9}; + return {.high = 7, .low = 9}; } Zone *find_zone(uint32_t index) override { - return index == 0 ? &zones_[0] : nullptr; + return index == 0 ? zones_.data() : nullptr; } private: @@ -142,11 +145,11 @@ class DioLifetimeTest : public ::testing::Test return frame; } - Frame *read(uint32_t size) + Frame *read(uint32_t size, uint64_t created_ms = 0, bool probe = false) { flatbuffers::FlatBufferBuilder builder; crowdb::rpc::proto::FBInt128 disk(7, 9); - builder.Finish(proto::CreateFBDiskReadRequest(builder, 19, 0, &disk, 0, 0, size)); + builder.Finish(proto::CreateFBDiskReadRequest(builder, 19, 0, &disk, 0, 0, size, 0, created_ms, probe)); auto *frame = new Frame; frame->request_id = 19; frame->control.assign(builder.GetBufferPointer(), builder.GetBufferPointer() + builder.GetSize()); @@ -154,6 +157,24 @@ class DioLifetimeTest : public ::testing::Test } }; +TEST_F(DioLifetimeTest, ExpiredProtectedReadDoesNotSubmitDiskIo) +{ + EXPECT_EQ(service->handle_read(read(4096, 1), connection.get()), nullptr); + EXPECT_TRUE(engine.pending.empty()); +} + +TEST_F(DioLifetimeTest, ProtectedReadProbeNeverAllocatesOrSubmitsDataIo) +{ + const auto now = static_cast<uint64_t>( + std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch()) + .count()); + EXPECT_EQ(service->handle_read(read(0, now, true), connection.get()), nullptr); + EXPECT_TRUE(engine.pending.empty()); + EXPECT_EQ(service->handle_read(read(4096, now), connection.get()), nullptr); + ASSERT_EQ(engine.pending.size(), 1U); + engine.complete(4096); +} + TEST_F(DioLifetimeTest, DiskOwnsExactPayloadUntilCompletionWithoutReplacementAdmission) { RegisteredBufferPool buffers(4096, 4); diff --git a/app/crowdb-diskio/tests/ordered_drain_test.cpp b/app/crowdb-diskio/tests/ordered_drain_test.cpp new file mode 100644 index 000000000..380c465c5 --- /dev/null +++ b/app/crowdb-diskio/tests/ordered_drain_test.cpp @@ -0,0 +1,31 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +#include "common/aligned_writer.h" + +#include <gtest/gtest.h> + +#include <cerrno> +#include <chrono> + +namespace crowdb::diskio +{ +TEST(OrderedDiskDrain, ZeroByteBarrierWaitsForActiveWriteAndExpiredLateWriteCannotFollow) +{ + test::TestRecordingEngine engine(8192, true); + auto disk = std::make_shared<test::TestDisk>(&engine, 1); + AlignedWriter writer; + std::vector<uint8_t> data(32, 9); + std::vector<int> completions; + writer.submit_ordered(disk, 0, data.data(), data.size(), 0, [&](int result) { completions.push_back(result); }); + writer.submit_ordered(disk, 0, nullptr, 0, 0, [&](int result) { completions.push_back(result); }); + EXPECT_TRUE(completions.empty()); + EXPECT_EQ(engine.pending_writes.size(), 1U); + writer.submit_ordered( + disk, 32, data.data(), data.size(), 0, [&](int result) { completions.push_back(result); }, + std::chrono::steady_clock::now() - std::chrono::seconds(1)); + engine.complete_one_write(); + EXPECT_EQ(completions, (std::vector<int>{32, 0, -ETIMEDOUT})); + EXPECT_EQ(engine.write_sizes.size(), 1U); + EXPECT_TRUE(engine.pending_writes.empty()); +} +} // namespace crowdb::diskio diff --git a/app/crowdb-kv-server/Cargo.toml b/app/crowdb-kv-server/Cargo.toml deleted file mode 100644 index 06389f473..000000000 --- a/app/crowdb-kv-server/Cargo.toml +++ /dev/null @@ -1,109 +0,0 @@ -[package] -name = "crowdb-kv-server" -version.workspace = true -edition.workspace = true -rust-version.workspace = true -license.workspace = true -repository.workspace = true -description = "CROWDB server binary: single-group, multi-group, and cluster modes." -autotests = false - -[[bin]] -name = "crowdb-kv-server" -path = "src/main.rs" - -[lints] -workspace = true - -[features] -default = [] - -[dependencies] -crowdb-kv = { path = "../../lib/crowdb-kv" } -crowdb-kv-client = { path = "../../lib/crowdb-kv-client" } -crowdb-common = { path = "../../lib/crowdb-common/rust" } -crowdb-protocol = { path = "../../lib/crowdb-protocol", features = ["schema"] } -crowdb-tree-ffi = { path = "../../lib/crowdb-tree/ffi" } -crowdb-rpc-ffi = { path = "../../lib/crowdb-rpc/ffi" } -tokio = { workspace = true, features = ["rt-multi-thread", "macros", "time", "signal"] } -tracing = { workspace = true } -clap = { version = "4", features = ["derive", "env"] } -axum = "0.7" -serde = { version = "1", features = ["derive"] } -serde_json = "1" -dashmap = "6" -arc-swap = "1.9" -utoipa = "5" -bytes = "1" -sha2 = "0.10" -uuid = { version = "1", features = ["v4"] } - -[dev-dependencies] -crowdb-kv = { path = "../../lib/crowdb-kv", features = ["test-util"] } -reqwest = { version = "0.12", features = ["json"] } -crowdb-rpc-ffi = { path = "../../lib/crowdb-rpc/ffi", features = ["test-util"] } -crowdb-test-harness = { path = "../../lib/crowdb-test-harness" } - -[[test]] -name = "keepalive_recovery_test" -path = "tests/keepalive_recovery_test.rs" - -[[test]] -name = "group_creation_test" -path = "tests/group_creation_test.rs" - -[[test]] -name = "member_routes_test" -path = "tests/member_routes_test.rs" - -[[test]] -name = "chunk_dynamic_monitor_test" -path = "tests/chunk_dynamic_monitor_test.rs" - -[[test]] -name = "group0_discovery_test" -path = "tests/group0_discovery_test.rs" - -[[test]] -name = "cli_parse_test" -path = "tests/cli_parse_test.rs" - - - -[[test]] -name = "monitor_supervisor_test" -path = "tests/monitor_supervisor_test.rs" - -[[test]] -name = "group0_control_plane_test" -path = "tests/group0_control_plane_test.rs" - -[[test]] -name = "domain_monitor_test" -path = "tests/domain_monitor_test.rs" - -[[test]] -name = "chunk_kv_balance_planning_test" -path = "tests/chunk_kv_balance_planning_test.rs" - -[[test]] -name = "chunk_slot_monitor_test" -path = "tests/chunk_slot_monitor_test.rs" - -[[test]] -name = "reconcile_test" -path = "tests/reconcile_test.rs" - -[[test]] -name = "restore_test" -path = "tests/restore_test.rs" - - -[[test]] -name = "startup_test" -path = "tests/startup_test.rs" - - -[[test]] -name = "diskdb_placement_test" -path = "tests/diskdb_placement_test.rs" diff --git a/app/crowdb-kv-server/conf/crowdb_kv_server_config.toml b/app/crowdb-kv-server/conf/crowdb_kv_server_config.toml deleted file mode 100644 index 815121855..000000000 --- a/app/crowdb-kv-server/conf/crowdb_kv_server_config.toml +++ /dev/null @@ -1,8 +0,0 @@ -# Canonical crowdb-kv-server startup tunables. -# Runtime paths are derived from --root and are not file settings. - -wal_early_ack = true -async_engine_apply = true - -[server] -rpc_workers = 2 diff --git a/app/crowdb-kv-server/src/background/domain_monitor.rs b/app/crowdb-kv-server/src/background/domain_monitor.rs deleted file mode 100644 index fafc63bbd..000000000 --- a/app/crowdb-kv-server/src/background/domain_monitor.rs +++ /dev/null @@ -1,336 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -//! Descriptor-driven, leader-tenure-fenced domain monitor supervision. - -use std::collections::{HashMap, HashSet}; -use std::future::Future; -use std::pin::Pin; -use std::sync::Arc; -use std::time::Duration; - -use bytes::Bytes; -use crowdb_kv::cluster::group_operations::{ - KvGroupOperationError, KvGroupOperations, KvGroupScanRequest, KvReadConsistency, -}; -use crowdb_protocol::chunk_kv::DomainMonitorDescriptor; -use crowdb_protocol::key::{DomainMonitorKey, TextKey}; -use tracing::{debug, info, warn}; - -use crate::group0_control_plane::Group0ControlPlane; -use crate::store_registry::KvStoreRegistry; - -mod chunk_kv; -mod chunkdb; -mod chunkdb_dynamic; -mod diskdb; -mod diskdb_placement; - -pub use chunk_kv::ChunkKvRangeMonitorDriver; -pub use chunkdb::ChunkdbRangeMonitorDriver; -pub use chunkdb_dynamic::ChunkdbDynamicMonitorDriver; -pub use diskdb::DiskdbOwnershipMonitorDriver; -pub use diskdb_placement::DiskdbPlacementMonitorDriver; - -/// Run one monitor child at a time and reconstruct it after an unexpected -/// return or panic. -pub fn spawn_restarting_monitor<F, Fut>( - mut stop_rx: tokio::sync::watch::Receiver<bool>, - make_monitor: F, -) -> tokio::task::JoinHandle<()> -where - F: Fn(tokio::sync::watch::Receiver<bool>) -> Fut + Send + Sync + 'static, - Fut: Future<Output = ()> + Send + 'static, -{ - tokio::spawn(async move { - loop { - if *stop_rx.borrow() { - break; - } - let mut child = tokio::spawn(make_monitor(stop_rx.clone())); - tokio::select! { - result = &mut child => { - if !*stop_rx.borrow() { - warn!(?result, "domain monitor task exited; restarting"); - tokio::task::yield_now().await; - } - } - changed = stop_rx.changed() => { - if changed.is_err() || *stop_rx.borrow() { - let _ = child.await; - break; - } - } - } - } - }) -} - -pub type DomainMonitorFuture<'a> = Pin<Box<dyn Future<Output = Result<(), String>> + Send + 'a>>; - -/// One compiled monitor implementation selected by a persisted descriptor. -pub trait DomainMonitorDriver: Send + Sync { - fn domain(&self) -> &'static str; - fn driver_version(&self) -> u32; - fn tick<'a>( - &'a self, - control: &'a Group0ControlPlane, - descriptor: &'a DomainMonitorDescriptor, - ) -> DomainMonitorFuture<'a>; -} - -#[derive(Default)] -pub struct DomainMonitorDrivers { - drivers: HashMap<(String, u32), Arc<dyn DomainMonitorDriver>>, -} - -impl DomainMonitorDrivers { - #[must_use] - pub fn new(drivers: Vec<Arc<dyn DomainMonitorDriver>>) -> Self { - let drivers = drivers - .into_iter() - .map(|driver| ((driver.domain().to_string(), driver.driver_version()), driver)) - .collect(); - Self { drivers } - } - - fn find(&self, descriptor: &DomainMonitorDescriptor) -> Option<Arc<dyn DomainMonitorDriver>> { - self.drivers - .get(&(descriptor.domain.clone(), descriptor.driver_version)) - .cloned() - } -} - -pub struct DomainMonitorSupervisorHandle { - stop_tx: tokio::sync::watch::Sender<bool>, - supervisor: Option<tokio::task::JoinHandle<()>>, -} - -impl DomainMonitorSupervisorHandle { - pub fn stop(&self) { - let _ = self.stop_tx.send(true); - } - - pub async fn stop_and_wait(mut self) { - self.stop(); - if let Some(supervisor) = self.supervisor.take() { - let _ = supervisor.await; - } - } -} - -impl Drop for DomainMonitorSupervisorHandle { - fn drop(&mut self) { - let _ = self.stop_tx.send(true); - } -} - -struct RunningMonitor { - descriptor: DomainMonitorDescriptor, - stop_tx: tokio::sync::watch::Sender<bool>, - supervisor: tokio::task::JoinHandle<()>, -} - -/// Observe persisted descriptors on this replica and prepare one supervised -/// task for every supported domain. Driver ticks acquire their own local -/// group-0 leader tenure; followers therefore stay prepared but idle. -#[must_use] -pub fn spawn_domain_monitor_supervisor( - registry: Arc<KvStoreRegistry>, - drivers: DomainMonitorDrivers, - discovery_interval: Duration, -) -> DomainMonitorSupervisorHandle { - let drivers = Arc::new(drivers); - let (stop_tx, mut stop_rx) = tokio::sync::watch::channel(false); - let supervisor = tokio::spawn(async move { - let mut running: HashMap<String, RunningMonitor> = HashMap::new(); - let mut interval = tokio::time::interval(discovery_interval); - interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); - loop { - tokio::select! { - _ = interval.tick() => { - match discover_descriptors(®istry).await { - Ok(descriptors) => reconcile_descriptors( - ®istry, - &drivers, - descriptors, - &mut running, - ), - Err(error) => warn!(%error, "domain monitor descriptor discovery failed"), - } - } - changed = stop_rx.changed() => { - if changed.is_err() || *stop_rx.borrow() { - break; - } - } - } - } - for (_, monitor) in running.drain() { - let _ = monitor.stop_tx.send(true); - let _ = monitor.supervisor.await; - } - }); - DomainMonitorSupervisorHandle { - stop_tx, - supervisor: Some(supervisor), - } -} - -async fn discover_descriptors( - registry: &KvStoreRegistry, -) -> Result<Vec<DomainMonitorDescriptor>, KvGroupOperationError> { - let store = registry - .get_store(0) - .ok_or_else(|| KvGroupOperationError::Unavailable("store 0 is not hosted locally".into()))?; - let group = store - .get_group(0) - .ok_or_else(|| KvGroupOperationError::Unavailable("group 0 is not hosted locally".into()))?; - let operations = KvGroupOperations::new(group, 1024 * 1024); - let prefix = Bytes::from(DomainMonitorKey::text_prefix_all()); - let mut start_after = Bytes::new(); - let mut scan_cutoff = 0; - let mut descriptors = Vec::new(); - loop { - let scan = operations - .scan(&KvGroupScanRequest { - prefix: prefix.clone(), - start_after: start_after.clone(), - end_key: Bytes::new(), - limit: 128, - consistency: KvReadConsistency::MinAppliedSlot(0), - keys_only: false, - count_only: false, - deadline_ms: 0, - bounded: true, - requested_scan_cutoff: scan_cutoff, - direction: crowdb_kv::kv::ScanDirection::Forward, - }) - .await?; - scan_cutoff = scan.scan_cutoff; - for item in &scan.items { - let descriptor: DomainMonitorDescriptor = serde_json::from_slice(&item.value) - .map_err(|error| KvGroupOperationError::Internal(error.to_string()))?; - descriptor - .validate() - .map_err(|error| KvGroupOperationError::Internal(error.to_string()))?; - let path = std::str::from_utf8(&item.key) - .map_err(|error| KvGroupOperationError::Internal(error.to_string()))?; - let key = DomainMonitorKey::from_path(path) - .map_err(|error| KvGroupOperationError::Internal(error.to_string()))?; - if key.domain != descriptor.domain { - return Err(KvGroupOperationError::Internal(format!( - "monitor descriptor key domain {} does not match value {}", - key.domain, descriptor.domain - ))); - } - descriptors.push(descriptor); - } - let Some(last) = scan.items.last() else { - break; - }; - if !scan.truncated { - break; - } - start_after = last.key.clone(); - } - Ok(descriptors) -} - -fn reconcile_descriptors( - registry: &Arc<KvStoreRegistry>, - drivers: &DomainMonitorDrivers, - descriptors: Vec<DomainMonitorDescriptor>, - running: &mut HashMap<String, RunningMonitor>, -) { - let observed: HashSet<_> = descriptors - .iter() - .map(|descriptor| descriptor.domain.clone()) - .collect(); - let removed: Vec<_> = running - .keys() - .filter(|domain| !observed.contains(*domain)) - .cloned() - .collect(); - for domain in removed { - if let Some(monitor) = running.remove(&domain) { - let _ = monitor.stop_tx.send(true); - monitor.supervisor.abort(); - info!(%domain, "domain monitor descriptor removed; task stopped"); - } - } - - for descriptor in descriptors { - if let Some(existing) = running.get(&descriptor.domain) { - if existing.descriptor != descriptor { - warn!(domain = %descriptor.domain, "domain monitor descriptor changed unexpectedly; retaining original task fail-closed"); - } - continue; - } - let Some(driver) = drivers.find(&descriptor) else { - warn!( - domain = %descriptor.domain, - driver_version = descriptor.driver_version, - "unsupported domain monitor descriptor" - ); - continue; - }; - let monitor = spawn_driver(registry, descriptor.clone(), driver); - running.insert(descriptor.domain.clone(), monitor); - } -} - -fn spawn_driver( - registry: &Arc<KvStoreRegistry>, - descriptor: DomainMonitorDescriptor, - driver: Arc<dyn DomainMonitorDriver>, -) -> RunningMonitor { - let (stop_tx, stop_rx) = tokio::sync::watch::channel(false); - let registry = Arc::clone(registry); - let child_descriptor = descriptor.clone(); - let supervisor = spawn_restarting_monitor(stop_rx, move |mut child_stop| { - let registry = Arc::clone(®istry); - let descriptor = child_descriptor.clone(); - let driver = Arc::clone(&driver); - async move { - let mut interval = tokio::time::interval(Duration::from_millis(descriptor.heartbeat_interval_ms)); - interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); - loop { - tokio::select! { - _ = interval.tick() => run_driver_tick(®istry, driver.as_ref(), &descriptor).await, - changed = child_stop.changed() => { - if changed.is_err() || *child_stop.borrow() { - break; - } - } - } - } - } - }); - RunningMonitor { - descriptor, - stop_tx, - supervisor, - } -} - -async fn run_driver_tick( - registry: &KvStoreRegistry, - driver: &dyn DomainMonitorDriver, - descriptor: &DomainMonitorDescriptor, -) { - let Some(store) = registry.get_store(0) else { - return; - }; - match Group0ControlPlane::acquire(&store).await { - Ok(control) => { - if let Err(error) = driver.tick(&control, descriptor).await { - warn!(domain = %descriptor.domain, %error, "domain monitor tick failed"); - } - } - Err(KvGroupOperationError::NotLeader { .. }) => { - debug!(domain = %descriptor.domain, "domain monitor idle on group-0 follower"); - } - Err(error) => warn!(domain = %descriptor.domain, %error, "domain monitor leader fence unavailable"), - } -} diff --git a/app/crowdb-kv-server/src/lib.rs b/app/crowdb-kv-server/src/lib.rs deleted file mode 100644 index ad87c785f..000000000 --- a/app/crowdb-kv-server/src/lib.rs +++ /dev/null @@ -1,17 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -//! `crowdb-kv-server` library — exposes the binary's modules so integration -//! tests under `tests/` can exercise CLI parsing, the management router, -//! and the registry without spawning a process. -//! -//! The binary entry (`main.rs`) imports from this lib via -//! `use crowdb_kv_server::{cli, mgmt, startup, store_registry};`. - -pub mod background; -pub mod cli; -pub mod engine_collector; -pub mod group0_control_plane; -pub mod mgmt; -pub mod recovery; -pub mod store_registry; diff --git a/app/crowdb-kv-server/src/mgmt/group_creation.rs b/app/crowdb-kv-server/src/mgmt/group_creation.rs deleted file mode 100644 index b55412479..000000000 --- a/app/crowdb-kv-server/src/mgmt/group_creation.rs +++ /dev/null @@ -1,249 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -//! Durable creation ownership precedes ordinary-group WAL and engine effects. - -use std::{ - fs::{self, File, OpenOptions}, - io::{Read, Write}, - os::unix::fs::{OpenOptionsExt, PermissionsExt}, - path::{Path, PathBuf}, -}; - -use axum::{ - extract::{Path as AxumPath, State}, - http::StatusCode, - Json, -}; -use crowdb_protocol::mgmt::{AddGroupRequest, GroupCreationStatus}; - -use super::{err_json, system_bootstrap::ManagementError, RegistryArc}; - -pub(super) async fn status( - State(state): State<RegistryArc>, - AxumPath((store, group)): AxumPath<(u64, u64)>, -) -> Result<Json<GroupCreationStatus>, ManagementError> { - read(&state.config.config_root, store, group)? - .map(Json) - .ok_or_else(|| err_json(StatusCode::NOT_FOUND, "group creation ownership is absent")) -} - -pub(super) async fn accept( - state: &RegistryArc, - store: u64, - request: &AddGroupRequest, -) -> Result<(), ManagementError> { - if store == 0 && request.group_id == 0 { - return if request.deployment.is_none() { - Ok(()) - } else { - Err(err_json( - StatusCode::BAD_REQUEST, - "system group must use system bootstrap ownership", - )) - }; - } - let expected = GroupCreationStatus { - version: 1, - store_id: store, - group_id: request.group_id, - replica_id: request.replica_id, - deployment: request.deployment.clone(), - retired: false, - }; - validate(&expected)?; - let root = &state.config.config_root; - if let Some(current) = read(root, store, request.group_id)? { - return matches(¤t, &expected); - } - if state - .get_store(store) - .is_some_and(|store| store.get_group(request.group_id).is_some()) - { - return if request.deployment.is_none() { - Ok(()) - } else { - Err(err_json( - StatusCode::CONFLICT, - "existing group has no matching deployment ownership", - )) - }; - } - let persisted = crowdb_kv::cluster::node_config::NodeConfigStore::new(root) - .load() - .await - .map_err(storage_error)?; - if persisted - .group(store, request.group_id) - .is_some_and(|group| group.replica_id != request.replica_id) - { - return Err(err_json( - StatusCode::CONFLICT, - "persisted group replica identity differs", - )); - } - if request.deployment.is_some() - && (persisted.group(store, request.group_id).is_some() - || crate::recovery::startup::store_crowdb_tree_path( - &state.config.data_root, - store, - request.group_id, - ) - .exists() - || crate::recovery::startup::store_wal_root(&state.config.wal_root, store) - .join(format!("group{}", request.group_id)) - .exists()) - { - return Err(err_json( - StatusCode::CONFLICT, - "existing group has no matching deployment ownership", - )); - } - fs::create_dir_all(root).map_err(storage_error)?; - publish(root, &expected, false)?; - let current = read(root, store, request.group_id)? - .ok_or_else(|| err_json(StatusCode::CONFLICT, "group creation ownership disappeared"))?; - matches(¤t, &expected) -} - -pub(super) fn retire(state: &RegistryArc, store: u64, group: u64) -> Result<(), ManagementError> { - if let Some(mut current) = read(&state.config.config_root, store, group)? { - if !current.retired { - current.retired = true; - publish(&state.config.config_root, ¤t, true)?; - } - } - Ok(()) -} - -pub(super) fn finish_removal(state: &RegistryArc, store: u64, group: u64) -> Result<(), ManagementError> { - let root = &state.config.config_root; - if read(root, store, group)?.is_some_and(|current| current.deployment.is_none() && current.retired) { - fs::remove_file(path(root, store, group)).map_err(storage_error)?; - File::open(root) - .and_then(|file| file.sync_all()) - .map_err(storage_error)?; - } - Ok(()) -} - -pub(crate) fn check_restore(root: &Path, store: u64, group: u64, replica: u64) -> std::io::Result<()> { - let status = read(root, store, group).map_err(|_| { - std::io::Error::new( - std::io::ErrorKind::InvalidData, - "invalid durable group creation ownership", - ) - })?; - if status.is_some_and(|status| status.retired || status.replica_id != replica) { - return Err(std::io::Error::new( - std::io::ErrorKind::PermissionDenied, - "group creation is retired or has another replica identity", - )); - } - Ok(()) -} - -fn matches(current: &GroupCreationStatus, expected: &GroupCreationStatus) -> Result<(), ManagementError> { - if current != expected { - return Err(err_json( - StatusCode::CONFLICT, - "group creation is retired or owned by another operation", - )); - } - Ok(()) -} - -fn path(root: &Path, store: u64, group: u64) -> PathBuf { - root.join(format!("group-creation-{store}-{group}.json")) -} - -fn read(root: &Path, store: u64, group: u64) -> Result<Option<GroupCreationStatus>, ManagementError> { - let path = path(root, store, group); - let metadata = match fs::symlink_metadata(&path) { - Ok(metadata) => metadata, - Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None), - Err(error) => return Err(storage_error(error)), - }; - if !metadata.file_type().is_file() - || metadata.len() > 4096 - || metadata.permissions().mode() & 0o777 != 0o600 - { - return Err(err_json( - StatusCode::CONFLICT, - "invalid durable group creation ownership", - )); - } - let mut bytes = Vec::new(); - File::open(path) - .map_err(storage_error)? - .take(4097) - .read_to_end(&mut bytes) - .map_err(storage_error)?; - if bytes.len() > 4096 { - return Err(err_json( - StatusCode::CONFLICT, - "group ownership exceeds its size bound", - )); - } - let status: GroupCreationStatus = serde_json::from_slice(&bytes) - .map_err(|_| err_json(StatusCode::CONFLICT, "invalid durable group creation ownership"))?; - validate(&status)?; - if status.store_id != store || status.group_id != group { - return Err(err_json( - StatusCode::CONFLICT, - "group creation ownership path differs", - )); - } - Ok(Some(status)) -} - -fn validate(status: &GroupCreationStatus) -> Result<(), ManagementError> { - let valid_identity = status.deployment.as_ref().map_or(true, |identity| { - [&identity.cluster_id, &identity.operation_id] - .iter() - .all(|value| { - uuid::Uuid::parse_str(value).is_ok_and(|id| !id.is_nil() && id.to_string() == value.as_str()) - }) - }); - if status.version != 1 || status.replica_id == 0 || !valid_identity { - return Err(err_json(StatusCode::CONFLICT, "invalid group creation identity")); - } - Ok(()) -} - -fn publish(root: &Path, status: &GroupCreationStatus, replace: bool) -> Result<(), ManagementError> { - let temporary = root.join(format!(".group-creation-{}.tmp", uuid::Uuid::new_v4())); - let result = (|| { - let mut file = OpenOptions::new() - .create_new(true) - .write(true) - .mode(0o600) - .open(&temporary)?; - file.write_all(&serde_json::to_vec(status).map_err(std::io::Error::other)?)?; - file.sync_all()?; - let target = path(root, status.store_id, status.group_id); - if replace { - fs::rename(&temporary, target)?; - } else { - match fs::hard_link(&temporary, target) { - Ok(()) => {} - Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => {} - Err(error) => return Err(error), - } - } - File::open(root)?.sync_all() - })(); - match fs::remove_file(temporary) { - Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} - Err(error) if result.is_ok() => return Err(storage_error(error)), - Ok(()) | Err(_) => {} - } - result.map_err(storage_error) -} - -fn storage_error(error: impl std::fmt::Display) -> ManagementError { - err_json( - StatusCode::INTERNAL_SERVER_ERROR, - format!("group ownership persistence failed: {error}"), - ) -} diff --git a/app/crowdb-kv-server/src/mgmt/system_bootstrap.rs b/app/crowdb-kv-server/src/mgmt/system_bootstrap.rs deleted file mode 100644 index 4a5cef094..000000000 --- a/app/crowdb-kv-server/src/mgmt/system_bootstrap.rs +++ /dev/null @@ -1,228 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -//! Durable system-store ownership, published before any store creation. - -use std::fs::{self, File, OpenOptions}; -use std::io::{Read, Write}; -use std::os::unix::fs::{OpenOptionsExt, PermissionsExt}; -use std::path::Path; -use std::sync::atomic::{AtomicU64, Ordering}; -use std::sync::Arc; - -use axum::{extract::State, http::StatusCode, Json}; -use crowdb_protocol::mgmt::{SystemBootstrapIdentity, SystemPrepareRequest}; -use serde::{Deserialize, Serialize}; - -use super::{err_json, ErrorResponse, RegistryArc}; - -pub(super) type ManagementError = (StatusCode, Json<ErrorResponse>); -const OWNERSHIP_FILE: &str = "system-bootstrap.json"; -const RETIRED_FILE: &str = "system-bootstrap-cleanup.json"; -static NEXT_TEMP: AtomicU64 = AtomicU64::new(0); - -#[derive(Debug, PartialEq, Eq, Serialize, Deserialize)] -#[serde(deny_unknown_fields)] -struct AcceptedBootstrap { - version: u32, - replica_id: u64, - identity: SystemBootstrapIdentity, -} - -pub(super) struct BootstrapExecution(Arc<crate::store_registry::KvStoreRegistry>); - -impl Drop for BootstrapExecution { - fn drop(&mut self) { - self.0.bootstrap_executing.store(false, Ordering::Release); - } -} - -pub(super) fn begin(state: &RegistryArc) -> Result<BootstrapExecution, ManagementError> { - state - .bootstrap_executing - .compare_exchange(false, true, Ordering::Acquire, Ordering::Relaxed) - .map_err(|_| { - err_json( - StatusCode::CONFLICT, - "node group initialization is executing; retry the same operation", - ) - })?; - Ok(BootstrapExecution(Arc::clone(&state.registry))) -} - -pub(super) async fn prepare( - State(state): State<RegistryArc>, - Json(req): Json<SystemPrepareRequest>, -) -> Result<Json<SystemPrepareRequest>, ManagementError> { - let _execution = begin(&state)?; - accept(&state, req.replica_id, Some(&req.bootstrap))?; - Ok(Json(req)) -} - -pub(super) fn has_owner(state: &RegistryArc) -> Result<bool, ManagementError> { - match fs::symlink_metadata(state.config.config_root.join(OWNERSHIP_FILE)) { - Ok(_) => Ok(true), - Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(false), - Err(error) => Err(storage_error(&error)), - } -} - -pub(super) fn accept( - state: &RegistryArc, - replica_id: u64, - identity: Option<&SystemBootstrapIdentity>, -) -> Result<(), ManagementError> { - let root = &state.config.config_root; - let path = root.join(OWNERSHIP_FILE); - if root.join(RETIRED_FILE).exists() { - return Err(err_json(StatusCode::CONFLICT, "system cleanup has not completed")); - } - let Some(identity) = identity else { - return if has_owner(state)? { - Err(err_json( - StatusCode::CONFLICT, - "system store belongs to an identified bootstrap operation", - )) - } else { - Ok(()) - }; - }; - validate(replica_id, identity)?; - if root - .join(format!("system-bootstrap-retired-{}.json", identity.operation_id)) - .exists() - { - return Err(err_json( - StatusCode::CONFLICT, - "bootstrap operation was explicitly retired", - )); - } - if root.join(RETIRED_FILE).exists() { - return Err(err_json(StatusCode::CONFLICT, "system cleanup has not completed")); - } - let accepted = AcceptedBootstrap { - version: 1, - replica_id, - identity: identity.clone(), - }; - if has_owner(state)? { - return verify(&path, &accepted); - } - if state.contains_store(0) { - return Err(err_json( - StatusCode::CONFLICT, - "existing system store has no matching bootstrap ownership", - )); - } - fs::create_dir_all(root).map_err(|error| storage_error(&error))?; - let temporary = root.join(format!( - ".system-bootstrap-{}-{}.tmp", - std::process::id(), - NEXT_TEMP.fetch_add(1, Ordering::Relaxed) - )); - let result = (|| { - let mut file = OpenOptions::new() - .write(true) - .create_new(true) - .mode(0o600) - .open(&temporary) - .map_err(|error| storage_error(&error))?; - let bytes = - serde_json::to_vec(&accepted).map_err(|error| storage_error(&std::io::Error::other(error)))?; - file.write_all(&bytes).map_err(|error| storage_error(&error))?; - file.sync_all().map_err(|error| storage_error(&error))?; - match fs::hard_link(&temporary, &path) { - Ok(()) => {} - Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => {} - Err(error) => return Err(storage_error(&error)), - } - File::open(root) - .and_then(|directory| directory.sync_all()) - .map_err(|error| storage_error(&error))?; - verify(&path, &accepted) - })(); - let cleanup = fs::remove_file(&temporary); - if result.is_ok() { - cleanup.map_err(|error| storage_error(&error))?; - } - result -} - -fn validate(replica_id: u64, identity: &SystemBootstrapIdentity) -> Result<(), ManagementError> { - fn uuid(text: &str) -> bool { - text.len() == 36 - && text.bytes().enumerate().all(|(index, byte)| { - if [8, 13, 18, 23].contains(&index) { - byte == b'-' - } else { - byte.is_ascii_hexdigit() - } - }) - && text - .bytes() - .any(|byte| matches!(byte, b'1'..=b'9' | b'a'..=b'f' | b'A'..=b'F')) - } - if replica_id == 0 - || !uuid(&identity.cluster_id) - || !uuid(&identity.operation_id) - || identity.configuration_digest.len() != 64 - || !identity - .configuration_digest - .bytes() - .all(|byte| byte.is_ascii_hexdigit()) - { - return Err(err_json( - StatusCode::BAD_REQUEST, - "bootstrap requires nonzero replica, cluster/operation UUIDs and SHA-256 configuration digest", - )); - } - Ok(()) -} - -fn verify(path: &Path, expected: &AcceptedBootstrap) -> Result<(), ManagementError> { - let metadata = fs::symlink_metadata(path).map_err(|error| storage_error(&error))?; - if !metadata.file_type().is_file() - || metadata.len() > 4096 - || metadata.permissions().mode() & 0o777 != 0o600 - { - return Err(err_json( - StatusCode::CONFLICT, - "invalid durable system bootstrap ownership", - )); - } - let mut bytes = Vec::with_capacity(4096); - File::open(path) - .map_err(|error| storage_error(&error))? - .take(4097) - .read_to_end(&mut bytes) - .map_err(|error| storage_error(&error))?; - let actual: AcceptedBootstrap = serde_json::from_slice(&bytes) - .map_err(|_| err_json(StatusCode::CONFLICT, "invalid durable system bootstrap ownership"))?; - if &actual != expected { - return Err(err_json( - StatusCode::CONFLICT, - "system store is reserved by a conflicting bootstrap operation", - )); - } - Ok(()) -} - -fn storage_error(error: &std::io::Error) -> ManagementError { - err_json( - StatusCode::INTERNAL_SERVER_ERROR, - format!("system bootstrap persistence failed: {error}"), - ) -} - -mod cleanup; -pub(super) use cleanup::cleanup; - -/// # Errors -/// Reports unreadable cleanup markers instead of restarting a retired system store. -pub fn system_cleanup_pending(root: &Path) -> std::io::Result<bool> { - match fs::symlink_metadata(root.join(RETIRED_FILE)) { - Ok(_) => Ok(true), - Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(false), - Err(error) => Err(error), - } -} diff --git a/app/crowdb-kv-server/src/mgmt/topology.rs b/app/crowdb-kv-server/src/mgmt/topology.rs deleted file mode 100644 index 277cdd899..000000000 --- a/app/crowdb-kv-server/src/mgmt/topology.rs +++ /dev/null @@ -1,145 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -//! Topology export and metrics endpoints. - -use axum::extract::{Query, State}; -use axum::response::IntoResponse; -use serde::Deserialize; -use utoipa::ToSchema; - -use crowdb_kv::cluster::status::StoreStatus; -use crowdb_protocol::mgmt::{MetricField, MetricPoint, MetricsResponse, TopologyResponse}; - -use super::RegistryArc; - -/// `GET /topology` (alias `/top`) — full hierarchy with per-remote RPC -/// metrics and cheap kv-store stats. -#[utoipa::path( - get, - path = "/topology", - tag = "management", - responses((status = 200, description = "Cluster topology status", body = TopologyResponse)) - )] -pub(super) async fn export_topology(State(state): State<RegistryArc>) -> impl IntoResponse { - let stores: Vec<StoreStatus> = state - .stores_snapshot() - .values() - .map(|store| store.status()) - .collect(); - let body = serde_json::to_string_pretty(&TopologyResponse { stores }).unwrap(); - ([("content-type", "application/json")], body) -} - -/// Query params for `GET /metrics`. -#[derive(Deserialize, ToSchema)] -pub(crate) struct MetricsQuery { - /// Metric name prefix filter (e.g. `s.1.g.2.`). Default empty = all. - #[serde(default)] - prefix: String, -} - -/// `GET /metrics` — structured snapshot of all registry metrics matching -/// the `prefix` query param. Does not reset window state. Intended for -/// the GUI Inspector and script/scrape consumers. -#[utoipa::path( - get, - path = "/metrics", - tag = "management", - params(("prefix" = Option<String>, Query, description = "Metric name prefix filter")), - responses((status = 200, description = "Metric snapshot", body = MetricsResponse)) - )] -pub(super) async fn metrics( - State(state): State<RegistryArc>, - Query(q): Query<MetricsQuery>, -) -> impl IntoResponse { - let timestamp = crowdb_kv::metrics::iso8601_now(); - let window_secs = 5.0; // approximate — snapshot path does not track elapsed - let metrics: Vec<MetricPoint> = state - .metrics_registry - .as_ref() - .map(|reg| { - let reg = reg.lock().unwrap(); - reg.snapshot_struct(&q.prefix, window_secs) - .iter() - .map(metric_point_to_dto) - .collect() - }) - .unwrap_or_default(); - let body = serde_json::to_string_pretty(&MetricsResponse { - window_secs, - timestamp, - metrics, - }) - .unwrap(); - ([("content-type", "application/json")], body) -} - -#[allow(clippy::cast_precision_loss)] -fn metric_point_to_dto(p: &crowdb_kv::metrics::MetricPoint) -> MetricPoint { - use crowdb_kv::metrics::MetricPoint as KvMetricPoint; - let kind = p.kind().to_string(); - let fields = match p { - KvMetricPoint::Counter { - count, tps, total, .. - } => vec![("count", *count as f64), ("tps", *tps), ("total", *total as f64)], - KvMetricPoint::Gauge { value, .. } => vec![("value", *value as f64)], - KvMetricPoint::Bandwidth { - count, - avg_size, - rate, - total_bytes, - .. - } => vec![ - ("count", *count as f64), - ("avg_size", *avg_size as f64), - ("rate", *rate as f64), - ("total_bytes", *total_bytes as f64), - ], - KvMetricPoint::Histogram { - count, - avg_ns, - p50_ns, - p99_ns, - max_ns, - total, - .. - } => vec![ - ("count", *count as f64), - ("avg_ns", *avg_ns), - ("p50_ns", *p50_ns as f64), - ("p99_ns", *p99_ns as f64), - ("max_ns", *max_ns as f64), - ("total", *total as f64), - ], - KvMetricPoint::Summary { - count, - avg_ns, - max_ns, - total, - .. - } => vec![ - ("count", *count as f64), - ("avg_ns", *avg_ns as f64), - ("max_ns", *max_ns as f64), - ("total", *total as f64), - ], - }; - MetricPoint { - name: match p { - KvMetricPoint::Counter { name, .. } - | KvMetricPoint::Gauge { name, .. } - | KvMetricPoint::Bandwidth { name, .. } - | KvMetricPoint::Histogram { name, .. } - | KvMetricPoint::Summary { name, .. } => name.clone(), - }, - kind, - fields: fields - .into_iter() - .map(|(key, value)| MetricField { - key: key.to_string(), - value, - }) - .collect(), - } -} diff --git a/app/crowdb-kv-server/src/recovery/startup.rs b/app/crowdb-kv-server/src/recovery/startup.rs deleted file mode 100644 index 932116dc6..000000000 --- a/app/crowdb-kv-server/src/recovery/startup.rs +++ /dev/null @@ -1,246 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -use std::io; -use std::path::{Path, PathBuf}; -use std::sync::Arc; - -use tracing::info; - -use crowdb_kv::cluster::group::PxGroup; -use crowdb_kv::cluster::group_config::{GroupConfigStore, PxGroupConfig, PxGroupMember}; -use crowdb_kv::cluster::group_election::LeaderElection; -use crowdb_kv::cluster::local_replica::{PxLocalReplica, PxLocalReplicaRole}; -use crowdb_kv::cluster::node_config::NodeConfigStore; -use crowdb_kv::common::config::{CrowDBConfig, WalConfig}; -use crowdb_kv::kv::{CrowdbTreeBackend, CrowdbTreeConfig, CrowdbTreeEngine, KVEngine}; -use crowdb_kv::wal::replay::replay_group; -use crowdb_kv::wal::{IoBackend, WalEngine}; - -/// Load persisted group config and apply it to the group. -/// -/// Reads from `node-config.json` (the per-node config cache). If the -/// group entry is present, the group is seeded with the durable -/// membership so it does not start as a `quorum=1` singleton in the -/// restore window. The node config store is also set on the group so -/// future `persist_config` calls write to the same file. -async fn maybe_apply_persisted_config( - group: &mut PxGroup, - config_root: &Path, - store_id: u64, -) -> io::Result<()> { - let node_store = NodeConfigStore::new(config_root); - match node_store.load_group(store_id, group.group_id()).await { - Ok(Some(config)) => { - if config.group_id == group.group_id() { - group.apply_config(&config); - } - } - Ok(None) => { - // Fall back to legacy per-group config file for migration. - let legacy_store = GroupConfigStore::new(config_root, store_id, group.group_id()); - match legacy_store.load().await { - Ok(Some(config)) => { - if config.group_id == group.group_id() { - group.apply_config(&config); - } - } - Ok(None) => {} - Err(e) => { - return Err(e); - } - } - } - Err(e) => { - return Err(e); - } - } - group.set_node_config_store(node_store, store_id, group.group_id()); - Ok(()) -} - -async fn persist_creation_identity( - config_root: &Path, - store_id: u64, - group_id: u64, - replica_id: u64, -) -> io::Result<()> { - let store = NodeConfigStore::new(config_root); - if let Some(existing) = store.load().await?.group(store_id, group_id) { - if existing.replica_id != replica_id { - return Err(io::Error::new( - io::ErrorKind::InvalidInput, - "requested replica identity conflicts with persisted group", - )); - } - return Ok(()); - } - let seed = GroupConfigStore::new(config_root, store_id, group_id) - .load() - .await? - .unwrap_or_else(|| PxGroupConfig { - group_id, - members: vec![PxGroupMember { - replica_id, - endpoint: String::new(), - voting: true, - }], - ..PxGroupConfig::default() - }); - // A WAL directory makes the group discoverable after a crash. Persist its - // local identity first, so restore cannot substitute the CLI default. - store.save_group(store_id, &seed, replica_id).await -} - -#[must_use] -pub fn store_wal_root(wal_root: &Path, store_id: u64) -> PathBuf { - wal_root.join(format!("store{store_id}")) -} - -/// Durable per-group crowdb-tree directory path: `{data_root}/store{store_id}/group{group_id}`. -/// Both `TextPageStore` and `BlockPageStore` expect a directory path (`TextPageStore` creates a subdirectory -/// `{path}/{store_id}-{group_id}/`, `BlockPageStore` creates `.blk-*` files -/// directly in `path`). -#[must_use] -pub fn store_crowdb_tree_path(data_root: &Path, store_id: u64, group_id: u64) -> PathBuf { - data_root - .join(format!("store{store_id}")) - .join(format!("group{group_id}")) -} - -/// Open (creating on first boot) the durable crowdb-tree engine backing -/// `(store_id, group_id)`'s learner, boxed for [`PxLearner::with_engine`] -/// via [`PxLocalReplica::restore_from_replay_with_engine`]. -/// -/// # Errors -/// -/// Returns an I/O error if the parent directory cannot be created, or if -/// `CrowdbTreeEngine::open` fails (e.g. a corrupt or unreadable file). -async fn open_crowdb_tree_engine( - data_root: &Path, - store_id: u64, - group_id: u64, - backend: CrowdbTreeBackend, - log_dir: &str, -) -> io::Result<Box<dyn KVEngine>> { - let path = store_crowdb_tree_path(data_root, store_id, group_id); - if let Some(parent) = path.parent() { - tokio::fs::create_dir_all(parent).await?; - } - tokio::fs::create_dir_all(&path).await?; - let opt = CrowdbTreeConfig { - path: Some(path.to_string_lossy().into_owned()), - backend, - store_id: u32::try_from(store_id).unwrap_or(0), - group_id: u32::try_from(group_id).unwrap_or(0), - log_dir: log_dir.to_string(), - log_file_prefix: "crowdb-kv-server-tree".to_string(), - ..Default::default() - }; - info!( - store_id, - group_id, - backend = ?backend, - path = %path.display(), - "opening crowdb-tree engine" - ); - // `CrowdbTreeEngine::open` is a synchronous FFI call; called here inline - // (not `spawn_blocking`) consistent with `CrowdbTreeEngine`'s own - // documented policy of calling the still-fully-synchronous crowdb-tree - // core directly rather than adding a thread-pool hop with no genuine - // asynchrony behind it (see `crowdb_kv::kv::CrowdbTreeEngine`'s docs). This - // runs once per group at boot, not on a hot path. - let engine = CrowdbTreeEngine::open(&opt).map_err(|e| { - io::Error::other(format!( - "CrowdbTreeEngine::open({}) failed: {e:?}", - path.display() - )) - })?; - Ok(Box::new(engine)) -} - -/// Create a live group by replaying any existing WAL, restoring the local -/// replica state, attaching a fresh `WalEngine`, and seeding the next proposal -/// slot / segment id. -/// -/// # Errors -/// -/// Returns any I/O or replay/restore error encountered while scanning the -/// existing WAL, creating the new WAL engine, opening the durable crowdb-tree -/// engine, or rebuilding the local replica. -#[allow(clippy::too_many_arguments)] -pub async fn create_group_with_wal( - store_id: u64, - group_id: u64, - replica_id: u64, - initial_role: PxLocalReplicaRole, - config: &CrowDBConfig, - wal_backend: Arc<IoBackend>, - crowtree_backend: CrowdbTreeBackend, -) -> io::Result<PxGroup> { - crate::mgmt::group_creation::check_restore(&config.config_root, store_id, group_id, replica_id)?; - persist_creation_identity(&config.config_root, store_id, group_id, replica_id).await?; - let mut wal_config = WalConfig::with_root(store_wal_root(&config.wal_root, store_id)); - if std::env::var("CROWDB_KV_WAL_TEXT").as_deref() == Ok("1") { - wal_config.wal_record_format = crowdb_kv::wal::WalRecordFormat::TextLine; - } - wal_config.wal_skip_fsync = config.wal_skip_fsync; - let mut replay = replay_group(&wal_backend, &wal_config.wal_disks, group_id).await?; - let wal = WalEngine::create_from_replay( - wal_backend, - wal_config, - group_id, - replay.max_segment_id, - std::mem::take(&mut replay.index), - ) - .await?; - - let mut local_replica = { - let engine = open_crowdb_tree_engine( - &config.data_root, - store_id, - group_id, - crowtree_backend, - &config.log_dir, - ) - .await?; - wal.set_snapshot_slot(engine.resume_from_slot()); - PxLocalReplica::restore_from_replay_with_engine(replica_id, initial_role, &replay, engine).await? - }; - local_replica.set_wal(wal); - - let mut group = PxGroup::new(group_id, local_replica); - maybe_apply_persisted_config(&mut group, &config.config_root, store_id).await?; - // If the caller initialized the group as Leader, the proposal leadership - // gate (current_term == proposing_term) will not open until the term is - // stamped. The election driver handles this on a real win, but groups - // created/restored as Leader (e.g., single-replica management API) may - // serve proposals before the driver runs, so stamp it synchronously here. - if initial_role == PxLocalReplicaRole::Leader { - let term = group.local_replica().current_term_snapshot(); - group.stamp_proposing_term(term); - } - group.set_from_config(config); - info!( - store_id, - group_id, - max_inflight = config.max_inflight(), - reject_on_window_full = config.reject_on_inflight_window_full(), - coalesce_max_keys = config.paxos.coalesce_max_keys, - skip_fsync = config.wal_skip_fsync, - wal_early_ack = config.wal_early_ack, - "group created with config" - ); - let next_slot = group - .local_replica() - .highest_seen_slot() - .max(group.local_replica().last_chosen_slot()) - .max(group.local_replica().contiguous_applied()) - .saturating_add(1) - .max(1); - group.set_next_slot(next_slot); - if let Err(error) = super::member_routes::apply(&group, &config.config_root, store_id) { - tracing::debug!(store_id, group_id, %error, "restored member routes are blocked pending verification"); - } - Ok(group) -} diff --git a/app/crowdb-kv-server/tests/common/group_creation.rs b/app/crowdb-kv-server/tests/common/group_creation.rs deleted file mode 100644 index f27d8f227..000000000 --- a/app/crowdb-kv-server/tests/common/group_creation.rs +++ /dev/null @@ -1,99 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -use crowdb_kv::{ - cluster::{KvServer, PxKvStore}, - common::config::CrowDBConfig, -}; -use crowdb_kv_server::{ - mgmt::{self, operation_registry::AppState}, - store_registry::KvStoreRegistry, -}; -use serde_json::{json, Value}; -use std::{path::PathBuf, sync::Arc}; - -pub struct TestGroupCreation { - _root: crowdb_test_harness::test_dirs::TestDir, - pub registry: Arc<KvStoreRegistry>, - pub store: Arc<PxKvStore>, - pub origin: String, - server: tokio::task::JoinHandle<()>, -} - -impl TestGroupCreation { - pub async fn start(block_data: bool) -> Self { - let root = crowdb_test_harness::test_dirs::tempdir_in_test_data("native-group-creation"); - let config = CrowDBConfig { - data_root: root.path().join("data"), - wal_root: root.path().join("wal"), - config_root: root.path().join("config"), - crowtree_backend: "file".into(), - wal_backend: "file".into(), - ..CrowDBConfig::for_tests() - }; - if block_data { - std::fs::write(&config.data_root, b"blocked").unwrap(); - } - let registry = Arc::new(KvStoreRegistry::with_config(config)); - let store = Arc::new(PxKvStore::new(0, "127.0.0.1:0".parse().unwrap())); - store.start().await.unwrap(); - registry.add_store(0, &store); - let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); - let origin = format!("http://{}", listener.local_addr().unwrap()); - let router = mgmt::router(AppState::new(Arc::clone(®istry))); - let server = tokio::spawn(async move { - axum::serve(listener, router).await.unwrap(); - }); - Self { - _root: root, - registry, - store, - origin, - server, - } - } - - pub fn ownership(&self) -> PathBuf { - self.registry.config.config_root.join("group-creation-0-1.json") - } - - pub async fn request( - &self, - method: reqwest::Method, - path: &str, - body: Option<&Value>, - ) -> (reqwest::StatusCode, Vec<u8>) { - let client = reqwest::Client::builder() - .no_proxy() - .timeout(std::time::Duration::from_secs(5)) - .build() - .unwrap(); - let mut request = client.request(method, format!("{}{path}", self.origin)); - if let Some(body) = body { - request = request.json(body); - } - let response = request.send().await.unwrap(); - let status = response.status(); - let bytes = response.bytes().await.unwrap().to_vec(); - (status, bytes) - } - - pub async fn create(&self, body: &Value, expected: reqwest::StatusCode) { - let (status, bytes) = self - .request(reqwest::Method::POST, "/stores/0/groups", Some(body)) - .await; - assert_eq!(status, expected, "{}", String::from_utf8_lossy(&bytes)); - } -} - -impl Drop for TestGroupCreation { - fn drop(&mut self) { - self.server.abort(); - self.store.stop(); - } -} - -pub fn request() -> Value { - json!({"group_id":1,"replica_id":7,"initial_role":"leader","start_election":false, - "deployment":{"cluster_id":uuid::Uuid::new_v4().to_string(),"operation_id":uuid::Uuid::new_v4().to_string()}}) -} diff --git a/app/crowdb-kv-server/tests/common/system_bootstrap.rs b/app/crowdb-kv-server/tests/common/system_bootstrap.rs deleted file mode 100644 index 5bcad0b50..000000000 --- a/app/crowdb-kv-server/tests/common/system_bootstrap.rs +++ /dev/null @@ -1,127 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -use crowdb_kv::common::config::CrowDBConfig; -use crowdb_kv_server::{ - mgmt::{self, operation_registry::AppState}, - store_registry::KvStoreRegistry, -}; -use crowdb_protocol::mgmt::{SystemBootstrapIdentity, SystemPrepareRequest}; -use reqwest::StatusCode; -use serde_json::Value; -use std::{path::PathBuf, sync::Arc, time::Duration}; - -pub struct TestSystemBootstrap { - _root: crowdb_test_harness::test_dirs::TestDir, - pub registry: Arc<KvStoreRegistry>, - pub origin: String, - server: tokio::task::JoinHandle<()>, -} - -impl TestSystemBootstrap { - pub async fn start() -> Self { - let root = crowdb_test_harness::test_dirs::tempdir_in_test_data("system-prepare"); - let config = CrowDBConfig { - data_root: root.path().join("data"), - wal_root: root.path().join("wal"), - config_root: root.path().join("config"), - ..CrowDBConfig::for_tests() - }; - let registry = Arc::new(KvStoreRegistry::with_config(config)); - let (origin, server) = Self::serve(®istry).await; - Self { - _root: root, - registry, - origin, - server, - } - } - - async fn serve(registry: &Arc<KvStoreRegistry>) -> (String, tokio::task::JoinHandle<()>) { - let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); - let origin = format!("http://{}", listener.local_addr().unwrap()); - let router = mgmt::router(AppState::new(registry.clone())); - let server = tokio::spawn(async move { - axum::serve(listener, router).await.unwrap(); - }); - (origin, server) - } - - pub async fn restart(&mut self) { - self.server.abort(); - self.registry = Arc::new(KvStoreRegistry::with_config(self.registry.config.clone())); - (self.origin, self.server) = Self::serve(&self.registry).await; - } - - pub fn ownership(&self) -> PathBuf { - self.registry.config.config_root.join("system-bootstrap.json") - } - - pub async fn request(&self, path: &str, body: &Value) -> StatusCode { - let response = reqwest::Client::builder() - .no_proxy() - .timeout(Duration::from_secs(3)) - .build() - .unwrap() - .post(format!("{}{path}", self.origin)) - .json(body) - .send() - .await - .unwrap(); - let status = response.status(); - let bytes = response.bytes().await.unwrap(); - assert!( - matches!(status, StatusCode::OK | StatusCode::CONFLICT), - "{path}: {status}: {}", - String::from_utf8_lossy(&bytes) - ); - status - } - - pub async fn prepare(&self, replica_id: u64, bootstrap: &SystemBootstrapIdentity) -> StatusCode { - self.request( - "/system/prepare", - &serde_json::to_value(SystemPrepareRequest { - replica_id, - bootstrap: bootstrap.clone(), - }) - .unwrap(), - ) - .await - } - - pub fn assert_no_execution(&self) { - assert_eq!(self.registry.store_count(), 0); - assert!(!self.registry.config.data_root.exists()); - assert!(!self.registry.config.wal_root.exists()); - } -} - -impl Drop for TestSystemBootstrap { - fn drop(&mut self) { - self.server.abort(); - } -} - -pub fn identity() -> SystemBootstrapIdentity { - SystemBootstrapIdentity { - cluster_id: uuid::Uuid::new_v4().to_string(), - operation_id: uuid::Uuid::new_v4().to_string(), - configuration_digest: "a".repeat(64), - } -} - -pub async fn proposal( - nodes: &[TestSystemBootstrap], - selected: [usize; 3], - identity: &SystemBootstrapIdentity, -) -> bool { - let (first, second, third) = tokio::join!( - nodes[selected[0]].prepare(1, identity), - nodes[selected[1]].prepare(2, identity), - nodes[selected[2]].prepare(3, identity) - ); - [first, second, third] - .iter() - .all(|status| *status == StatusCode::OK) -} diff --git a/app/crowdb-kv-server/tests/startup_test.rs b/app/crowdb-kv-server/tests/startup_test.rs deleted file mode 100644 index 170380d1a..000000000 --- a/app/crowdb-kv-server/tests/startup_test.rs +++ /dev/null @@ -1,264 +0,0 @@ -// Copyright 2026-present Gian <crow.db@outlook.com> -// Licensed under the Apache License, Version 2.0. - -use std::sync::Arc; - -use bytes::Bytes; -use crowdb_kv::cluster::group::ProposeResult; -use crowdb_kv::cluster::group_election::LeaderElection; -use crowdb_kv::cluster::local_replica::PxLocalReplicaRole; -use crowdb_kv::common::config::{CrowDBConfig, WalConfig}; -use crowdb_kv::kv::CrowdbTreeBackend; -use crowdb_kv::paxos::roles::{PxBallot, PxLogEntry}; -use crowdb_kv::wal::record::WALRecord; -use crowdb_kv::wal::replay::replay_group; -use crowdb_kv::wal::{IoBackend, WalEngine}; -use crowdb_kv_server::recovery::startup::{create_group_with_wal, store_wal_root}; - -#[path = "startup_test/reclamation.rs"] -mod reclamation; - -fn encode_put_payload(key: &[u8], value: &[u8]) -> Vec<u8> { - let mut buf = Vec::new(); - buf.extend_from_slice(&1u16.to_le_bytes()); // 1 op (u16 LE) - buf.push(0); // kind = Put - let key_len = u32::try_from(key.len()).expect("key length exceeds u32"); - buf.extend_from_slice(&key_len.to_le_bytes()); - buf.extend_from_slice(key); - let value_len = u32::try_from(value.len()).expect("value length exceeds u32"); - buf.extend_from_slice(&value_len.to_le_bytes()); - buf.extend_from_slice(value); - buf -} - -#[tokio::test] -async fn incomplete_group_creation_retains_replica_identity_before_wal_is_discoverable() { - let temp = crowdb_test_harness::test_dirs::tempdir_in_test_data("startup-identity"); - let blocked_tree = temp.path().join("blocked-tree"); - std::fs::write(&blocked_tree, b"not a directory").unwrap(); - let config = CrowDBConfig { - wal_root: temp.path().join("wal"), - config_root: temp.path().join("conf"), - data_root: blocked_tree, - ..CrowDBConfig::for_tests() - }; - let result = create_group_with_wal( - 0, - 1, - 2, - PxLocalReplicaRole::Leader, - &config, - Arc::new(IoBackend::detect()), - CrowdbTreeBackend::File, - ) - .await; - assert!(result.is_err()); - let groups = crowdb_kv_server::recovery::restore::scan_local_groups(&config.wal_root) - .await - .unwrap(); - assert!(groups - .iter() - .any(|group| group.store_id == 0 && group.group_id == 1)); - let persisted = crowdb_kv::cluster::node_config::NodeConfigStore::new(&config.config_root) - .load() - .await - .unwrap(); - assert_eq!( - persisted.group(0, 1).map(|group| group.replica_id), - Some(2), - "an interrupted creation must never restore using CLI replica 1" - ); -} - -#[tokio::test] -async fn create_group_with_wal_restores_and_resumes_at_next_slot() { - let temp = crowdb_test_harness::test_dirs::tempdir_in_test_data("startup"); - let wal_root = temp.path().join("wal-root"); - let config_root = temp.path().join("conf-root"); - let backend = Arc::new(IoBackend::detect()); - let store_id = 9; - let group_id = 11; - let replica_id = 7; - - let config = WalConfig::with_root(store_wal_root(&wal_root, store_id)); - let wal = WalEngine::create(backend.clone(), config.clone(), group_id) - .await - .unwrap(); - - wal.append(&WALRecord::from_promised( - group_id, - 2, - 1, - PxBallot::new(2, replica_id), - )) - .await - .unwrap(); - - let accepted_entry = PxLogEntry { - slot: 2, - ballot: PxBallot::new(3, replica_id), - term: 3, - payload: Bytes::from(encode_put_payload(b"restore-key", b"restore-value")), - }; - wal.append(&WALRecord::from_accepted(group_id, &accepted_entry)) - .await - .unwrap(); - wal.append(&WALRecord::from_vote_granted(group_id, 5, 99)) - .await - .unwrap(); - wal.seal_all().await.unwrap(); - - let data_root = temp.path().join("data-root"); - let config = CrowDBConfig { - wal_root: wal_root.clone(), - config_root: config_root.clone(), - data_root: data_root.clone(), - ..CrowDBConfig::for_tests() - }; - let group = create_group_with_wal( - store_id, - group_id, - replica_id, - PxLocalReplicaRole::Leader, - &config, - backend.clone(), - CrowdbTreeBackend::File, - ) - .await - .unwrap(); - - let replica = group.local_replica(); - assert_eq!(replica.current_term(), 5); - assert_eq!(replica.voted_for(), Some(99)); - assert_eq!(replica.accepted_at(2).await, Some(accepted_entry.clone())); - assert_eq!(replica.promised_at(1).await, Some(PxBallot::new(2, replica_id))); - assert_eq!( - replica.learner.engine_get(b"restore-key").await.map(|(_, v)| v), - Some(b"restore-value".to_vec()), - "WAL replay applies accepted slot 2 to the learner" - ); - - replica.become_leader(); - group.stamp_proposing_term(replica.current_term()); - - let result = group - .propose(encode_put_payload(b"new-key", b"new-value"), Some(55), Some(1)) - .await; - match result { - ProposeResult::Chosen { slot } => assert_eq!(slot, 3), - other => panic!("expected chosen proposal after restore, got {other:?}"), - } - - let replay_wal_dir = store_wal_root(&config.wal_root, store_id); - let replay = replay_group(&backend, &[replay_wal_dir], group_id).await.unwrap(); - assert!(replay.records.iter().any(|record| { - record.slot == 3 && matches!(record.record_type, crowdb_kv::wal::record::RecordType::Accepted) - })); -} - -/// Durable crowdb-tree engine end-to-end: a group backed by a durable -/// `CrowdbTreeEngine` file survives a simulated process restart (drop the -/// group, then call `create_group_with_wal` again against the same -/// `wal_root`/`data_root`) with its KV state intact -- via full WAL replay -/// into a fresh `CrowdbTreeEngine::open` at the same file -/// (`PxLocalReplica::restore_from_replay_with_engine`), not by any -/// resume-from-last-applied-slot shortcut (not implemented; see /// #20's note on why that needs separate, careful frontier-seeding work). -/// Parameterized over [`CrowdbTreeBackend`] so the same -/// scenario covers both the explicit file backend and the raw `O_DIRECT` -/// block-device backend. -async fn crowdb_tree_engine_persists_across_restart(crowtree_backend: CrowdbTreeBackend) { - let temp = crowdb_test_harness::test_dirs::tempdir_in_test_data("startup"); - let wal_root = temp.path().join("wal-root"); - let config_root = temp.path().join("conf-root"); - let data_root = temp.path().join("data-root"); - let backend = Arc::new(IoBackend::detect()); - let store_id = 21; - let group_id = 5; - let replica_id = 1; - - let config = CrowDBConfig { - wal_root: wal_root.clone(), - config_root: config_root.clone(), - data_root: data_root.clone(), - ..CrowDBConfig::for_tests() - }; - let group = create_group_with_wal( - store_id, - group_id, - replica_id, - PxLocalReplicaRole::Leader, - &config, - backend.clone(), - crowtree_backend, - ) - .await - .unwrap(); - - // The durable crowdb-tree file was created under data_root, not left at the - // default in-memory (no file) path. - let ct_path = crowdb_kv_server::recovery::startup::store_crowdb_tree_path(&data_root, store_id, group_id); - assert!( - ct_path.exists(), - "expected a durable crowdb-tree file at {}", - ct_path.display() - ); - - group.local_replica().become_leader(); - group.stamp_proposing_term(group.local_replica().current_term()); - let result = group - .propose(encode_put_payload(b"ct-key", b"ct-value"), Some(1), Some(1)) - .await; - match result { - ProposeResult::Chosen { slot } => assert_eq!(slot, 1), - other => panic!("expected chosen proposal, got {other:?}"), - } - assert_eq!( - group - .local_replica() - .learner - .engine_get(b"ct-key") - .await - .map(|(_, v)| v), - Some(b"ct-value".to_vec()) - ); - - // Simulate a process restart: drop the group (closes the crowdb-tree file - // handle via `Crowdbtree`'s `Drop`), then rebuild from the same WAL + - // crowdb-tree file. - drop(group); - - let restarted = create_group_with_wal( - store_id, - group_id, - replica_id, - PxLocalReplicaRole::Leader, - &config, - backend.clone(), - crowtree_backend, - ) - .await - .unwrap(); - - assert_eq!( - restarted - .local_replica() - .learner - .engine_get(b"ct-key") - .await - .map(|(_, v)| v), - Some(b"ct-value".to_vec()), - "crowdb-tree-backed KV state must survive a simulated restart" - ); -} - -#[tokio::test] -async fn create_group_with_wal_crowdb_tree_engine_persists_across_restart() { - crowdb_tree_engine_persists_across_restart(CrowdbTreeBackend::File).await; -} - -/// : same scenario, through `BlockPageStore` (`O_DIRECT`) -/// instead of the default `FilePageStore`. -#[tokio::test] -async fn create_group_with_wal_crowdb_tree_block_backend_persists_across_restart() { - crowdb_tree_engine_persists_across_restart(CrowdbTreeBackend::Block).await; -} diff --git a/app/crowdb-paxos-kv-server/Cargo.toml b/app/crowdb-paxos-kv-server/Cargo.toml new file mode 100644 index 000000000..b2f398706 --- /dev/null +++ b/app/crowdb-paxos-kv-server/Cargo.toml @@ -0,0 +1,125 @@ +[package] +name = "crowdb-paxos-kv-server" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true +description = "CROWDB server binary: single-group, multi-group, and cluster modes." +autotests = false + +[[bin]] +name = "crowdb-paxos-kv-server" +path = "src/main.rs" + +[lints] +workspace = true + +[features] +default = [] + +[dependencies] +crowdb-paxos-kv = { path = "../../lib/crowdb-paxos-kv" } +crowdb-paxos-kv-client = { path = "../../lib/crowdb-paxos-kv-client" } +crowdb-common = { path = "../../lib/crowdb-common/rust" } +crowdb-protocol = { path = "../../lib/crowdb-protocol", features = ["schema"] } +crowdb-tree-ffi = { path = "../../lib/crowdb-tree/ffi" } +crowdb-rpc-ffi = { path = "../../lib/crowdb-rpc/ffi" } +tokio = { workspace = true, features = ["rt-multi-thread", "macros", "time", "signal"] } +tracing = { workspace = true } +clap = { version = "4", features = ["derive", "env"] } +axum = "0.7" +serde = { version = "1", features = ["derive"] } +serde_json = "1" +dashmap = "6" +arc-swap = "1.9" +utoipa = "5" +bytes = "1" +sha2 = "0.10" +uuid = { version = "1", features = ["v4"] } + +[dev-dependencies] +crowdb-paxos-kv = { path = "../../lib/crowdb-paxos-kv", features = ["test-util"] } +reqwest = { version = "0.12", features = ["json"] } +crowdb-rpc-ffi = { path = "../../lib/crowdb-rpc/ffi", features = ["test-util"] } +crowdb-test-harness = { path = "../../lib/crowdb-test-harness" } + +[[test]] +name = "keepalive_recovery_test" +path = "tests/keepalive_recovery_test.rs" + +[[test]] +name = "group_creation_test" +path = "tests/group_creation_test.rs" + +[[test]] +name = "member_routes_test" +path = "tests/member_routes_test.rs" + +[[test]] +name = "chunk_dynamic_monitor_test" +path = "tests/chunk_dynamic_monitor_test.rs" + +[[test]] +name = "group0_discovery_test" +path = "tests/group0_discovery_test.rs" + +[[test]] +name = "cli_parse_test" +path = "tests/cli_parse_test.rs" + + + +[[test]] +name = "monitor_supervisor_test" +path = "tests/monitor_supervisor_test.rs" + +[[test]] +name = "group0_control_plane_test" +path = "tests/group0_control_plane_test.rs" + +[[test]] +name = "domain_monitor_test" +path = "tests/domain_monitor_test.rs" + +[[test]] +name = "chunk_kv_balance_planning_test" +path = "tests/chunk_kv_balance_planning_test.rs" + +[[test]] +name = "chunk_slot_monitor_test" +path = "tests/chunk_slot_monitor_test.rs" + +[[test]] +name = "reconcile_test" +path = "tests/reconcile_test.rs" + +[[test]] +name = "restore_test" +path = "tests/restore_test.rs" + + +[[test]] +name = "startup_test" +path = "tests/startup_test.rs" + + +[[test]] +name = "diskdb_placement_test" +path = "tests/diskdb_placement_test.rs" + +[[test]] +name = "tree_runtime_host_test" +path = "tests/tree_runtime_host_test.rs" + +[[test]] +name = "tree_config_test" +path = "tests/tree_config_test.rs" + +[[test]] +name = "engine_resource_metrics_test" +path = "tests/engine_resource_metrics_test.rs" + +[[test]] +name = "runtime_shutdown_test" +path = "tests/runtime_shutdown_test.rs" diff --git a/app/crowdb-kv-server/build.rs b/app/crowdb-paxos-kv-server/build.rs similarity index 100% rename from app/crowdb-kv-server/build.rs rename to app/crowdb-paxos-kv-server/build.rs diff --git a/app/crowdb-paxos-kv-server/conf/crowdb_paxos_kv_server_config.toml b/app/crowdb-paxos-kv-server/conf/crowdb_paxos_kv_server_config.toml new file mode 100644 index 000000000..088b36b18 --- /dev/null +++ b/app/crowdb-paxos-kv-server/conf/crowdb_paxos_kv_server_config.toml @@ -0,0 +1,199 @@ +# Copyright 2026-present Gian <crow.db@outlook.com> +# Licensed under the Apache License, Version 2.0. +# +# Paxos KV Server +# crowdb-paxos-kv-server --root <data-dir> --config <this-file> +# Read at startup. Paths, backends and log locations are supplied through the CLI. +# These are the existing defaults. Find maximum sustainable flush throughput with +# ample resources first, then reduce resources using the measured curves. +# Sizes are integer bytes; 1 MiB = 1,048,576 bytes. Tune threads, queues and budgets +# together for the whole process and all of its Trees. +# --election-profile replaces the entire [election] table below. Omit that flag +# when tuning maintenance intervals in this file. + +# Preserve the default ACK/apply modes when introducing a configuration file. +wal_early_ack = true +async_engine_apply = true + +# Keep Tree field additions, removals, renames and explanations in sync with: +# app/crowdb-chunk-kv-server/conf/crowdb_chunk_kv_server_config.toml +# Each server owns its values independently; no shared include or override is loaded. + +# Process-wide CPU/backend pools. Check CPU quotas and utilization when scaling. +# Resource diagnostics sample every 100 ms and flush at the server metrics-log +# interval. Each host/Tree resource reports its limit, current usage, average +# held sampled usage, and sampled maximum for that log window. The maximum is +# not an exact transient peak; sample count, age, gaps and errors show coverage. +# Averages retain fractional counts: worker avg=0.750 means 75% of one worker. +# Reserved capacity and allocated storage are reported separately. Budget +# denial counters distinguish Tree limits from shared host limits. Queue-slot +# usage includes prepaid progress registrations even when no job is runnable. +# Tree cache rows cover decoded bytes, page slots and request slots; they +# overlap the storage charged to budget partitions and must not be added to it. +[tree_runtime.host] +workers = 4 +# Workers for synchronous PageStore calls; async concurrency also depends on the backend. +backend_workers = 2 +# Larger queues retain more work without increasing service rate. Recheck +# lifecycle/Routing reservations when increasing queue capacity. +cpu_queue_limit = 256 +backend_queue_limit = 256 +# Maximum pending waiters per resource partition. +max_waiters_per_class = 256 +# Retained queue/control reservation charged to host Routing, not an extra partition. +lifecycle_bytes = 4_194_304 + +# Shared limits for all Trees in this process. Resource classes cannot borrow +# from each other. These limits do not bound total process RSS. +[tree_runtime.host.limits] +# Source capture, run/range construction and complete-batch handoff. +handoff_bytes = 268_435_456 +# Routing, publication, cache/load and controls, including lifecycle reservations. +routing_bytes = 536_870_912 +# Consolidation, split/merge and checkpoint/writeback work. +reclaim_bytes = 134_217_728 +# Ordinary writes, payload owners and retained COW allocations. +writer_bytes = 536_870_912 +# Independent progress capacity for chosen/replay/gap apply work. +gap_repair_bytes = 268_435_456 +# In-flight I/O tickets. More tickets alone do not parallelize a serial write pipeline. +io_operations = 128 + +# Per-Tree limits. All Trees still compete for the shared host resources above. +[tree_runtime.tree] +# The producer window must fit this CPU queue. Local queues must fit host queues. +# This also bounds ordinary apply/backend operation owners; replay has two spare slots. +cpu_queue_limit = 64 +backend_queue_limit = 64 +# Fixed queue/core/cache descriptors, charged to this Tree's Routing partition. +# Increase this together with cpu_queue_limit and tuning.cache_max_pages; a +# larger Routing limit alone does not enlarge this retained control reserve. +lifecycle_bytes = 1_048_576 + +[tree_runtime.tree.limits] +# Same categories as the host limits. Check both levels when changing a partition. +handoff_bytes = 67_108_864 +routing_bytes = 268_435_456 +# 128 MiB is an adjustable default, not a fixed checkpoint memory target. +reclaim_bytes = 134_217_728 +writer_bytes = 268_435_456 +gap_repair_bytes = 268_435_456 +io_operations = 64 + +[tree] +# Durable frame size; the current public format supports at most 64 KiB. +# A logical leaf may exceed one frame. +# Must be at least 512 bytes and satisfy backend IU alignment. Logical working +# set size is independent of durable frame size. +frame_bytes = 65_536 +# Decoded-cache byte watermark; tuning.cache_max_pages also limits page count. +buffer_pool_bytes = 67_108_864 +# 0 = frame_bytes/4. Values larger than this spill to overflow; exactly 16 KiB +# remains inline with the default frame size. +max_inline_value = 0 +# LZ4 trades CPU for storage bandwidth/space. Keep compression fixed in comparisons. +compression_lz4 = false +# Local block allocation/reclamation unit; ignored by file/mem-block and injected Chunk storage. +block_size = 67_108_864 +# 0 = backend default: block 4 KiB, mem-block 1 byte, Chunk 64 KiB. A nonzero value +# must be a power of two that divides frame_bytes. +iu_size = 0 + +# Engine work granularity. Select these values from measurements. +[tree.tuning] +# Soft per-source rotation thresholds. Crossing either requests rotation; +# 0 requests rotation after every nonempty write. +# The source_max_* hard admission capacities below also apply. +memtable_flush_bytes = 16_777_216 +memtable_flush_entries = 100_000 +# Includes the writable source; the current read/handoff window requires 2..64. +max_memtable_count = 10 +# Per-source admitted version/key capacities, including writes not yet applied. +# Raising soft thresholds alone may not enlarge batches if these limits bind first. +source_max_entries = 4_096 +source_max_key_bytes = 1_048_576 +# Additional accepted/replay allowance. Size it for accepted apply bursts. +# Only the next contiguous slot may use gap source capacity; higher holes use ordinary capacity or wait. +source_max_gap_entries = 1_024 +source_max_gap_key_bytes = 1_048_576 +# 0 = min(host workers, Tree io_operations). Explicit values must fit both limits. +flush_parallelism = 0 +# Per-handoff-run entry/key/work-byte limits. Value owners are retained independently. +run_max_entries = 128 +# Effective key capacity is max(run_max_key_bytes, max_key_size). +run_max_key_bytes = 262_144 +run_max_bytes = 1_048_576 +# Key-arena allocation granularity; must not exceed run_max_key_bytes. +run_key_block_bytes = 1_024 +# Initial run-install window; later windows double. Balances early visible +# progress against prefix-array copying cost. +install_run_window = 64 +# Visits per step/pass; the step size must not exceed the pass limit. +maintenance_visits_per_step = 64 +maintenance_max_visits = 4_096 +# Candidates/steps per pass. Larger limits increase work and reservations; +# they do not repair starvation caused by an incorrect scan cursor. +maintenance_max_candidates = 256 +maintenance_max_steps = 1_024 +# Local scratch per maintenance operation; independent of the total Reclaim budget. +maintenance_metadata_bytes = 1_048_576 +# Decoded-cache page limit. Check this together with buffer_pool_bytes. +cache_max_pages = 256 +# Split/merge thresholds for the physical primary record region, not logical leaf limits. +# Split is capped by frame/format capacity. Merge=0 disables merging; merged +# output must also fit the hysteresis target of about 75% of the split threshold. +leaf_split_bytes = 65_502 +leaf_merge_bytes = 16_384 +# Inner separator limit; encoded output must also fit the frame byte capacity. +inner_max_keys = 256 +# 0 = max(1, inner_max_keys/4). +inner_merge_keys = 0 +# Maximum key size; 0 = frame_bytes/2. Oversized keys are rejected before admission. +max_key_size = 0 + +[paxos] +# Upstream proposal window, held through native apply even after an async Chosen +# reply; must not exceed tree_runtime.tree.cpu_queue_limit. +max_inflight_proposals = 32 +# Keys per proposal; 0 disables coalescing. The current descriptor limit is 64; +# this must also fit the source gap allowance. +coalesce_max_keys = 32 +# Skip draining a pending batch when at least this many other coalesced rounds exist. +coalesce_drain_threshold = 1 +# false = await a window permit; true = return Busy when full. Changes backpressure +# behavior without increasing service rate. +reject_on_inflight_window_full = false + +[server] +# RPC I/O workers are separate from Tree CPU and backend workers. +rpc_workers = 2 +# Connections per replica peer. +peer_pool_size = 2 +# Enqueue to I/O workers to coalesce writev calls; adds wakeup latency. +event_write = false +# Per-connection send queue. More queued work does not increase service rate. +send_queue_capacity = 4_096 +# Unary scan payload limit for mixed workloads; independent of checkpoint frame size. +scan_byte_budget = 3_670_016 + +[election] +# Watchdog/idle maintenance interval; source rotation also signals maintenance. +maintenance_tick_ms = 30_000 +# Any of the next three thresholds triggers a durable snapshot. Frequent snapshots +# add overhead; infrequent snapshots retain dirty state and WAL longer. +snapshot_slot_threshold = 100_000 +snapshot_time_threshold_ms = 600_000 +# 0 disables the flush-count trigger. +snapshot_flush_count_threshold = 10 +# Maintenance-forced WAL flush interval. 0 disables this trigger, not the WAL +# writer's own flush pipeline. +wal_flush_interval_ms = 60_000 +# Sparse-block relocation interval; 0 disables relocation, not snapshot folding. +merge_gc_interval_ms = 0 + +[wal] +# Segment rotation/GC granularity. +wal_segment_size = 67_108_864 +# WAL batch size and low-traffic flush watchdog; affect consensus input cost. +wal_flush_batch_bytes = 65_536 +wal_flush_watchdog_ms = 100 diff --git a/app/crowdb-kv-server/examples/openapi-export.rs b/app/crowdb-paxos-kv-server/examples/openapi-export.rs similarity index 91% rename from app/crowdb-kv-server/examples/openapi-export.rs rename to app/crowdb-paxos-kv-server/examples/openapi-export.rs index d1ae55940..feebdf83e 100644 --- a/app/crowdb-kv-server/examples/openapi-export.rs +++ b/app/crowdb-paxos-kv-server/examples/openapi-export.rs @@ -11,7 +11,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> { if let Some(parent) = out.parent() { std::fs::create_dir_all(parent)?; } - let json = crowdb_kv_server::mgmt::openapi_json(); + let json = crowdb_paxos_kv_server::mgmt::openapi_json(); std::fs::write(out, serde_json::to_string_pretty(&json)?)?; println!("{}", out.display()); Ok(()) diff --git a/app/crowdb-kv-server/src/background.rs b/app/crowdb-paxos-kv-server/src/background.rs similarity index 100% rename from app/crowdb-kv-server/src/background.rs rename to app/crowdb-paxos-kv-server/src/background.rs diff --git a/app/crowdb-kv-server/src/background/discovery.rs b/app/crowdb-paxos-kv-server/src/background/discovery.rs similarity index 100% rename from app/crowdb-kv-server/src/background/discovery.rs rename to app/crowdb-paxos-kv-server/src/background/discovery.rs diff --git a/app/crowdb-paxos-kv-server/src/background/domain_monitor.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor.rs new file mode 100644 index 000000000..0d6d9880a --- /dev/null +++ b/app/crowdb-paxos-kv-server/src/background/domain_monitor.rs @@ -0,0 +1,336 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Descriptor-driven, leader-tenure-fenced domain monitor supervision. + +use std::collections::{HashMap, HashSet}; +use std::future::Future; +use std::pin::Pin; +use std::sync::Arc; +use std::time::Duration; + +use bytes::Bytes; +use crowdb_paxos_kv::cluster::group_operations::{ + KvGroupOperationError, KvGroupOperations, KvGroupScanRequest, KvReadConsistency, +}; +use crowdb_protocol::chunk_kv::DomainMonitorDescriptor; +use crowdb_protocol::key::{DomainMonitorKey, TextKey}; +use tracing::{debug, info, warn}; + +use crate::group0_control_plane::Group0ControlPlane; +use crate::store_registry::PaxosKvStoreRegistry; + +mod chunk_kv; +mod chunkdb; +mod chunkdb_dynamic; +mod diskdb; +mod diskdb_placement; + +pub use chunk_kv::ChunkKvRangeMonitorDriver; +pub use chunkdb::ChunkdbRangeMonitorDriver; +pub use chunkdb_dynamic::ChunkdbDynamicMonitorDriver; +pub use diskdb::DiskdbOwnershipMonitorDriver; +pub use diskdb_placement::DiskdbPlacementMonitorDriver; + +/// Run one monitor child at a time and reconstruct it after an unexpected +/// return or panic. +pub fn spawn_restarting_monitor<F, Fut>( + mut stop_rx: tokio::sync::watch::Receiver<bool>, + make_monitor: F, +) -> tokio::task::JoinHandle<()> +where + F: Fn(tokio::sync::watch::Receiver<bool>) -> Fut + Send + Sync + 'static, + Fut: Future<Output = ()> + Send + 'static, +{ + tokio::spawn(async move { + loop { + if *stop_rx.borrow() { + break; + } + let mut child = tokio::spawn(make_monitor(stop_rx.clone())); + tokio::select! { + result = &mut child => { + if !*stop_rx.borrow() { + warn!(?result, "domain monitor task exited; restarting"); + tokio::task::yield_now().await; + } + } + changed = stop_rx.changed() => { + if changed.is_err() || *stop_rx.borrow() { + let _ = child.await; + break; + } + } + } + } + }) +} + +pub type DomainMonitorFuture<'a> = Pin<Box<dyn Future<Output = Result<(), String>> + Send + 'a>>; + +/// One compiled monitor implementation selected by a persisted descriptor. +pub trait DomainMonitorDriver: Send + Sync { + fn domain(&self) -> &'static str; + fn driver_version(&self) -> u32; + fn tick<'a>( + &'a self, + control: &'a Group0ControlPlane, + descriptor: &'a DomainMonitorDescriptor, + ) -> DomainMonitorFuture<'a>; +} + +#[derive(Default)] +pub struct DomainMonitorDrivers { + drivers: HashMap<(String, u32), Arc<dyn DomainMonitorDriver>>, +} + +impl DomainMonitorDrivers { + #[must_use] + pub fn new(drivers: Vec<Arc<dyn DomainMonitorDriver>>) -> Self { + let drivers = drivers + .into_iter() + .map(|driver| ((driver.domain().to_string(), driver.driver_version()), driver)) + .collect(); + Self { drivers } + } + + fn find(&self, descriptor: &DomainMonitorDescriptor) -> Option<Arc<dyn DomainMonitorDriver>> { + self.drivers + .get(&(descriptor.domain.clone(), descriptor.driver_version)) + .cloned() + } +} + +pub struct DomainMonitorSupervisorHandle { + stop_tx: tokio::sync::watch::Sender<bool>, + supervisor: Option<tokio::task::JoinHandle<()>>, +} + +impl DomainMonitorSupervisorHandle { + pub fn stop(&self) { + let _ = self.stop_tx.send(true); + } + + pub async fn stop_and_wait(mut self) { + self.stop(); + if let Some(supervisor) = self.supervisor.take() { + let _ = supervisor.await; + } + } +} + +impl Drop for DomainMonitorSupervisorHandle { + fn drop(&mut self) { + let _ = self.stop_tx.send(true); + } +} + +struct RunningMonitor { + descriptor: DomainMonitorDescriptor, + stop_tx: tokio::sync::watch::Sender<bool>, + supervisor: tokio::task::JoinHandle<()>, +} + +/// Observe persisted descriptors on this replica and prepare one supervised +/// task for every supported domain. Driver ticks acquire their own local +/// group-0 leader tenure; followers therefore stay prepared but idle. +#[must_use] +pub fn spawn_domain_monitor_supervisor( + registry: Arc<PaxosKvStoreRegistry>, + drivers: DomainMonitorDrivers, + discovery_interval: Duration, +) -> DomainMonitorSupervisorHandle { + let drivers = Arc::new(drivers); + let (stop_tx, mut stop_rx) = tokio::sync::watch::channel(false); + let supervisor = tokio::spawn(async move { + let mut running: HashMap<String, RunningMonitor> = HashMap::new(); + let mut interval = tokio::time::interval(discovery_interval); + interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); + loop { + tokio::select! { + _ = interval.tick() => { + match discover_descriptors(®istry).await { + Ok(descriptors) => reconcile_descriptors( + ®istry, + &drivers, + descriptors, + &mut running, + ), + Err(error) => warn!(%error, "domain monitor descriptor discovery failed"), + } + } + changed = stop_rx.changed() => { + if changed.is_err() || *stop_rx.borrow() { + break; + } + } + } + } + for (_, monitor) in running.drain() { + let _ = monitor.stop_tx.send(true); + let _ = monitor.supervisor.await; + } + }); + DomainMonitorSupervisorHandle { + stop_tx, + supervisor: Some(supervisor), + } +} + +async fn discover_descriptors( + registry: &PaxosKvStoreRegistry, +) -> Result<Vec<DomainMonitorDescriptor>, KvGroupOperationError> { + let store = registry + .get_store(0) + .ok_or_else(|| KvGroupOperationError::Unavailable("store 0 is not hosted locally".into()))?; + let group = store + .get_group(0) + .ok_or_else(|| KvGroupOperationError::Unavailable("group 0 is not hosted locally".into()))?; + let operations = KvGroupOperations::new(group, 1024 * 1024); + let prefix = Bytes::from(DomainMonitorKey::text_prefix_all()); + let mut start_after = Bytes::new(); + let mut scan_cutoff = 0; + let mut descriptors = Vec::new(); + loop { + let scan = operations + .scan(&KvGroupScanRequest { + prefix: prefix.clone(), + start_after: start_after.clone(), + end_key: Bytes::new(), + limit: 128, + consistency: KvReadConsistency::MinAppliedSlot(0), + keys_only: false, + count_only: false, + deadline_ms: 0, + bounded: true, + requested_scan_cutoff: scan_cutoff, + direction: crowdb_paxos_kv::kv::ScanDirection::Forward, + }) + .await?; + scan_cutoff = scan.scan_cutoff; + for item in &scan.items { + let descriptor: DomainMonitorDescriptor = serde_json::from_slice(&item.value) + .map_err(|error| KvGroupOperationError::Internal(error.to_string()))?; + descriptor + .validate() + .map_err(|error| KvGroupOperationError::Internal(error.to_string()))?; + let path = std::str::from_utf8(&item.key) + .map_err(|error| KvGroupOperationError::Internal(error.to_string()))?; + let key = DomainMonitorKey::from_path(path) + .map_err(|error| KvGroupOperationError::Internal(error.to_string()))?; + if key.domain != descriptor.domain { + return Err(KvGroupOperationError::Internal(format!( + "monitor descriptor key domain {} does not match value {}", + key.domain, descriptor.domain + ))); + } + descriptors.push(descriptor); + } + let Some(last) = scan.items.last() else { + break; + }; + if !scan.truncated { + break; + } + start_after = last.key.clone(); + } + Ok(descriptors) +} + +fn reconcile_descriptors( + registry: &Arc<PaxosKvStoreRegistry>, + drivers: &DomainMonitorDrivers, + descriptors: Vec<DomainMonitorDescriptor>, + running: &mut HashMap<String, RunningMonitor>, +) { + let observed: HashSet<_> = descriptors + .iter() + .map(|descriptor| descriptor.domain.clone()) + .collect(); + let removed: Vec<_> = running + .keys() + .filter(|domain| !observed.contains(*domain)) + .cloned() + .collect(); + for domain in removed { + if let Some(monitor) = running.remove(&domain) { + let _ = monitor.stop_tx.send(true); + monitor.supervisor.abort(); + info!(%domain, "domain monitor descriptor removed; task stopped"); + } + } + + for descriptor in descriptors { + if let Some(existing) = running.get(&descriptor.domain) { + if existing.descriptor != descriptor { + warn!(domain = %descriptor.domain, "domain monitor descriptor changed unexpectedly; retaining original task fail-closed"); + } + continue; + } + let Some(driver) = drivers.find(&descriptor) else { + warn!( + domain = %descriptor.domain, + driver_version = descriptor.driver_version, + "unsupported domain monitor descriptor" + ); + continue; + }; + let monitor = spawn_driver(registry, descriptor.clone(), driver); + running.insert(descriptor.domain.clone(), monitor); + } +} + +fn spawn_driver( + registry: &Arc<PaxosKvStoreRegistry>, + descriptor: DomainMonitorDescriptor, + driver: Arc<dyn DomainMonitorDriver>, +) -> RunningMonitor { + let (stop_tx, stop_rx) = tokio::sync::watch::channel(false); + let registry = Arc::clone(registry); + let child_descriptor = descriptor.clone(); + let supervisor = spawn_restarting_monitor(stop_rx, move |mut child_stop| { + let registry = Arc::clone(®istry); + let descriptor = child_descriptor.clone(); + let driver = Arc::clone(&driver); + async move { + let mut interval = tokio::time::interval(Duration::from_millis(descriptor.heartbeat_interval_ms)); + interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); + loop { + tokio::select! { + _ = interval.tick() => run_driver_tick(®istry, driver.as_ref(), &descriptor).await, + changed = child_stop.changed() => { + if changed.is_err() || *child_stop.borrow() { + break; + } + } + } + } + } + }); + RunningMonitor { + descriptor, + stop_tx, + supervisor, + } +} + +async fn run_driver_tick( + registry: &PaxosKvStoreRegistry, + driver: &dyn DomainMonitorDriver, + descriptor: &DomainMonitorDescriptor, +) { + let Some(store) = registry.get_store(0) else { + return; + }; + match Group0ControlPlane::acquire(&store).await { + Ok(control) => { + if let Err(error) = driver.tick(&control, descriptor).await { + warn!(domain = %descriptor.domain, %error, "domain monitor tick failed"); + } + } + Err(KvGroupOperationError::NotLeader { .. }) => { + debug!(domain = %descriptor.domain, "domain monitor idle on group-0 follower"); + } + Err(error) => warn!(domain = %descriptor.domain, %error, "domain monitor leader fence unavailable"), + } +} diff --git a/app/crowdb-kv-server/src/background/domain_monitor/chunk_kv.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv.rs similarity index 99% rename from app/crowdb-kv-server/src/background/domain_monitor/chunk_kv.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv.rs index 7c757f980..db4176a31 100644 --- a/app/crowdb-kv-server/src/background/domain_monitor/chunk_kv.rs +++ b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv.rs @@ -7,7 +7,7 @@ use std::collections::{HashMap, HashSet}; use std::time::{SystemTime, UNIX_EPOCH}; use bytes::Bytes; -use crowdb_kv::cluster::group_operations::{KvGroupOperationError, KvGroupScanItem}; +use crowdb_paxos_kv::cluster::group_operations::{KvGroupOperationError, KvGroupScanItem}; use crowdb_protocol::chunk_kv::{ AuthorityReleaseProof, ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogPartitionState, DomainFailurePolicy, DomainMonitorDescriptor, Id128, OwnerDescriptor, ServingAssignment, ServingGrant, SplitPhase, diff --git a/app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/balance.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/balance.rs similarity index 99% rename from app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/balance.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/balance.rs index 61a3700f3..e143a1830 100644 --- a/app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/balance.rs +++ b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/balance.rs @@ -6,7 +6,7 @@ use std::collections::{HashMap, HashSet}; use bytes::Bytes; -use crowdb_kv::cluster::group_operations::KvGroupOperationError; +use crowdb_paxos_kv::cluster::group_operations::KvGroupOperationError; use crowdb_protocol::chunk_kv::{ ChunkKvRangeBalancePolicy, ChunkKvRangeCatalogEntry, DomainMonitorDescriptor, Id128, OwnerDescriptor, SplitPhase, TransferPhase, TransferTransition, diff --git a/app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/balance/materialization.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/balance/materialization.rs similarity index 100% rename from app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/balance/materialization.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/balance/materialization.rs diff --git a/app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/balance/observation.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/balance/observation.rs similarity index 100% rename from app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/balance/observation.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/balance/observation.rs diff --git a/app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/balance/selection.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/balance/selection.rs similarity index 100% rename from app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/balance/selection.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/balance/selection.rs diff --git a/app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/balance/selection/candidates.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/balance/selection/candidates.rs similarity index 100% rename from app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/balance/selection/candidates.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/balance/selection/candidates.rs diff --git a/app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/balance/selection/diagnostics.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/balance/selection/diagnostics.rs similarity index 100% rename from app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/balance/selection/diagnostics.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/balance/selection/diagnostics.rs diff --git a/app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/balance/split.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/balance/split.rs similarity index 100% rename from app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/balance/split.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/balance/split.rs diff --git a/app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/catalog.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/catalog.rs similarity index 99% rename from app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/catalog.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/catalog.rs index c340cc986..85cfcabd5 100644 --- a/app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/catalog.rs +++ b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/catalog.rs @@ -4,7 +4,7 @@ //! Immutable chunk-KV range-catalog loading and transition cutover. use bytes::Bytes; -use crowdb_kv::cluster::group_operations::KvGroupOperationError; +use crowdb_paxos_kv::cluster::group_operations::KvGroupOperationError; use crowdb_protocol::chunk_kv::{ AuthorityReleaseProof, ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogHead, ChunkKvRangeCatalogPage, ChunkKvRangeCatalogPageRef, ChunkKvRangeCatalogPartitionState, SplitPhase, SplitTransition, diff --git a/app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/catalog/endpoints.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/catalog/endpoints.rs similarity index 100% rename from app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/catalog/endpoints.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/catalog/endpoints.rs diff --git a/app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/grants.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/grants.rs similarity index 97% rename from app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/grants.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/grants.rs index aa95956f9..530db8cf8 100644 --- a/app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/grants.rs +++ b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/grants.rs @@ -6,7 +6,7 @@ use super::{catalog, operation_error, read_transfers, wall_time_ms}; use crate::group0_control_plane::Group0ControlPlane; use bytes::Bytes; -use crowdb_kv::cluster::group_operations::KvGroupMutation; +use crowdb_paxos_kv::cluster::group_operations::KvGroupMutation; use crowdb_protocol::{ chunk_kv::{ChunkKvRangeCatalogPartitionState, DomainMonitorDescriptor, ServingAssignment, ServingGrant}, chunk_stream::{StreamProcessLease, StreamPurpose}, @@ -109,7 +109,7 @@ async fn publish( if lease.identity.purpose != StreamPurpose::Wal || lease.identity.instance_id != instance_id { return Err("WAL process lease does not match its key".into()); } - if lease.identity.process_epoch != process_epoch || now >= lease.expires_at_ms { + if lease.revoked || lease.identity.process_epoch != process_epoch || now >= lease.expires_at_ms { return Ok(()); } let mut grant = ServingGrant { diff --git a/app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/liveness.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/liveness.rs similarity index 100% rename from app/crowdb-kv-server/src/background/domain_monitor/chunk_kv/liveness.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/chunk_kv/liveness.rs diff --git a/app/crowdb-kv-server/src/background/domain_monitor/chunkdb.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunkdb.rs similarity index 100% rename from app/crowdb-kv-server/src/background/domain_monitor/chunkdb.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/chunkdb.rs diff --git a/app/crowdb-kv-server/src/background/domain_monitor/chunkdb_dynamic.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunkdb_dynamic.rs similarity index 98% rename from app/crowdb-kv-server/src/background/domain_monitor/chunkdb_dynamic.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/chunkdb_dynamic.rs index 3c97172ff..ffeda12ea 100644 --- a/app/crowdb-kv-server/src/background/domain_monitor/chunkdb_dynamic.rs +++ b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunkdb_dynamic.rs @@ -8,7 +8,7 @@ use super::{DomainMonitorDriver, DomainMonitorFuture}; mod owners; mod snapshot; use crate::group0_control_plane::Group0ControlPlane; -use crowdb_kv::cluster::group_operations::KvGroupMutation; +use crowdb_paxos_kv::cluster::group_operations::KvGroupMutation; use crowdb_protocol::chunk_kv::{DomainFailurePolicy, DomainMonitorDescriptor}; use crowdb_protocol::chunk_slot::{ ChunkSlot, ChunkSlotAuthority, ChunkSlotBinding, ChunkSlotBitmap, ChunkSlotMap, diff --git a/app/crowdb-kv-server/src/background/domain_monitor/chunkdb_dynamic/owners.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunkdb_dynamic/owners.rs similarity index 100% rename from app/crowdb-kv-server/src/background/domain_monitor/chunkdb_dynamic/owners.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/chunkdb_dynamic/owners.rs diff --git a/app/crowdb-kv-server/src/background/domain_monitor/chunkdb_dynamic/snapshot.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/chunkdb_dynamic/snapshot.rs similarity index 100% rename from app/crowdb-kv-server/src/background/domain_monitor/chunkdb_dynamic/snapshot.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/chunkdb_dynamic/snapshot.rs diff --git a/app/crowdb-kv-server/src/background/domain_monitor/diskdb.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/diskdb.rs similarity index 100% rename from app/crowdb-kv-server/src/background/domain_monitor/diskdb.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/diskdb.rs diff --git a/app/crowdb-kv-server/src/background/domain_monitor/diskdb_placement.rs b/app/crowdb-paxos-kv-server/src/background/domain_monitor/diskdb_placement.rs similarity index 99% rename from app/crowdb-kv-server/src/background/domain_monitor/diskdb_placement.rs rename to app/crowdb-paxos-kv-server/src/background/domain_monitor/diskdb_placement.rs index e8660dd5c..b28413359 100644 --- a/app/crowdb-kv-server/src/background/domain_monitor/diskdb_placement.rs +++ b/app/crowdb-paxos-kv-server/src/background/domain_monitor/diskdb_placement.rs @@ -6,7 +6,7 @@ use super::{DomainMonitorDriver, DomainMonitorFuture}; use crate::group0_control_plane::Group0ControlPlane; use bytes::Bytes; -use crowdb_kv::cluster::group_operations::KvGroupMutation; +use crowdb_paxos_kv::cluster::group_operations::KvGroupMutation; use crowdb_protocol::chunk_kv::{DomainFailurePolicy, DomainMonitorDescriptor}; use crowdb_protocol::common::{BindMapValue, InstanceValue, OwnerMapValue}; use crowdb_protocol::key::{BindMapKey, DiskGroupKey, InstanceKey, OwnerMapKey, RackKey, TextKey}; diff --git a/app/crowdb-kv-server/src/background/identity.rs b/app/crowdb-paxos-kv-server/src/background/identity.rs similarity index 97% rename from app/crowdb-kv-server/src/background/identity.rs rename to app/crowdb-paxos-kv-server/src/background/identity.rs index 20fba4614..a5d027a8e 100644 --- a/app/crowdb-kv-server/src/background/identity.rs +++ b/app/crowdb-paxos-kv-server/src/background/identity.rs @@ -33,7 +33,7 @@ pub fn load_or_create( Err(error) if error.kind() == io::ErrorKind::NotFound => {} Err(error) => return Err(error), } - let generated = crowdb_kv_client::new_client_id(); + let generated = crowdb_paxos_kv_client::new_client_id(); let stored = StoredIdentity { version: 1, instance_id: requested.unwrap_or(generated), diff --git a/app/crowdb-kv-server/src/background/keepalive.rs b/app/crowdb-paxos-kv-server/src/background/keepalive.rs similarity index 96% rename from app/crowdb-kv-server/src/background/keepalive.rs rename to app/crowdb-paxos-kv-server/src/background/keepalive.rs index 784109955..cc42f74c6 100644 --- a/app/crowdb-kv-server/src/background/keepalive.rs +++ b/app/crowdb-paxos-kv-server/src/background/keepalive.rs @@ -9,12 +9,12 @@ use std::sync::Arc; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient, ServiceRegistryClient}; use crowdb_protocol::common::HostedGroup; use tokio::task::JoinHandle; use tracing::{debug, info, info_span, warn, Instrument}; -use crate::store_registry::KvStoreRegistry; +use crate::store_registry::PaxosKvStoreRegistry; /// Keep-alive loop handle. Call `stop` to unregister and terminate the /// background task. @@ -30,7 +30,7 @@ impl KeepAliveLoop { /// `interval_secs`. It reads `hosted_stores` / `hosted_groups` /// from the registry each tick so the record reflects live state. pub fn spawn( - registry: Arc<KvStoreRegistry>, + registry: Arc<PaxosKvStoreRegistry>, identity: crowdb_protocol::common::KvServerIdentity, rpc_endpoint: String, group0_endpoint: &str, @@ -56,7 +56,7 @@ impl KeepAliveLoop { mgmt_seeds.insert(0, rpc_endpoint.clone()); } let handle = tokio::spawn(async move { - let kv_client = CrowdbKvClient::new(ClientConfig::new(mgmt_seeds.clone())); + let kv_client = PaxosKvClient::new(ClientConfig::new(mgmt_seeds.clone())); if bootstrap_local { kv_client.seed_leader(0, 0, ep); } @@ -166,7 +166,7 @@ impl KeepAliveLoop { } async fn refresh_discovery( - registry: &KvStoreRegistry, + registry: &PaxosKvStoreRegistry, service: &ServiceRegistryClient, bootstrap_seeds: &[String], current: &mut Vec<String>, @@ -219,7 +219,7 @@ impl Drop for KeepAliveLoop { } /// Collect hosted store IDs and group IDs from the registry. -fn hosted_summary(registry: &KvStoreRegistry) -> (Vec<u64>, Vec<HostedGroup>) { +fn hosted_summary(registry: &PaxosKvStoreRegistry) -> (Vec<u64>, Vec<HostedGroup>) { let stores: Vec<u64> = registry.store_ids(); let mut groups: Vec<HostedGroup> = Vec::new(); for &sid in &stores { diff --git a/app/crowdb-kv-server/src/cli.rs b/app/crowdb-paxos-kv-server/src/cli.rs similarity index 95% rename from app/crowdb-kv-server/src/cli.rs rename to app/crowdb-paxos-kv-server/src/cli.rs index d7e63ad0a..7987280e8 100644 --- a/app/crowdb-kv-server/src/cli.rs +++ b/app/crowdb-paxos-kv-server/src/cli.rs @@ -2,15 +2,15 @@ // Licensed under the Apache License, Version 2.0. use clap::Parser; -use crowdb_protocol::KV_SERVER_MGMT_BASE; +use crowdb_protocol::PAXOS_KV_SERVER_MGMT_BASE; -/// `CrowDB` server — reference implementation wrapping the `crowdb_kv` library. +/// `CrowDB` server — reference implementation wrapping the `crowdb_paxos_kv` library. #[derive(Parser, Debug)] #[allow(clippy::struct_excessive_bools)] -#[command(name = "crowdb-kv-server", about = "CrowDB server daemon")] +#[command(name = "crowdb-paxos-kv-server", about = "CrowDB server daemon")] pub struct Cli { - /// HTTP management API listen port. Default: 10000. - #[arg(long, default_value_t = KV_SERVER_MGMT_BASE, value_parser = clap::value_parser!(u16).range(1..))] + /// HTTP management API listen port. Default: 10135. + #[arg(long, default_value_t = PAXOS_KV_SERVER_MGMT_BASE, value_parser = clap::value_parser!(u16).range(1..))] pub management_port: u16, /// HTTP management API bind address. @@ -18,7 +18,7 @@ pub struct Cli { pub management_addr: String, /// Reachable address advertised in the Group 0 service registry. - #[arg(long, env = "CROWDB_KV_MANAGEMENT_ADVERTISE_ADDR")] + #[arg(long, env = "CROWDB_PAXOS_KV_MANAGEMENT_ADVERTISE_ADDR")] pub management_advertise_addr: Option<std::net::IpAddr>, /// Node root directory. Derives `wal_root`=`<root>`/waldata, @@ -31,11 +31,11 @@ pub struct Cli { pub node_recovery_root: Option<std::path::PathBuf>, /// Optional TOML config for startup tunables in first-boot and restore modes. - /// When omitted, tunables use `CrowDBConfig::default()`. + /// When omitted, tunables use `PaxosKvConfig::default()`. #[arg(long)] pub config: Option<std::path::PathBuf>, - /// Port pool for crowdb-rpc `PxKvStore` listeners (comma/range format, e.g. "10100,10101,10110..10120"). + /// Port pool for crowdb-rpc `PxKvStore` listeners (comma/range format, e.g. "11135,11136,11145..11155"). #[arg(long)] pub ports: Option<String>, @@ -194,9 +194,9 @@ impl Cli { /// Returns the first complete-config validation error. pub fn apply_config_overrides( &self, - config: &mut crowdb_kv::common::config::CrowDBConfig, + config: &mut crowdb_paxos_kv::common::config::PaxosKvConfig, ) -> Result<(), String> { - use crowdb_kv::common::config::PxElectionConfig; + use crowdb_paxos_kv::common::config::PxElectionConfig; if let Some(profile) = self.election_profile.as_deref() { config.election = match profile { diff --git a/app/crowdb-kv-server/src/engine_collector.rs b/app/crowdb-paxos-kv-server/src/engine_collector.rs similarity index 91% rename from app/crowdb-kv-server/src/engine_collector.rs rename to app/crowdb-paxos-kv-server/src/engine_collector.rs index de35606dd..8f9387cc4 100644 --- a/app/crowdb-kv-server/src/engine_collector.rs +++ b/app/crowdb-paxos-kv-server/src/engine_collector.rs @@ -11,11 +11,13 @@ use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::{Arc, Mutex}; -use crate::store_registry::KvStoreRegistry; +use crate::store_registry::PaxosKvStoreRegistry; use arc_swap::ArcSwap; -use crowdb_kv::cluster::px_kv_store::PxKvStore; -use crowdb_kv::kv::CrowdbTreeEngine; -use crowdb_kv::metrics::{Counter, Gauge, MetricsRegistry, MetricsRunner}; +use crowdb_paxos_kv::cluster::px_kv_store::PxKvStore; +use crowdb_paxos_kv::kv::CrowdbTreeEngine; +use crowdb_paxos_kv::metrics::{Counter, Gauge, MetricsRegistry, MetricsRunner}; + +mod resources; /// Registered Rust handles for one (store, group): Paxos gauges + /// snapshot.pages.c bridge counter. @@ -113,10 +115,11 @@ fn read_paxos_gauges_per_group(store: &Arc<PxKvStore>) -> std::collections::Hash #[allow(clippy::too_many_lines)] pub fn setup_engine_collector( registry: &Arc<Mutex<MetricsRegistry>>, - store_registry: &Arc<KvStoreRegistry>, + store_registry: &Arc<PaxosKvStoreRegistry>, runner: &mut MetricsRunner, ) { type Key = (u64, u64); + runner.set_sampler(resources::Resources::new(Arc::clone(store_registry))); let mut handles: Vec<(Key, Arc<EngineHandles>)> = Vec::new(); let mut rpc_handles: Vec<(u64, Arc<RpcTransportHandles>)> = Vec::new(); @@ -218,6 +221,14 @@ pub fn setup_engine_collector( let replica = group.local_replica(); let engine = replica.learner.engine(); if let Some(e) = engine.as_any().downcast_ref::<CrowdbTreeEngine>() { + if let Err(error) = e.capture_work_metrics() { + tracing::warn!( + store_id = *store_id, + group_id = key.1, + ?error, + "Tree metrics capture failed; retaining the prior window" + ); + } let s = e.stats(); let current = s.snapshot_pages_total; let previous = hd.last_snapshot_pages.swap(current, Ordering::Relaxed); @@ -269,11 +280,16 @@ pub fn setup_engine_collector( let stores2 = Arc::clone(store_registry); runner.set_cpp_flush( move |writer, window_secs, timestamp, rust_width, count_w, tps_w| { - for store in stores2.stores_snapshot().values() { + for (store_id, store) in stores2.stores_snapshot().iter() { store.for_each_group(|group| { let replica = group.local_replica(); let engine = replica.learner.engine(); if let Some(e) = engine.as_any().downcast_ref::<CrowdbTreeEngine>() { + let prefix = format!("s.{store_id}.g.{}", group.group_id()); + let work = e.format_work_metrics_window(&prefix, timestamp); + if !work.is_empty() { + let _ = std::io::Write::write_all(writer, work.as_bytes()); + } let cpp_max = e.max_name_len(); let shared_width = rust_width.max(cpp_max); let str = diff --git a/app/crowdb-paxos-kv-server/src/engine_collector/resources.rs b/app/crowdb-paxos-kv-server/src/engine_collector/resources.rs new file mode 100644 index 000000000..115f00eb9 --- /dev/null +++ b/app/crowdb-paxos-kv-server/src/engine_collector/resources.rs @@ -0,0 +1,55 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use std::sync::Arc; +use std::time::Instant; + +use crowdb_common::metrics::MetricsSampler; +use crowdb_paxos_kv::kv::CrowdbTreeEngine; +use crowdb_tree_ffi::runtime::{ResourceUsageLog, ResourceUsageSnapshot}; + +use crate::store_registry::PaxosKvStoreRegistry; + +pub(super) struct Resources { + stores: Arc<PaxosKvStoreRegistry>, + windows: ResourceUsageLog, +} + +impl Resources { + pub fn new(stores: Arc<PaxosKvStoreRegistry>) -> Self { + Self { + stores, + windows: ResourceUsageLog::default(), + } + } + + fn observe(&mut self, name: &str, value: &Result<ResourceUsageSnapshot, crowdb_tree_ffi::CtError>) { + match value { + Ok(value) => self.windows.observe(name, *value, Instant::now()), + Err(_) => self.windows.sample_failed(name), + } + } +} + +impl MetricsSampler for Resources { + fn sample(&mut self) { + self.windows.begin_sample(); + self.observe("host.tree", &self.stores.tree_runtime.resource_usage()); + let stores = self.stores.stores_snapshot(); + for (store_id, store) in stores.iter() { + store.for_each_group(|group| { + let replica = group.local_replica(); + let engine = replica.learner.engine(); + if let Some(engine) = engine.as_any().downcast_ref::<CrowdbTreeEngine>() { + let name = format!("s.{store_id}.g.{}.tree", group.group_id()); + self.observe(&name, &engine.handle().resource_usage()); + } + }); + } + self.windows.end_sample(Instant::now()); + } + + fn flush(&mut self, writer: &mut dyn std::io::Write, timestamp: &str) -> std::io::Result<()> { + self.windows.flush(writer, timestamp, Instant::now()) + } +} diff --git a/app/crowdb-kv-server/src/group0_control_plane.rs b/app/crowdb-paxos-kv-server/src/group0_control_plane.rs similarity index 97% rename from app/crowdb-kv-server/src/group0_control_plane.rs rename to app/crowdb-paxos-kv-server/src/group0_control_plane.rs index 846a45f5a..693f05ffa 100644 --- a/app/crowdb-kv-server/src/group0_control_plane.rs +++ b/app/crowdb-paxos-kv-server/src/group0_control_plane.rs @@ -8,11 +8,11 @@ use std::sync::Arc; use std::time::{SystemTime, UNIX_EPOCH}; use bytes::Bytes; -use crowdb_kv::cluster::group_operations::{ +use crowdb_paxos_kv::cluster::group_operations::{ KvGroupMutation, KvGroupOperationError, KvGroupOperations, KvGroupRead, KvGroupScanItem, KvGroupScanRequest, KvReadConsistency, KvRequestIdentity, }; -use crowdb_kv::cluster::PxKvStore; +use crowdb_paxos_kv::cluster::PxKvStore; static CONTROL_PLANE_INSTANCE_NONCE: AtomicU64 = AtomicU64::new(1); @@ -106,7 +106,7 @@ impl Group0ControlPlane { deadline_ms: 0, bounded: true, requested_scan_cutoff: scan_cutoff, - direction: crowdb_kv::kv::ScanDirection::Forward, + direction: crowdb_paxos_kv::kv::ScanDirection::Forward, }) .await?; Ok((scan.items, scan.truncated, scan.scan_cutoff)) diff --git a/app/crowdb-paxos-kv-server/src/lib.rs b/app/crowdb-paxos-kv-server/src/lib.rs new file mode 100644 index 000000000..bd5b57515 --- /dev/null +++ b/app/crowdb-paxos-kv-server/src/lib.rs @@ -0,0 +1,18 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! `crowdb-paxos-kv-server` library — exposes the binary's modules so integration +//! tests under `tests/` can exercise CLI parsing, the management router, +//! and the registry without spawning a process. +//! +//! The binary entry (`main.rs`) imports from this lib via +//! `use crowdb_paxos_kv_server::{cli, mgmt, startup, store_registry};`. + +pub mod background; +pub mod cli; +pub mod engine_collector; +pub mod group0_control_plane; +pub mod mgmt; +pub mod recovery; +pub mod shutdown; +pub mod store_registry; diff --git a/app/crowdb-kv-server/src/main.rs b/app/crowdb-paxos-kv-server/src/main.rs similarity index 76% rename from app/crowdb-kv-server/src/main.rs rename to app/crowdb-paxos-kv-server/src/main.rs index e60dc75d9..c7c8fabc7 100644 --- a/app/crowdb-kv-server/src/main.rs +++ b/app/crowdb-paxos-kv-server/src/main.rs @@ -1,9 +1,9 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -//! `crowdb-kv-server` — `CrowDB` daemon entry point. +//! `crowdb-paxos-kv-server` — `CrowDB` daemon entry point. //! -//! Wraps the `crowdb_kv` library into a runnable server process with: +//! Wraps the `crowdb_paxos_kv` library into a runnable server process with: //! - CLI-driven startup of `PxKvStore` instances with `PxGroup`s. //! - HTTP management API for runtime topology control. //! - Graceful shutdown on SIGINT/SIGTERM. @@ -13,18 +13,18 @@ use std::net::SocketAddr; use std::sync::Arc; use clap::Parser; -use tracing::{debug, info, warn}; +use tracing::{debug, error, info, warn}; -use crowdb_kv::cluster::kv_server::KvServer; -use crowdb_kv::cluster::local_replica::PxLocalReplicaRole; -use crowdb_kv::cluster::px_kv_store::PxKvStore; -use crowdb_kv::common::config::{CrowDBConfig, ServerConfig}; -use crowdb_kv::metrics::MetricsRunner; +use crowdb_paxos_kv::cluster::local_replica::PxLocalReplicaRole; +use crowdb_paxos_kv::cluster::paxos_kv_server::PaxosKvServer; +use crowdb_paxos_kv::cluster::px_kv_store::PxKvStore; +use crowdb_paxos_kv::common::config::PaxosKvConfig; +use crowdb_paxos_kv::metrics::MetricsRunner; -use crowdb_kv_server::cli::{parse_id_list, parse_port_list, Cli}; -use crowdb_kv_server::mgmt::{self, persisted_port_for_store}; -use crowdb_kv_server::recovery::startup::create_group_with_wal; -use crowdb_kv_server::store_registry::KvStoreRegistry; +use crowdb_paxos_kv_server::cli::{parse_id_list, parse_port_list, Cli}; +use crowdb_paxos_kv_server::mgmt::{self, persisted_port_for_store}; +use crowdb_paxos_kv_server::recovery::startup::create_group_with_wal_on_runtime; +use crowdb_paxos_kv_server::store_registry::PaxosKvStoreRegistry; #[tokio::main] #[allow(clippy::too_many_lines)] @@ -46,21 +46,21 @@ async fn main() { let _guards = if args.log { crowdb_common::logging::init_file_and_console_logging_split( &log_dir, - "crowdb-kv-server", + "crowdb-paxos-kv-server", args.log_max_file_mb, args.log_max_files, - crowdb_kv::common::logging::CROWDB_KV_DEFAULT_FILTER, + crowdb_paxos_kv::common::logging::CROWDB_PAXOS_KV_DEFAULT_FILTER, "warn", ) - .expect("failed to initialize crowdb-kv-server logging") + .expect("failed to initialize crowdb-paxos-kv-server logging") } else { - crowdb_kv::common::logging::init_file_logging( + crowdb_paxos_kv::common::logging::init_file_logging( &log_dir, - "crowdb-kv-server", + "crowdb-paxos-kv-server", args.log_max_file_mb, args.log_max_files, ) - .expect("failed to initialize crowdb-kv-server logging") + .expect("failed to initialize crowdb-paxos-kv-server logging") }; // Initialize the C++ spdlog async logger as a process-global resource. @@ -71,7 +71,7 @@ async fn main() { &cpp_level, args.log_max_file_mb, args.log_max_files, - "crowdb-kv-server-tree", + "crowdb-paxos-kv-server-tree", ); // Initialize the crowdb-rpc C++ spdlog logger (connection failures, @@ -82,7 +82,7 @@ async fn main() { &cpp_level, args.log_max_file_mb, args.log_max_files, - "crowdb-kv-server-rpc", + "crowdb-paxos-kv-server-rpc", ); // Optional err-to-stderr mirror for the C++ stacks. @@ -91,7 +91,7 @@ async fn main() { crowdb_rpc_ffi::add_log_stderr(stderr_level); } - info!("crowdb-kv-server starting..."); + info!("crowdb-paxos-kv-server starting..."); // Metrics runner: periodic flush to a dedicated metrics log file. let mut metrics_runner = create_metrics_runner( @@ -127,13 +127,13 @@ async fn main() { // omitted, use defaults. Paths are always derived from // --root via apply_root (fixed on-disk layout). let mut config = match args.config.as_ref() { - Some(path) => CrowDBConfig::load_from_file(path) + Some(path) => PaxosKvConfig::load_from_file(path) .unwrap_or_else(|e| panic!("failed to load config from {}: {e}", path.display())), - None => CrowDBConfig::default(), + None => PaxosKvConfig::default(), }; config.apply_root(&args.root); if let Some(root) = &args.node_recovery_root { - crowdb_kv_server::recovery::member_routes::configure(&config.config_root, root) + crowdb_paxos_kv_server::recovery::member_routes::configure(&config.config_root, root) .expect("invalid monitor member recovery source"); } @@ -145,7 +145,7 @@ async fn main() { .unwrap_or_else(|e| panic!("invalid config after CLI overrides: {e}")); let service_identity = (args.keepalive_interval > 0).then(|| { - crowdb_kv_server::background::identity::load_or_create( + crowdb_paxos_kv_server::background::identity::load_or_create( &config.config_root, args.instance_id, args.node_id, @@ -154,21 +154,25 @@ async fn main() { }); let registry = Arc::new( - KvStoreRegistry::try_with_config(config.clone()) + PaxosKvStoreRegistry::try_with_config(config.clone()) .unwrap_or_else(|error| panic!("failed to initialize WAL backend: {error}")) .with_metrics_registry(metrics_runner.as_ref().map_or_else( - || Arc::new(std::sync::Mutex::new(crowdb_kv::metrics::MetricsRegistry::new())), + || { + Arc::new(std::sync::Mutex::new( + crowdb_paxos_kv::metrics::MetricsRegistry::new(), + )) + }, |r| r.registry().clone(), )), ); - let _member_routes = crowdb_kv_server::recovery::member_routes::start(Arc::clone(®istry)); + let member_routes = crowdb_paxos_kv_server::recovery::member_routes::start(Arc::clone(®istry)); // Spawn a config file watcher for diff logging. Only when --config is // passed; with no toml there is nothing to watch. let watcher_old_config = Arc::new(std::sync::Mutex::new(config)); let _config_watcher = match args.config.as_ref() { Some(path) => { - match crowdb_common::config::watch::<CrowDBConfig, _>(std::path::Path::new(path), move |new| { + match crowdb_common::config::watch::<PaxosKvConfig, _>(std::path::Path::new(path), move |new| { let mut old = watcher_old_config.lock().unwrap(); crowdb_common::config::log_diff(&*old, new); *old = new.clone(); @@ -206,9 +210,9 @@ async fn main() { std::process::exit(1); }); - let router = mgmt::router(crowdb_kv_server::mgmt::operation_registry::AppState::new( - registry.clone(), - )); + let app_state = crowdb_paxos_kv_server::mgmt::operation_registry::AppState::new(registry.clone()); + let operation_cleanup = app_state.start_operation_cleanup(); + let router = mgmt::router(app_state); let listener = tokio::net::TcpListener::bind(mgmt_addr) .await .unwrap_or_else(|e| { @@ -243,12 +247,18 @@ async fn main() { // - First-boot mode: no group 0 on disk. Use --stores/--groups CLI // args (if given) to create stores; otherwise boot empty so the // operator can call POST /system/init. - let mut local_groups = crowdb_kv_server::recovery::restore::scan_local_groups(®istry.config.wal_root) - .await - .unwrap_or_else(|e| { - warn!(error = %e, wal_root = %registry.config.wal_root.display(), "scan_local_groups failed; treating as empty"); - Vec::new() - }); + let mut local_groups = + match crowdb_paxos_kv_server::recovery::restore::scan_local_groups(®istry.config.wal_root).await { + Ok(groups) => groups, + Err(error) => { + error!( + error = %error, + wal_root = %registry.config.wal_root.display(), + "scan_local_groups failed; refusing to continue startup" + ); + std::process::exit(1); + } + }; let cleanup_pending = mgmt::system_cleanup_pending(®istry.config.config_root) .expect("cannot inspect system cleanup fence"); if cleanup_pending { @@ -257,14 +267,18 @@ async fn main() { if local_groups.is_empty() { info!("first-boot mode: no group 0 on disk"); if let Some(b) = bootstrap.as_ref().filter(|_| !cleanup_pending) { - create_and_start_stores( + if let Err(error) = create_and_start_stores( &b.store_ids, &b.group_ids, b.replica_id, b.ports.clone(), registry.clone(), ) - .await; + .await + { + error!(%error, "first-boot store creation failed; refusing to continue with partial state"); + std::process::exit(1); + } } // Reconcile is a no-op without group 0; skip the call. } else { @@ -275,8 +289,17 @@ async fn main() { if bootstrap.is_some() { warn!("restore mode: --stores/--groups ignored (local disk is the source of truth)"); } - crowdb_kv_server::recovery::restore::load_local_groups(&local_groups, args.replica, ®istry).await; - crowdb_kv_server::recovery::reconcile::reconcile_with_group0(®istry).await; + if let Err(error) = crowdb_paxos_kv_server::recovery::restore::load_local_groups( + &local_groups, + args.replica, + ®istry, + ) + .await + { + error!(%error, "restore failed; refusing to continue with partial local state"); + std::process::exit(1); + } + crowdb_paxos_kv_server::recovery::reconcile::reconcile_with_group0(®istry).await; } // Start the keep-alive loop (registers under /srv/kv-server/<id>). @@ -297,20 +320,22 @@ async fn main() { .and_then(|s| s.listen_addr().map(|a| a.to_string())) .or_else(|| registry.first_port().map(|p| format!("{display_ip}:{p}"))) .unwrap_or_else(|| format!("http://{display_addr}")); - Some(crowdb_kv_server::background::keepalive::KeepAliveLoop::spawn( - registry.clone(), - identity, - mgmt_endpoint, - &group0_ep, - args.group0_management_seeds.clone(), - registry - .config - .node_root - .as_ref() - .map(|p| p.to_string_lossy().into_owned()) - .unwrap_or_default(), - args.keepalive_interval, - )) + Some( + crowdb_paxos_kv_server::background::keepalive::KeepAliveLoop::spawn( + registry.clone(), + identity, + mgmt_endpoint, + &group0_ep, + args.group0_management_seeds.clone(), + registry + .config + .node_root + .as_ref() + .map(|p| p.to_string_lossy().into_owned()) + .unwrap_or_default(), + args.keepalive_interval, + ), + ) } else { None }; @@ -319,7 +344,7 @@ async fn main() { // their compiled drivers prepared. Driver ticks acquire an exact local // leader tenure, so followers remain idle without loopback RPC. let domain_monitors = if args.binding_monitor_interval > 0 { - use crowdb_kv_server::background::domain_monitor::{ + use crowdb_paxos_kv_server::background::domain_monitor::{ spawn_domain_monitor_supervisor, ChunkKvRangeMonitorDriver, ChunkdbDynamicMonitorDriver, ChunkdbRangeMonitorDriver, DiskdbOwnershipMonitorDriver, DomainMonitorDrivers, }; @@ -330,7 +355,7 @@ async fn main() { Arc::new(ChunkdbDynamicMonitorDriver), Arc::new(ChunkKvRangeMonitorDriver::new()), Arc::new(DiskdbOwnershipMonitorDriver::new()), - Arc::new(crowdb_kv_server::background::domain_monitor::DiskdbPlacementMonitorDriver), + Arc::new(crowdb_paxos_kv_server::background::domain_monitor::DiskdbPlacementMonitorDriver), ]), std::time::Duration::from_secs(args.binding_monitor_interval), )) @@ -344,7 +369,7 @@ async fn main() { // appear in the metrics log alongside the Rust-side metrics. if let Some(ref mut runner) = metrics_runner { let reg = runner.registry().clone(); - crowdb_kv_server::engine_collector::setup_engine_collector(®, ®istry, runner); + crowdb_paxos_kv_server::engine_collector::setup_engine_collector(®, ®istry, runner); runner.start(); info!(interval_secs = args.metrics_interval, "metrics runner started"); } @@ -357,6 +382,9 @@ async fn main() { tracing::error!("management HTTP server error: {e}"); }); + operation_cleanup.abort(); + let _ = operation_cleanup.await; + // Graceful cascade shutdown (PxKvStore → PxGroup → replicas) if let Some(ref mut runner) = metrics_runner { runner.stop().await; @@ -370,7 +398,7 @@ async fn main() { monitors.stop_and_wait().await; info!("domain monitor supervisor stopped"); } - graceful_shutdown(registry).await; + graceful_shutdown(registry, member_routes, metrics_runner).await; } /// Create a metrics runner if interval > 0, else None. Does NOT start @@ -384,9 +412,13 @@ fn create_metrics_runner( if interval_secs == 0 { return None; } - let metrics_file = - crowdb_kv::common::logging::open_metrics_log(log_dir, "crowdb-kv-server", max_file_mb, max_files) - .expect("failed to open metrics log file"); + let metrics_file = crowdb_paxos_kv::common::logging::open_metrics_log( + log_dir, + "crowdb-paxos-kv-server", + max_file_mb, + max_files, + ) + .expect("failed to open metrics log file"); let runner = MetricsRunner::new(metrics_file, interval_secs); Some(runner) } @@ -470,8 +502,8 @@ async fn create_and_start_stores( group_ids: &[u64], replica_id: u64, ports: Vec<u16>, - registry: Arc<KvStoreRegistry>, -) { + registry: Arc<PaxosKvStoreRegistry>, +) -> Result<(), String> { debug!( store_count = store_ids.len(), group_count = group_ids.len(), @@ -515,7 +547,7 @@ async fn create_and_start_stores( store_id, group_id, replica_id, "creating PxGroup with local replica" ); - let group = match create_group_with_wal( + let group = match create_group_with_wal_on_runtime( store_id, group_id, replica_id, @@ -523,21 +555,22 @@ async fn create_and_start_stores( ®istry.config, registry.wal_backend.clone(), registry.crowtree_backend, + registry.tree_runtime.clone(), ) .await { Ok(group) => group, Err(e) => { - tracing::error!(s = store_id, g = group_id, error = %e, "failed to create WAL-backed group"); - continue; + return Err(format!( + "failed to create WAL-backed store {store_id} group {group_id}: {e}" + )); } }; store.add_group(group); } if let Err(e) = store.start().await { - tracing::error!(s = store_id, port, error = %e, "failed to start store, skipping"); - continue; + return Err(format!("failed to start store {store_id} on port {port}: {e}")); } // Wire the shared transport into all existing remote replicas. @@ -556,12 +589,17 @@ async fn create_and_start_stores( store_count = registry.store_count(), "all stores started, management API ready" ); + Ok(()) } /// Gracefully cascade-shutdown every store via [`PxKvStore::shutdown`]. /// /// Continues on errors; aggregates `critical:` messages to the operator. -async fn graceful_shutdown(registry: Arc<KvStoreRegistry>) { +async fn graceful_shutdown( + registry: Arc<PaxosKvStoreRegistry>, + member_routes: crowdb_paxos_kv_server::recovery::member_routes::Worker, + metrics_runner: Option<MetricsRunner>, +) { info!( store_count = registry.store_count(), "initiating graceful shutdown of crowdb-rpc stores" @@ -572,28 +610,24 @@ async fn graceful_shutdown(registry: Arc<KvStoreRegistry>) { crowdb_tree_ffi::ct_flush_logging(); crowdb_rpc_ffi::flush_logging(); - let mut total_errors = 0usize; - for (store_id, store) in registry.stores_snapshot().iter() { - let report = store - .shutdown(std::time::Duration::from_millis( - ServerConfig::DEFAULT.shutdown_timeout_ms, - )) - .await; - if !report.is_clean() { - total_errors += report.errors.len(); - for err in &report.errors { - tracing::error!(s = *store_id, "{err}"); - } - } - } - if total_errors == 0 { - info!("crowdb-kv-server shut down cleanly"); + let report = crowdb_paxos_kv_server::shutdown::shutdown_stores(®istry).await; + if report.is_clean() { + info!("crowdb-paxos-kv-server shut down cleanly"); } else { + for error in &report.errors { + tracing::error!(error = %error, "server shutdown stage failed"); + } tracing::warn!( - error_count = total_errors, - "crowdb-kv-server shut down with errors (see critical: logs above)" + error_count = report.errors.len(), + "crowdb-paxos-kv-server shut down with errors" ); } + member_routes.stop().await; + drop(metrics_runner); + registry.tree_runtime.stop_admission(); + // Drop stores and their Trees before the shared runtime and loggers. An + // incomplete accepted drain above never reaches this destruction point. + drop(registry); // Final flush + stop the C++ spdlog async loggers. All Crowdbtree // instances are now dropped (or about to be), so this is safe. diff --git a/app/crowdb-kv-server/src/mgmt.rs b/app/crowdb-paxos-kv-server/src/mgmt.rs similarity index 96% rename from app/crowdb-kv-server/src/mgmt.rs rename to app/crowdb-paxos-kv-server/src/mgmt.rs index 749a931cc..ed29e5d14 100644 --- a/app/crowdb-kv-server/src/mgmt.rs +++ b/app/crowdb-paxos-kv-server/src/mgmt.rs @@ -15,6 +15,7 @@ pub(crate) mod group_creation; mod group_ops; pub mod operation_registry; mod replica_ops; +mod runtime_shutdown; mod store_ops; mod system_bootstrap; pub use system_bootstrap::system_cleanup_pending; @@ -28,9 +29,9 @@ use axum::{Json, Router}; use serde::Serialize; use utoipa::{OpenApi, ToSchema}; -use crowdb_kv::cluster::group_config::GroupConfigStore; -use crowdb_kv::cluster::node_config::NodeConfigStore; -use crowdb_kv::cluster::status::{GroupStatus, RemoteStatus, ReplicaStatus, StoreStatus}; +use crowdb_paxos_kv::cluster::group_config::GroupConfigStore; +use crowdb_paxos_kv::cluster::node_config::NodeConfigStore; +use crowdb_paxos_kv::cluster::status::{GroupStatus, RemoteStatus, ReplicaStatus, StoreStatus}; use crowdb_protocol::mgmt::{ AddGroupRequest, AddStoreRequest, GroupSummary, HealthResponse, MetricField, MetricPoint, MetricsResponse, RemoteListResponse, RemoteReplicaInfo, StepDownRequest, StepDownResult, StoreDetail, @@ -125,7 +126,9 @@ pub async fn persisted_port_for_store(config_root: &std::path::Path, store_id: u } /// Extract the first non-zero port from a list of group members. -fn extract_port_from_members(members: &[crowdb_kv::cluster::group_config::PxGroupMember]) -> Option<u16> { +fn extract_port_from_members( + members: &[crowdb_paxos_kv::cluster::group_config::PxGroupMember], +) -> Option<u16> { members .iter() .find_map(|m| m.endpoint.rsplit(':').next().and_then(|p| p.parse::<u16>().ok())) diff --git a/app/crowdb-kv-server/src/mgmt/discovery.rs b/app/crowdb-paxos-kv-server/src/mgmt/discovery.rs similarity index 100% rename from app/crowdb-kv-server/src/mgmt/discovery.rs rename to app/crowdb-paxos-kv-server/src/mgmt/discovery.rs diff --git a/app/crowdb-paxos-kv-server/src/mgmt/group_creation.rs b/app/crowdb-paxos-kv-server/src/mgmt/group_creation.rs new file mode 100644 index 000000000..c44d69ce3 --- /dev/null +++ b/app/crowdb-paxos-kv-server/src/mgmt/group_creation.rs @@ -0,0 +1,249 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Durable creation ownership precedes ordinary-group WAL and engine effects. + +use std::{ + fs::{self, File, OpenOptions}, + io::{Read, Write}, + os::unix::fs::{OpenOptionsExt, PermissionsExt}, + path::{Path, PathBuf}, +}; + +use axum::{ + extract::{Path as AxumPath, State}, + http::StatusCode, + Json, +}; +use crowdb_protocol::mgmt::{AddGroupRequest, GroupCreationStatus}; + +use super::{err_json, system_bootstrap::ManagementError, RegistryArc}; + +pub(super) async fn status( + State(state): State<RegistryArc>, + AxumPath((store, group)): AxumPath<(u64, u64)>, +) -> Result<Json<GroupCreationStatus>, ManagementError> { + read(&state.config.config_root, store, group)? + .map(Json) + .ok_or_else(|| err_json(StatusCode::NOT_FOUND, "group creation ownership is absent")) +} + +pub(super) async fn accept( + state: &RegistryArc, + store: u64, + request: &AddGroupRequest, +) -> Result<(), ManagementError> { + if store == 0 && request.group_id == 0 { + return if request.deployment.is_none() { + Ok(()) + } else { + Err(err_json( + StatusCode::BAD_REQUEST, + "system group must use system bootstrap ownership", + )) + }; + } + let expected = GroupCreationStatus { + version: 1, + store_id: store, + group_id: request.group_id, + replica_id: request.replica_id, + deployment: request.deployment.clone(), + retired: false, + }; + validate(&expected)?; + let root = &state.config.config_root; + if let Some(current) = read(root, store, request.group_id)? { + return matches(¤t, &expected); + } + if state + .get_store(store) + .is_some_and(|store| store.get_group(request.group_id).is_some()) + { + return if request.deployment.is_none() { + Ok(()) + } else { + Err(err_json( + StatusCode::CONFLICT, + "existing group has no matching deployment ownership", + )) + }; + } + let persisted = crowdb_paxos_kv::cluster::node_config::NodeConfigStore::new(root) + .load() + .await + .map_err(storage_error)?; + if persisted + .group(store, request.group_id) + .is_some_and(|group| group.replica_id != request.replica_id) + { + return Err(err_json( + StatusCode::CONFLICT, + "persisted group replica identity differs", + )); + } + if request.deployment.is_some() + && (persisted.group(store, request.group_id).is_some() + || crate::recovery::startup::store_crowdb_tree_path( + &state.config.data_root, + store, + request.group_id, + ) + .exists() + || crate::recovery::startup::store_wal_root(&state.config.wal_root, store) + .join(format!("group{}", request.group_id)) + .exists()) + { + return Err(err_json( + StatusCode::CONFLICT, + "existing group has no matching deployment ownership", + )); + } + fs::create_dir_all(root).map_err(storage_error)?; + publish(root, &expected, false)?; + let current = read(root, store, request.group_id)? + .ok_or_else(|| err_json(StatusCode::CONFLICT, "group creation ownership disappeared"))?; + matches(¤t, &expected) +} + +pub(super) fn retire(state: &RegistryArc, store: u64, group: u64) -> Result<(), ManagementError> { + if let Some(mut current) = read(&state.config.config_root, store, group)? { + if !current.retired { + current.retired = true; + publish(&state.config.config_root, ¤t, true)?; + } + } + Ok(()) +} + +pub(super) fn finish_removal(state: &RegistryArc, store: u64, group: u64) -> Result<(), ManagementError> { + let root = &state.config.config_root; + if read(root, store, group)?.is_some_and(|current| current.deployment.is_none() && current.retired) { + fs::remove_file(path(root, store, group)).map_err(storage_error)?; + File::open(root) + .and_then(|file| file.sync_all()) + .map_err(storage_error)?; + } + Ok(()) +} + +pub(crate) fn check_restore(root: &Path, store: u64, group: u64, replica: u64) -> std::io::Result<()> { + let status = read(root, store, group).map_err(|_| { + std::io::Error::new( + std::io::ErrorKind::InvalidData, + "invalid durable group creation ownership", + ) + })?; + if status.is_some_and(|status| status.retired || status.replica_id != replica) { + return Err(std::io::Error::new( + std::io::ErrorKind::PermissionDenied, + "group creation is retired or has another replica identity", + )); + } + Ok(()) +} + +fn matches(current: &GroupCreationStatus, expected: &GroupCreationStatus) -> Result<(), ManagementError> { + if current != expected { + return Err(err_json( + StatusCode::CONFLICT, + "group creation is retired or owned by another operation", + )); + } + Ok(()) +} + +fn path(root: &Path, store: u64, group: u64) -> PathBuf { + root.join(format!("group-creation-{store}-{group}.json")) +} + +fn read(root: &Path, store: u64, group: u64) -> Result<Option<GroupCreationStatus>, ManagementError> { + let path = path(root, store, group); + let metadata = match fs::symlink_metadata(&path) { + Ok(metadata) => metadata, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(error) => return Err(storage_error(error)), + }; + if !metadata.file_type().is_file() + || metadata.len() > 4096 + || metadata.permissions().mode() & 0o777 != 0o600 + { + return Err(err_json( + StatusCode::CONFLICT, + "invalid durable group creation ownership", + )); + } + let mut bytes = Vec::new(); + File::open(path) + .map_err(storage_error)? + .take(4097) + .read_to_end(&mut bytes) + .map_err(storage_error)?; + if bytes.len() > 4096 { + return Err(err_json( + StatusCode::CONFLICT, + "group ownership exceeds its size bound", + )); + } + let status: GroupCreationStatus = serde_json::from_slice(&bytes) + .map_err(|_| err_json(StatusCode::CONFLICT, "invalid durable group creation ownership"))?; + validate(&status)?; + if status.store_id != store || status.group_id != group { + return Err(err_json( + StatusCode::CONFLICT, + "group creation ownership path differs", + )); + } + Ok(Some(status)) +} + +fn validate(status: &GroupCreationStatus) -> Result<(), ManagementError> { + let valid_identity = status.deployment.as_ref().map_or(true, |identity| { + [&identity.cluster_id, &identity.operation_id] + .iter() + .all(|value| { + uuid::Uuid::parse_str(value).is_ok_and(|id| !id.is_nil() && id.to_string() == value.as_str()) + }) + }); + if status.version != 1 || status.replica_id == 0 || !valid_identity { + return Err(err_json(StatusCode::CONFLICT, "invalid group creation identity")); + } + Ok(()) +} + +fn publish(root: &Path, status: &GroupCreationStatus, replace: bool) -> Result<(), ManagementError> { + let temporary = root.join(format!(".group-creation-{}.tmp", uuid::Uuid::new_v4())); + let result = (|| { + let mut file = OpenOptions::new() + .create_new(true) + .write(true) + .mode(0o600) + .open(&temporary)?; + file.write_all(&serde_json::to_vec(status).map_err(std::io::Error::other)?)?; + file.sync_all()?; + let target = path(root, status.store_id, status.group_id); + if replace { + fs::rename(&temporary, target)?; + } else { + match fs::hard_link(&temporary, target) { + Ok(()) => {} + Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => {} + Err(error) => return Err(error), + } + } + File::open(root)?.sync_all() + })(); + match fs::remove_file(temporary) { + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + Err(error) if result.is_ok() => return Err(storage_error(error)), + Ok(()) | Err(_) => {} + } + result.map_err(storage_error) +} + +fn storage_error(error: impl std::fmt::Display) -> ManagementError { + err_json( + StatusCode::INTERNAL_SERVER_ERROR, + format!("group ownership persistence failed: {error}"), + ) +} diff --git a/app/crowdb-kv-server/src/mgmt/group_ops.rs b/app/crowdb-paxos-kv-server/src/mgmt/group_ops.rs similarity index 95% rename from app/crowdb-kv-server/src/mgmt/group_ops.rs rename to app/crowdb-paxos-kv-server/src/mgmt/group_ops.rs index f0800c4b5..ad8f91db6 100644 --- a/app/crowdb-kv-server/src/mgmt/group_ops.rs +++ b/app/crowdb-paxos-kv-server/src/mgmt/group_ops.rs @@ -12,8 +12,8 @@ use serde::{Deserialize, Serialize}; use tracing::{debug, info, info_span, Instrument}; use utoipa::ToSchema; -use crowdb_kv::cluster::group_election::LeaderElection; -use crowdb_kv::cluster::local_replica::PxLocalReplicaRole; +use crowdb_paxos_kv::cluster::group_election::LeaderElection; +use crowdb_paxos_kv::cluster::local_replica::PxLocalReplicaRole; use crowdb_protocol::mgmt::{ AddGroupInitialRole, AddGroupRequest, GroupSummary, JoinGroupRequest, StepDownRequest, StepDownResult, }; @@ -235,7 +235,7 @@ pub(super) async fn add_group( AddGroupInitialRole::Leader => PxLocalReplicaRole::Leader, AddGroupInitialRole::Follower => PxLocalReplicaRole::Follower, }; - let group = crate::recovery::startup::create_group_with_wal( + let group = crate::recovery::startup::create_group_with_wal_on_runtime( sid, req.group_id, req.replica_id, @@ -243,6 +243,7 @@ pub(super) async fn add_group( &state.config, state.wal_backend.clone(), state.crowtree_backend, + state.tree_runtime.clone(), ) .await .map_err(|e| { @@ -384,7 +385,7 @@ pub(super) async fn join_snapshot( peer_endpoint = %req.peer_endpoint, "joining PxGroup via snapshot pull" ); - let group = crate::recovery::startup::create_group_with_wal( + let group = crate::recovery::startup::create_group_with_wal_on_runtime( sid, gid, req.replica_id, @@ -392,6 +393,7 @@ pub(super) async fn join_snapshot( &state.config, state.wal_backend.clone(), state.crowtree_backend, + state.tree_runtime.clone(), ) .await .map_err(|e| { @@ -423,7 +425,7 @@ pub(super) async fn join_snapshot( } }; // The frontier moved from 0 to `at_slot` (or further, if a concurrent - // catch-up already advanced it) after `create_group_with_wal` computed + // catch-up already advanced it) after `create_group_with_wal_on_runtime` computed // `next_slot` from a still-empty replica; recompute so a future // proposal (once this replica becomes leader-eligible) doesn't clash // with an already-applied slot. @@ -474,14 +476,17 @@ pub(super) async fn remove_group( .get_store(sid) .ok_or_else(|| err_json(StatusCode::NOT_FOUND, format!("store {sid} not found")))?; + let membership_guard = store.membership_guard(gid); + let _membership_guard = membership_guard.lock().await; + // Load the group only after serializing membership changes. Otherwise a + // request that waited behind a prior mutation could retain an old Arc and + // then remove or overwrite the newly published topology. let group = store.get_group(gid).ok_or_else(|| { err_json( StatusCode::NOT_FOUND, format!("group {gid} not found in store {sid}"), ) })?; - let membership_guard = store.membership_guard(gid); - let _membership_guard = membership_guard.lock().await; let expected_epoch = super::replica_ops::check_expected_epoch(&headers, group.membership_epoch())?; if expected_epoch == Some(group.membership_epoch()) { return Err(err_json( @@ -493,6 +498,7 @@ pub(super) async fn remove_group( )); } info!(s = sid, g = gid, "removing PxGroup via management API"); + super::runtime_shutdown::before_group_removal(&state, &group).await?; super::group_creation::retire(&state, sid, gid)?; if !store.remove_group(gid) { return Err(err_json( @@ -519,7 +525,8 @@ pub(super) async fn remove_group( } // Update node-config.json so the group does not resurrect on restart. - let node_config_store = crowdb_kv::cluster::node_config::NodeConfigStore::new(&state.config.config_root); + let node_config_store = + crowdb_paxos_kv::cluster::node_config::NodeConfigStore::new(&state.config.config_root); if let Err(e) = node_config_store.remove_group(sid, gid).await { tracing::warn!(s = sid, g = gid, error = %e, "failed to update node_config; continuing"); } @@ -643,7 +650,7 @@ pub(super) async fn group_readiness( if group.local_replica().voting { voting_count += 1; - if status.local_replica.status != crowdb_kv::cluster::status::StatusLevel::Unhealthy { + if status.local_replica.status == crowdb_paxos_kv::cluster::status::StatusLevel::Ok { reachable_count += 1; } } @@ -651,7 +658,7 @@ pub(super) async fn group_readiness( for remote in &status.remotes { if remote.voting { voting_count += 1; - if remote.status != crowdb_kv::cluster::status::StatusLevel::Unhealthy { + if remote.status == crowdb_paxos_kv::cluster::status::StatusLevel::Ok { reachable_count += 1; } } @@ -741,13 +748,12 @@ pub(super) async fn flush_group( ) })?; - // `KVEngine::flush` forces a freeze + drain of every memtable up to - // `contiguous_slot` into L1 (in-memory only; never touches the page - // store). Cheap when L0 is empty. Used by the bench's - // `--flush-after-prepopulate` flag to produce a clean L1-only scan - // baseline; also useful as an admin drain. - group.local_replica().learner.engine().flush(); - info!(s = sid, g = gid, "engine flush requested via management API"); + let outcome = crowdb_paxos_kv::kv::flush_engine(group.local_replica().learner.engine_arc()) + .await + .map_err(|error| err_json(StatusCode::SERVICE_UNAVAILABLE, error))?; + outcome + .status() + .map_err(|error| err_json(StatusCode::SERVICE_UNAVAILABLE, error.to_owned()))?; Ok(Json(FlushResult { store_id: sid, group_id: gid, @@ -764,7 +770,7 @@ pub(super) type WipeResult = crowdb_protocol::mgmt::WipeResult; /// group0 sysdata + store/group/replica topology. /// /// Flow: resolve → failed-WAL reject → unwired no-op → step down if -/// leader → delete on-disk WAL + engine dirs → `create_group_with_wal` +/// leader → delete on-disk WAL + engine dirs → `create_group_with_wal_on_runtime` /// (replays empty WAL → fresh state, restores remotes from /// node-config) → quorum-1 self-election → `store.add_group` (replaces /// old group, cancels old tenure, spawns fresh election driver). The @@ -829,6 +835,8 @@ pub(super) async fn wipe_user_data( })); } + super::runtime_shutdown::before_group_removal(&state, &group).await?; + // Step down if leader so the group stops accepting writes before // the data is destroyed. `accepted: false` (not leader) is fine. let step = group.step_down_if_leader("wipe-user-data"); @@ -848,7 +856,7 @@ pub(super) async fn wipe_user_data( // above, the old group's Arc refcount reaches zero (assuming no // in-flight tasks hold strong clones — they observe cancel within // one RPC timeout) and the old WalEngine + MemBlockDevice + engine - // are freed. Doing this BEFORE create_group_with_wal ensures the old + // are freed. Doing this BEFORE create_group_with_wal_on_runtime ensures the old // group's memory is released even if WAL replay is slow. store.remove_group(gid); @@ -856,16 +864,16 @@ pub(super) async fn wipe_user_data( // not tokio::fs::remove_dir_all: the mem-block backend stores // segments in-memory (BTreeMap<PathBuf, Vec<u8>>), and // tokio::fs::remove_dir_all is a no-op on it. Without this, the old - // WAL segments persist and create_group_with_wal replays them, + // WAL segments persist and create_group_with_wal_on_runtime replays them, // causing multi-second stalls and preventing the wipe from actually // wiping anything. wipe_wal_and_engine_dirs(&state, sid, gid).await?; - // Recreate the group with a fresh WAL + engine. `create_group_with_wal` + // Recreate the group with a fresh WAL + engine. `create_group_with_wal_on_runtime` // replays the now-empty WAL (→ slot 0), creates fresh WalEngine + // engine, and `maybe_apply_persisted_config` restores the remote // membership from node-config.json (which the wipe did not touch). - let new_group = crate::recovery::startup::create_group_with_wal( + let new_group = crate::recovery::startup::create_group_with_wal_on_runtime( sid, gid, replica_id, @@ -873,6 +881,7 @@ pub(super) async fn wipe_user_data( &state.config, state.wal_backend.clone(), state.crowtree_backend, + state.tree_runtime.clone(), ) .await .map_err(|e| { diff --git a/app/crowdb-kv-server/src/mgmt/operation_registry.rs b/app/crowdb-paxos-kv-server/src/mgmt/operation_registry.rs similarity index 80% rename from app/crowdb-kv-server/src/mgmt/operation_registry.rs rename to app/crowdb-paxos-kv-server/src/mgmt/operation_registry.rs index a21aa8254..6778a03f3 100644 --- a/app/crowdb-kv-server/src/mgmt/operation_registry.rs +++ b/app/crowdb-paxos-kv-server/src/mgmt/operation_registry.rs @@ -153,6 +153,26 @@ impl OperationRegistry { true }); } + + /// Start the process-local cleanup loop for completed operations. + /// + /// Pending and running operations are retained across cleanup ticks. The + /// returned task is owned by the server and must be aborted during + /// shutdown before the registry is dropped. + pub(crate) fn start_cleanup( + self: &Arc<Self>, + interval: Duration, + ttl: Duration, + ) -> tokio::task::JoinHandle<()> { + let registry = Arc::clone(self); + tokio::spawn(async move { + let mut ticker = tokio::time::interval(interval); + loop { + ticker.tick().await; + registry.cleanup_expired(ttl); + } + }) + } } impl Default for OperationRegistry { @@ -169,16 +189,16 @@ fn now_ms() -> u64 { /// Wrapper for shared state passed to HTTP handlers — bundles the store /// registry and the operation registry. Implements `Deref` to -/// `KvStoreRegistry` so existing handlers that call `state.stores`, +/// `PaxosKvStoreRegistry` so existing handlers that call `state.stores`, /// `state.get_store()`, etc. work unchanged. #[derive(Clone)] pub struct AppState { - pub(crate) registry: Arc<crate::store_registry::KvStoreRegistry>, + pub(crate) registry: Arc<crate::store_registry::PaxosKvStoreRegistry>, pub(crate) operations: Arc<OperationRegistry>, } impl std::ops::Deref for AppState { - type Target = crate::store_registry::KvStoreRegistry; + type Target = crate::store_registry::PaxosKvStoreRegistry; fn deref(&self) -> &Self::Target { &self.registry } @@ -186,19 +206,28 @@ impl std::ops::Deref for AppState { impl AppState { #[must_use] - pub fn new(registry: Arc<crate::store_registry::KvStoreRegistry>) -> Self { + pub fn new(registry: Arc<crate::store_registry::PaxosKvStoreRegistry>) -> Self { Self { registry, operations: Arc::new(OperationRegistry::new()), } } + /// Start cleanup for completed management operations. Operation IDs are + /// process-local and expire after the retention window; callers receive + /// 404 once a completed record has been collected. + #[must_use] + pub fn start_operation_cleanup(&self) -> tokio::task::JoinHandle<()> { + self.operations + .start_cleanup(Duration::from_secs(300), Duration::from_secs(3600)) + } + /// Construct with an explicit operation registry (for testing or when /// the registry must be shared across routers). #[must_use] #[allow(dead_code)] pub(crate) fn with_operations( - registry: Arc<crate::store_registry::KvStoreRegistry>, + registry: Arc<crate::store_registry::PaxosKvStoreRegistry>, operations: Arc<OperationRegistry>, ) -> Self { Self { registry, operations } diff --git a/app/crowdb-kv-server/src/mgmt/replica_ops.rs b/app/crowdb-paxos-kv-server/src/mgmt/replica_ops.rs similarity index 96% rename from app/crowdb-kv-server/src/mgmt/replica_ops.rs rename to app/crowdb-paxos-kv-server/src/mgmt/replica_ops.rs index b1cc1bd6b..333a4f06a 100644 --- a/app/crowdb-kv-server/src/mgmt/replica_ops.rs +++ b/app/crowdb-paxos-kv-server/src/mgmt/replica_ops.rs @@ -8,9 +8,9 @@ use axum::http::{HeaderMap, StatusCode}; use axum::Json; use tracing::{debug, info}; -use crowdb_kv::cluster::group_election::LeaderElection; -use crowdb_kv::cluster::kv_server::KvServer; -use crowdb_kv::cluster::remote_replica::PxRemoteReplica; +use crowdb_paxos_kv::cluster::group_election::LeaderElection; +use crowdb_paxos_kv::cluster::paxos_kv_server::PaxosKvServer; +use crowdb_paxos_kv::cluster::remote_replica::PxRemoteReplica; use crowdb_protocol::mgmt::{RemoteListResponse, RemoteReplicaInfo, TopologyResponse}; use super::{err_json, ErrorResponse, RegistryArc}; @@ -79,14 +79,18 @@ pub(super) async fn add_remote_replicas( let store = state .get_store(sid) .ok_or_else(|| err_json(StatusCode::NOT_FOUND, format!("store {sid} not found")))?; + let membership_guard = store.membership_guard(gid); + let _membership_guard = membership_guard.lock().await; + // Resolve the group after taking the membership guard. A concurrent + // membership mutation may have published a replacement while this + // request was waiting; continuing with the earlier Arc would apply the + // update to stale topology and can overwrite the newer membership. let group = store.get_group(gid).ok_or_else(|| { err_json( StatusCode::NOT_FOUND, format!("group {gid} not found in store {sid}"), ) })?; - let membership_guard = store.membership_guard(gid); - let _membership_guard = membership_guard.lock().await; let expected_epoch = check_expected_epoch(&headers, group.membership_epoch())?; let local_id = group.local_replica().id; @@ -186,14 +190,14 @@ pub(super) async fn remove_remote_replica( let store = state .get_store(sid) .ok_or_else(|| err_json(StatusCode::NOT_FOUND, format!("store {sid} not found")))?; + let membership_guard = store.membership_guard(gid); + let _membership_guard = membership_guard.lock().await; let group = store.get_group(gid).ok_or_else(|| { err_json( StatusCode::NOT_FOUND, format!("group {gid} not found in store {sid}"), ) })?; - let membership_guard = store.membership_guard(gid); - let _membership_guard = membership_guard.lock().await; let expected_epoch = check_expected_epoch(&headers, group.membership_epoch())?; let local_id = group.local_replica().id; @@ -306,14 +310,14 @@ pub(super) async fn batch_add_remote_replicas( let store = state .get_store(sid) .ok_or_else(|| err_json(StatusCode::NOT_FOUND, format!("store {sid} not found")))?; + let membership_guard = store.membership_guard(gid); + let _membership_guard = membership_guard.lock().await; let group = store.get_group(gid).ok_or_else(|| { err_json( StatusCode::NOT_FOUND, format!("group {gid} not found in store {sid}"), ) })?; - let membership_guard = store.membership_guard(gid); - let _membership_guard = membership_guard.lock().await; let expected_epoch = check_expected_epoch(&headers, group.membership_epoch())?; let local_id = group.local_replica().id; @@ -433,7 +437,10 @@ pub(super) fn check_expected_epoch( Ok(Some(expected)) } -fn remotes_match_existing(group: &crowdb_kv::cluster::group::PxGroup, remotes: &[RemoteReplicaInfo]) -> bool { +fn remotes_match_existing( + group: &crowdb_paxos_kv::cluster::group::PxGroup, + remotes: &[RemoteReplicaInfo], +) -> bool { let existing = group.remote_replica_info(); remotes.iter().all(|requested| { existing.iter().any(|(id, endpoint, voting)| { diff --git a/app/crowdb-paxos-kv-server/src/mgmt/runtime_shutdown.rs b/app/crowdb-paxos-kv-server/src/mgmt/runtime_shutdown.rs new file mode 100644 index 000000000..f34c91922 --- /dev/null +++ b/app/crowdb-paxos-kv-server/src/mgmt/runtime_shutdown.rs @@ -0,0 +1,28 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +use super::{err_json, ErrorResponse, RegistryArc}; +use axum::{http::StatusCode, Json}; +use crowdb_paxos_kv::cluster::group::PxGroup; +use std::time::Duration; + +pub(super) async fn before_group_removal( + state: &RegistryArc, + group: &PxGroup, +) -> Result<(), (StatusCode, Json<ErrorResponse>)> { + if !group.local_replica().learner.engine().requires_contiguous_apply() { + return Ok(()); + } + let report = group + .shutdown(Duration::from_millis(state.config.server.shutdown_timeout_ms)) + .await; + if report.is_clean() { + return Ok(()); + } + Err(err_json( + StatusCode::SERVICE_UNAVAILABLE, + format!( + "runtime group retained; shutdown incomplete: {}", + report.errors.join("; ") + ), + )) +} diff --git a/app/crowdb-kv-server/src/mgmt/store_ops.rs b/app/crowdb-paxos-kv-server/src/mgmt/store_ops.rs similarity index 90% rename from app/crowdb-kv-server/src/mgmt/store_ops.rs rename to app/crowdb-paxos-kv-server/src/mgmt/store_ops.rs index ec6d89446..be48c8c7d 100644 --- a/app/crowdb-kv-server/src/mgmt/store_ops.rs +++ b/app/crowdb-paxos-kv-server/src/mgmt/store_ops.rs @@ -11,9 +11,8 @@ use axum::http::StatusCode; use axum::Json; use tracing::{debug, info}; -use crowdb_kv::cluster::kv_server::KvServer; -use crowdb_kv::cluster::px_kv_store::PxKvStore; -use crowdb_kv::common::config::ServerConfig; +use crowdb_paxos_kv::cluster::paxos_kv_server::PaxosKvServer; +use crowdb_paxos_kv::cluster::px_kv_store::PxKvStore; use crowdb_protocol::mgmt::{AddStoreRequest, StoreDetail, StoreListResponse, StoreSummary}; use super::{err_json, ErrorResponse, RegistryArc}; @@ -184,11 +183,15 @@ pub(super) async fn remove_store( ) -> Result<StatusCode, (StatusCode, Json<ErrorResponse>)> { info!(s = sid, "removing PxKvStore via management API"); let store = state - .remove_store(sid) + .get_store(sid) .ok_or_else(|| err_json(StatusCode::NOT_FOUND, format!("store {sid} not found")))?; + let runtime = store.has_runtime_engine(); + if !runtime { + let _ = state.remove_store(sid); + } let report = store .shutdown(std::time::Duration::from_millis( - ServerConfig::DEFAULT.shutdown_timeout_ms, + state.config.server.shutdown_timeout_ms, )) .await; if !report.is_clean() { @@ -197,6 +200,19 @@ pub(super) async fn remove_store( } } + if runtime { + if !report.is_clean() { + return Err(err_json( + StatusCode::SERVICE_UNAVAILABLE, + format!( + "runtime store retained; shutdown incomplete: {}", + report.errors.join("; ") + ), + )); + } + let _ = state.remove_store(sid); + } + // Delete the engine store dir (cascades all group subdirs). let engine_store_dir = state.config.data_root.join(format!("store{sid}")); if let Err(e) = tokio::fs::remove_dir_all(&engine_store_dir).await { @@ -214,7 +230,8 @@ pub(super) async fn remove_store( } // Update node-config.json so the store does not resurrect on restart. - let node_config_store = crowdb_kv::cluster::node_config::NodeConfigStore::new(&state.config.config_root); + let node_config_store = + crowdb_paxos_kv::cluster::node_config::NodeConfigStore::new(&state.config.config_root); if let Err(e) = node_config_store.remove_store(sid).await { tracing::warn!(s = sid, error = %e, "failed to update node_config; continuing"); } diff --git a/app/crowdb-paxos-kv-server/src/mgmt/system_bootstrap.rs b/app/crowdb-paxos-kv-server/src/mgmt/system_bootstrap.rs new file mode 100644 index 000000000..383426002 --- /dev/null +++ b/app/crowdb-paxos-kv-server/src/mgmt/system_bootstrap.rs @@ -0,0 +1,228 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Durable system-store ownership, published before any store creation. + +use std::fs::{self, File, OpenOptions}; +use std::io::{Read, Write}; +use std::os::unix::fs::{OpenOptionsExt, PermissionsExt}; +use std::path::Path; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::Arc; + +use axum::{extract::State, http::StatusCode, Json}; +use crowdb_protocol::mgmt::{SystemBootstrapIdentity, SystemPrepareRequest}; +use serde::{Deserialize, Serialize}; + +use super::{err_json, ErrorResponse, RegistryArc}; + +pub(super) type ManagementError = (StatusCode, Json<ErrorResponse>); +const OWNERSHIP_FILE: &str = "system-bootstrap.json"; +const RETIRED_FILE: &str = "system-bootstrap-cleanup.json"; +static NEXT_TEMP: AtomicU64 = AtomicU64::new(0); + +#[derive(Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct AcceptedBootstrap { + version: u32, + replica_id: u64, + identity: SystemBootstrapIdentity, +} + +pub(super) struct BootstrapExecution(Arc<crate::store_registry::PaxosKvStoreRegistry>); + +impl Drop for BootstrapExecution { + fn drop(&mut self) { + self.0.bootstrap_executing.store(false, Ordering::Release); + } +} + +pub(super) fn begin(state: &RegistryArc) -> Result<BootstrapExecution, ManagementError> { + state + .bootstrap_executing + .compare_exchange(false, true, Ordering::Acquire, Ordering::Relaxed) + .map_err(|_| { + err_json( + StatusCode::CONFLICT, + "node group initialization is executing; retry the same operation", + ) + })?; + Ok(BootstrapExecution(Arc::clone(&state.registry))) +} + +pub(super) async fn prepare( + State(state): State<RegistryArc>, + Json(req): Json<SystemPrepareRequest>, +) -> Result<Json<SystemPrepareRequest>, ManagementError> { + let _execution = begin(&state)?; + accept(&state, req.replica_id, Some(&req.bootstrap))?; + Ok(Json(req)) +} + +pub(super) fn has_owner(state: &RegistryArc) -> Result<bool, ManagementError> { + match fs::symlink_metadata(state.config.config_root.join(OWNERSHIP_FILE)) { + Ok(_) => Ok(true), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(false), + Err(error) => Err(storage_error(&error)), + } +} + +pub(super) fn accept( + state: &RegistryArc, + replica_id: u64, + identity: Option<&SystemBootstrapIdentity>, +) -> Result<(), ManagementError> { + let root = &state.config.config_root; + let path = root.join(OWNERSHIP_FILE); + if root.join(RETIRED_FILE).exists() { + return Err(err_json(StatusCode::CONFLICT, "system cleanup has not completed")); + } + let Some(identity) = identity else { + return if has_owner(state)? { + Err(err_json( + StatusCode::CONFLICT, + "system store belongs to an identified bootstrap operation", + )) + } else { + Ok(()) + }; + }; + validate(replica_id, identity)?; + if root + .join(format!("system-bootstrap-retired-{}.json", identity.operation_id)) + .exists() + { + return Err(err_json( + StatusCode::CONFLICT, + "bootstrap operation was explicitly retired", + )); + } + if root.join(RETIRED_FILE).exists() { + return Err(err_json(StatusCode::CONFLICT, "system cleanup has not completed")); + } + let accepted = AcceptedBootstrap { + version: 1, + replica_id, + identity: identity.clone(), + }; + if has_owner(state)? { + return verify(&path, &accepted); + } + if state.contains_store(0) { + return Err(err_json( + StatusCode::CONFLICT, + "existing system store has no matching bootstrap ownership", + )); + } + fs::create_dir_all(root).map_err(|error| storage_error(&error))?; + let temporary = root.join(format!( + ".system-bootstrap-{}-{}.tmp", + std::process::id(), + NEXT_TEMP.fetch_add(1, Ordering::Relaxed) + )); + let result = (|| { + let mut file = OpenOptions::new() + .write(true) + .create_new(true) + .mode(0o600) + .open(&temporary) + .map_err(|error| storage_error(&error))?; + let bytes = + serde_json::to_vec(&accepted).map_err(|error| storage_error(&std::io::Error::other(error)))?; + file.write_all(&bytes).map_err(|error| storage_error(&error))?; + file.sync_all().map_err(|error| storage_error(&error))?; + match fs::hard_link(&temporary, &path) { + Ok(()) => {} + Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => {} + Err(error) => return Err(storage_error(&error)), + } + File::open(root) + .and_then(|directory| directory.sync_all()) + .map_err(|error| storage_error(&error))?; + verify(&path, &accepted) + })(); + let cleanup = fs::remove_file(&temporary); + if result.is_ok() { + cleanup.map_err(|error| storage_error(&error))?; + } + result +} + +fn validate(replica_id: u64, identity: &SystemBootstrapIdentity) -> Result<(), ManagementError> { + fn uuid(text: &str) -> bool { + text.len() == 36 + && text.bytes().enumerate().all(|(index, byte)| { + if [8, 13, 18, 23].contains(&index) { + byte == b'-' + } else { + byte.is_ascii_hexdigit() + } + }) + && text + .bytes() + .any(|byte| matches!(byte, b'1'..=b'9' | b'a'..=b'f' | b'A'..=b'F')) + } + if replica_id == 0 + || !uuid(&identity.cluster_id) + || !uuid(&identity.operation_id) + || identity.configuration_digest.len() != 64 + || !identity + .configuration_digest + .bytes() + .all(|byte| byte.is_ascii_hexdigit()) + { + return Err(err_json( + StatusCode::BAD_REQUEST, + "bootstrap requires nonzero replica, cluster/operation UUIDs and SHA-256 configuration digest", + )); + } + Ok(()) +} + +fn verify(path: &Path, expected: &AcceptedBootstrap) -> Result<(), ManagementError> { + let metadata = fs::symlink_metadata(path).map_err(|error| storage_error(&error))?; + if !metadata.file_type().is_file() + || metadata.len() > 4096 + || metadata.permissions().mode() & 0o777 != 0o600 + { + return Err(err_json( + StatusCode::CONFLICT, + "invalid durable system bootstrap ownership", + )); + } + let mut bytes = Vec::with_capacity(4096); + File::open(path) + .map_err(|error| storage_error(&error))? + .take(4097) + .read_to_end(&mut bytes) + .map_err(|error| storage_error(&error))?; + let actual: AcceptedBootstrap = serde_json::from_slice(&bytes) + .map_err(|_| err_json(StatusCode::CONFLICT, "invalid durable system bootstrap ownership"))?; + if &actual != expected { + return Err(err_json( + StatusCode::CONFLICT, + "system store is reserved by a conflicting bootstrap operation", + )); + } + Ok(()) +} + +fn storage_error(error: &std::io::Error) -> ManagementError { + err_json( + StatusCode::INTERNAL_SERVER_ERROR, + format!("system bootstrap persistence failed: {error}"), + ) +} + +mod cleanup; +pub(super) use cleanup::cleanup; + +/// # Errors +/// Reports unreadable cleanup markers instead of restarting a retired system store. +pub fn system_cleanup_pending(root: &Path) -> std::io::Result<bool> { + match fs::symlink_metadata(root.join(RETIRED_FILE)) { + Ok(_) => Ok(true), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(false), + Err(error) => Err(error), + } +} diff --git a/app/crowdb-kv-server/src/mgmt/system_bootstrap/cleanup.rs b/app/crowdb-paxos-kv-server/src/mgmt/system_bootstrap/cleanup.rs similarity index 98% rename from app/crowdb-kv-server/src/mgmt/system_bootstrap/cleanup.rs rename to app/crowdb-paxos-kv-server/src/mgmt/system_bootstrap/cleanup.rs index 0f1fc9ed9..028f37273 100644 --- a/app/crowdb-kv-server/src/mgmt/system_bootstrap/cleanup.rs +++ b/app/crowdb-paxos-kv-server/src/mgmt/system_bootstrap/cleanup.rs @@ -86,7 +86,7 @@ pub(crate) async fn cleanup( if state.contains_store(0) { super::super::store_ops::remove_store(State(state.clone()), axum::extract::Path(0)).await?; } - let node_config = crowdb_kv::cluster::node_config::NodeConfigStore::new(root); + let node_config = crowdb_paxos_kv::cluster::node_config::NodeConfigStore::new(root); node_config .remove_store(0) .await diff --git a/app/crowdb-kv-server/src/mgmt/system_init.rs b/app/crowdb-paxos-kv-server/src/mgmt/system_init.rs similarity index 94% rename from app/crowdb-kv-server/src/mgmt/system_init.rs rename to app/crowdb-paxos-kv-server/src/mgmt/system_init.rs index 7ab6e60fe..c3b979652 100644 --- a/app/crowdb-kv-server/src/mgmt/system_init.rs +++ b/app/crowdb-paxos-kv-server/src/mgmt/system_init.rs @@ -8,10 +8,10 @@ use axum::http::StatusCode; use axum::Json; use tracing::info; -use crowdb_kv::cluster::group_election::LeaderElection; -use crowdb_kv::cluster::kv_server::KvServer; -use crowdb_kv::cluster::local_replica::PxLocalReplicaRole; -use crowdb_kv::cluster::status::{StatusLevel, StoreStatus}; +use crowdb_paxos_kv::cluster::group_election::LeaderElection; +use crowdb_paxos_kv::cluster::local_replica::PxLocalReplicaRole; +use crowdb_paxos_kv::cluster::paxos_kv_server::PaxosKvServer; +use crowdb_paxos_kv::cluster::status::{StatusLevel, StoreStatus}; use crowdb_protocol::mgmt::{HealthResponse, SystemInitRequest, SystemInitResponse}; use super::{err_json, ErrorResponse, RegistryArc}; @@ -122,7 +122,7 @@ pub(super) async fn system_init( )); } - let group = crate::recovery::startup::create_group_with_wal( + let group = crate::recovery::startup::create_group_with_wal_on_runtime( SYSTEM_STORE_ID, SYSTEM_GROUP_ID, req.replica_id, @@ -134,6 +134,7 @@ pub(super) async fn system_init( &state.config, state.wal_backend.clone(), state.crowtree_backend, + state.tree_runtime.clone(), ) .await .map_err(|e| { diff --git a/app/crowdb-kv-server/src/mgmt/system_join.rs b/app/crowdb-paxos-kv-server/src/mgmt/system_join.rs similarity index 100% rename from app/crowdb-kv-server/src/mgmt/system_join.rs rename to app/crowdb-paxos-kv-server/src/mgmt/system_join.rs diff --git a/app/crowdb-kv-server/src/mgmt/system_store.rs b/app/crowdb-paxos-kv-server/src/mgmt/system_store.rs similarity index 94% rename from app/crowdb-kv-server/src/mgmt/system_store.rs rename to app/crowdb-paxos-kv-server/src/mgmt/system_store.rs index 2ae1e5e47..8d76e169f 100644 --- a/app/crowdb-kv-server/src/mgmt/system_store.rs +++ b/app/crowdb-paxos-kv-server/src/mgmt/system_store.rs @@ -5,8 +5,8 @@ use super::{err_json, system_bootstrap::ManagementError, RegistryArc}; use axum::http::StatusCode; -use crowdb_kv::cluster::kv_server::KvServer; -use crowdb_kv::cluster::px_kv_store::PxKvStore; +use crowdb_paxos_kv::cluster::paxos_kv_server::PaxosKvServer; +use crowdb_paxos_kv::cluster::px_kv_store::PxKvStore; use std::{net::SocketAddr, sync::Arc}; use tracing::info; diff --git a/app/crowdb-paxos-kv-server/src/mgmt/topology.rs b/app/crowdb-paxos-kv-server/src/mgmt/topology.rs new file mode 100644 index 000000000..f97a1e509 --- /dev/null +++ b/app/crowdb-paxos-kv-server/src/mgmt/topology.rs @@ -0,0 +1,145 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +//! Topology export and metrics endpoints. + +use axum::extract::{Query, State}; +use axum::response::IntoResponse; +use serde::Deserialize; +use utoipa::ToSchema; + +use crowdb_paxos_kv::cluster::status::StoreStatus; +use crowdb_protocol::mgmt::{MetricField, MetricPoint, MetricsResponse, TopologyResponse}; + +use super::RegistryArc; + +/// `GET /topology` (alias `/top`) — full hierarchy with per-remote RPC +/// metrics and cheap kv-store stats. +#[utoipa::path( + get, + path = "/topology", + tag = "management", + responses((status = 200, description = "Cluster topology status", body = TopologyResponse)) + )] +pub(super) async fn export_topology(State(state): State<RegistryArc>) -> impl IntoResponse { + let stores: Vec<StoreStatus> = state + .stores_snapshot() + .values() + .map(|store| store.status()) + .collect(); + let body = serde_json::to_string_pretty(&TopologyResponse { stores }).unwrap(); + ([("content-type", "application/json")], body) +} + +/// Query params for `GET /metrics`. +#[derive(Deserialize, ToSchema)] +pub(crate) struct MetricsQuery { + /// Metric name prefix filter (e.g. `s.1.g.2.`). Default empty = all. + #[serde(default)] + prefix: String, +} + +/// `GET /metrics` — structured snapshot of all registry metrics matching +/// the `prefix` query param. Does not reset window state. Intended for +/// the GUI Inspector and script/scrape consumers. +#[utoipa::path( + get, + path = "/metrics", + tag = "management", + params(("prefix" = Option<String>, Query, description = "Metric name prefix filter")), + responses((status = 200, description = "Metric snapshot", body = MetricsResponse)) + )] +pub(super) async fn metrics( + State(state): State<RegistryArc>, + Query(q): Query<MetricsQuery>, +) -> impl IntoResponse { + let timestamp = crowdb_paxos_kv::metrics::iso8601_now(); + let window_secs = 5.0; // approximate — snapshot path does not track elapsed + let metrics: Vec<MetricPoint> = state + .metrics_registry + .as_ref() + .map(|reg| { + let reg = reg.lock().unwrap(); + reg.snapshot_struct(&q.prefix, window_secs) + .iter() + .map(metric_point_to_dto) + .collect() + }) + .unwrap_or_default(); + let body = serde_json::to_string_pretty(&MetricsResponse { + window_secs, + timestamp, + metrics, + }) + .unwrap(); + ([("content-type", "application/json")], body) +} + +#[allow(clippy::cast_precision_loss)] +fn metric_point_to_dto(p: &crowdb_paxos_kv::metrics::MetricPoint) -> MetricPoint { + use crowdb_paxos_kv::metrics::MetricPoint as KvMetricPoint; + let kind = p.kind().to_string(); + let fields = match p { + KvMetricPoint::Counter { + count, tps, total, .. + } => vec![("count", *count as f64), ("tps", *tps), ("total", *total as f64)], + KvMetricPoint::Gauge { value, .. } => vec![("value", *value as f64)], + KvMetricPoint::Bandwidth { + count, + avg_size, + rate, + total_bytes, + .. + } => vec![ + ("count", *count as f64), + ("avg_size", *avg_size as f64), + ("rate", *rate as f64), + ("total_bytes", *total_bytes as f64), + ], + KvMetricPoint::Histogram { + count, + avg_ns, + p50_ns, + p99_ns, + max_ns, + total, + .. + } => vec![ + ("count", *count as f64), + ("avg_ns", *avg_ns), + ("p50_ns", *p50_ns as f64), + ("p99_ns", *p99_ns as f64), + ("max_ns", *max_ns as f64), + ("total", *total as f64), + ], + KvMetricPoint::Summary { + count, + avg_ns, + max_ns, + total, + .. + } => vec![ + ("count", *count as f64), + ("avg_ns", *avg_ns as f64), + ("max_ns", *max_ns as f64), + ("total", *total as f64), + ], + }; + MetricPoint { + name: match p { + KvMetricPoint::Counter { name, .. } + | KvMetricPoint::Gauge { name, .. } + | KvMetricPoint::Bandwidth { name, .. } + | KvMetricPoint::Histogram { name, .. } + | KvMetricPoint::Summary { name, .. } => name.clone(), + }, + kind, + fields: fields + .into_iter() + .map(|(key, value)| MetricField { + key: key.to_string(), + value, + }) + .collect(), + } +} diff --git a/app/crowdb-kv-server/src/recovery.rs b/app/crowdb-paxos-kv-server/src/recovery.rs similarity index 100% rename from app/crowdb-kv-server/src/recovery.rs rename to app/crowdb-paxos-kv-server/src/recovery.rs diff --git a/app/crowdb-kv-server/src/recovery/group_rebuild.rs b/app/crowdb-paxos-kv-server/src/recovery/group_rebuild.rs similarity index 94% rename from app/crowdb-kv-server/src/recovery/group_rebuild.rs rename to app/crowdb-paxos-kv-server/src/recovery/group_rebuild.rs index 3e3f9dc4e..01487bc73 100644 --- a/app/crowdb-kv-server/src/recovery/group_rebuild.rs +++ b/app/crowdb-paxos-kv-server/src/recovery/group_rebuild.rs @@ -13,10 +13,10 @@ //! `PxLocalReplica::new_inheriting_election_state` so a rebuild does //! not reset the cluster's election term and trigger a fresh race. -use crowdb_kv::cluster::group::PxGroup; -use crowdb_kv::cluster::group_election::LeaderElection; -use crowdb_kv::cluster::local_replica::PxLocalReplica; -use crowdb_kv::cluster::remote_replica::PxRemoteReplica; +use crowdb_paxos_kv::cluster::group::PxGroup; +use crowdb_paxos_kv::cluster::group_election::LeaderElection; +use crowdb_paxos_kv::cluster::local_replica::PxLocalReplica; +use crowdb_paxos_kv::cluster::remote_replica::PxRemoteReplica; /// Rebuild `group` with `new_remotes` merged into its remote list, /// applying the membership-epoch bump correctly. diff --git a/app/crowdb-kv-server/src/recovery/member_routes.rs b/app/crowdb-paxos-kv-server/src/recovery/member_routes.rs similarity index 94% rename from app/crowdb-kv-server/src/recovery/member_routes.rs rename to app/crowdb-paxos-kv-server/src/recovery/member_routes.rs index d4fab9d4a..3cec2b2f8 100644 --- a/app/crowdb-kv-server/src/recovery/member_routes.rs +++ b/app/crowdb-paxos-kv-server/src/recovery/member_routes.rs @@ -3,8 +3,8 @@ //! Monitor-owned endpoint evidence changes connections, never membership. -use crate::store_registry::KvStoreRegistry; -use crowdb_kv::cluster::group::PxGroup; +use crate::store_registry::PaxosKvStoreRegistry; +use crowdb_paxos_kv::cluster::group::PxGroup; use crowdb_protocol::mgmt::{ node::recovery::{MemberEndpoint, MemberEndpoints, RecoveryGroups}, GroupCreationStatus, SystemBootstrapIdentity, @@ -181,6 +181,14 @@ fn resolve( } pub struct Worker(tokio::task::JoinHandle<()>); +impl Worker { + /// Stop endpoint updates and release the registry after accepted work has + /// drained. The host awaits this before dropping Trees and native loggers. + pub async fn stop(mut self) { + self.0.abort(); + let _ = (&mut self.0).await; + } +} impl Drop for Worker { fn drop(&mut self) { self.0.abort(); @@ -188,7 +196,7 @@ impl Drop for Worker { } #[must_use] -pub fn start(registry: Arc<KvStoreRegistry>) -> Worker { +pub fn start(registry: Arc<PaxosKvStoreRegistry>) -> Worker { Worker(tokio::spawn(async move { let mut tick = tokio::time::interval(Duration::from_secs(1)); loop { diff --git a/app/crowdb-kv-server/src/recovery/reconcile.rs b/app/crowdb-paxos-kv-server/src/recovery/reconcile.rs similarity index 94% rename from app/crowdb-kv-server/src/recovery/reconcile.rs rename to app/crowdb-paxos-kv-server/src/recovery/reconcile.rs index 6e55c37df..9ab290e5c 100644 --- a/app/crowdb-kv-server/src/recovery/reconcile.rs +++ b/app/crowdb-paxos-kv-server/src/recovery/reconcile.rs @@ -24,16 +24,16 @@ use std::collections::HashMap; use tracing::{info, warn}; -use crowdb_kv::cluster::kv_store::KvStore; +use crowdb_paxos_kv::cluster::kv_store::KvStore; use super::group_rebuild::rebuild_group_with_new_remotes; -use crate::store_registry::KvStoreRegistry; +use crate::store_registry::PaxosKvStoreRegistry; /// Peers for one group, keyed by `(store_id, group_id)`. type PeersByGroup = HashMap<(u64, u64), Vec<(u64, String, bool)>>; /// Run reconciliation against group 0 if it is locally available. -pub async fn reconcile_with_group0(registry: &KvStoreRegistry) { +pub async fn reconcile_with_group0(registry: &PaxosKvStoreRegistry) { let Some(store0) = registry.get_store(0) else { info!("reconcile: store 0 not found locally; skipping"); return; @@ -115,7 +115,7 @@ pub struct ReconcileAction { /// Pure decision logic: given decoded group-0 replica records and the /// local registry, compute what to do for each local group. No side /// effects — the caller executes the plan via [`execute_reconcile`]. -pub fn plan_reconcile(records: &[ReplicaRecord], registry: &KvStoreRegistry) -> Vec<ReconcileAction> { +pub fn plan_reconcile(records: &[ReplicaRecord], registry: &PaxosKvStoreRegistry) -> Vec<ReconcileAction> { // Group records by (store_id, group_id) -> Vec<(replica_id, endpoint, voting)>. let mut by_group: PeersByGroup = HashMap::new(); for r in records { @@ -176,7 +176,7 @@ pub fn plan_reconcile(records: &[ReplicaRecord], registry: &KvStoreRegistry) -> /// Execute a reconcile plan: rebuild groups for seed actions, log warn /// for mismatch actions. -fn execute_reconcile(plan: &[ReconcileAction], registry: &KvStoreRegistry) { +fn execute_reconcile(plan: &[ReconcileAction], registry: &PaxosKvStoreRegistry) { let mut seeded = 0usize; let mut mismatches = 0usize; for action in plan { @@ -220,7 +220,7 @@ fn execute_reconcile(plan: &[ReconcileAction], registry: &KvStoreRegistry) { /// Prefix-scan group 0 for complete membership records. If no authoritative /// records exist yet, retain the old projection scan for pre-R229 groups. async fn scan_authoritative_records( - store0: &crowdb_kv::cluster::px_kv_store::PxKvStore, + store0: &crowdb_paxos_kv::cluster::px_kv_store::PxKvStore, ) -> Option<Vec<ReplicaRecord>> { let response = store0 .kv_scan( @@ -236,7 +236,7 @@ async fn scan_authoritative_records( 0, false, 0, - crowdb_kv::kv::ScanDirection::Forward, + crowdb_paxos_kv::kv::ScanDirection::Forward, 0, 0, ) @@ -278,7 +278,7 @@ async fn scan_authoritative_records( /// unreachable / not yet led); returns `Some(vec)` (possibly empty) on /// success. async fn scan_replica_records( - store0: &crowdb_kv::cluster::px_kv_store::PxKvStore, + store0: &crowdb_paxos_kv::cluster::px_kv_store::PxKvStore, ) -> Option<Vec<ReplicaRecord>> { let prefix = b"/kv/replica/"; let resp = store0 @@ -295,7 +295,7 @@ async fn scan_replica_records( 0, false, 0, - crowdb_kv::kv::ScanDirection::Forward, + crowdb_paxos_kv::kv::ScanDirection::Forward, 0, 0, ) diff --git a/app/crowdb-kv-server/src/recovery/restore.rs b/app/crowdb-paxos-kv-server/src/recovery/restore.rs similarity index 78% rename from app/crowdb-kv-server/src/recovery/restore.rs rename to app/crowdb-paxos-kv-server/src/recovery/restore.rs index a1e6f1971..76cbead13 100644 --- a/app/crowdb-kv-server/src/recovery/restore.rs +++ b/app/crowdb-paxos-kv-server/src/recovery/restore.rs @@ -6,7 +6,7 @@ //! When group 0 is present on disk (`<wal_root>/store0/group0`), the //! server boots in restore mode: it scans `<wal_root>` for every //! `store{S}/group{G}` directory, loads each via -//! [`crate::recovery::startup::create_group_with_wal`] (which replays the WAL, +//! [`crate::recovery::startup::create_group_with_wal_on_runtime`] (which replays the WAL, //! opens the crowdb-tree engine, and applies the persisted membership //! from `node-config.json` — including remote-replica endpoints), and //! starts the stores. No `--stores`/`--groups` CLI args are needed; @@ -20,16 +20,16 @@ use std::net::SocketAddr; use std::path::Path; use std::sync::Arc; -use tracing::{debug, info, warn}; +use tracing::{debug, info}; -use crowdb_kv::cluster::kv_server::KvServer; -use crowdb_kv::cluster::local_replica::PxLocalReplicaRole; -use crowdb_kv::cluster::node_config::NodeConfigStore; -use crowdb_kv::cluster::px_kv_store::PxKvStore; +use crowdb_paxos_kv::cluster::local_replica::PxLocalReplicaRole; +use crowdb_paxos_kv::cluster::node_config::NodeConfigStore; +use crowdb_paxos_kv::cluster::paxos_kv_server::PaxosKvServer; +use crowdb_paxos_kv::cluster::px_kv_store::PxKvStore; -use super::startup::create_group_with_wal; +use super::startup::create_group_with_wal_on_runtime; use crate::mgmt::persisted_port_for_store; -use crate::store_registry::KvStoreRegistry; +use crate::store_registry::PaxosKvStoreRegistry; /// A local `(store_id, group_id)` pair discovered by scanning `waldata`. #[derive(Clone, Copy, Debug, Eq, PartialEq)] @@ -90,18 +90,24 @@ pub fn group0_exists(wal_root: &Path) -> bool { /// /// Groups `local_groups` by `store_id`, creates each `PxKvStore` /// (port from the `--ports` pool, else `persisted_port_for_store`, -/// else OS-assigned 0), then calls [`create_group_with_wal`] per group +/// else OS-assigned 0), then calls [`create_group_with_wal_on_runtime`] per group /// with [`PxLocalReplicaRole::Follower`] (the election driver /// promotes), `add_group`, `store.start()`, and registers the store. -/// Reuses `create_and_start_stores`' skip-on-error policy: a failed -/// group is logged and skipped, the store still starts with its other -/// groups. +/// Restore is all-or-fail for the scanned local state: a failed group or +/// store start returns an error before the partially restored store is +/// published to the registry. This prevents a partial local restore from +/// being reported as a healthy server. /// /// The local `replica_id` for each group is read from `node-config.json` /// (where it was persisted when the group was first created). If the /// entry is missing (e.g. first boot after migrate, or stale config), /// falls back to the `replica_id` CLI arg. /// +/// # Errors +/// Returns an error when a persisted group identity cannot be read, a group +/// cannot be restored, or a store cannot be started. No partially restored +/// store is published. +/// /// # Panics /// Panics if the computed bind address `"0.0.0.0:{port}"` fails to /// parse as a `SocketAddr` (only possible if `port > 65535`, which @@ -109,8 +115,8 @@ pub fn group0_exists(wal_root: &Path) -> bool { pub async fn load_local_groups( local_groups: &[LocalGroup], replica_id: u64, - registry: &Arc<KvStoreRegistry>, -) { + registry: &Arc<PaxosKvStoreRegistry>, +) -> Result<(), String> { // Group by store_id, preserving scan (sorted) order. let mut by_store: Vec<(u64, Vec<u64>)> = Vec::new(); for lg in local_groups { @@ -154,6 +160,12 @@ pub async fn load_local_groups( // fall back to the CLI arg if not found. let effective_replica_id = persisted_replica_id(®istry.config.config_root, store_id, group_id) .await + .map_err(|error| { + format!( + "restore: failed to read persisted replica identity for store {store_id} \ + group {group_id}: {error}" + ) + })? .unwrap_or(replica_id); debug!( s = store_id, @@ -161,7 +173,7 @@ pub async fn load_local_groups( replica = effective_replica_id, "restore: creating group from disk" ); - let group = match create_group_with_wal( + let group = match create_group_with_wal_on_runtime( store_id, group_id, effective_replica_id, @@ -169,26 +181,24 @@ pub async fn load_local_groups( ®istry.config, registry.wal_backend.clone(), registry.crowtree_backend, + registry.tree_runtime.clone(), ) .await { Ok(group) => group, - Err(e) => { - warn!( - store_id, - group_id, - error = %e, - "restore: failed to load group from disk; skipping" - ); - continue; + Err(error) => { + return Err(format!( + "restore: failed to load store {store_id} group {group_id} from disk: {error}" + )); } }; store.add_group(group); } if let Err(e) = store.start().await { - warn!(s = store_id, port, error = %e, "restore: failed to start store; skipping"); - continue; + return Err(format!( + "restore: failed to start store {store_id} on port {port}: {e}" + )); } info!( @@ -199,6 +209,8 @@ pub async fn load_local_groups( ); registry.add_store(store_id, &store); } + + Ok(()) } /// Parse `store{S}` → `S`. Returns `None` if the name does not match. @@ -213,8 +225,12 @@ fn parse_group_dir(name: &str) -> Option<u64> { /// Read the persisted `replica_id` for `(store_id, group_id)` from /// `node-config.json`. Returns `None` if the file or entry is missing. -async fn persisted_replica_id(config_root: &Path, store_id: u64, group_id: u64) -> Option<u64> { +/// +/// A present but unreadable or invalid file is an error. Falling back to the +/// CLI replica id in that case could restore a WAL with the wrong identity and +/// make a damaged node appear healthy. +async fn persisted_replica_id(config_root: &Path, store_id: u64, group_id: u64) -> io::Result<Option<u64>> { let store = NodeConfigStore::new(config_root); - let config = store.load().await.ok()?; - config.group(store_id, group_id).map(|g| g.replica_id) + let config = store.load().await?; + Ok(config.group(store_id, group_id).map(|g| g.replica_id)) } diff --git a/app/crowdb-paxos-kv-server/src/recovery/startup.rs b/app/crowdb-paxos-kv-server/src/recovery/startup.rs new file mode 100644 index 000000000..7729fd201 --- /dev/null +++ b/app/crowdb-paxos-kv-server/src/recovery/startup.rs @@ -0,0 +1,291 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use std::io; +use std::path::{Path, PathBuf}; +use std::sync::Arc; + +use tracing::info; + +use crowdb_paxos_kv::cluster::group::PxGroup; +use crowdb_paxos_kv::cluster::group_config::{GroupConfigStore, PxGroupConfig, PxGroupMember}; +use crowdb_paxos_kv::cluster::group_election::LeaderElection; +use crowdb_paxos_kv::cluster::local_replica::{PxLocalReplica, PxLocalReplicaRole}; +use crowdb_paxos_kv::cluster::node_config::NodeConfigStore; +use crowdb_paxos_kv::common::config::PaxosKvConfig; +use crowdb_paxos_kv::kv::{CrowdbTreeBackend, CrowdbTreeConfig, CrowdbTreeEngine, KVEngine}; +use crowdb_paxos_kv::wal::replay::replay_group; +use crowdb_paxos_kv::wal::{IoBackend, WalEngine}; + +/// Load persisted group config and apply it to the group. +/// +/// Reads from `node-config.json` (the per-node config cache). If the +/// group entry is present, the group is seeded with the durable +/// membership so it does not start as a `quorum=1` singleton in the +/// restore window. The node config store is also set on the group so +/// future `persist_config` calls write to the same file. +async fn maybe_apply_persisted_config( + group: &mut PxGroup, + config_root: &Path, + store_id: u64, +) -> io::Result<()> { + let node_store = NodeConfigStore::new(config_root); + match node_store.load_group(store_id, group.group_id()).await { + Ok(Some(config)) => { + if config.group_id == group.group_id() { + group.apply_config(&config); + } + } + Ok(None) => { + // Fall back to legacy per-group config file for migration. + let legacy_store = GroupConfigStore::new(config_root, store_id, group.group_id()); + match legacy_store.load().await { + Ok(Some(config)) => { + if config.group_id == group.group_id() { + group.apply_config(&config); + } + } + Ok(None) => {} + Err(e) => { + return Err(e); + } + } + } + Err(e) => { + return Err(e); + } + } + group.set_node_config_store(node_store, store_id, group.group_id()); + Ok(()) +} + +async fn persist_creation_identity( + config_root: &Path, + store_id: u64, + group_id: u64, + replica_id: u64, +) -> io::Result<()> { + let store = NodeConfigStore::new(config_root); + if let Some(existing) = store.load().await?.group(store_id, group_id) { + if existing.replica_id != replica_id { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "requested replica identity conflicts with persisted group", + )); + } + return Ok(()); + } + let seed = GroupConfigStore::new(config_root, store_id, group_id) + .load() + .await? + .unwrap_or_else(|| PxGroupConfig { + group_id, + members: vec![PxGroupMember { + replica_id, + endpoint: String::new(), + voting: true, + }], + ..PxGroupConfig::default() + }); + // A WAL directory makes the group discoverable after a crash. Persist its + // local identity first, so restore cannot substitute the CLI default. + store.save_group(store_id, &seed, replica_id).await +} + +#[must_use] +pub fn store_wal_root(wal_root: &Path, store_id: u64) -> PathBuf { + wal_root.join(format!("store{store_id}")) +} + +/// Durable per-group crowdb-tree directory path: `{data_root}/store{store_id}/group{group_id}`. +/// Both `TextPageStore` and `BlockPageStore` expect a directory path (`TextPageStore` creates a subdirectory +/// `{path}/{store_id}-{group_id}/`, `BlockPageStore` creates `.blk-*` files +/// directly in `path`). +#[must_use] +pub fn store_crowdb_tree_path(data_root: &Path, store_id: u64, group_id: u64) -> PathBuf { + data_root + .join(format!("store{store_id}")) + .join(format!("group{group_id}")) +} + +/// Open (creating on first boot) the durable crowdb-tree engine backing +/// `(store_id, group_id)`'s learner, boxed for [`PxLearner::with_engine`] +/// via [`PxLocalReplica::restore_from_replay_with_engine`]. +/// +/// # Errors +/// +/// Returns an I/O error if the parent directory cannot be created, or if +/// `CrowdbTreeEngine::open` fails (e.g. a corrupt or unreadable file). +async fn open_crowdb_tree_engine( + data_root: &Path, + store_id: u64, + group_id: u64, + backend: CrowdbTreeBackend, + config: &PaxosKvConfig, + runtime: Arc<crowdb_tree_ffi::TreeRuntime>, +) -> io::Result<Box<dyn KVEngine>> { + let path = store_crowdb_tree_path(data_root, store_id, group_id); + if let Some(parent) = path.parent() { + tokio::fs::create_dir_all(parent).await?; + } + tokio::fs::create_dir_all(&path).await?; + let mut opt = CrowdbTreeConfig { + path: Some(path.to_string_lossy().into_owned()), + backend, + store_id: u32::try_from(store_id).unwrap_or(0), + group_id: u32::try_from(group_id).unwrap_or(0), + log_dir: config.log_dir.clone(), + runtime: Some(runtime), + runtime_binding: config.tree_runtime.binding_options(), + log_file_prefix: "crowdb-paxos-kv-server-tree".to_string(), + ..Default::default() + }; + config.tree.apply_to(&mut opt); + info!( + store_id, + group_id, + backend = ?backend, + path = %path.display(), + "opening crowdb-tree engine" + ); + // Open waits synchronously for recovery on the shared native runtime. + // This boot-time call runs once per group; subsequent reads, apply and + // maintenance use native asynchronous operation futures. + let engine = CrowdbTreeEngine::open(&opt).map_err(|e| { + io::Error::other(format!( + "CrowdbTreeEngine::open({}) failed: {e:?}", + path.display() + )) + })?; + Ok(Box::new(engine)) +} + +/// Create a live group by replaying any existing WAL, restoring the local +/// replica state, attaching a fresh `WalEngine`, and seeding the next proposal +/// slot / segment id using a standalone host. Production groups share the +/// registry host through `create_group_with_wal_on_runtime`. +/// +/// # Errors +/// +/// Returns any I/O or replay/restore error encountered while scanning the +/// existing WAL, creating the new WAL engine, opening the durable crowdb-tree +/// engine, or rebuilding the local replica. +#[allow(clippy::too_many_arguments)] +pub async fn create_group_with_wal( + store_id: u64, + group_id: u64, + replica_id: u64, + initial_role: PxLocalReplicaRole, + config: &PaxosKvConfig, + wal_backend: Arc<IoBackend>, + crowtree_backend: CrowdbTreeBackend, +) -> io::Result<PxGroup> { + config + .validate_tree_config() + .map_err(|error| io::Error::new(io::ErrorKind::InvalidInput, error))?; + let runtime = Arc::new( + crowdb_tree_ffi::TreeRuntime::new(config.tree_runtime.runtime_options()) + .map_err(|error| io::Error::other(format!("standalone Tree runtime: {error}")))?, + ); + create_group_with_wal_on_runtime( + store_id, + group_id, + replica_id, + initial_role, + config, + wal_backend, + crowtree_backend, + runtime, + ) + .await +} + +/// Create or restore a group with its host-owned shared runtime. +/// # Errors +/// Returns an error for invalid limits, backend recovery failures or failed +/// accepted-log application. +#[allow(clippy::too_many_arguments)] +pub async fn create_group_with_wal_on_runtime( + store_id: u64, + group_id: u64, + replica_id: u64, + initial_role: PxLocalReplicaRole, + config: &PaxosKvConfig, + wal_backend: Arc<IoBackend>, + crowtree_backend: CrowdbTreeBackend, + runtime: Arc<crowdb_tree_ffi::TreeRuntime>, +) -> io::Result<PxGroup> { + config + .validate_tree_config() + .map_err(|error| io::Error::new(io::ErrorKind::InvalidInput, error))?; + crate::mgmt::group_creation::check_restore(&config.config_root, store_id, group_id, replica_id)?; + persist_creation_identity(&config.config_root, store_id, group_id, replica_id).await?; + // Preserve the typed WAL tunables from the effective server config while + // deriving the per-store root. Rebuilding a fresh `WalConfig` here would + // silently discard segment, flush, retention, and alignment settings. + let mut wal_config = config.wal.clone(); + wal_config.set_root(store_wal_root(&config.wal_root, store_id)); + if std::env::var("CROWDB_PAXOS_KV_WAL_TEXT").as_deref() == Ok("1") { + wal_config.wal_record_format = crowdb_paxos_kv::wal::WalRecordFormat::TextLine; + } + wal_config.wal_skip_fsync = config.wal_skip_fsync; + let mut replay = replay_group(&wal_backend, &wal_config.wal_disks, group_id).await?; + let wal = WalEngine::create_from_replay( + wal_backend, + wal_config, + group_id, + replay.max_segment_id, + std::mem::take(&mut replay.index), + ) + .await?; + + let mut local_replica = { + let engine = open_crowdb_tree_engine( + &config.data_root, + store_id, + group_id, + crowtree_backend, + config, + runtime, + ) + .await?; + wal.set_snapshot_slot(engine.resume_from_slot()); + PxLocalReplica::restore_from_replay_with_engine(replica_id, initial_role, &replay, engine).await? + }; + local_replica.set_wal(wal); + + let mut group = PxGroup::new(group_id, local_replica); + maybe_apply_persisted_config(&mut group, &config.config_root, store_id).await?; + // If the caller initialized the group as Leader, the proposal leadership + // gate (current_term == proposing_term) will not open until the term is + // stamped. The election driver handles this on a real win, but groups + // created/restored as Leader (e.g., single-replica management API) may + // serve proposals before the driver runs, so stamp it synchronously here. + if initial_role == PxLocalReplicaRole::Leader { + let term = group.local_replica().current_term_snapshot(); + group.stamp_proposing_term(term); + } + group.set_from_config(config); + info!( + store_id, + group_id, + max_inflight = config.max_inflight(), + reject_on_window_full = config.reject_on_inflight_window_full(), + coalesce_max_keys = config.paxos.coalesce_max_keys, + skip_fsync = config.wal_skip_fsync, + wal_early_ack = config.wal_early_ack, + "group created with config" + ); + let next_slot = group + .local_replica() + .highest_seen_slot() + .max(group.local_replica().last_chosen_slot()) + .max(group.local_replica().contiguous_applied()) + .saturating_add(1) + .max(1); + group.set_next_slot(next_slot); + if let Err(error) = super::member_routes::apply(&group, &config.config_root, store_id) { + tracing::debug!(store_id, group_id, %error, "restored member routes are blocked pending verification"); + } + Ok(group) +} diff --git a/app/crowdb-paxos-kv-server/src/shutdown.rs b/app/crowdb-paxos-kv-server/src/shutdown.rs new file mode 100644 index 000000000..5affb9731 --- /dev/null +++ b/app/crowdb-paxos-kv-server/src/shutdown.rs @@ -0,0 +1,47 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +//! Host-wide shutdown keeps every consensus listener alive until every +//! accepted runtime prefix has drained, including peers hosted in this process. +use crate::store_registry::PaxosKvStoreRegistry; +use crowdb_paxos_kv::common::OperationReport; +use std::time::Duration; + +/// Perform one bounded shutdown attempt. An incomplete accepted drain returns +/// an error before any store is torn down, so callers may repair and retry. +pub async fn shutdown_stores_once(registry: &PaxosKvStoreRegistry) -> OperationReport { + let stores = registry.stores_snapshot(); + let timeout = Duration::from_millis(registry.config.server.shutdown_timeout_ms); + for store in stores.values() { + store.close_runtime_proposals(); + } + let mut report = OperationReport::new(); + for store in stores.values() { + report.merge(store.prepare_runtime_shutdown(timeout).await); + } + if !report.is_clean() { + return report; + } + for store in stores.values() { + report.merge(store.shutdown(timeout).await); + } + report +} + +/// Complete host shutdown. A timed-out accepted prefix is reported while the +/// process retains its communication/repair owners and awaits real progress. +/// Backend checkpoint failures remain explicit terminal errors in the result. +pub async fn shutdown_stores(registry: &PaxosKvStoreRegistry) -> OperationReport { + loop { + let report = shutdown_stores_once(registry).await; + let stores = registry.stores_snapshot(); + if !stores.values().any(|store| store.runtime_shutdown_pending()) { + return report; + } + for error in &report.errors { + tracing::error!(error = %error, "shutdown incomplete; consensus remains live; next step: restore peer progress to drain accepted work"); + } + for store in stores.values() { + store.await_runtime_shutdown_ready().await; + } + } +} diff --git a/app/crowdb-kv-server/src/store_registry.rs b/app/crowdb-paxos-kv-server/src/store_registry.rs similarity index 86% rename from app/crowdb-kv-server/src/store_registry.rs rename to app/crowdb-paxos-kv-server/src/store_registry.rs index 1c4ab0d3d..dc130a395 100644 --- a/app/crowdb-kv-server/src/store_registry.rs +++ b/app/crowdb-paxos-kv-server/src/store_registry.rs @@ -2,11 +2,11 @@ // Licensed under the Apache License, Version 2.0. use arc_swap::ArcSwap; -use crowdb_kv::cluster::px_kv_store::PxKvStore; -use crowdb_kv::common::config::CrowDBConfig; -use crowdb_kv::kv::CrowdbTreeBackend; -use crowdb_kv::metrics::MetricsRegistry; -use crowdb_kv::wal::IoBackend; +use crowdb_paxos_kv::cluster::px_kv_store::PxKvStore; +use crowdb_paxos_kv::common::config::PaxosKvConfig; +use crowdb_paxos_kv::kv::CrowdbTreeBackend; +use crowdb_paxos_kv::metrics::MetricsRegistry; +use crowdb_paxos_kv::wal::IoBackend; use std::collections::HashMap; use std::sync::Arc; use std::sync::Mutex; @@ -35,15 +35,17 @@ pub(crate) fn parse_crowtree_backend(s: &str) -> CrowdbTreeBackend { } } -pub struct KvStoreRegistry { +pub struct PaxosKvStoreRegistry { stores: ArcSwap<HashMap<u64, Arc<PxKvStore>>>, pub(crate) bootstrap_executing: std::sync::atomic::AtomicBool, /// Unified cluster configuration (all sub-configs + flags + paths). - pub config: CrowDBConfig, + pub config: PaxosKvConfig, /// Parsed WAL I/O backend (derived from `config.wal_backend`). pub wal_backend: Arc<IoBackend>, /// Parsed crowdb-tree storage backend (derived from `config.crowtree_backend`). pub crowtree_backend: CrowdbTreeBackend, + /// Shared by normal startup, restore, management creation and recreation. + pub tree_runtime: Arc<crowdb_tree_ffi::TreeRuntime>, /// Port pool for KV server listeners, populated from `--ports` CLI arg. /// Used by `add_store` as a fallback before `persisted_port_for_store`. port_pool: Mutex<Vec<u16>>, @@ -54,16 +56,16 @@ pub struct KvStoreRegistry { pub rpc_workers: u32, } -impl Default for KvStoreRegistry { +impl Default for PaxosKvStoreRegistry { fn default() -> Self { Self::new() } } -impl KvStoreRegistry { +impl PaxosKvStoreRegistry { #[must_use] pub(crate) fn new() -> Self { - Self::with_config(CrowDBConfig::default()) + Self::with_config(PaxosKvConfig::default()) } /// Construct a registry from an already validated configuration. @@ -72,7 +74,7 @@ impl KvStoreRegistry { /// Panics when the configured backend cannot initialize. Startup code /// should use [`Self::try_with_config`] to return an actionable error. #[must_use] - pub fn with_config(config: CrowDBConfig) -> Self { + pub fn with_config(config: PaxosKvConfig) -> Self { Self::try_with_config(config).expect("default WAL backend must initialize") } @@ -81,7 +83,14 @@ impl KvStoreRegistry { /// # Errors /// Returns an actionable initialization error when `uring` was explicitly /// selected but liburing or the running kernel cannot create a ring. - pub fn try_with_config(config: CrowDBConfig) -> std::io::Result<Self> { + pub fn try_with_config(config: PaxosKvConfig) -> std::io::Result<Self> { + config + .validate_tree_config() + .map_err(|error| std::io::Error::new(std::io::ErrorKind::InvalidInput, error))?; + let tree_runtime = Arc::new( + crowdb_tree_ffi::TreeRuntime::new(config.tree_runtime.runtime_options()) + .map_err(|error| std::io::Error::other(format!("requested Tree runtime: {error}")))?, + ); let wal_backend = Arc::new(parse_wal_backend(&config.wal_backend).map_err(|error| { std::io::Error::new( error.kind(), @@ -95,6 +104,7 @@ impl KvStoreRegistry { bootstrap_executing: std::sync::atomic::AtomicBool::new(false), wal_backend, crowtree_backend, + tree_runtime, config, port_pool: Mutex::new(Vec::new()), metrics_registry: None, diff --git a/app/crowdb-kv-server/tests/chunk_dynamic_monitor_test.rs b/app/crowdb-paxos-kv-server/tests/chunk_dynamic_monitor_test.rs similarity index 95% rename from app/crowdb-kv-server/tests/chunk_dynamic_monitor_test.rs rename to app/crowdb-paxos-kv-server/tests/chunk_dynamic_monitor_test.rs index af0f2b310..89998c2b1 100644 --- a/app/crowdb-kv-server/tests/chunk_dynamic_monitor_test.rs +++ b/app/crowdb-paxos-kv-server/tests/chunk_dynamic_monitor_test.rs @@ -4,7 +4,7 @@ #[path = "common/dynamic_slots.rs"] mod dynamic_slots; use bytes::Bytes; -use crowdb_kv_server::background::domain_monitor::{ChunkdbDynamicMonitorDriver, DomainMonitorDriver}; +use crowdb_paxos_kv_server::background::domain_monitor::{ChunkdbDynamicMonitorDriver, DomainMonitorDriver}; use crowdb_protocol::key::{ChunkSlotMapHeadKey, TextKey}; use dynamic_slots::{control, descriptor, entry, initialize, TestDynamicOwners}; @@ -137,8 +137,8 @@ async fn corrupt_dynamic_snapshot_is_rejected_without_overwriting_it() { #[tokio::test] async fn replacement_group_zero_tenure_resumes_and_obsolete_control_cannot_publish() { - use crowdb_kv::cluster::group_election::LeaderElection; - use crowdb_kv_server::group0_control_plane::Group0ControlPlane; + use crowdb_paxos_kv::cluster::group_election::LeaderElection; + use crowdb_paxos_kv_server::group0_control_plane::Group0ControlPlane; let (store, control) = control().await; initialize(&control).await; for owner in 1..=4 { diff --git a/app/crowdb-kv-server/tests/chunk_kv_balance_planning_test.rs b/app/crowdb-paxos-kv-server/tests/chunk_kv_balance_planning_test.rs similarity index 97% rename from app/crowdb-kv-server/tests/chunk_kv_balance_planning_test.rs rename to app/crowdb-paxos-kv-server/tests/chunk_kv_balance_planning_test.rs index 9f9503ab8..74dffbd6c 100644 --- a/app/crowdb-kv-server/tests/chunk_kv_balance_planning_test.rs +++ b/app/crowdb-paxos-kv-server/tests/chunk_kv_balance_planning_test.rs @@ -11,9 +11,9 @@ mod committed_transfer; mod recovering_owner; use bytes::Bytes; -use crowdb_kv::cluster::{group::PxGroup, PxKvStore, PxLocalReplica, PxLocalReplicaRole}; -use crowdb_kv_server::background::domain_monitor::{ChunkKvRangeMonitorDriver, DomainMonitorDriver}; -use crowdb_kv_server::group0_control_plane::Group0ControlPlane; +use crowdb_paxos_kv::cluster::{group::PxGroup, PxKvStore, PxLocalReplica, PxLocalReplicaRole}; +use crowdb_paxos_kv_server::background::domain_monitor::{ChunkKvRangeMonitorDriver, DomainMonitorDriver}; +use crowdb_paxos_kv_server::group0_control_plane::Group0ControlPlane; use crowdb_protocol::chunk_kv::{ ChunkKvRangeBalancePolicy, DomainFailurePolicy, DomainMonitorDescriptor, Id128, TransferTransition, }; diff --git a/app/crowdb-kv-server/tests/chunk_kv_balance_planning_test/committed_transfer.rs b/app/crowdb-paxos-kv-server/tests/chunk_kv_balance_planning_test/committed_transfer.rs similarity index 100% rename from app/crowdb-kv-server/tests/chunk_kv_balance_planning_test/committed_transfer.rs rename to app/crowdb-paxos-kv-server/tests/chunk_kv_balance_planning_test/committed_transfer.rs diff --git a/app/crowdb-kv-server/tests/chunk_kv_balance_planning_test/recovering_owner.rs b/app/crowdb-paxos-kv-server/tests/chunk_kv_balance_planning_test/recovering_owner.rs similarity index 99% rename from app/crowdb-kv-server/tests/chunk_kv_balance_planning_test/recovering_owner.rs rename to app/crowdb-paxos-kv-server/tests/chunk_kv_balance_planning_test/recovering_owner.rs index 3fb2585a2..6d55907b3 100644 --- a/app/crowdb-kv-server/tests/chunk_kv_balance_planning_test/recovering_owner.rs +++ b/app/crowdb-paxos-kv-server/tests/chunk_kv_balance_planning_test/recovering_owner.rs @@ -45,6 +45,7 @@ async fn renewing_recovery_process_is_not_dead_when_discovery_is_stale() { prior_exclusion_deadline_ms: now - 1, grant_sequence: 0, grant_expires_at_ms: 0, + revoked: false, }; lease.validate().unwrap(); for expiry in [now + 60_000, now - 1] { diff --git a/app/crowdb-kv-server/tests/chunk_slot_monitor_test.rs b/app/crowdb-paxos-kv-server/tests/chunk_slot_monitor_test.rs similarity index 95% rename from app/crowdb-kv-server/tests/chunk_slot_monitor_test.rs rename to app/crowdb-paxos-kv-server/tests/chunk_slot_monitor_test.rs index 0f8b122f7..4d0131306 100644 --- a/app/crowdb-kv-server/tests/chunk_slot_monitor_test.rs +++ b/app/crowdb-paxos-kv-server/tests/chunk_slot_monitor_test.rs @@ -4,10 +4,10 @@ use std::sync::Arc; use bytes::Bytes; -use crowdb_kv::cluster::group::PxGroup; -use crowdb_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; -use crowdb_kv_server::background::domain_monitor::{ChunkdbRangeMonitorDriver, DomainMonitorDriver}; -use crowdb_kv_server::group0_control_plane::Group0ControlPlane; +use crowdb_paxos_kv::cluster::group::PxGroup; +use crowdb_paxos_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; +use crowdb_paxos_kv_server::background::domain_monitor::{ChunkdbRangeMonitorDriver, DomainMonitorDriver}; +use crowdb_paxos_kv_server::group0_control_plane::Group0ControlPlane; use crowdb_protocol::chunk_kv::{DomainFailurePolicy, DomainMonitorDescriptor}; use crowdb_protocol::chunk_slot::{ChunkSlot, ChunkSlotBinding, ChunkSlotMapHead, ChunkStorageGroup}; use crowdb_protocol::common::InstanceValue; diff --git a/app/crowdb-kv-server/tests/cli_parse_test.rs b/app/crowdb-paxos-kv-server/tests/cli_parse_test.rs similarity index 85% rename from app/crowdb-kv-server/tests/cli_parse_test.rs rename to app/crowdb-paxos-kv-server/tests/cli_parse_test.rs index 711ad4d8b..38c13c7f3 100644 --- a/app/crowdb-kv-server/tests/cli_parse_test.rs +++ b/app/crowdb-paxos-kv-server/tests/cli_parse_test.rs @@ -2,13 +2,13 @@ // Licensed under the Apache License, Version 2.0. //! CLI argument parsing tests. Migrated from the inline `#[cfg(test)] mod -//! tests` in `crowdb-kv-server/src/cli.rs` per `.windsurf/workflows/coding.md` +//! tests` in `crowdb-paxos-kv-server/src/cli.rs` per `.windsurf/workflows/coding.md` //! §3 ("integration tests only — do not add new inline test modules"). use clap::Parser; -use crowdb_kv::common::config::CrowDBConfig; -use crowdb_kv_server::cli::{parse_id_list, parse_port_list, Cli}; -use crowdb_kv_server::store_registry::KvStoreRegistry; +use crowdb_paxos_kv::common::config::PaxosKvConfig; +use crowdb_paxos_kv_server::cli::{parse_id_list, parse_port_list, Cli}; +use crowdb_paxos_kv_server::store_registry::PaxosKvStoreRegistry; #[test] fn parse_single() { @@ -65,7 +65,7 @@ fn parse_port_rejects_zero() { #[test] fn parse_root_cli_option() { let cli = Cli::parse_from([ - "crowdb-kv-server", + "crowdb-paxos-kv-server", "--root", "/data/N-1", "--management-port", @@ -78,7 +78,7 @@ fn parse_root_cli_option() { assert_eq!(cli.kv_backend, "block"); assert_eq!(cli.wal_backend, "block-device"); assert!(!cli.no_fsync); - let defaults = CrowDBConfig::default(); + let defaults = PaxosKvConfig::default(); assert_eq!(defaults.crowtree_backend, cli.kv_backend); assert_eq!(defaults.wal_backend, cli.wal_backend); assert!(!defaults.wal_skip_fsync); @@ -87,7 +87,7 @@ fn parse_root_cli_option() { #[test] fn parse_root_is_required() { // Omitting --root should fail (clap rejects the invocation). - let result = Cli::try_parse_from(["crowdb-kv-server", "--management-port", "10000"]); + let result = Cli::try_parse_from(["crowdb-paxos-kv-server", "--management-port", "10000"]); assert!(result.is_err()); } @@ -96,7 +96,7 @@ fn loaded_configuration_uses_durable_block_defaults() { let root = crowdb_test_harness::test_dirs::tempdir_in_test_data("block-default-config"); let path = root.path().join("server.toml"); std::fs::write(&path, "").unwrap(); - let loaded = CrowDBConfig::load_from_file(&path).unwrap(); + let loaded = PaxosKvConfig::load_from_file(&path).unwrap(); assert_eq!(loaded.crowtree_backend, "block"); assert_eq!(loaded.wal_backend, "block-device"); assert!(!loaded.wal_skip_fsync); @@ -105,7 +105,7 @@ fn loaded_configuration_uses_durable_block_defaults() { #[test] fn management_port_rejects_zero() { let result = Cli::try_parse_from([ - "crowdb-kv-server", + "crowdb-paxos-kv-server", "--root", ".crowdb-runtime/ephemeral/cli-parse/n1", "--management-port", @@ -117,7 +117,7 @@ fn management_port_rejects_zero() { #[test] fn wal_backend_accepts_explicit_uring() { let cli = Cli::parse_from([ - "crowdb-kv-server", + "crowdb-paxos-kv-server", "--root", ".crowdb-runtime/ephemeral/cli-parse/n1", "--wal-backend", @@ -125,7 +125,7 @@ fn wal_backend_accepts_explicit_uring() { ]); assert_eq!(cli.wal_backend, "uring"); let invalid = Cli::try_parse_from([ - "crowdb-kv-server", + "crowdb-paxos-kv-server", "--root", ".crowdb-runtime/ephemeral/cli-parse/n1", "--wal-backend", @@ -136,9 +136,9 @@ fn wal_backend_accepts_explicit_uring() { #[test] fn explicit_uring_initialization_never_silently_falls_back() { - let mut config = CrowDBConfig::for_tests(); + let mut config = PaxosKvConfig::for_tests(); config.wal_backend = "uring".to_owned(); - match KvStoreRegistry::try_with_config(config) { + match PaxosKvStoreRegistry::try_with_config(config) { Ok(registry) => assert_eq!(format!("{:?}", registry.wal_backend), "Uring"), Err(error) => assert!( error.to_string().contains("requested WAL backend 'uring'"), @@ -150,7 +150,7 @@ fn explicit_uring_initialization_never_silently_falls_back() { #[test] fn rpc_workers_is_only_an_explicit_override() { let defaults = Cli::parse_from([ - "crowdb-kv-server", + "crowdb-paxos-kv-server", "--root", ".crowdb-runtime/ephemeral/cli-parse/n1", ]); @@ -161,7 +161,7 @@ fn rpc_workers_is_only_an_explicit_override() { assert_eq!(defaults.send_queue_capacity, None); let overridden = Cli::parse_from([ - "crowdb-kv-server", + "crowdb-paxos-kv-server", "--root", ".crowdb-runtime/ephemeral/cli-parse/n1", "--rpc-workers", @@ -170,7 +170,7 @@ fn rpc_workers_is_only_an_explicit_override() { assert_eq!(overridden.rpc_workers, Some(7)); let invalid = Cli::try_parse_from([ - "crowdb-kv-server", + "crowdb-paxos-kv-server", "--root", ".crowdb-runtime/ephemeral/cli-parse/n1", "--rpc-workers", @@ -182,15 +182,15 @@ fn rpc_workers_is_only_an_explicit_override() { #[test] fn production_tree_backend_defaults_to_block() { let defaults = Cli::parse_from([ - "crowdb-kv-server", + "crowdb-paxos-kv-server", "--root", ".crowdb-runtime/ephemeral/cli-parse/n1", ]); assert_eq!(defaults.kv_backend, "block"); - assert_eq!(CrowDBConfig::default().crowtree_backend, "block"); + assert_eq!(PaxosKvConfig::default().crowtree_backend, "block"); let file = Cli::parse_from([ - "crowdb-kv-server", + "crowdb-paxos-kv-server", "--root", ".crowdb-runtime/ephemeral/cli-parse/n1", "--kv-backend", @@ -202,11 +202,11 @@ fn production_tree_backend_defaults_to_block() { #[test] fn config_values_survive_absent_cli_and_explicit_values_override() { let defaults = Cli::parse_from([ - "crowdb-kv-server", + "crowdb-paxos-kv-server", "--root", ".crowdb-runtime/ephemeral/cli-parse/n1", ]); - let mut config = CrowDBConfig::default(); + let mut config = PaxosKvConfig::default(); config.paxos.max_inflight_proposals = 19; config.server.peer_pool_size = 7; config.server.send_queue_capacity = 1234; @@ -224,7 +224,7 @@ fn config_values_survive_absent_cli_and_explicit_values_override() { assert!(config.server.event_write); let overrides = Cli::parse_from([ - "crowdb-kv-server", + "crowdb-paxos-kv-server", "--root", ".crowdb-runtime/ephemeral/cli-parse/n1", "--max-inflight", diff --git a/app/crowdb-kv-server/tests/common/balance_distribution.rs b/app/crowdb-paxos-kv-server/tests/common/balance_distribution.rs similarity index 98% rename from app/crowdb-kv-server/tests/common/balance_distribution.rs rename to app/crowdb-paxos-kv-server/tests/common/balance_distribution.rs index 875077518..5da4c13b2 100644 --- a/app/crowdb-kv-server/tests/common/balance_distribution.rs +++ b/app/crowdb-paxos-kv-server/tests/common/balance_distribution.rs @@ -2,7 +2,7 @@ // Licensed under the Apache License, Version 2.0. use bytes::Bytes; -use crowdb_kv_server::group0_control_plane::Group0ControlPlane; +use crowdb_paxos_kv_server::group0_control_plane::Group0ControlPlane; use crowdb_protocol::{ chunk_kv::{ ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogHead, ChunkKvRangeCatalogPage, diff --git a/app/crowdb-kv-server/tests/common/balance_planning.rs b/app/crowdb-paxos-kv-server/tests/common/balance_planning.rs similarity index 98% rename from app/crowdb-kv-server/tests/common/balance_planning.rs rename to app/crowdb-paxos-kv-server/tests/common/balance_planning.rs index d98251c4b..45f5deb0c 100644 --- a/app/crowdb-kv-server/tests/common/balance_planning.rs +++ b/app/crowdb-paxos-kv-server/tests/common/balance_planning.rs @@ -2,7 +2,7 @@ // Licensed under the Apache License, Version 2.0. use bytes::Bytes; -use crowdb_kv_server::group0_control_plane::Group0ControlPlane; +use crowdb_paxos_kv_server::group0_control_plane::Group0ControlPlane; use crowdb_protocol::chunk_kv::{ ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogHead, ChunkKvRangeCatalogPage, ChunkKvRangeCatalogPageRef, ChunkKvRangeCatalogPartitionState, HostedPartition, Id128, KeyRange, OwnerDescriptor, PartitionArtifact, diff --git a/app/crowdb-kv-server/tests/common/catalog_publication.rs b/app/crowdb-paxos-kv-server/tests/common/catalog_publication.rs similarity index 99% rename from app/crowdb-kv-server/tests/common/catalog_publication.rs rename to app/crowdb-paxos-kv-server/tests/common/catalog_publication.rs index 5599d7a60..b0adaac12 100644 --- a/app/crowdb-kv-server/tests/common/catalog_publication.rs +++ b/app/crowdb-paxos-kv-server/tests/common/catalog_publication.rs @@ -2,7 +2,7 @@ // Licensed under the Apache License, Version 2.0. use super::*; -use crowdb_kv_server::background::domain_monitor::ChunkKvRangeMonitorDriver; +use crowdb_paxos_kv_server::background::domain_monitor::ChunkKvRangeMonitorDriver; struct TestPublication { control: Group0ControlPlane, diff --git a/app/crowdb-kv-server/tests/common/dynamic_slots.rs b/app/crowdb-paxos-kv-server/tests/common/dynamic_slots.rs similarity index 96% rename from app/crowdb-kv-server/tests/common/dynamic_slots.rs rename to app/crowdb-paxos-kv-server/tests/common/dynamic_slots.rs index 73b785636..7e4f1f746 100644 --- a/app/crowdb-kv-server/tests/common/dynamic_slots.rs +++ b/app/crowdb-paxos-kv-server/tests/common/dynamic_slots.rs @@ -4,9 +4,9 @@ use std::sync::Arc; use bytes::Bytes; -use crowdb_kv::cluster::group::PxGroup; -use crowdb_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; -use crowdb_kv_server::group0_control_plane::Group0ControlPlane; +use crowdb_paxos_kv::cluster::group::PxGroup; +use crowdb_paxos_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; +use crowdb_paxos_kv_server::group0_control_plane::Group0ControlPlane; use crowdb_protocol::chunk_kv::{DomainFailurePolicy, DomainMonitorDescriptor}; use crowdb_protocol::chunk_slot::{ChunkSlot, ChunkSlotBinding, ChunkSlotMapHead, ChunkStorageGroup}; use crowdb_protocol::common::InstanceValue; diff --git a/app/crowdb-kv-server/tests/common/endpoint_publication.rs b/app/crowdb-paxos-kv-server/tests/common/endpoint_publication.rs similarity index 98% rename from app/crowdb-kv-server/tests/common/endpoint_publication.rs rename to app/crowdb-paxos-kv-server/tests/common/endpoint_publication.rs index c87bfe346..0bc31b444 100644 --- a/app/crowdb-kv-server/tests/common/endpoint_publication.rs +++ b/app/crowdb-paxos-kv-server/tests/common/endpoint_publication.rs @@ -2,7 +2,7 @@ // Licensed under the Apache License, Version 2.0. use super::*; -use crowdb_kv_server::background::domain_monitor::ChunkKvRangeMonitorDriver; +use crowdb_paxos_kv_server::background::domain_monitor::ChunkKvRangeMonitorDriver; #[tokio::test] async fn recovered_owner_address_advances_catalog_without_transferring_data() { diff --git a/app/crowdb-kv-server/tests/common/grant_isolation.rs b/app/crowdb-paxos-kv-server/tests/common/grant_isolation.rs similarity index 98% rename from app/crowdb-kv-server/tests/common/grant_isolation.rs rename to app/crowdb-paxos-kv-server/tests/common/grant_isolation.rs index d8e2c251a..eccf1707b 100644 --- a/app/crowdb-kv-server/tests/common/grant_isolation.rs +++ b/app/crowdb-paxos-kv-server/tests/common/grant_isolation.rs @@ -2,7 +2,7 @@ // Licensed under the Apache License, Version 2.0. use super::*; -use crowdb_kv_server::background::domain_monitor::ChunkKvRangeMonitorDriver; +use crowdb_paxos_kv_server::background::domain_monitor::ChunkKvRangeMonitorDriver; use crowdb_protocol::{ chunk_stream::{StreamProcessLease, StreamPurpose}, key::StreamProcessLeaseKey, diff --git a/app/crowdb-kv-server/tests/common/grant_process.rs b/app/crowdb-paxos-kv-server/tests/common/grant_process.rs similarity index 97% rename from app/crowdb-kv-server/tests/common/grant_process.rs rename to app/crowdb-paxos-kv-server/tests/common/grant_process.rs index 830f0c73b..a291094b4 100644 --- a/app/crowdb-kv-server/tests/common/grant_process.rs +++ b/app/crowdb-paxos-kv-server/tests/common/grant_process.rs @@ -2,7 +2,7 @@ // Licensed under the Apache License, Version 2.0. use super::*; -use crowdb_kv_server::background::domain_monitor::ChunkKvRangeMonitorDriver; +use crowdb_paxos_kv_server::background::domain_monitor::ChunkKvRangeMonitorDriver; use crowdb_protocol::{ chunk_stream::{StreamProcessLease, StreamPurpose}, key::StreamProcessLeaseKey, diff --git a/app/crowdb-paxos-kv-server/tests/common/group_creation.rs b/app/crowdb-paxos-kv-server/tests/common/group_creation.rs new file mode 100644 index 000000000..88e908dbd --- /dev/null +++ b/app/crowdb-paxos-kv-server/tests/common/group_creation.rs @@ -0,0 +1,99 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use crowdb_paxos_kv::{ + cluster::{PaxosKvServer, PxKvStore}, + common::config::PaxosKvConfig, +}; +use crowdb_paxos_kv_server::{ + mgmt::{self, operation_registry::AppState}, + store_registry::PaxosKvStoreRegistry, +}; +use serde_json::{json, Value}; +use std::{path::PathBuf, sync::Arc}; + +pub struct TestGroupCreation { + _root: crowdb_test_harness::test_dirs::TestDir, + pub registry: Arc<PaxosKvStoreRegistry>, + pub store: Arc<PxKvStore>, + pub origin: String, + server: tokio::task::JoinHandle<()>, +} + +impl TestGroupCreation { + pub async fn start(block_data: bool) -> Self { + let root = crowdb_test_harness::test_dirs::tempdir_in_test_data("native-group-creation"); + let config = PaxosKvConfig { + data_root: root.path().join("data"), + wal_root: root.path().join("wal"), + config_root: root.path().join("config"), + crowtree_backend: "file".into(), + wal_backend: "file".into(), + ..PaxosKvConfig::for_tests() + }; + if block_data { + std::fs::write(&config.data_root, b"blocked").unwrap(); + } + let registry = Arc::new(PaxosKvStoreRegistry::with_config(config)); + let store = Arc::new(PxKvStore::new(0, "127.0.0.1:0".parse().unwrap())); + store.start().await.unwrap(); + registry.add_store(0, &store); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let origin = format!("http://{}", listener.local_addr().unwrap()); + let router = mgmt::router(AppState::new(Arc::clone(®istry))); + let server = tokio::spawn(async move { + axum::serve(listener, router).await.unwrap(); + }); + Self { + _root: root, + registry, + store, + origin, + server, + } + } + + pub fn ownership(&self) -> PathBuf { + self.registry.config.config_root.join("group-creation-0-1.json") + } + + pub async fn request( + &self, + method: reqwest::Method, + path: &str, + body: Option<&Value>, + ) -> (reqwest::StatusCode, Vec<u8>) { + let client = reqwest::Client::builder() + .no_proxy() + .timeout(std::time::Duration::from_secs(5)) + .build() + .unwrap(); + let mut request = client.request(method, format!("{}{path}", self.origin)); + if let Some(body) = body { + request = request.json(body); + } + let response = request.send().await.unwrap(); + let status = response.status(); + let bytes = response.bytes().await.unwrap().to_vec(); + (status, bytes) + } + + pub async fn create(&self, body: &Value, expected: reqwest::StatusCode) { + let (status, bytes) = self + .request(reqwest::Method::POST, "/stores/0/groups", Some(body)) + .await; + assert_eq!(status, expected, "{}", String::from_utf8_lossy(&bytes)); + } +} + +impl Drop for TestGroupCreation { + fn drop(&mut self) { + self.server.abort(); + self.store.stop(); + } +} + +pub fn request() -> Value { + json!({"group_id":1,"replica_id":7,"initial_role":"leader","start_election":false, + "deployment":{"cluster_id":uuid::Uuid::new_v4().to_string(),"operation_id":uuid::Uuid::new_v4().to_string()}}) +} diff --git a/app/crowdb-kv-server/tests/common/process_lease.rs b/app/crowdb-paxos-kv-server/tests/common/process_lease.rs similarity index 94% rename from app/crowdb-kv-server/tests/common/process_lease.rs rename to app/crowdb-paxos-kv-server/tests/common/process_lease.rs index f867005a8..a34cadd1b 100644 --- a/app/crowdb-kv-server/tests/common/process_lease.rs +++ b/app/crowdb-paxos-kv-server/tests/common/process_lease.rs @@ -2,7 +2,7 @@ // Licensed under the Apache License, Version 2.0. use bytes::Bytes; -use crowdb_kv_server::group0_control_plane::Group0ControlPlane; +use crowdb_paxos_kv_server::group0_control_plane::Group0ControlPlane; use crowdb_protocol::{ chunk_stream::{StreamProcessIdentity, StreamProcessLease, StreamPurpose}, key::{StreamProcessLeaseKey, TextKey}, @@ -39,6 +39,7 @@ impl TestProcessLease { prior_exclusion_deadline_ms: 0, grant_sequence: 0, grant_expires_at_ms: 0, + revoked: false, }; control .compare_and_put( diff --git a/app/crowdb-paxos-kv-server/tests/common/runtime_shutdown.rs b/app/crowdb-paxos-kv-server/tests/common/runtime_shutdown.rs new file mode 100644 index 000000000..52a225507 --- /dev/null +++ b/app/crowdb-paxos-kv-server/tests/common/runtime_shutdown.rs @@ -0,0 +1,155 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +use bytes::Bytes; +use crowdb_paxos_kv::{ + cluster::{ + group::PxGroup, + replica::{HeartbeatRequestPayload, ReplicaHandler}, + PaxosKvServer, PxKvStore, PxLocalReplicaRole, + }, + common::config::PaxosKvConfig, + paxos::roles::{PxAcceptReply, PxBallot, PxLogEntry}, +}; +use crowdb_paxos_kv_server::{ + mgmt::{self, operation_registry::AppState}, + recovery::startup::{create_group_with_wal_on_runtime, store_crowdb_tree_path, store_wal_root}, + store_registry::PaxosKvStoreRegistry, +}; +use std::{path::PathBuf, sync::Arc, time::Duration}; +pub const DEADLINE: Duration = Duration::from_secs(3); +pub struct TestRuntimeShutdown { + _root: crowdb_test_harness::test_dirs::TestDir, + pub registry: Arc<PaxosKvStoreRegistry>, + pub store: Arc<PxKvStore>, + pub group: Arc<PxGroup>, + pub engine_marker: PathBuf, + pub wal_marker: PathBuf, + origin: String, + http: tokio::task::JoinHandle<()>, +} +impl TestRuntimeShutdown { + pub async fn start() -> Self { + let root = crowdb_test_harness::test_dirs::tempdir_in_test_data("runtime-host-shutdown"); + let mut config = PaxosKvConfig { + wal_backend: "file".into(), + crowtree_backend: "mem-block".into(), + wal_root: root.path().join("wal"), + data_root: root.path().join("data"), + config_root: root.path().join("config"), + ..PaxosKvConfig::for_tests() + }; + config.server.shutdown_timeout_ms = 20; + config.tree_runtime.host.workers = Some(1); + let registry = Arc::new(PaxosKvStoreRegistry::try_with_config(config).unwrap()); + let group = create_group_with_wal_on_runtime( + 1, + 1, + 1, + PxLocalReplicaRole::Follower, + ®istry.config, + registry.wal_backend.clone(), + registry.crowtree_backend, + registry.tree_runtime.clone(), + ) + .await + .unwrap(); + let store = Arc::new(PxKvStore::new(1, "127.0.0.1:0".parse().unwrap())); + store.add_group_without_election(group); + store.start().await.unwrap(); + registry.add_store(1, &store); + let group = store.get_group(1).unwrap(); + accept(&group, 2).await; + group + .local_replica() + .on_heartbeat( + HeartbeatRequestPayload { + term: 0, + leader_id: 2, + prev_log_slot: 0, + prev_log_term: 0, + committed_safe_slot: 2, + lease_grant_until_ms_mono: 0, + t_send_ms_mono: 0, + gc_before_slot: 0, + gc_membership_epoch: 0, + }, + 1, + ) + .await + .unwrap(); + let engine_dir = store_crowdb_tree_path(®istry.config.data_root, 1, 1); + let wal_dir = store_wal_root(®istry.config.wal_root, 1).join("group1"); + std::fs::create_dir_all(&engine_dir).unwrap(); + std::fs::create_dir_all(&wal_dir).unwrap(); + let engine_marker = engine_dir.join("retained-marker"); + let wal_marker = wal_dir.join("retained-marker"); + std::fs::write(&engine_marker, b"engine retained").unwrap(); + std::fs::write(&wal_marker, b"WAL retained").unwrap(); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let origin = format!("http://{}", listener.local_addr().unwrap()); + let router = mgmt::router(AppState::new(registry.clone())); + let http = tokio::spawn(async move { + axum::serve(listener, router).await.unwrap(); + }); + Self { + _root: root, + registry, + store, + group, + engine_marker, + wal_marker, + origin, + http, + } + } + pub async fn request(&self, method: reqwest::Method, path: &str) -> (reqwest::StatusCode, String) { + let client = reqwest::Client::builder() + .no_proxy() + .timeout(DEADLINE) + .build() + .unwrap(); + let result = client + .request(method, format!("{}{path}", self.origin)) + .send() + .await + .unwrap(); + (result.status(), result.text().await.unwrap()) + } + pub fn assert_retained(&self) { + assert!(self.registry.get_store(1).is_some()); + assert!(self.store.get_group(1).is_some()); + assert!(self.engine_marker.exists()); + assert!(self.wal_marker.exists()); + assert_eq!(self.group.local_replica().contiguous_applied(), 0); + assert_eq!(self.group.local_replica().learner.engine().resume_from_slot(), 0); + } + pub async fn repair(&self) { + accept(&self.group, 1).await; + tokio::time::timeout(DEADLINE, self.group.local_replica().await_apply_fence(2)) + .await + .unwrap(); + } +} +impl Drop for TestRuntimeShutdown { + fn drop(&mut self) { + self.http.abort(); + self.store.stop(); + } +} +async fn accept(group: &PxGroup, slot: u64) { + let mut payload = vec![1, 0, 0]; + payload.extend_from_slice(&8u32.to_le_bytes()); + payload.extend_from_slice(&slot.to_le_bytes()); + payload.extend_from_slice(&8u32.to_le_bytes()); + payload.extend_from_slice(&slot.to_le_bytes()); + let entry = PxLogEntry { + slot, + term: 0, + ballot: PxBallot::new(1, 1), + payload: Bytes::from(payload), + }; + assert!(matches!( + group.local_replica().on_accept(&entry).await, + PxAcceptReply::Accepted { .. } + )); +} diff --git a/app/crowdb-kv-server/tests/common/split_materialization.rs b/app/crowdb-paxos-kv-server/tests/common/split_materialization.rs similarity index 100% rename from app/crowdb-kv-server/tests/common/split_materialization.rs rename to app/crowdb-paxos-kv-server/tests/common/split_materialization.rs diff --git a/app/crowdb-kv-server/tests/common/split_materialization/transfer.rs b/app/crowdb-paxos-kv-server/tests/common/split_materialization/transfer.rs similarity index 100% rename from app/crowdb-kv-server/tests/common/split_materialization/transfer.rs rename to app/crowdb-paxos-kv-server/tests/common/split_materialization/transfer.rs diff --git a/app/crowdb-paxos-kv-server/tests/common/system_bootstrap.rs b/app/crowdb-paxos-kv-server/tests/common/system_bootstrap.rs new file mode 100644 index 000000000..079aa41a8 --- /dev/null +++ b/app/crowdb-paxos-kv-server/tests/common/system_bootstrap.rs @@ -0,0 +1,127 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use crowdb_paxos_kv::common::config::PaxosKvConfig; +use crowdb_paxos_kv_server::{ + mgmt::{self, operation_registry::AppState}, + store_registry::PaxosKvStoreRegistry, +}; +use crowdb_protocol::mgmt::{SystemBootstrapIdentity, SystemPrepareRequest}; +use reqwest::StatusCode; +use serde_json::Value; +use std::{path::PathBuf, sync::Arc, time::Duration}; + +pub struct TestSystemBootstrap { + _root: crowdb_test_harness::test_dirs::TestDir, + pub registry: Arc<PaxosKvStoreRegistry>, + pub origin: String, + server: tokio::task::JoinHandle<()>, +} + +impl TestSystemBootstrap { + pub async fn start() -> Self { + let root = crowdb_test_harness::test_dirs::tempdir_in_test_data("system-prepare"); + let config = PaxosKvConfig { + data_root: root.path().join("data"), + wal_root: root.path().join("wal"), + config_root: root.path().join("config"), + ..PaxosKvConfig::for_tests() + }; + let registry = Arc::new(PaxosKvStoreRegistry::with_config(config)); + let (origin, server) = Self::serve(®istry).await; + Self { + _root: root, + registry, + origin, + server, + } + } + + async fn serve(registry: &Arc<PaxosKvStoreRegistry>) -> (String, tokio::task::JoinHandle<()>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let origin = format!("http://{}", listener.local_addr().unwrap()); + let router = mgmt::router(AppState::new(registry.clone())); + let server = tokio::spawn(async move { + axum::serve(listener, router).await.unwrap(); + }); + (origin, server) + } + + pub async fn restart(&mut self) { + self.server.abort(); + self.registry = Arc::new(PaxosKvStoreRegistry::with_config(self.registry.config.clone())); + (self.origin, self.server) = Self::serve(&self.registry).await; + } + + pub fn ownership(&self) -> PathBuf { + self.registry.config.config_root.join("system-bootstrap.json") + } + + pub async fn request(&self, path: &str, body: &Value) -> StatusCode { + let response = reqwest::Client::builder() + .no_proxy() + .timeout(Duration::from_secs(3)) + .build() + .unwrap() + .post(format!("{}{path}", self.origin)) + .json(body) + .send() + .await + .unwrap(); + let status = response.status(); + let bytes = response.bytes().await.unwrap(); + assert!( + matches!(status, StatusCode::OK | StatusCode::CONFLICT), + "{path}: {status}: {}", + String::from_utf8_lossy(&bytes) + ); + status + } + + pub async fn prepare(&self, replica_id: u64, bootstrap: &SystemBootstrapIdentity) -> StatusCode { + self.request( + "/system/prepare", + &serde_json::to_value(SystemPrepareRequest { + replica_id, + bootstrap: bootstrap.clone(), + }) + .unwrap(), + ) + .await + } + + pub fn assert_no_execution(&self) { + assert_eq!(self.registry.store_count(), 0); + assert!(!self.registry.config.data_root.exists()); + assert!(!self.registry.config.wal_root.exists()); + } +} + +impl Drop for TestSystemBootstrap { + fn drop(&mut self) { + self.server.abort(); + } +} + +pub fn identity() -> SystemBootstrapIdentity { + SystemBootstrapIdentity { + cluster_id: uuid::Uuid::new_v4().to_string(), + operation_id: uuid::Uuid::new_v4().to_string(), + configuration_digest: "a".repeat(64), + } +} + +pub async fn proposal( + nodes: &[TestSystemBootstrap], + selected: [usize; 3], + identity: &SystemBootstrapIdentity, +) -> bool { + let (first, second, third) = tokio::join!( + nodes[selected[0]].prepare(1, identity), + nodes[selected[1]].prepare(2, identity), + nodes[selected[2]].prepare(3, identity) + ); + [first, second, third] + .iter() + .all(|status| *status == StatusCode::OK) +} diff --git a/app/crowdb-kv-server/tests/diskdb_placement_test.rs b/app/crowdb-paxos-kv-server/tests/diskdb_placement_test.rs similarity index 92% rename from app/crowdb-kv-server/tests/diskdb_placement_test.rs rename to app/crowdb-paxos-kv-server/tests/diskdb_placement_test.rs index 495cc9ba3..db376bed2 100644 --- a/app/crowdb-kv-server/tests/diskdb_placement_test.rs +++ b/app/crowdb-paxos-kv-server/tests/diskdb_placement_test.rs @@ -1,12 +1,12 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. use bytes::Bytes; -use crowdb_kv::cluster::group::PxGroup; -use crowdb_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; -use crowdb_kv::common::config::CrowDBConfig; -use crowdb_kv_server::background::domain_monitor::{DiskdbPlacementMonitorDriver, DomainMonitorDriver}; -use crowdb_kv_server::group0_control_plane::Group0ControlPlane; -use crowdb_kv_server::store_registry::KvStoreRegistry; +use crowdb_paxos_kv::cluster::group::PxGroup; +use crowdb_paxos_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; +use crowdb_paxos_kv::common::config::PaxosKvConfig; +use crowdb_paxos_kv_server::background::domain_monitor::{DiskdbPlacementMonitorDriver, DomainMonitorDriver}; +use crowdb_paxos_kv_server::group0_control_plane::Group0ControlPlane; +use crowdb_paxos_kv_server::store_registry::PaxosKvStoreRegistry; use crowdb_protocol::chunk_kv::{DomainFailurePolicy, DomainMonitorDescriptor}; use crowdb_protocol::common::{BindMapValue, InstanceValue, OwnerMapValue}; use crowdb_protocol::key::{BindMapKey, DiskGroupKey, InstanceKey, OwnerMapKey, RackKey, TextKey}; @@ -29,8 +29,8 @@ fn descriptor() -> DomainMonitorDescriptor { } } -fn registry_with_group_zero(role: PxLocalReplicaRole) -> (Arc<KvStoreRegistry>, Arc<PxKvStore>) { - let registry = Arc::new(KvStoreRegistry::with_config(CrowDBConfig::for_tests())); +fn registry_with_group_zero(role: PxLocalReplicaRole) -> (Arc<PaxosKvStoreRegistry>, Arc<PxKvStore>) { + let registry = Arc::new(PaxosKvStoreRegistry::with_config(PaxosKvConfig::for_tests())); let store = Arc::new(PxKvStore::new(0, "127.0.0.1:0".parse().unwrap())); store.add_group(PxGroup::new(0, PxLocalReplica::new(1, role))); registry.add_store(0, &store); diff --git a/app/crowdb-kv-server/tests/domain_monitor_test.rs b/app/crowdb-paxos-kv-server/tests/domain_monitor_test.rs similarity index 97% rename from app/crowdb-kv-server/tests/domain_monitor_test.rs rename to app/crowdb-paxos-kv-server/tests/domain_monitor_test.rs index 1b1307719..3dd46c045 100644 --- a/app/crowdb-kv-server/tests/domain_monitor_test.rs +++ b/app/crowdb-paxos-kv-server/tests/domain_monitor_test.rs @@ -6,15 +6,15 @@ use std::sync::Arc; use std::time::Duration; use bytes::Bytes; -use crowdb_kv::cluster::group::PxGroup; -use crowdb_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; -use crowdb_kv::common::config::CrowDBConfig; -use crowdb_kv_server::background::domain_monitor::{ +use crowdb_paxos_kv::cluster::group::PxGroup; +use crowdb_paxos_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; +use crowdb_paxos_kv::common::config::PaxosKvConfig; +use crowdb_paxos_kv_server::background::domain_monitor::{ spawn_domain_monitor_supervisor, DiskdbOwnershipMonitorDriver, DomainMonitorDriver, DomainMonitorDrivers, DomainMonitorFuture, }; -use crowdb_kv_server::group0_control_plane::Group0ControlPlane; -use crowdb_kv_server::store_registry::KvStoreRegistry; +use crowdb_paxos_kv_server::group0_control_plane::Group0ControlPlane; +use crowdb_paxos_kv_server::store_registry::PaxosKvStoreRegistry; use crowdb_protocol::chunk_kv::{ AuthorityReleaseProof, ChunkKvRangeBalancePolicy, ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogHead, ChunkKvRangeCatalogPage, ChunkKvRangeCatalogPageRef, ChunkKvRangeCatalogPartitionState, HostedPartition, @@ -86,8 +86,8 @@ fn descriptor() -> DomainMonitorDescriptor { } } -fn registry_with_group_zero(role: PxLocalReplicaRole) -> (Arc<KvStoreRegistry>, Arc<PxKvStore>) { - let registry = Arc::new(KvStoreRegistry::with_config(CrowDBConfig::for_tests())); +fn registry_with_group_zero(role: PxLocalReplicaRole) -> (Arc<PaxosKvStoreRegistry>, Arc<PxKvStore>) { + let registry = Arc::new(PaxosKvStoreRegistry::with_config(PaxosKvConfig::for_tests())); let store = Arc::new(PxKvStore::new(0, "127.0.0.1:0".parse().unwrap())); store.add_group(PxGroup::new(0, PxLocalReplica::new(1, role))); registry.add_store(0, &store); @@ -208,7 +208,7 @@ async fn chunk_kv_driver_publishes_ready_transfer_then_issues_matching_grant() { #[allow(clippy::too_many_lines)] async fn assert_ready_transfer_publication() -> (Group0ControlPlane, TransferTransition) { - use crowdb_kv_server::background::domain_monitor::ChunkKvRangeMonitorDriver; + use crowdb_paxos_kv_server::background::domain_monitor::ChunkKvRangeMonitorDriver; let (_, store) = registry_with_group_zero(PxLocalReplicaRole::Leader); let control = Group0ControlPlane::acquire(&store).await.unwrap(); @@ -455,7 +455,7 @@ async fn assert_ready_transfer_publication() -> (Group0ControlPlane, TransferTra #[tokio::test] async fn chunk_kv_driver_plans_dead_owner_recovery_once_and_waits_for_target() { - use crowdb_kv_server::background::domain_monitor::ChunkKvRangeMonitorDriver; + use crowdb_paxos_kv_server::background::domain_monitor::ChunkKvRangeMonitorDriver; let (_, store) = registry_with_group_zero(PxLocalReplicaRole::Leader); let control = Group0ControlPlane::acquire(&store).await.unwrap(); @@ -556,7 +556,7 @@ async fn assert_chunk_kv_split_plan( target_partitions_per_owner: u32, target_partition_bytes: u64, ) -> (Group0ControlPlane, SplitTransition) { - use crowdb_kv_server::background::domain_monitor::ChunkKvRangeMonitorDriver; + use crowdb_paxos_kv_server::background::domain_monitor::ChunkKvRangeMonitorDriver; let (_, store) = registry_with_group_zero(PxLocalReplicaRole::Leader); let control = Group0ControlPlane::acquire(&store).await.unwrap(); @@ -753,7 +753,7 @@ async fn chunk_kv_driver_plans_size_driven_split() { #[tokio::test] async fn chunk_kv_driver_enables_child_owner_balance_from_policy() { - use crowdb_kv_server::background::domain_monitor::ChunkKvRangeMonitorDriver; + use crowdb_paxos_kv_server::background::domain_monitor::ChunkKvRangeMonitorDriver; let (_, store) = registry_with_group_zero(PxLocalReplicaRole::Leader); let control = Group0ControlPlane::acquire(&store).await.unwrap(); diff --git a/app/crowdb-paxos-kv-server/tests/engine_resource_metrics_test.rs b/app/crowdb-paxos-kv-server/tests/engine_resource_metrics_test.rs new file mode 100644 index 000000000..48157acf9 --- /dev/null +++ b/app/crowdb-paxos-kv-server/tests/engine_resource_metrics_test.rs @@ -0,0 +1,64 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use std::sync::Arc; +use std::time::Duration; + +use crowdb_common::metrics::MetricsRunner; +use crowdb_paxos_kv::cluster::{px_kv_store::PxKvStore, PxLocalReplicaRole}; +use crowdb_paxos_kv::common::config::PaxosKvConfig; +use crowdb_paxos_kv::kv::CrowdbTreeEngine; +use crowdb_paxos_kv_server::engine_collector::setup_engine_collector; +use crowdb_paxos_kv_server::recovery::startup::create_group_with_wal_on_runtime; +use crowdb_paxos_kv_server::store_registry::PaxosKvStoreRegistry; + +#[tokio::test] +async fn service_collector_logs_host_and_tree_resource_windows() { + let root = crowdb_test_harness::test_dirs::tempdir_in_test_data("paxos-tree-resources"); + let config = PaxosKvConfig { + wal_backend: "mem-block".into(), + crowtree_backend: "mem-block".into(), + wal_root: root.path().join("wal"), + config_root: root.path().join("config"), + data_root: root.path().join("tree"), + log_dir: root.path().join("log").to_string_lossy().into_owned(), + ..Default::default() + }; + let registry = Arc::new(PaxosKvStoreRegistry::try_with_config(config).unwrap()); + let store = Arc::new(PxKvStore::new(1, "127.0.0.1:0".parse().unwrap())); + let group = create_group_with_wal_on_runtime( + 1, + 1, + 1, + PxLocalReplicaRole::Leader, + ®istry.config, + registry.wal_backend.clone(), + registry.crowtree_backend, + registry.tree_runtime.clone(), + ) + .await + .unwrap(); + let engine = group.local_replica().learner.engine_arc(); + store.add_group_without_election(group); + registry.add_store(1, &store); + let dir = root.path().join("metrics"); + let file = crowdb_common::logging::open_metrics_log(&dir, "test", 10, 2).unwrap(); + let mut runner = MetricsRunner::new(file, 1); + setup_engine_collector(&runner.registry().clone(), ®istry, &mut runner); + runner.start(); + let tree = engine.as_any().downcast_ref::<CrowdbTreeEngine>().unwrap(); + tokio::time::timeout(Duration::from_secs(5), async { + while tree.work_metrics_window().is_none() { + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .unwrap(); + runner.stop().await; + let path = std::fs::read_dir(&dir).unwrap().next().unwrap().unwrap().path(); + let log = std::fs::read_to_string(path).unwrap(); + assert!(log.contains("host.tree.resource.cpu_workers"), "{log}"); + assert!(log.contains("s.1.g.1.tree.resource.routing"), "{log}"); + assert!(log.contains("reserved_sample_max="), "{log}"); + assert!(log.contains("max_sample_gap_ns="), "{log}"); +} diff --git a/app/crowdb-kv-server/tests/group0_control_plane_test.rs b/app/crowdb-paxos-kv-server/tests/group0_control_plane_test.rs similarity index 85% rename from app/crowdb-kv-server/tests/group0_control_plane_test.rs rename to app/crowdb-paxos-kv-server/tests/group0_control_plane_test.rs index 423c6f0c4..c2cb3d1c7 100644 --- a/app/crowdb-kv-server/tests/group0_control_plane_test.rs +++ b/app/crowdb-paxos-kv-server/tests/group0_control_plane_test.rs @@ -4,11 +4,11 @@ use std::sync::Arc; use bytes::Bytes; -use crowdb_kv::cluster::group::PxGroup; -use crowdb_kv::cluster::group_election::LeaderElection; -use crowdb_kv::cluster::group_operations::KvGroupOperationError; -use crowdb_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; -use crowdb_kv_server::group0_control_plane::Group0ControlPlane; +use crowdb_paxos_kv::cluster::group::PxGroup; +use crowdb_paxos_kv::cluster::group_election::LeaderElection; +use crowdb_paxos_kv::cluster::group_operations::KvGroupOperationError; +use crowdb_paxos_kv::cluster::{PxKvStore, PxLocalReplica, PxLocalReplicaRole}; +use crowdb_paxos_kv_server::group0_control_plane::Group0ControlPlane; fn group_zero_store() -> Arc<PxKvStore> { let store = Arc::new(PxKvStore::new(0, "127.0.0.1:0".parse().unwrap())); diff --git a/app/crowdb-kv-server/tests/group0_discovery_test.rs b/app/crowdb-paxos-kv-server/tests/group0_discovery_test.rs similarity index 92% rename from app/crowdb-kv-server/tests/group0_discovery_test.rs rename to app/crowdb-paxos-kv-server/tests/group0_discovery_test.rs index e1a96efc4..c5b04766c 100644 --- a/app/crowdb-kv-server/tests/group0_discovery_test.rs +++ b/app/crowdb-paxos-kv-server/tests/group0_discovery_test.rs @@ -5,7 +5,7 @@ use crowdb_test_harness::test_dirs::TestDir; #[test] fn persisted_identity_rejects_conflicting_configuration_and_corruption() { - use crowdb_kv_server::background::identity::load_or_create; + use crowdb_paxos_kv_server::background::identity::load_or_create; let root = TestDir::new("service-identity").unwrap(); let first = load_or_create(root.path(), Some(42), Some(2)).unwrap(); diff --git a/app/crowdb-kv-server/tests/group_creation_test.rs b/app/crowdb-paxos-kv-server/tests/group_creation_test.rs similarity index 98% rename from app/crowdb-kv-server/tests/group_creation_test.rs rename to app/crowdb-paxos-kv-server/tests/group_creation_test.rs index ae4b09aba..94a9b2b48 100644 --- a/app/crowdb-kv-server/tests/group_creation_test.rs +++ b/app/crowdb-paxos-kv-server/tests/group_creation_test.rs @@ -7,7 +7,7 @@ mod common; mod system_bootstrap; use common::{request, TestGroupCreation}; -use crowdb_kv::cluster::{group::PxGroup, PxLocalReplica, PxLocalReplicaRole}; +use crowdb_paxos_kv::cluster::{group::PxGroup, PxLocalReplica, PxLocalReplicaRole}; use reqwest::{Method, StatusCode}; use serde_json::Value; @@ -91,7 +91,7 @@ async fn explicit_removal_keeps_the_deployment_tombstone_and_restore_cannot_recr true ); fixture.create(&body, StatusCode::CONFLICT).await; - let restored = crowdb_kv_server::recovery::startup::create_group_with_wal( + let restored = crowdb_paxos_kv_server::recovery::startup::create_group_with_wal( 0, 1, 7, diff --git a/app/crowdb-kv-server/tests/keepalive_recovery_test.rs b/app/crowdb-paxos-kv-server/tests/keepalive_recovery_test.rs similarity index 87% rename from app/crowdb-kv-server/tests/keepalive_recovery_test.rs rename to app/crowdb-paxos-kv-server/tests/keepalive_recovery_test.rs index ff7f45341..bc7e93f81 100644 --- a/app/crowdb-kv-server/tests/keepalive_recovery_test.rs +++ b/app/crowdb-paxos-kv-server/tests/keepalive_recovery_test.rs @@ -2,10 +2,12 @@ // Licensed under the Apache License, Version 2.0. use axum::{extract::State, routing::get, Json, Router}; -use crowdb_kv::cluster::{group::PxGroup, KvServer, PxKvStore, PxLocalReplica, PxLocalReplicaRole}; -use crowdb_kv::common::config::CrowDBConfig; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, GetOutcome, ReadMode}; -use crowdb_kv_server::{background::keepalive::KeepAliveLoop, store_registry::KvStoreRegistry}; +use crowdb_paxos_kv::cluster::{ + group::PxGroup, PaxosKvServer, PxKvStore, PxLocalReplica, PxLocalReplicaRole, +}; +use crowdb_paxos_kv::common::config::PaxosKvConfig; +use crowdb_paxos_kv_client::{ClientConfig, GetOutcome, PaxosKvClient, ReadMode}; +use crowdb_paxos_kv_server::{background::keepalive::KeepAliveLoop, store_registry::PaxosKvStoreRegistry}; use crowdb_protocol::common::KvServerIdentity; use std::{sync::Arc, time::Duration}; @@ -32,7 +34,7 @@ async fn slow_registration_does_not_add_its_latency_to_the_first_heartbeat_inter PxLocalReplica::new(1, PxLocalReplicaRole::Leader), )); store.start().await.unwrap(); - let registry = Arc::new(KvStoreRegistry::with_config(CrowDBConfig::for_tests())); + let registry = Arc::new(PaxosKvStoreRegistry::with_config(PaxosKvConfig::for_tests())); registry.add_store(0, &store); let delay = Arc::new(TestDelayedTopology { store: store.clone(), @@ -64,9 +66,9 @@ async fn slow_registration_does_not_add_its_latency_to_the_first_heartbeat_inter // Inject discovery latency after register() has stamped its first value. tokio::time::sleep(Duration::from_millis(900)).await; delay.release.notify_one(); - let client = CrowdbKvClient::new(ClientConfig::new(Vec::new())); + let client = PaxosKvClient::new(ClientConfig::new(Vec::new())); client.seed_leader(0, 0, store.listen_addr().unwrap().to_string()); - let observations = crowdb_kv_client::ServiceRegistryClient::new(client); + let observations = crowdb_paxos_kv_client::ServiceRegistryClient::new(client); let initial = tokio::time::timeout(Duration::from_millis(250), async { loop { if let Some(value) = observations.read_instance("kv-server", 7).await.unwrap() { @@ -104,7 +106,7 @@ async fn a_recovered_singleton_registers_its_current_address_with_unreachable_re PxLocalReplica::new(1, PxLocalReplicaRole::Leader), )); store.start().await.unwrap(); - let registry = Arc::new(KvStoreRegistry::with_config(CrowDBConfig::for_tests())); + let registry = Arc::new(PaxosKvStoreRegistry::with_config(PaxosKvConfig::for_tests())); registry.add_store(0, &store); let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); let current = format!("http://{}", listener.local_addr().unwrap()); @@ -126,7 +128,7 @@ async fn a_recovered_singleton_registers_its_current_address_with_unreachable_re String::new(), 1, ); - let client = CrowdbKvClient::new(ClientConfig::new(vec![current.clone()])); + let client = PaxosKvClient::new(ClientConfig::new(vec![current.clone()])); client.seed_leader(0, 0, store.listen_addr().unwrap().to_string()); let value = tokio::time::timeout(Duration::from_secs(3), async { loop { diff --git a/app/crowdb-kv-server/tests/member_routes_test.rs b/app/crowdb-paxos-kv-server/tests/member_routes_test.rs similarity index 95% rename from app/crowdb-kv-server/tests/member_routes_test.rs rename to app/crowdb-paxos-kv-server/tests/member_routes_test.rs index 4b8f65d72..ba14e119b 100644 --- a/app/crowdb-kv-server/tests/member_routes_test.rs +++ b/app/crowdb-paxos-kv-server/tests/member_routes_test.rs @@ -1,8 +1,8 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -use crowdb_kv::cluster::{group::PxGroup, PxLocalReplica, PxLocalReplicaRole, PxRemoteReplica}; -use crowdb_kv_server::recovery::member_routes::{apply, configure}; +use crowdb_paxos_kv::cluster::{group::PxGroup, PxLocalReplica, PxLocalReplicaRole, PxRemoteReplica}; +use crowdb_paxos_kv_server::recovery::member_routes::{apply, configure}; use crowdb_protocol::mgmt::{ node::recovery::{MemberEndpoint, MemberEndpoints, RecoveryGroup, RecoveryGroups, RecoveryReplica}, SystemBootstrapIdentity, diff --git a/app/crowdb-kv-server/tests/monitor_supervisor_test.rs b/app/crowdb-paxos-kv-server/tests/monitor_supervisor_test.rs similarity index 94% rename from app/crowdb-kv-server/tests/monitor_supervisor_test.rs rename to app/crowdb-paxos-kv-server/tests/monitor_supervisor_test.rs index d8ac1c5f7..f683420a5 100644 --- a/app/crowdb-kv-server/tests/monitor_supervisor_test.rs +++ b/app/crowdb-paxos-kv-server/tests/monitor_supervisor_test.rs @@ -4,7 +4,7 @@ use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::Arc; -use crowdb_kv_server::background::domain_monitor::spawn_restarting_monitor; +use crowdb_paxos_kv_server::background::domain_monitor::spawn_restarting_monitor; #[tokio::test] async fn failed_monitor_task_is_restarted_and_shutdown_is_drained() { diff --git a/app/crowdb-kv-server/tests/reconcile_test.rs b/app/crowdb-paxos-kv-server/tests/reconcile_test.rs similarity index 89% rename from app/crowdb-kv-server/tests/reconcile_test.rs rename to app/crowdb-paxos-kv-server/tests/reconcile_test.rs index 8bce17559..b6e1220d1 100644 --- a/app/crowdb-kv-server/tests/reconcile_test.rs +++ b/app/crowdb-paxos-kv-server/tests/reconcile_test.rs @@ -5,20 +5,20 @@ use std::sync::Arc; -use crowdb_kv::cluster::group::PxGroup; -use crowdb_kv::cluster::local_replica::{PxLocalReplica, PxLocalReplicaRole}; -use crowdb_kv::cluster::px_kv_store::PxKvStore; -use crowdb_kv::common::config::CrowDBConfig; -use crowdb_kv_server::recovery::reconcile::{expand_membership_records, plan_reconcile, ReplicaRecord}; -use crowdb_kv_server::store_registry::KvStoreRegistry; +use crowdb_paxos_kv::cluster::group::PxGroup; +use crowdb_paxos_kv::cluster::local_replica::{PxLocalReplica, PxLocalReplicaRole}; +use crowdb_paxos_kv::cluster::px_kv_store::PxKvStore; +use crowdb_paxos_kv::common::config::PaxosKvConfig; +use crowdb_paxos_kv_server::recovery::reconcile::{expand_membership_records, plan_reconcile, ReplicaRecord}; +use crowdb_paxos_kv_server::store_registry::PaxosKvStoreRegistry; use crowdb_protocol::kv_membership::{GroupMember, GroupMembership, GroupMembershipState}; // ── Unit tests: plan_reconcile (pure decision logic) ──────────── /// Build a registry with one store (id 0) containing a group (`gid`) /// whose local replica is `rid` and has `remotes` wired. -fn registry_with_group(gid: u64, rid: u64, remotes: &[(u64, &str, bool)]) -> Arc<KvStoreRegistry> { - let registry = Arc::new(KvStoreRegistry::with_config(CrowDBConfig::for_tests())); +fn registry_with_group(gid: u64, rid: u64, remotes: &[(u64, &str, bool)]) -> Arc<PaxosKvStoreRegistry> { + let registry = Arc::new(PaxosKvStoreRegistry::with_config(PaxosKvConfig::for_tests())); let store = PxKvStore::new(0, "0.0.0.0:0".parse().unwrap()); let local = PxLocalReplica::new(rid, PxLocalReplicaRole::Follower); let mut group = PxGroup::new(gid, local); @@ -27,7 +27,7 @@ fn registry_with_group(gid: u64, rid: u64, remotes: &[(u64, &str, bool)]) -> Arc remotes .iter() .map(|(id, ep, v)| { - crowdb_kv::cluster::remote_replica::PxRemoteReplica::new(*id, (*ep).to_string()) + crowdb_paxos_kv::cluster::remote_replica::PxRemoteReplica::new(*id, (*ep).to_string()) .with_voting(*v) }) .collect(), diff --git a/app/crowdb-kv-server/tests/restore_test.rs b/app/crowdb-paxos-kv-server/tests/restore_test.rs similarity index 89% rename from app/crowdb-kv-server/tests/restore_test.rs rename to app/crowdb-paxos-kv-server/tests/restore_test.rs index 03946382d..9647eece0 100644 --- a/app/crowdb-kv-server/tests/restore_test.rs +++ b/app/crowdb-paxos-kv-server/tests/restore_test.rs @@ -3,7 +3,7 @@ //! Tests for restore mode (R104): local-disk scan + restart rebuild. -use crowdb_kv_server::recovery::restore::{group0_exists, scan_local_groups}; +use crowdb_paxos_kv_server::recovery::restore::{group0_exists, scan_local_groups}; // ── Unit tests: scan_local_groups / group0_exists ─────────────── @@ -35,15 +35,15 @@ async fn scan_local_groups_multi_store() { assert_eq!( groups, vec![ - crowdb_kv_server::recovery::restore::LocalGroup { + crowdb_paxos_kv_server::recovery::restore::LocalGroup { store_id: 0, group_id: 0 }, - crowdb_kv_server::recovery::restore::LocalGroup { + crowdb_paxos_kv_server::recovery::restore::LocalGroup { store_id: 1, group_id: 1 }, - crowdb_kv_server::recovery::restore::LocalGroup { + crowdb_paxos_kv_server::recovery::restore::LocalGroup { store_id: 1, group_id: 2 }, diff --git a/app/crowdb-paxos-kv-server/tests/runtime_shutdown_test.rs b/app/crowdb-paxos-kv-server/tests/runtime_shutdown_test.rs new file mode 100644 index 000000000..f8f8622e7 --- /dev/null +++ b/app/crowdb-paxos-kv-server/tests/runtime_shutdown_test.rs @@ -0,0 +1,104 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. +#[path = "common/runtime_shutdown.rs"] +mod common; +use common::{TestRuntimeShutdown, DEADLINE}; +use crowdb_paxos_kv::cluster::{ + group::PxGroup, PaxosKvServer, PxKvStore, PxLocalReplica, PxLocalReplicaRole, +}; +use crowdb_paxos_kv::paxos::roles::{PxBallot, PxPrepareReply}; +use crowdb_paxos_kv::rpc::ReplicaDestination; +use crowdb_paxos_kv_server::shutdown::{shutdown_stores, shutdown_stores_once}; +use reqwest::{Method, StatusCode}; +use std::sync::Arc; + +async fn removal_preserves_accepted_work(method: Method, path: &str) { + let fixture = TestRuntimeShutdown::start().await; + let (status, body) = fixture.request(method.clone(), path).await; + assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE, "{body}"); + assert!(body.contains("shutdown incomplete"), "{body}"); + fixture.assert_retained(); + fixture.repair().await; + let (status, body) = fixture.request(method, path).await; + assert_eq!(status, StatusCode::OK, "{body}"); + assert_eq!( + fixture.group.local_replica().learner.engine().resume_from_slot(), + 2 + ); + assert!(!fixture.engine_marker.exists()); + assert!(!fixture.wal_marker.exists()); + if fixture.registry.get_store(1).is_some() { + assert!(fixture.store.shutdown(DEADLINE).await.is_clean()); + } +} +#[tokio::test(flavor = "current_thread")] +async fn store_removal_retains_registry_and_wal_until_accepted_gap_drains() { + removal_preserves_accepted_work(Method::DELETE, "/stores/1").await; +} +#[tokio::test(flavor = "current_thread")] +async fn group_removal_retains_registry_and_wal_until_accepted_gap_drains() { + removal_preserves_accepted_work(Method::DELETE, "/stores/1/groups/1").await; +} +#[tokio::test(flavor = "current_thread")] +async fn wipe_retains_registry_and_wal_until_accepted_gap_drains() { + removal_preserves_accepted_work(Method::POST, "/stores/1/groups/1/wipe-user-data").await; +} +#[tokio::test(flavor = "current_thread")] +async fn host_timeout_retains_all_peer_servers_and_waits_for_real_progress() { + let fixture = TestRuntimeShutdown::start().await; + let peer = Arc::new(PxKvStore::new(2, "127.0.0.1:0".parse().unwrap())); + peer.add_group_without_election(PxGroup::new( + 2, + PxLocalReplica::new(2, PxLocalReplicaRole::Follower), + )); + peer.start().await.unwrap(); + fixture.registry.add_store(2, &peer); + let report = shutdown_stores_once(&fixture.registry).await; + assert!(!report.is_clean()); + fixture.assert_retained(); + let shutdown = tokio::spawn({ + let registry = fixture.registry.clone(); + async move { shutdown_stores(®istry).await } + }); + let endpoint = peer.listen_addr().unwrap().to_string(); + let reply = tokio::time::timeout( + DEADLINE, + fixture.store.rpc_transport().unwrap().send_prepare( + ReplicaDestination { + endpoint: &endpoint, + replica_id: 2, + }, + 2, + PxBallot::new(1, 1), + 0, + 2, + 0, + ), + ) + .await + .unwrap() + .unwrap(); + assert!(matches!(reply, PxPrepareReply::Promised { .. })); + assert!(!shutdown.is_finished()); + fixture.repair().await; + let report = tokio::time::timeout(DEADLINE, shutdown).await.unwrap().unwrap(); + assert!(report.is_clean(), "{:?}", report.errors); + assert_eq!( + fixture.group.local_replica().learner.engine().resume_from_slot(), + 2 + ); +} + +#[tokio::test(flavor = "current_thread")] +async fn member_route_shutdown_releases_its_registry_before_returning() { + let registry = Arc::new( + crowdb_paxos_kv_server::store_registry::PaxosKvStoreRegistry::with_config( + crowdb_paxos_kv::common::config::PaxosKvConfig::for_tests(), + ), + ); + let weak = Arc::downgrade(®istry); + let worker = crowdb_paxos_kv_server::recovery::member_routes::start(registry.clone()); + drop(registry); + worker.stop().await; + assert!(weak.upgrade().is_none()); +} diff --git a/app/crowdb-paxos-kv-server/tests/startup_test.rs b/app/crowdb-paxos-kv-server/tests/startup_test.rs new file mode 100644 index 000000000..8a955a733 --- /dev/null +++ b/app/crowdb-paxos-kv-server/tests/startup_test.rs @@ -0,0 +1,377 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use std::sync::Arc; + +use bytes::Bytes; +use crowdb_paxos_kv::cluster::group::ProposeResult; +use crowdb_paxos_kv::cluster::group_election::LeaderElection; +use crowdb_paxos_kv::cluster::local_replica::PxLocalReplicaRole; +use crowdb_paxos_kv::common::config::{PaxosKvConfig, WalConfig}; +use crowdb_paxos_kv::kv::CrowdbTreeBackend; +use crowdb_paxos_kv::paxos::roles::{PxBallot, PxLogEntry}; +use crowdb_paxos_kv::wal::record::WALRecord; +use crowdb_paxos_kv::wal::replay::replay_group; +use crowdb_paxos_kv::wal::{IoBackend, WalEngine}; +use crowdb_paxos_kv_server::recovery::restore::{load_local_groups, LocalGroup}; +use crowdb_paxos_kv_server::recovery::startup::{create_group_with_wal, store_wal_root}; +use crowdb_paxos_kv_server::store_registry::PaxosKvStoreRegistry; + +#[path = "startup_test/reclamation.rs"] +mod reclamation; + +fn encode_put_payload(key: &[u8], value: &[u8]) -> Vec<u8> { + let mut buf = Vec::new(); + buf.extend_from_slice(&1u16.to_le_bytes()); // 1 op (u16 LE) + buf.push(0); // kind = Put + let key_len = u32::try_from(key.len()).expect("key length exceeds u32"); + buf.extend_from_slice(&key_len.to_le_bytes()); + buf.extend_from_slice(key); + let value_len = u32::try_from(value.len()).expect("value length exceeds u32"); + buf.extend_from_slice(&value_len.to_le_bytes()); + buf.extend_from_slice(value); + buf +} + +#[tokio::test] +async fn incomplete_group_creation_retains_replica_identity_before_wal_is_discoverable() { + let temp = crowdb_test_harness::test_dirs::tempdir_in_test_data("startup-identity"); + let blocked_tree = temp.path().join("blocked-tree"); + std::fs::write(&blocked_tree, b"not a directory").unwrap(); + let config = PaxosKvConfig { + wal_root: temp.path().join("wal"), + config_root: temp.path().join("conf"), + data_root: blocked_tree, + ..PaxosKvConfig::for_tests() + }; + let result = create_group_with_wal( + 0, + 1, + 2, + PxLocalReplicaRole::Leader, + &config, + Arc::new(IoBackend::detect()), + CrowdbTreeBackend::File, + ) + .await; + assert!(result.is_err()); + let groups = crowdb_paxos_kv_server::recovery::restore::scan_local_groups(&config.wal_root) + .await + .unwrap(); + assert!(groups + .iter() + .any(|group| group.store_id == 0 && group.group_id == 1)); + let persisted = crowdb_paxos_kv::cluster::node_config::NodeConfigStore::new(&config.config_root) + .load() + .await + .unwrap(); + assert_eq!( + persisted.group(0, 1).map(|group| group.replica_id), + Some(2), + "an interrupted creation must never restore using CLI replica 1" + ); +} + +#[tokio::test] +async fn restore_rejects_corrupt_node_config_instead_of_using_cli_replica() { + let temp = crowdb_test_harness::test_dirs::tempdir_in_test_data("startup-corrupt-config"); + let config = PaxosKvConfig { + wal_root: temp.path().join("wal"), + config_root: temp.path().join("conf"), + data_root: temp.path().join("data"), + ..PaxosKvConfig::for_tests() + }; + let backend = Arc::new(IoBackend::detect()); + + create_group_with_wal( + 0, + 1, + 7, + PxLocalReplicaRole::Follower, + &config, + backend, + CrowdbTreeBackend::File, + ) + .await + .unwrap(); + tokio::fs::write(config.config_root.join("node-config.json"), b"{broken") + .await + .unwrap(); + + let registry = Arc::new(PaxosKvStoreRegistry::with_config(config)); + let error = load_local_groups( + &[LocalGroup { + store_id: 0, + group_id: 1, + }], + 99, + ®istry, + ) + .await + .expect_err("corrupt node config must block restore"); + assert!( + error.contains("failed to read persisted replica identity"), + "{error}" + ); +} + +#[tokio::test] +async fn startup_passes_wal_segment_tuning_to_the_group_writer() { + let temp = crowdb_test_harness::test_dirs::tempdir_in_test_data("startup-wal-tuning"); + let mut config = PaxosKvConfig { + wal_root: temp.path().join("wal"), + config_root: temp.path().join("conf"), + data_root: temp.path().join("data"), + ..PaxosKvConfig::for_tests() + }; + config.wal.wal_segment_size = 512; + + let group = create_group_with_wal( + 0, + 1, + 1, + PxLocalReplicaRole::Follower, + &config, + Arc::new(IoBackend::detect()), + CrowdbTreeBackend::File, + ) + .await + .unwrap(); + let wal = group.local_replica().wal().unwrap().clone(); + for slot in 1..=8 { + let entry = PxLogEntry { + slot, + ballot: PxBallot::new(slot, 1), + term: slot, + payload: Bytes::from(vec![u8::try_from(slot).unwrap(); 1024]), + }; + wal.append(&WALRecord::from_accepted(group.group_id(), &entry)) + .await + .unwrap(); + } + wal.seal_all().await.unwrap(); + + let replay = replay_group( + &Arc::new(IoBackend::detect()), + &[store_wal_root(&config.wal_root, 0)], + 1, + ) + .await + .unwrap(); + assert!( + replay.max_segment_id > 1, + "configured 512-byte segments must rotate" + ); +} + +#[tokio::test] +async fn create_group_with_wal_restores_and_resumes_at_next_slot() { + let temp = crowdb_test_harness::test_dirs::tempdir_in_test_data("startup"); + let wal_root = temp.path().join("wal-root"); + let config_root = temp.path().join("conf-root"); + let backend = Arc::new(IoBackend::detect()); + let store_id = 9; + let group_id = 11; + let replica_id = 7; + + let config = WalConfig::with_root(store_wal_root(&wal_root, store_id)); + let wal = WalEngine::create(backend.clone(), config.clone(), group_id) + .await + .unwrap(); + + wal.append(&WALRecord::from_promised( + group_id, + 2, + 1, + PxBallot::new(2, replica_id), + )) + .await + .unwrap(); + + // Replay may apply slot 2 only after a real accepted entry completes slot 1. + // A promise alone does not establish an empty slot. + wal.append(&WALRecord::from_accepted( + group_id, + &PxLogEntry { + slot: 1, + ballot: PxBallot::new(2, replica_id), + term: 2, + payload: Bytes::new(), + }, + )) + .await + .unwrap(); + + let accepted_entry = PxLogEntry { + slot: 2, + ballot: PxBallot::new(3, replica_id), + term: 3, + payload: Bytes::from(encode_put_payload(b"restore-key", b"restore-value")), + }; + wal.append(&WALRecord::from_accepted(group_id, &accepted_entry)) + .await + .unwrap(); + wal.append(&WALRecord::from_vote_granted(group_id, 5, 99)) + .await + .unwrap(); + wal.seal_all().await.unwrap(); + + let data_root = temp.path().join("data-root"); + let config = PaxosKvConfig { + wal_root: wal_root.clone(), + config_root: config_root.clone(), + data_root: data_root.clone(), + ..PaxosKvConfig::for_tests() + }; + let group = create_group_with_wal( + store_id, + group_id, + replica_id, + PxLocalReplicaRole::Leader, + &config, + backend.clone(), + CrowdbTreeBackend::File, + ) + .await + .unwrap(); + + let replica = group.local_replica(); + assert_eq!(replica.current_term(), 5); + assert_eq!(replica.voted_for(), Some(99)); + assert_eq!(replica.accepted_at(2).await, Some(accepted_entry.clone())); + assert_eq!(replica.promised_at(1).await, Some(PxBallot::new(2, replica_id))); + assert_eq!( + replica.learner.engine_get(b"restore-key").await.map(|(_, v)| v), + Some(b"restore-value".to_vec()), + "WAL replay applies accepted slot 2 to the learner" + ); + + replica.become_leader(); + group.stamp_proposing_term(replica.current_term()); + + let result = group + .propose(encode_put_payload(b"new-key", b"new-value"), Some(55), Some(1)) + .await; + match result { + ProposeResult::Chosen { slot } => assert_eq!(slot, 3), + other => panic!("expected chosen proposal after restore, got {other:?}"), + } + + let replay_wal_dir = store_wal_root(&config.wal_root, store_id); + let replay = replay_group(&backend, &[replay_wal_dir], group_id).await.unwrap(); + assert!(replay.records.iter().any(|record| { + record.slot == 3 + && matches!( + record.record_type, + crowdb_paxos_kv::wal::record::RecordType::Accepted + ) + })); +} + +/// Durable crowdb-tree engine end-to-end: a group backed by a durable +/// `CrowdbTreeEngine` file survives a simulated process restart (drop the +/// group, then call `create_group_with_wal` again against the same +/// `wal_root`/`data_root`) with its KV state intact -- via full WAL replay +/// into a fresh `CrowdbTreeEngine::open` at the same file +/// (`PxLocalReplica::restore_from_replay_with_engine`), not by any +/// resume-from-last-applied-slot shortcut (not implemented; see /// #20's note on why that needs separate, careful frontier-seeding work). +/// Parameterized over [`CrowdbTreeBackend`] so the same +/// scenario covers both the explicit file backend and the raw `O_DIRECT` +/// block-device backend. +async fn crowdb_tree_engine_persists_across_restart(crowtree_backend: CrowdbTreeBackend) { + let temp = crowdb_test_harness::test_dirs::tempdir_in_test_data("startup"); + let wal_root = temp.path().join("wal-root"); + let config_root = temp.path().join("conf-root"); + let data_root = temp.path().join("data-root"); + let backend = Arc::new(IoBackend::detect()); + let store_id = 21; + let group_id = 5; + let replica_id = 1; + + let config = PaxosKvConfig { + wal_root: wal_root.clone(), + config_root: config_root.clone(), + data_root: data_root.clone(), + ..PaxosKvConfig::for_tests() + }; + let group = create_group_with_wal( + store_id, + group_id, + replica_id, + PxLocalReplicaRole::Leader, + &config, + backend.clone(), + crowtree_backend, + ) + .await + .unwrap(); + + // The durable crowdb-tree file was created under data_root, not left at the + // default in-memory (no file) path. + let ct_path = + crowdb_paxos_kv_server::recovery::startup::store_crowdb_tree_path(&data_root, store_id, group_id); + assert!( + ct_path.exists(), + "expected a durable crowdb-tree file at {}", + ct_path.display() + ); + + group.local_replica().become_leader(); + group.stamp_proposing_term(group.local_replica().current_term()); + let result = group + .propose(encode_put_payload(b"ct-key", b"ct-value"), Some(1), Some(1)) + .await; + match result { + ProposeResult::Chosen { slot } => assert_eq!(slot, 1), + other => panic!("expected chosen proposal, got {other:?}"), + } + assert_eq!( + group + .local_replica() + .learner + .engine_get(b"ct-key") + .await + .map(|(_, v)| v), + Some(b"ct-value".to_vec()) + ); + + // Simulate a process restart: drop the group (closes the crowdb-tree file + // handle via `Crowdbtree`'s `Drop`), then rebuild from the same WAL + + // crowdb-tree file. + drop(group); + + let restarted = create_group_with_wal( + store_id, + group_id, + replica_id, + PxLocalReplicaRole::Leader, + &config, + backend.clone(), + crowtree_backend, + ) + .await + .unwrap(); + + assert_eq!( + restarted + .local_replica() + .learner + .engine_get(b"ct-key") + .await + .map(|(_, v)| v), + Some(b"ct-value".to_vec()), + "crowdb-tree-backed KV state must survive a simulated restart" + ); +} + +#[tokio::test] +async fn create_group_with_wal_crowdb_tree_engine_persists_across_restart() { + crowdb_tree_engine_persists_across_restart(CrowdbTreeBackend::File).await; +} + +/// : same scenario, through `BlockPageStore` (`O_DIRECT`) +/// instead of the default `FilePageStore`. +#[tokio::test] +async fn create_group_with_wal_crowdb_tree_block_backend_persists_across_restart() { + crowdb_tree_engine_persists_across_restart(CrowdbTreeBackend::Block).await; +} diff --git a/app/crowdb-kv-server/tests/startup_test/reclamation.rs b/app/crowdb-paxos-kv-server/tests/startup_test/reclamation.rs similarity index 96% rename from app/crowdb-kv-server/tests/startup_test/reclamation.rs rename to app/crowdb-paxos-kv-server/tests/startup_test/reclamation.rs index b999207c7..7ff16a367 100644 --- a/app/crowdb-kv-server/tests/startup_test/reclamation.rs +++ b/app/crowdb-paxos-kv-server/tests/startup_test/reclamation.rs @@ -2,16 +2,16 @@ // Licensed under the Apache License, Version 2.0. use super::*; -use crowdb_kv::wal::gc::run_gc_pass; +use crowdb_paxos_kv::wal::gc::run_gc_pass; #[tokio::test] async fn restored_wal_keeps_old_segments_visible_to_safe_reclamation() { let temp = crowdb_test_harness::test_dirs::tempdir_in_test_data("startup-wal-reclamation"); - let config = CrowDBConfig { + let config = PaxosKvConfig { wal_root: temp.path().join("wal"), config_root: temp.path().join("conf"), data_root: temp.path().join("data"), - ..CrowDBConfig::for_tests() + ..PaxosKvConfig::for_tests() }; let backend = Arc::new(IoBackend::File); let mut wal_config = WalConfig::with_root(store_wal_root(&config.wal_root, 0)); diff --git a/app/crowdb-paxos-kv-server/tests/tree_config_test.rs b/app/crowdb-paxos-kv-server/tests/tree_config_test.rs new file mode 100644 index 000000000..a1c550f06 --- /dev/null +++ b/app/crowdb-paxos-kv-server/tests/tree_config_test.rs @@ -0,0 +1,103 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use crowdb_paxos_kv::common::config::PaxosKvConfig; +use crowdb_tree_ffi::{Config, Crowdbtree, CtError}; + +const CANONICAL: &str = include_str!("../conf/crowdb_paxos_kv_server_config.toml"); + +fn load(text: &str) -> Result<PaxosKvConfig, String> { + let root = crowdb_test_harness::test_dirs::tempdir_in_test_data("tree-config"); + let path = root.path().join("server.toml"); + std::fs::write(&path, text).unwrap(); + PaxosKvConfig::load_from_file(&path) +} + +#[test] +fn canonical_config_preserves_engine_and_runtime_defaults() { + let config = load(CANONICAL).unwrap(); + let defaults = PaxosKvConfig::default(); + assert_eq!( + serde_json::to_value(&config.tree).unwrap(), + serde_json::to_value(&defaults.tree).unwrap() + ); + let host = config.tree_runtime.runtime_options(); + let expected = defaults.tree_runtime.runtime_options(); + assert_eq!(host.workers, expected.workers); + assert_eq!(host.backend_workers, expected.backend_workers); + assert_eq!(host.backend_queue_limit, expected.backend_queue_limit); + assert_eq!(host.limits.reclaim_bytes, expected.limits.reclaim_bytes); + assert_eq!( + config.tree_runtime.binding_options().backend_queue_limit, + defaults.tree_runtime.binding_options().backend_queue_limit + ); + assert!(config.wal_early_ack); + assert!(config.async_engine_apply); +} + +#[test] +fn toml_key_limit_is_enforced_by_native_admission() { + let config = load(&CANONICAL.replace("max_key_size = 0", "max_key_size = 4")).unwrap(); + let mut options = Config::default(); + config.tree.apply_to(&mut options); + let tree = Crowdbtree::open(&options).unwrap(); + assert_eq!(tree.put(b"abcde", b"value"), Err(CtError::InvalidArgument)); + tree.put(b"abcd", b"value").unwrap(); + assert_eq!( + tree.get(b"abcd").unwrap().map(|(_, value)| value), + Some(b"value".to_vec()) + ); +} + +#[test] +fn enlarged_source_capacity_is_included_in_scan_admission() { + let config = + load(&CANONICAL.replace("source_max_entries = 4_096", "source_max_entries = 8_192")).unwrap(); + let mut options = Config::default(); + config.tree.apply_to(&mut options); + let tree = Crowdbtree::open(&options).unwrap(); + for index in 0..4200 { + tree.put(format!("key{index:05}").as_bytes(), b"value").unwrap(); + } + assert_eq!(tree.frozen_table_count(), 0); + let (entries, truncated) = tree.scan(b"", b"", b"", 0, 1024 * 1024, true, 0, false).unwrap(); + assert!(!truncated); + assert_eq!(entries.len(), 4200); + assert_eq!(entries.first().unwrap().key.as_ref(), b"key00000"); + assert_eq!(entries.last().unwrap().key.as_ref(), b"key04199"); +} + +#[test] +fn incompatible_or_unknown_work_windows_are_rejected_on_load() { + for (input, field) in [ + ( + CANONICAL.replace("flush_parallelism = 0", "flush_parallelism = 5"), + "flush_parallelism", + ), + ( + CANONICAL.replace("cache_max_pages = 256", "cache_max_pages = 0"), + "cache_max_pages", + ), + ( + CANONICAL.replace("source_max_gap_entries = 1_024", "source_max_gap_entries = 16"), + "coalesce_max_keys", + ), + ( + CANONICAL.replace("run_max_entries = 128", "run_entries_typo = 128"), + "run_entries_typo", + ), + ] { + assert!(load(&input).unwrap_err().contains(field), "{field}"); + } +} + +#[test] +fn partial_tables_keep_omitted_defaults() { + let config = load("[paxos]\ncoalesce_max_keys = 16\n[election]\nmaintenance_tick_ms = 3000\n[wal]\nwal_flush_batch_bytes = 131072\n").unwrap(); + assert_eq!(config.paxos.coalesce_max_keys, 16); + assert_eq!(config.paxos.max_inflight_proposals, 32); + assert_eq!(config.election.snapshot_time_threshold_ms, 600_000); + assert_eq!(config.wal.wal_segment_size, 64 * 1024 * 1024); + assert!(config.wal_early_ack); + assert!(config.async_engine_apply); +} diff --git a/app/crowdb-paxos-kv-server/tests/tree_runtime_host_test.rs b/app/crowdb-paxos-kv-server/tests/tree_runtime_host_test.rs new file mode 100644 index 000000000..1e534f3f9 --- /dev/null +++ b/app/crowdb-paxos-kv-server/tests/tree_runtime_host_test.rs @@ -0,0 +1,96 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +use std::sync::Arc; + +use bytes::Bytes; +use crowdb_paxos_kv::cluster::PxLocalReplicaRole; +use crowdb_paxos_kv::common::config::PaxosKvConfig; +use crowdb_paxos_kv_server::recovery::startup::{create_group_with_wal, create_group_with_wal_on_runtime}; +use crowdb_paxos_kv_server::store_registry::PaxosKvStoreRegistry; + +fn config(root: &std::path::Path) -> PaxosKvConfig { + let mut config = PaxosKvConfig { + wal_backend: "mem-block".to_owned(), + crowtree_backend: "mem-block".to_owned(), + wal_root: root.join("wal"), + config_root: root.join("config"), + data_root: root.join("tree"), + log_dir: root.join("log").to_string_lossy().into_owned(), + ..Default::default() + }; + config.tree_runtime.host.workers = Some(1); + config +} + +#[test] +fn incompatible_proposal_window_is_rejected_before_startup() { + let mut config = PaxosKvConfig::default(); + config.tree_runtime.tree.cpu_queue_limit = Some(16); + let Err(error) = PaxosKvStoreRegistry::try_with_config(config) else { + panic!("proposal window must fit its paid Tree lane"); + }; + assert_eq!(error.kind(), std::io::ErrorKind::InvalidInput); + assert!(error.to_string().contains("max_inflight_proposals")); +} + +#[tokio::test(flavor = "current_thread")] +async fn standalone_constructor_uses_runtime_without_a_mode_flag() { + let root = crowdb_test_harness::test_dirs::tempdir_in_test_data("tree-runtime-host"); + let registry = PaxosKvStoreRegistry::try_with_config(config(root.path())).unwrap(); + let result = create_group_with_wal( + 1, + 1, + 1, + PxLocalReplicaRole::Leader, + ®istry.config, + Arc::clone(®istry.wal_backend), + registry.crowtree_backend, + ) + .await; + let group = result.unwrap(); + let engine = group.local_replica().learner.engine(); + assert!(engine.requires_contiguous_apply()); + let permit = engine.admit_proposal(Bytes::new()).await.unwrap().unwrap(); + permit.apply(1).await.unwrap(); +} + +#[tokio::test(flavor = "current_thread")] +async fn every_group_uses_the_same_registry_owned_native_host() { + let root = crowdb_test_harness::test_dirs::tempdir_in_test_data("tree-runtime-host"); + let registry = PaxosKvStoreRegistry::try_with_config(config(root.path())).unwrap(); + let mut groups = Vec::new(); + for group_id in 1..=2 { + groups.push( + create_group_with_wal_on_runtime( + 1, + group_id, + 1, + PxLocalReplicaRole::Leader, + ®istry.config, + Arc::clone(®istry.wal_backend), + registry.crowtree_backend, + registry.tree_runtime.clone(), + ) + .await + .unwrap(), + ); + } + for group in &groups { + let engine = group.local_replica().learner.engine(); + assert!(engine.requires_contiguous_apply()); + let permit = engine.admit_proposal(Bytes::new()).await.unwrap().unwrap(); + permit.apply(1).await.unwrap(); + } + registry.tree_runtime.stop_admission(); + for group in &groups { + // This one host transition must be visible in both independent Trees. + assert!(group + .local_replica() + .learner + .engine() + .admit_proposal(Bytes::new()) + .await + .is_err()); + } +} diff --git a/app/crowdb-web/Cargo.toml b/app/crowdb-web/Cargo.toml index 67c848a14..f22c271f8 100644 --- a/app/crowdb-web/Cargo.toml +++ b/app/crowdb-web/Cargo.toml @@ -23,7 +23,7 @@ crowdb-chunkdb-client = { path = "../../lib/crowdb-chunkdb-client" } crowdb-common = { path = "../../lib/crowdb-common/rust" } crowdb-console-shared = { path = "../../lib/crowdb-console-shared" } crowdb-diskdb-client = { path = "../../lib/crowdb-diskdb-client" } -crowdb-kv-client = { path = "../../lib/crowdb-kv-client" } +crowdb-paxos-kv-client = { path = "../../lib/crowdb-paxos-kv-client" } crowdb-monitor = { path = "../../container/crowdb-monitor" } crowdb-protocol = { path = "../../lib/crowdb-protocol" } bincode = "1" @@ -56,6 +56,6 @@ tokio = { workspace = true, features = ["rt-multi-thread", "macros", "time", "ne reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json"] } serde_json = "1" tower = { version = "0.5", features = ["util"] } -crowdb-kv = { path = "../../lib/crowdb-kv" } +crowdb-paxos-kv = { path = "../../lib/crowdb-paxos-kv" } crowdb-rpc-ffi = { path = "../../lib/crowdb-rpc/ffi", features = ["test-util"] } crowdb-test-harness = { path = "../../lib/crowdb-test-harness" } diff --git a/app/crowdb-web/src/chunk.rs b/app/crowdb-web/src/chunk.rs index 3249407fa..13036e054 100644 --- a/app/crowdb-web/src/chunk.rs +++ b/app/crowdb-web/src/chunk.rs @@ -14,7 +14,7 @@ use axum::http::StatusCode; use axum::Json; use crowdb_chunkdb_client::ChunkdbClient; use crowdb_console_shared::ops::hardware; -use crowdb_kv_client::{RangeBindingClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{RangeBindingClient, ServiceRegistryClient}; use crowdb_protocol::chunkdb::rpc::{Chunk, ListChunksRequest, QueryChunkRequest}; use crowdb_protocol::common::ChunkId; use serde::Deserialize; diff --git a/app/crowdb-web/src/chunk/slots.rs b/app/crowdb-web/src/chunk/slots.rs index edbb34239..1c19b973d 100644 --- a/app/crowdb-web/src/chunk/slots.rs +++ b/app/crowdb-web/src/chunk/slots.rs @@ -4,7 +4,7 @@ //! Bounded owner-specific windows over validated fixed slot maps. use axum::extract::{Query, State}; -use crowdb_kv_client::ChunkSlotMapClient; +use crowdb_paxos_kv_client::ChunkSlotMapClient; use crowdb_protocol::chunk_slot::{ChunkSlotBitmap, ChunkStorageGroup, CHUNK_SLOT_COUNT}; use serde::Deserialize; use serde_json::json; diff --git a/app/crowdb-web/src/chunk_kv.rs b/app/crowdb-web/src/chunk_kv.rs index 43a516434..efe2d893a 100644 --- a/app/crowdb-web/src/chunk_kv.rs +++ b/app/crowdb-web/src/chunk_kv.rs @@ -14,7 +14,7 @@ use axum::{ http::StatusCode, Json, }; -use crowdb_kv_client::{CrowdbKvClient, GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{GetOutcome, PaxosKvClient, ReadMode}; use crowdb_protocol::chunk_kv::{ChunkKvRangeCatalogHead, ChunkKvRangeCatalogPage}; use crowdb_protocol::key::{ChunkKvRangeCatalogHeadKey, ChunkKvRangeCatalogPageKey, TextKey}; use serde::{de::DeserializeOwned, Deserialize}; @@ -48,7 +48,7 @@ pub(crate) async fn catalog( .map_err(|_| err_502("Chunk-KV catalog observation timed out"))? } -async fn read<T: DeserializeOwned>(kv: &CrowdbKvClient, key: &str, max_bytes: usize) -> Result<T, Failure> { +async fn read<T: DeserializeOwned>(kv: &PaxosKvClient, key: &str, max_bytes: usize) -> Result<T, Failure> { let value = match kv .get(0, 0, key.as_bytes(), ReadMode::Linearizable, None) .await diff --git a/app/crowdb-web/src/chunk_kv/balance.rs b/app/crowdb-web/src/chunk_kv/balance.rs index 8aac86b6f..5cc015209 100644 --- a/app/crowdb-web/src/chunk_kv/balance.rs +++ b/app/crowdb-web/src/chunk_kv/balance.rs @@ -3,7 +3,7 @@ //! Read the monitor's bounded diagnostic snapshot without scanning catalog/data. -use crowdb_kv_client::{CrowdbKvClient, GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{GetOutcome, PaxosKvClient, ReadMode}; use crowdb_protocol::chunk_kv::{ balance::{BalanceObservation, MAX_OBSERVATION_BYTES, OBSERVATION_KEY}, ChunkKvRangeCatalogEntry, @@ -16,7 +16,7 @@ pub(super) struct Observation { reason: String, } -pub(super) async fn observe(kv: &CrowdbKvClient, generation: u64) -> Observation { +pub(super) async fn observe(kv: &PaxosKvClient, generation: u64) -> Observation { match load(kv, generation).await { Ok(value) => Observation { reason: value.reason.clone(), @@ -26,7 +26,7 @@ pub(super) async fn observe(kv: &CrowdbKvClient, generation: u64) -> Observation } } -async fn load(kv: &CrowdbKvClient, generation: u64) -> Result<BalanceObservation, String> { +async fn load(kv: &PaxosKvClient, generation: u64) -> Result<BalanceObservation, String> { let outcome = kv .get(0, 0, OBSERVATION_KEY.as_bytes(), ReadMode::Linearizable, None) .await diff --git a/app/crowdb-web/src/chunk_kv/runtime/discovery.rs b/app/crowdb-web/src/chunk_kv/runtime/discovery.rs index 68adcaa0f..92fc1f394 100644 --- a/app/crowdb-web/src/chunk_kv/runtime/discovery.rs +++ b/app/crowdb-web/src/chunk_kv/runtime/discovery.rs @@ -6,7 +6,7 @@ use std::time::{SystemTime, UNIX_EPOCH}; use crowdb_console_shared::config::ServiceType; -use crowdb_kv_client::{GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{GetOutcome, ReadMode}; use crowdb_protocol::{common::InstanceValue, key::InstanceKey}; use super::super::Failure; diff --git a/app/crowdb-web/src/corr_id.rs b/app/crowdb-web/src/corr_id.rs index 7d5d56188..7a1ceea21 100644 --- a/app/crowdb-web/src/corr_id.rs +++ b/app/crowdb-web/src/corr_id.rs @@ -1,14 +1,14 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -//! Axum middleware that propagates the `x-crowdb-kv-corr-id` header. +//! Axum middleware that propagates the `x-crowdb-paxos-kv-corr-id` header. //! //! For every inbound request: -//! 1. Read `x-crowdb-kv-corr-id` from the request, or generate a new one +//! 1. Read `x-crowdb-paxos-kv-corr-id` from the request, or generate a new one //! via `shared::corr_id::new()` if absent. //! 2. Open a `shared::corr_id::scope` so any outbound client call made //! inside the handler attaches the same id to its -//! `x-crowdb-kv-corr-id` header (see `ConsoleClient` / +//! `x-crowdb-paxos-kv-corr-id` header (see `ConsoleClient` / //! `ServerClient` plumbing). //! 3. Add the id to the response headers so the SPA / CLI can echo it //! in user-facing error messages and operators can grep one log diff --git a/app/crowdb-web/src/diskdb/lifecycle.rs b/app/crowdb-web/src/diskdb/lifecycle.rs index 09c77c58c..d38438157 100644 --- a/app/crowdb-web/src/diskdb/lifecycle.rs +++ b/app/crowdb-web/src/diskdb/lifecycle.rs @@ -20,15 +20,9 @@ use crowdb_console_shared::lifecycle::{self, DiskdbDeployRequest}; #[derive(Debug, Deserialize)] pub struct DeployDiskdbBody { pub rpc_port: u16, - /// Main listener port. When omitted, derived from `rpc_port`. - #[serde(default)] - pub listen_port: Option<u16>, - /// Internal HTTP health port. When omitted, derived from `rpc_port`. + /// HTTP listener; omitted values use the protocol default. #[serde(default)] pub http_port: Option<u16>, - /// Explicit RPC listener for policies with independent listener ranges. - #[serde(default)] - pub rpc_listen_port: Option<u16>, /// Normal `DiskDB` heartbeat cadence, retained across service restart. #[serde(default)] pub keepalive_interval_secs: Option<std::num::NonZeroU32>, @@ -36,24 +30,16 @@ pub struct DeployDiskdbBody { pub(super) fn validate_diskdb_ports( body: &DeployDiskdbBody, -) -> Result<(u16, u16, u16), (StatusCode, Json<ErrorBody>)> { - let listen_port = body.listen_port.unwrap_or(body.rpc_port); - let http_port = body.http_port.unwrap_or_else(|| body.rpc_port.saturating_add(1)); - let rpc_listen_port = body - .rpc_listen_port - .unwrap_or_else(|| body.rpc_port.saturating_add(2)); - let ports = [listen_port, http_port, rpc_listen_port]; - if ports.contains(&0) || (body.rpc_listen_port.is_none() && body.rpc_port > 65_533) { - return Err(err_400("DiskDB ports must be in the range 1..=65535")); - } - if ports[0] == ports[1] || ports[0] == ports[2] || ports[1] == ports[2] { - return Err(err_400("DiskDB listener ports must be distinct")); +) -> Result<(u16, u16), (StatusCode, Json<ErrorBody>)> { + let http = body.http_port.unwrap_or(crowdb_protocol::DISKDB_HTTP_BASE); + if http == 0 || body.rpc_port == 0 || http == body.rpc_port { + return Err(err_400("DiskDB HTTP and RPC ports must be distinct and nonzero")); } - Ok((listen_port, http_port, rpc_listen_port)) + Ok((http, body.rpc_port)) } #[cfg(target_os = "macos")] -fn ensure_local_ports_free(ports: [u16; 3]) -> Result<(), (StatusCode, Json<ErrorBody>)> { +fn ensure_local_ports_free(ports: [u16; 2]) -> Result<(), (StatusCode, Json<ErrorBody>)> { for port in ports { if std::net::TcpListener::bind(("127.0.0.1", port)).is_err() { return Err(err_409(format!("DiskDB listener port {port} is unavailable"))); @@ -105,10 +91,10 @@ async fn deploy_diskdb( "Group 0 is not ready; deploy Paxos-KV and initialize Group 0 before starting DiskDB", )); } - let (listen_port, http_port, rpc_listen_port) = validate_diskdb_ports(&body)?; + let (http_port, rpc_listen_port) = validate_diskdb_ports(&body)?; #[cfg(target_os = "macos")] - ensure_local_ports_free([listen_port, http_port, rpc_listen_port])?; - let _ports = crate::services::defaults::claim_ports(&state, &[listen_port, http_port, rpc_listen_port])?; + ensure_local_ports_free([http_port, rpc_listen_port])?; + let _ports = crate::services::defaults::claim_ports(&state, &[http_port, rpc_listen_port])?; let node = { let cfg = state.config.read().unwrap(); // Check for existing diskdb instance on this node. @@ -155,7 +141,6 @@ async fn deploy_diskdb( free_batch_enabled: None, free_flush_max_batch: None, server_id: format!("diskdb-{node_id}"), - listen_port, http_port, rpc_port: rpc_listen_port, kv_server_mgmt_seeds, @@ -176,7 +161,7 @@ async fn deploy_diskdb( url: deployed.endpoint.clone(), node_id: Some(node_id), rpc_url: Some(deployed.endpoint.clone()), - rest_port: None, + rest_port: Some(http_port), rpc_port: Some(body.rpc_port), auto_start: true, binary: None, @@ -247,8 +232,10 @@ async fn restart_diskdb( let (entry, node) = diskdb_restart_inputs(&state, node_id)?; let rpc_port = entry - .rpc_port - .or_else(|| entry.rpc_url.as_deref().and_then(crate::mgmt::port_of)) + .rpc_url + .as_deref() + .and_then(crate::mgmt::port_of) + .or(entry.rpc_port) .ok_or_else(|| err_500(format!("diskdb entry has malformed rpc_url: {:?}", entry.rpc_url)))?; // Stop existing process. @@ -269,10 +256,21 @@ async fn restart_diskdb( .collect() }; - // Backward-compat: derive from the old paired-port scheme. - let listen_port = rpc_port; - let http_port = rpc_port.saturating_add(1); - let rpc_listen_port = rpc_port.saturating_add(2); + // Retain the configured listeners across restart. + let (http_port, rpc_listen_port) = validate_diskdb_ports(&DeployDiskdbBody { + rpc_port, + http_port: entry.rest_port.or_else(|| { + state + .config + .read() + .unwrap() + .local_launches + .get(&entry.id) + .and_then(|launch| launch.readiness_url.as_deref()) + .and_then(crate::mgmt::port_of) + }), + keepalive_interval_secs: None, + })?; let req = DiskdbDeployRequest { instance_id: None, metrics_interval: None, @@ -283,7 +281,6 @@ async fn restart_diskdb( free_batch_enabled: None, free_flush_max_batch: None, server_id: format!("diskdb-{node_id}"), - listen_port, http_port, rpc_port: rpc_listen_port, kv_server_mgmt_seeds, @@ -303,7 +300,7 @@ async fn restart_diskdb( url: deployed.endpoint.clone(), node_id: Some(node_id), rpc_url: Some(deployed.endpoint.clone()), - rest_port: None, + rest_port: Some(http_port), rpc_port: Some(rpc_port), auto_start: entry.auto_start, binary: None, diff --git a/app/crowdb-web/src/diskdb/proxy.rs b/app/crowdb-web/src/diskdb/proxy.rs index 63097731b..6d5297941 100644 --- a/app/crowdb-web/src/diskdb/proxy.rs +++ b/app/crowdb-web/src/diskdb/proxy.rs @@ -14,7 +14,7 @@ use serde::{Deserialize, Serialize}; use tracing::warn; use crowdb_diskdb_client::{DiskdbClient, DiskdbRpcTransport}; -use crowdb_kv_client::ServiceRegistryClient; +use crowdb_paxos_kv_client::ServiceRegistryClient; use crowdb_protocol::common::{DiskGroupUsageSummary, DiskdbExtra, HwStatus, InstanceValue}; use crowdb_protocol::common_type::InstanceId; use crowdb_protocol::diskdb::rpc::{ @@ -354,7 +354,7 @@ pub async fn http_diskdb_usage( /// Returns `502` if group-0 is not available. pub async fn http_hardware_capacity( State(state): State<AppState>, -) -> Result<Json<crowdb_kv_client::HardwareCapacitySummary>, (StatusCode, Json<ErrorBody>)> { +) -> Result<Json<crowdb_paxos_kv_client::HardwareCapacitySummary>, (StatusCode, Json<ErrorBody>)> { let hw = build_hardware_client(&state) .await .ok_or_else(|| err_502("no group-0 endpoint; cluster not initialized"))?; @@ -612,7 +612,7 @@ pub async fn http_set_disk_group_owner( hw.set_owner(rack_id, node_id, dg_id, instance_id, body.lease_expiry_ms) .await .map_err(|e| match e { - crowdb_kv_client::Error::OwnerConflict { .. } => ( + crowdb_paxos_kv_client::Error::OwnerConflict { .. } => ( StatusCode::CONFLICT, Json(ErrorBody { error: format!("set_owner: {e}"), diff --git a/app/crowdb-web/src/diskdb/remote.rs b/app/crowdb-web/src/diskdb/remote.rs index 542aa022c..145370260 100644 --- a/app/crowdb-web/src/diskdb/remote.rs +++ b/app/crowdb-web/src/diskdb/remote.rs @@ -15,7 +15,7 @@ pub(super) async fn deploy( node_id: u64, body: &DeployDiskdbBody, ) -> Result<(StatusCode, Json<DiskdbDeployResult>), crate::services::Failure> { - let (_, _, rpc) = super::lifecycle::validate_diskdb_ports(body)?; + let (_, rpc) = super::lifecycle::validate_diskdb_ports(body)?; let binding = crate::node::cluster_binding(state)?.ok_or_else(|| err_409("Node is unbound"))?; let mut node = state .config @@ -65,9 +65,9 @@ pub fn configuration( seeds: &[String], body: &DeployDiskdbBody, ) -> Result<String, crate::services::Failure> { - let (listen, http, rpc) = super::lifecycle::validate_diskdb_ports(body)?; - let mut configuration = format!("[server]\nrpc_workers = 2\nlisten_addr = {:?}\nhttp_listen_addr = {:?}\nrpc_listen_addr = {:?}\ninstance_id = {:?}\nkv_server_mgmt_seeds = {:?}\n", - format!("{}:{listen}", node.host), format!("{}:{http}", node.host), format!("{}:{rpc}", node.host), node.id.to_string(), seeds); + let (http, rpc) = super::lifecycle::validate_diskdb_ports(body)?; + let mut configuration = format!("[server]\nrpc_workers = 2\nhttp_listen_addr = {:?}\nrpc_listen_addr = {:?}\ninstance_id = {:?}\nkv_server_mgmt_seeds = {:?}\n", + format!("{}:{http}", node.host), format!("{}:{rpc}", node.host), node.id.to_string(), seeds); if let Some(interval) = body.keepalive_interval_secs { writeln!(configuration, "\n[heartbeat]\ninterval_secs = {interval}") .expect("writing configuration to a String cannot fail"); diff --git a/app/crowdb-web/src/error.rs b/app/crowdb-web/src/error.rs index eaeb2382c..31f02d7f7 100644 --- a/app/crowdb-web/src/error.rs +++ b/app/crowdb-web/src/error.rs @@ -59,7 +59,7 @@ pub fn map_config_err(e: Error) -> (StatusCode, Json<ErrorBody>) { Error::UpstreamRpc { ref status, .. } if status.contains("HTTP 409") => { (StatusCode::CONFLICT, Json(ErrorBody { error: msg })) } - Error::KvClient(crowdb_kv_client::Error::MembershipConflict { .. }) => { + Error::KvClient(crowdb_paxos_kv_client::Error::MembershipConflict { .. }) => { (StatusCode::CONFLICT, Json(ErrorBody { error: msg })) } Error::NotFound { .. } => (StatusCode::NOT_FOUND, Json(ErrorBody { error: msg })), diff --git a/app/crowdb-web/src/kv.rs b/app/crowdb-web/src/kv.rs index b403cd05f..f1a62a4c3 100644 --- a/app/crowdb-web/src/kv.rs +++ b/app/crowdb-web/src/kv.rs @@ -11,7 +11,7 @@ use axum::http::StatusCode; use axum::Json; use crowdb_console_shared::cluster::{GroupHealth, NodeId}; use crowdb_console_shared::ops; -use crowdb_kv_client::{GetOutcome, ScanOutcome}; +use crowdb_paxos_kv_client::{GetOutcome, ScanOutcome}; use hex; use serde::{Deserialize, Serialize}; use std::collections::{HashMap, HashSet}; diff --git a/app/crowdb-web/src/lib.rs b/app/crowdb-web/src/lib.rs index 510e09246..c1c6af70d 100644 --- a/app/crowdb-web/src/lib.rs +++ b/app/crowdb-web/src/lib.rs @@ -476,7 +476,7 @@ pub fn router(state: AppState) -> axum::Router { health::require_editable, )) .with_state(state) - // Propagate `x-crowdb-kv-corr-id` through every request: read it + // Propagate `x-crowdb-paxos-kv-corr-id` through every request: read it // (or mint one), open a task-local scope so outbound clients // attach it to their own headers, echo it back on the response. .layer(axum::middleware::from_fn(corr_id::corr_id_layer)) diff --git a/app/crowdb-web/src/lifecycle.rs b/app/crowdb-web/src/lifecycle.rs index 0d54dec0a..0bc59a807 100644 --- a/app/crowdb-web/src/lifecycle.rs +++ b/app/crowdb-web/src/lifecycle.rs @@ -220,7 +220,7 @@ pub async fn http_remove_node( let t0 = std::time::Instant::now(); // Cascade-stop the server process and drop its deployment record + // topology before removing the node, so a direct DELETE /api/nodes/:id - // does not orphan a running crowdb-kv-server. No-op when no server is + // does not orphan a running crowdb-paxos-kv-server. No-op when no server is // deployed (e.g. the UI already called DELETE .../server first). stop_and_remove_server_for_node(&state, id).await; let t_stop = t0.elapsed().as_millis(); @@ -620,7 +620,7 @@ pub async fn http_stop_node_server( /// deployed on the node. Does NOT persist — the caller persists. Does NOT /// remove the node itself. Used by both `DELETE /api/nodes/:id/server` /// and the node-delete cascade so a direct `DELETE /api/nodes/:id` does -/// not orphan a running crowdb-kv-server. +/// not orphan a running crowdb-paxos-kv-server. /// /// # Panics /// Panics if the `RwLock` is poisoned. @@ -945,13 +945,13 @@ async fn shutdown_kv_data(state: &AppState) -> Vec<String> { tracing::warn!(rack_id = rid, error = %e, "reset: remove_rack_cascade failed"); } } - let meta = crowdb_kv_client::KVClusterMetaClient::from_shared(hw.shared_kv()); + let meta = crowdb_paxos_kv_client::PaxosKvClusterMetaClient::from_shared(hw.shared_kv()); for sid in &stores { if let Err(e) = meta.remove_store(*sid).await { tracing::warn!(store_id = sid, error = %e, "reset: remove_store from sysdata failed"); } } - let svc = crowdb_kv_client::ServiceRegistryClient::from_shared(hw.shared_kv()); + let svc = crowdb_paxos_kv_client::ServiceRegistryClient::from_shared(hw.shared_kv()); match svc.read_all_diskdb_instances().await { Ok(instances) => { for (instance_id, _) in &instances { diff --git a/app/crowdb-web/src/lifecycle/kv_process.rs b/app/crowdb-web/src/lifecycle/kv_process.rs index a5f8f360f..8a8694825 100644 --- a/app/crowdb-web/src/lifecycle/kv_process.rs +++ b/app/crowdb-web/src/lifecycle/kv_process.rs @@ -62,10 +62,10 @@ pub struct DeployNodeServerBody { /// `--send-queue-capacity` value for per-connection send queue. #[serde(default)] send_queue_capacity: Option<u32>, - /// Optional `--config` JSON path passed to the spawned `crowdb-kv-server`. + /// Optional `--config` JSON path passed to the spawned `crowdb-paxos-kv-server`. #[serde(default)] config: Option<String>, - /// `--rpc-workers` value for the spawned `crowdb-kv-server`. + /// `--rpc-workers` value for the spawned `crowdb-paxos-kv-server`. #[serde(default)] rpc_workers: Option<u32>, } @@ -105,7 +105,7 @@ pub async fn http_get_node_server( Ok(Json(entry)) } -/// `POST /api/nodes/:node_id/server/deploy`. Spawn `crowdb-kv-server` on +/// `POST /api/nodes/:node_id/server/deploy`. Spawn `crowdb-paxos-kv-server` on /// the node (local fork for `ssh_user=""`, SSH otherwise), wait for /// health, persist the deployment record. /// @@ -179,9 +179,9 @@ async fn deploy_node_server( } /// `POST /api/nodes/:node_id/server/restart`. Stop the tracked -/// `crowdb-kv-server` process on this node (if any) and immediately +/// `crowdb-paxos-kv-server` process on this node (if any) and immediately /// re-deploy on the same ports recorded in the `ServerEntry`. The -/// binary path falls back to `CROWDB_KV_SERVER_BIN` / `"crowdb-kv-server"` +/// binary path falls back to `CROWDB_PAXOS_KV_SERVER_BIN` / `"crowdb-paxos-kv-server"` /// the same way the initial deploy does when no `binary` override /// is supplied. Returns the new `DeployResult`. /// @@ -313,7 +313,7 @@ fn deployment_request( group0_management_seeds: (*state.authority_seeds).clone(), election_profile: body .election_profile - .or_else(|| std::env::var("CROWDB_KV_SERVER_ELECTION_PROFILE").ok()), + .or_else(|| std::env::var("CROWDB_PAXOS_KV_SERVER_ELECTION_PROFILE").ok()), binary: body.binary.map(std::path::PathBuf::from), kv_backend: body.kv_backend, wal_backend: body.wal_backend, diff --git a/app/crowdb-web/src/main.rs b/app/crowdb-web/src/main.rs index 89e4eab1a..e418e4412 100644 --- a/app/crowdb-web/src/main.rs +++ b/app/crowdb-web/src/main.rs @@ -17,7 +17,7 @@ struct Args { #[arg(long, conflicts_with = "config")] bind: Option<String>, - /// Port for the web server (default: 9090) + /// Port for the web server (default: 6135) #[arg(long, conflicts_with = "config", value_parser = clap::value_parser!(u16).range(1..))] port: Option<u16>, @@ -49,7 +49,7 @@ struct Args { #[arg(long, conflicts_with = "config")] skip_startup_restore: bool, - /// Log directory. Default: ~/.crowdb-kv/log. + /// Log directory. Default: ~/.crowdb-paxos-kv/log. #[arg(long, conflicts_with = "config")] log_dir: Option<std::path::PathBuf>, @@ -90,14 +90,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> { let launch_registry = args.registry.as_deref().map(LaunchRegistry::load).transpose()?; let _log_guards = init_logging(&args, process_config.as_ref())?; - let bind = process_config.as_ref().map_or_else( - || args.bind.as_deref().unwrap_or("0.0.0.0"), - |config| config.bind.as_str(), - ); - let port = process_config - .as_ref() - .map_or_else(|| args.port.unwrap_or(9090), |config| config.port); - let addr: SocketAddr = format!("{bind}:{port}").parse()?; + let addr = listen_address(&args, process_config.as_ref())?; info!(%addr, "crowdb-web starting"); let mut test_namespace = if args.test_mode { @@ -183,6 +176,21 @@ async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> { Ok(()) } +fn listen_address( + args: &Args, + process_config: Option<&WebProcessConfig>, +) -> Result<SocketAddr, std::net::AddrParseError> { + let bind = process_config.map_or_else( + || args.bind.as_deref().unwrap_or("0.0.0.0"), + |config| config.bind.as_str(), + ); + let port = process_config.map_or_else( + || args.port.unwrap_or(crowdb_protocol::WEB_BASE), + |config| config.port, + ); + format!("{bind}:{port}").parse() +} + fn init_logging( args: &Args, process_config: Option<&WebProcessConfig>, diff --git a/app/crowdb-web/src/managed.rs b/app/crowdb-web/src/managed.rs index 9a30d54d8..cd22b56ff 100644 --- a/app/crowdb-web/src/managed.rs +++ b/app/crowdb-web/src/managed.rs @@ -6,8 +6,8 @@ use axum::extract::State; use axum::http::StatusCode; use axum::Json; use crowdb_console_shared::config::web::WebMode; -use crowdb_kv_client::CrowdbSysmdClient; use crowdb_monitor::{MonitorStatus, ServiceStatus, StatusStore}; +use crowdb_paxos_kv_client::CrowdbSysmdClient; use crowdb_protocol::common::{ReplicaValue, StoreValue}; use serde::Serialize; use serde_json::{json, Value}; @@ -110,7 +110,7 @@ async fn validate_live_store_nodes( sysmd: &CrowdbSysmdClient, stores: &[StoreValue], replicas: &[ReplicaValue], -) -> Result<(), crowdb_kv_client::Error> { +) -> Result<(), crowdb_paxos_kv_client::Error> { let mut live_nodes = BTreeSet::new(); for (_, instance) in sysmd.read_all_kv_server_instances().await? { let Some(node_id) = instance @@ -122,7 +122,7 @@ async fn validate_live_store_nodes( continue; }; if instance.rpc_endpoint.is_empty() || !live_nodes.insert(node_id) { - return Err(crowdb_kv_client::Error::Topology( + return Err(crowdb_paxos_kv_client::Error::Topology( "live KV management registration is ambiguous".into(), )); } @@ -133,7 +133,7 @@ async fn validate_live_store_nodes( .chain(replicas.iter().map(|replica| &replica.node_id)) .any(|node_id| !live_nodes.contains(node_id)) { - return Err(crowdb_kv_client::Error::Topology( + return Err(crowdb_paxos_kv_client::Error::Topology( "store node has no live KV management registration".into(), )); } @@ -164,7 +164,7 @@ async fn load_snapshot(state: &AppState) -> Result<ManagedSnapshot, SnapshotFail let disks = sysmd.list_all_disks().await?; let stores = sysmd.list_stores().await?; if racks.is_empty() || nodes.is_empty() || !stores.iter().any(|store| store.store_id == 0) { - return Err(crowdb_kv_client::Error::Topology("managed topology is incomplete".into())); + return Err(crowdb_paxos_kv_client::Error::Topology("managed topology is incomplete".into())); } let disk_group_nodes = disk_groups diff --git a/app/crowdb-web/src/mgmt.rs b/app/crowdb-web/src/mgmt.rs index 9ba883a38..8b831f59e 100644 --- a/app/crowdb-web/src/mgmt.rs +++ b/app/crowdb-web/src/mgmt.rs @@ -159,7 +159,7 @@ pub(crate) fn rpc_is_conflict(err: &SharedError) -> bool { } /// Return the bare `host:port` of the crowdb-rpc listener that hosts `store_id` -/// on `node_id`. Each `PxKvStore` on a `crowdb-kv-server` binds its own +/// on `node_id`. Each `PxKvStore` on a `crowdb-paxos-kv-server` binds its own /// random port, so the bootstrap `ServerEntry::rpc_url` only points at /// the store created at process start (id 1). Operator-created stores /// must be looked up via the monitor cache, which carries the actual @@ -193,7 +193,7 @@ pub(crate) fn strip_scheme(s: String) -> String { } } -/// Parse `port` out of a URL like `http://host:10000` or `host:10000`. +/// Parse `port` out of a URL like `http://host:10135` or `host:10135`. /// Returns `None` on any shape we don't recognise. #[must_use] pub(crate) fn port_of(url: &str) -> Option<u16> { @@ -219,11 +219,13 @@ pub(crate) fn remap_zero_host(addr: &str) -> String { /// leader instead of failing with "no seeds configured". /// Returns `None` when no group-0 endpoint is known (e.g. cluster not /// yet initialized). -pub(crate) async fn build_hardware_client(state: &AppState) -> Option<crowdb_kv_client::HardwareClient> { +pub(crate) async fn build_hardware_client( + state: &AppState, +) -> Option<crowdb_paxos_kv_client::HardwareClient> { if state.managed_mode { let kv = state.kv_client().await; kv.refresh_topology().await.ok()?; - return Some(crowdb_kv_client::HardwareClient::from_shared(kv)); + return Some(crowdb_paxos_kv_client::HardwareClient::from_shared(kv)); } if state.node_monitor_url.is_some() { let binding = crate::node::cluster_binding(state).ok()??; @@ -232,7 +234,7 @@ pub(crate) async fn build_hardware_client(state: &AppState) -> Option<crowdb_kv_ // Admitted remote services have no PID in this console's namespace. // A persisted binding supplies discovery before any browser warms the // process/topology observation cache, including after a web restart. - return Some(crowdb_kv_client::HardwareClient::from_shared(kv)); + return Some(crowdb_paxos_kv_client::HardwareClient::from_shared(kv)); } let snap = state.monitor_cache.snapshot().await; if snap.is_empty() { @@ -283,7 +285,7 @@ pub(crate) async fn build_hardware_client(state: &AppState) -> Option<crowdb_kv_ } } } - // Reuse the shared CrowdbKvClient — its topology cache persists + // Reuse the shared PaxosKvClient — its topology cache persists // across requests so the group-0 leader is already known from prior // op_context / clusterInit calls. Creating a new client here would // trigger a 5s leader-discovery retry on every poll, saturating the @@ -324,7 +326,7 @@ pub(crate) async fn build_hardware_client(state: &AppState) -> Option<crowdb_kv_ // refresh retry. Log at warn so this is visible. warn!("build_hardware_client: no rpc endpoint resolved; relying on shared client's cached leader"); } - Some(crowdb_kv_client::HardwareClient::from_shared(kv)) + Some(crowdb_paxos_kv_client::HardwareClient::from_shared(kv)) } /// Check whether group-0 (store 0) is available for a sysdata read. @@ -388,7 +390,7 @@ mod tests { NodeStore { node_id: 7, store_id: 0, - listen_addr: Some("127.0.0.1:10001".into()), + listen_addr: Some("127.0.0.1:10136".into()), groups: Vec::new(), }, ); diff --git a/app/crowdb-web/src/mgmt/group_ops.rs b/app/crowdb-web/src/mgmt/group_ops.rs index 646aac5e2..7f80421f9 100644 --- a/app/crowdb-web/src/mgmt/group_ops.rs +++ b/app/crowdb-web/src/mgmt/group_ops.rs @@ -180,7 +180,7 @@ fn project_group( gid: u64, members: Vec<ReplicaValue>, reports: Vec<Option<NodeGroup>>, - membership: Option<&crowdb_kv_client::GroupMembershipSnapshot>, + membership: Option<&crowdb_paxos_kv_client::GroupMembershipSnapshot>, ) -> GroupView { let mut replicas = Vec::with_capacity(members.len()); let mut observed = 0usize; diff --git a/app/crowdb-web/src/node/admission.rs b/app/crowdb-web/src/node/admission.rs index 69e67d691..d5ffe1c1e 100644 --- a/app/crowdb-web/src/node/admission.rs +++ b/app/crowdb-web/src/node/admission.rs @@ -343,7 +343,10 @@ pub(super) fn update_config( if config.server_for_node(record.node_id).is_none() { let (management_port, rpc_port) = setup .and_then(crowdb_protocol::mgmt::node::setup::NodeSetupChoices::paxos_kv_ports) - .unwrap_or((10000, 10100)); + .unwrap_or(( + crowdb_protocol::PAXOS_KV_SERVER_MGMT_BASE, + crowdb_protocol::PAXOS_KV_SERVER_RPC_BASE, + )); let mut server = ServerEntry::new( format!("kv-{}", record.node_id), format!("http://{}:{management_port}", bracket_host(&record.host)), diff --git a/app/crowdb-web/src/node/admission/recovery.rs b/app/crowdb-web/src/node/admission/recovery.rs index 9f3dbd26f..53539e7e5 100644 --- a/app/crowdb-web/src/node/admission/recovery.rs +++ b/app/crowdb-web/src/node/admission/recovery.rs @@ -13,7 +13,7 @@ use crowdb_console_shared::{ }, ops::OpContext, }; -use crowdb_kv_client::{GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{GetOutcome, ReadMode}; use crowdb_protocol::mgmt::node::{NodeBinding, NodeControl}; use std::path::Path; diff --git a/app/crowdb-web/src/node/authority.rs b/app/crowdb-web/src/node/authority.rs index c562d3a86..8a0a91ab7 100644 --- a/app/crowdb-web/src/node/authority.rs +++ b/app/crowdb-web/src/node/authority.rs @@ -9,7 +9,7 @@ use axum::{ Json, }; use crowdb_console_shared::deployment::PreparedBootstrap; -use crowdb_kv_client::{GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{GetOutcome, ReadMode}; use serde_json::{json, Value}; use crate::{ diff --git a/app/crowdb-web/src/node/authority/projection.rs b/app/crowdb-web/src/node/authority/projection.rs index 63d576306..bcac3da5e 100644 --- a/app/crowdb-web/src/node/authority/projection.rs +++ b/app/crowdb-web/src/node/authority/projection.rs @@ -30,22 +30,22 @@ pub(crate) fn project(servers: &mut Vec<ServerEntry>, bootstrap: Vec<ServerEntry let management_port = reqwest::Url::parse(&server.url) .ok() .and_then(|url| url.port_or_known_default()) - .unwrap_or(10000); + .unwrap_or(crowdb_protocol::PAXOS_KV_SERVER_MGMT_BASE); let rpc_port = server .rpc_url .as_deref() .and_then(|url| reqwest::Url::parse(&format!("http://{url}")).ok()) .and_then(|url| url.port_or_known_default()) - .unwrap_or(10100); + .unwrap_or(crowdb_protocol::PAXOS_KV_SERVER_RPC_BASE); server.url = format!("http://{host}:{management_port}"); server.rpc_url = Some(format!("{host}:{rpc_port}")); } else { let mut server = crowdb_console_shared::config::ServerEntry::new( format!("kv-{}", node.node_id), - format!("http://{host}:10000"), + format!("http://{host}:{}", crowdb_protocol::PAXOS_KV_SERVER_MGMT_BASE), ); server.node_id = Some(node.node_id); - server.rpc_url = Some(format!("{host}:10100")); + server.rpc_url = Some(format!("{host}:{}", crowdb_protocol::PAXOS_KV_SERVER_RPC_BASE)); servers.push(server); } } diff --git a/app/crowdb-web/src/node/auto_join/runner.rs b/app/crowdb-web/src/node/auto_join/runner.rs index ab4ae78e5..d22533b5b 100644 --- a/app/crowdb-web/src/node/auto_join/runner.rs +++ b/app/crowdb-web/src/node/auto_join/runner.rs @@ -260,7 +260,7 @@ async fn admit_candidate( discovery_id: id.clone(), rack_id: record.map_or(default_rack, |record| record.rack_id), ssh_user: record.map_or_else(|| "crowdb".into(), |record| record.ssh_user.clone()), - ssh_port: record.map_or(2222, |record| record.ssh_port), + ssh_port: record.map_or(crowdb_protocol::SSH_PORT, |record| record.ssh_port), ssh_password: password, preflight_operation_id: None, setup: None, diff --git a/app/crowdb-web/src/node/cluster_status.rs b/app/crowdb-web/src/node/cluster_status.rs index 3d8c19ed3..a28ebc5bd 100644 --- a/app/crowdb-web/src/node/cluster_status.rs +++ b/app/crowdb-web/src/node/cluster_status.rs @@ -12,7 +12,7 @@ use crowdb_console_shared::{ ops::{cluster::readiness, OpContext}, ConsoleConfig, }; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient}; use crowdb_protocol::mgmt::{ cluster::{ClusterStatus, DependencyKind, DependencyObservation, DependencyState, ReadinessReason}, node::observation::NodeRuntimeObservation, @@ -37,7 +37,7 @@ pub(crate) async fn read(State(state): State<AppState>) -> Result<Json<ClusterSt .binding .as_ref() .map_or_else(Vec::new, |binding| binding.management_seeds.clone()); - let client = Arc::new(CrowdbKvClient::new(ClientConfig::new(seeds.clone()))); + let client = Arc::new(PaxosKvClient::new(ClientConfig::new(seeds.clone()))); let context = OpContext::with_shared_client_preserving_hint(client, &seeds, ConsoleConfig::default()); let id = local.binding.as_ref().map_or_else( || "local-node/observation".into(), diff --git a/app/crowdb-web/src/node/storage/authority.rs b/app/crowdb-web/src/node/storage/authority.rs index cb40c72c2..8508fb91e 100644 --- a/app/crowdb-web/src/node/storage/authority.rs +++ b/app/crowdb-web/src/node/storage/authority.rs @@ -8,7 +8,7 @@ use crate::{ error::{err_400, err_404, err_409, err_502}, state::AppState, }; -use crowdb_kv_client::HardwareClient; +use crowdb_paxos_kv_client::HardwareClient; use crowdb_protocol::{key::DiskKey, DiskIdExt}; pub(super) async fn disk( diff --git a/app/crowdb-web/src/node/storage/lifecycle.rs b/app/crowdb-web/src/node/storage/lifecycle.rs index e75c12088..40080c536 100644 --- a/app/crowdb-web/src/node/storage/lifecycle.rs +++ b/app/crowdb-web/src/node/storage/lifecycle.rs @@ -12,7 +12,7 @@ use axum::{ extract::{Path, State}, Json, }; -use crowdb_kv_client::{Error, HardwareClient}; +use crowdb_paxos_kv_client::{Error, HardwareClient}; use crowdb_protocol::{common::HwStatus, disk_storage::DiskRemovalRecord, key::DiskKey, DiskIdExt}; use serde::{Deserialize, Serialize}; diff --git a/app/crowdb-web/src/node/target.rs b/app/crowdb-web/src/node/target.rs index 952b40eea..6b8c29d1b 100644 --- a/app/crowdb-web/src/node/target.rs +++ b/app/crowdb-web/src/node/target.rs @@ -5,7 +5,7 @@ use axum::{extract::State, http::StatusCode, Json}; use crowdb_console_shared::deployment::{registry, PreparedBootstrap, CLUSTER_KEY}; -use crowdb_kv_client::{GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{GetOutcome, ReadMode}; use crowdb_protocol::mgmt::cluster::ClusterTargetIdentity; use crate::{ diff --git a/app/crowdb-web/src/physical/disk.rs b/app/crowdb-web/src/physical/disk.rs index a97f8b50a..6b86cac2d 100644 --- a/app/crowdb-web/src/physical/disk.rs +++ b/app/crowdb-web/src/physical/disk.rs @@ -5,7 +5,7 @@ use axum::{http::StatusCode, Json}; use crowdb_console_shared::config::{DiskEntry, DiskGroupEntry}; -use crowdb_kv_client::HardwareClient; +use crowdb_paxos_kv_client::HardwareClient; use crate::{ error::{err_409, err_502, map_config_err, map_persist_err, ErrorBody}, diff --git a/app/crowdb-web/src/physical/node.rs b/app/crowdb-web/src/physical/node.rs index 74c656956..cb064af3b 100644 --- a/app/crowdb-web/src/physical/node.rs +++ b/app/crowdb-web/src/physical/node.rs @@ -4,7 +4,7 @@ //! Per-node store / group / remote primitives (A4). //! //! Key work: read endpoints served from the monitor cache, low-level -//! mutators that proxy to the upstream `crowdb-kv-server` management API +//! mutators that proxy to the upstream `crowdb-paxos-kv-server` management API //! and invalidate the affected node in the monitor cache on success. use crate::error::{err_500, map_err, ErrorBody}; diff --git a/app/crowdb-web/src/services/defaults.rs b/app/crowdb-web/src/services/defaults.rs index 976320edd..3245a06cd 100644 --- a/app/crowdb-web/src/services/defaults.rs +++ b/app/crowdb-web/src/services/defaults.rs @@ -8,6 +8,7 @@ use crate::{ }; use axum::{extract::State, Json}; use crowdb_console_shared::{config::ServiceType, ConsoleConfig}; +use crowdb_protocol::ServicePort; use serde_json::{json, Value}; use std::collections::{BTreeMap, HashSet}; @@ -28,11 +29,6 @@ fn occupied(config: &ConsoleConfig) -> HashSet<u16> { ports.extend(server.rest_port); ports.extend(server.access_origins.values().filter_map(|origin| port(origin))); ports.extend(server.rpc_port); - if server.service_type == ServiceType::Diskdb { - if let Some(rpc) = server.rpc_url.as_deref().and_then(port) { - ports.extend([rpc.saturating_sub(2), rpc.saturating_sub(1)]); - } - } } for launch in config.local_launches.values() { for name in [ @@ -51,15 +47,16 @@ fn available(port: u16) -> bool { std::net::TcpListener::bind((std::net::Ipv4Addr::UNSPECIFIED, port)).is_ok() } -fn next(used: &mut HashSet<u16>, start: u16, width: u16) -> Result<u16, Failure> { - for candidate in start..=32767 - width { - if (candidate..candidate + width).all(|port| !used.contains(&port) && available(port)) { - used.extend(candidate..candidate + width); +fn next(used: &mut HashSet<u16>, service: ServicePort) -> Result<u16, Failure> { + for instance in 0..service.range_size() { + let candidate = service.port(instance); + if !used.contains(&candidate) && available(candidate) { + used.insert(candidate); return Ok(candidate); } } Err(err_409( - "No free listener range is available; release a port before deploying", + "No free port remains in the service's default range; specify a free listener explicitly", )) } @@ -68,13 +65,45 @@ pub(super) async fn get(State(state): State<AppState>) -> Result<Json<Value>, Fa let mut used = occupied(&config); used.extend(super::plans::reserved_ports(&state)?); let mut result = BTreeMap::new(); - for (kind, base) in [ - ("paxos-kv", 19910), - ("diskdb", 29920), - ("chunkdb", 12010), - ("diskio", 13010), - ("chunk-kv", 15010), - ("access-server", 9091), + for (kind, listeners) in [ + ( + "paxos-kv", + vec![ + ("http_port", ServicePort::PaxosKvServerMgmt), + ("rpc_port", ServicePort::PaxosKvServerRpc), + ], + ), + ( + "diskdb", + vec![ + ("http_port", ServicePort::DiskdbHttp), + ("rpc_port", ServicePort::DiskdbRpc), + ], + ), + ( + "chunkdb", + vec![ + ("http_port", ServicePort::ChunkdbHttp), + ("rpc_port", ServicePort::ChunkdbRpc), + ], + ), + ("diskio", vec![("rpc_port", ServicePort::DiskioRpc)]), + ( + "chunk-kv", + vec![ + ("http_port", ServicePort::ChunkKvHttp), + ("rpc_port", ServicePort::ChunkKvRpc), + ], + ), + ( + "access-server", + vec![ + ("s3_port", ServicePort::AccessServerHttp), + ("http_port", ServicePort::AccessServerIcebergHttp), + ("health_port", ServicePort::AccessServerHealthHttp), + ("dataset_port", ServicePort::AccessServerDatasetHttp), + ], + ), ] { let instance = config .servers @@ -85,20 +114,9 @@ pub(super) async fn get(State(state): State<AppState>) -> Result<Json<Value>, Fa .checked_add(1) .filter(|id| i64::try_from(*id).is_ok()) .ok_or_else(|| err_409("Service instance ID space exhausted"))?; - let first = next(&mut used, base, if kind == "diskdb" { 3 } else { 1 })?; let mut value = json!({"instance_id":instance.to_string()}); - match kind { - "diskdb" | "diskio" => value["rpc_port"] = json!(first), - "access-server" => { - value["s3_port"] = json!(first); - value["http_port"] = json!(next(&mut used, base + 1, 1)?); - value["health_port"] = json!(next(&mut used, base + 2, 1)?); - value["dataset_port"] = json!(next(&mut used, base + 3, 1)?); - } - _ => { - value["http_port"] = json!(first); - value["rpc_port"] = json!(next(&mut used, base + 100, 1)?); - } + for (field, service) in listeners { + value[field] = json!(next(&mut used, service)?); } if kind == "chunkdb" { if let Some(server) = config diff --git a/app/crowdb-web/src/services/dependencies.rs b/app/crowdb-web/src/services/dependencies.rs index d438c5c4a..889cbf1e1 100644 --- a/app/crowdb-web/src/services/dependencies.rs +++ b/app/crowdb-web/src/services/dependencies.rs @@ -7,14 +7,14 @@ pub(crate) async fn group0_ready(state: &AppState) -> bool { if state.service_operations.load().contains("cluster/init") { return false; } - let sysmd = crowdb_kv_client::CrowdbSysmdClient::from_shared(state.kv_client().await); + let sysmd = crowdb_paxos_kv_client::CrowdbSysmdClient::from_shared(state.kv_client().await); tokio::time::timeout(std::time::Duration::from_secs(2), sysmd.get_group(0, 0)) .await .is_ok_and(|result| result.is_ok_and(|group| group.is_some())) } use axum::{extract::State, Json}; use crowdb_console_shared::config::ServiceType; -use crowdb_kv_client::{HardwareClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{HardwareClient, ServiceRegistryClient}; use serde_json::{json, Value}; use std::collections::HashSet; diff --git a/app/crowdb-web/src/services/deployment/chunk_slots.rs b/app/crowdb-web/src/services/deployment/chunk_slots.rs index 4995f673e..ffa4babf8 100644 --- a/app/crowdb-web/src/services/deployment/chunk_slots.rs +++ b/app/crowdb-web/src/services/deployment/chunk_slots.rs @@ -8,7 +8,7 @@ use crate::{ services::Failure, state::AppState, }; -use crowdb_kv_client::{ChunkSlotMapClient, CrowdbSysmdClient, GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{ChunkSlotMapClient, CrowdbSysmdClient, GetOutcome, ReadMode}; use crowdb_protocol::{ chunk_slot::{ChunkSlotBootstrap, ChunkStorageGroup}, key::{ChunkSlotMapHeadKey, TextKey}, diff --git a/app/crowdb-web/src/services/deployment/credentials.rs b/app/crowdb-web/src/services/deployment/credentials.rs index fe3963958..c6be8a4d8 100644 --- a/app/crowdb-web/src/services/deployment/credentials.rs +++ b/app/crowdb-web/src/services/deployment/credentials.rs @@ -13,7 +13,7 @@ pub(super) async fn prepare( root: &Path, spec: &LocalLaunchSpec, seeds: &[String], - kv: &crowdb_kv_client::CrowdbKvClient, + kv: &crowdb_paxos_kv_client::PaxosKvClient, ) -> Result<(), Failure> { let s3_endpoint = spec .env diff --git a/app/crowdb-web/src/services/deployment/geometry.rs b/app/crowdb-web/src/services/deployment/geometry.rs index 3d15b7ca6..8e2ddd119 100644 --- a/app/crowdb-web/src/services/deployment/geometry.rs +++ b/app/crowdb-web/src/services/deployment/geometry.rs @@ -1,7 +1,7 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -use crowdb_kv_client::HardwareClient; +use crowdb_paxos_kv_client::HardwareClient; use crowdb_protocol::chunk_allocation_geometry::uniform_allocation_unit; use crowdb_protocol::common::HwStatus; diff --git a/app/crowdb-web/src/services/deployment/launch.rs b/app/crowdb-web/src/services/deployment/launch.rs index c5282e5bf..2805b8175 100644 --- a/app/crowdb-web/src/services/deployment/launch.rs +++ b/app/crowdb-web/src/services/deployment/launch.rs @@ -189,15 +189,16 @@ pub(super) async fn chunk_kv_config( config["storage"]["stream_chunk_capacity_bytes"] = json!(bytes); } if let Some(group) = body.bootstrap_group_id { - let client = - crowdb_kv_client::CrowdbKvClient::new(crowdb_kv_client::ClientConfig::new(seeds.to_vec())); + let client = crowdb_paxos_kv_client::PaxosKvClient::new(crowdb_paxos_kv_client::ClientConfig::new( + seeds.to_vec(), + )); tokio::time::timeout( std::time::Duration::from_secs(5), client.get( store, group, b"console-deployment-probe", - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ), ) diff --git a/app/crowdb-web/src/services/deployment/planned.rs b/app/crowdb-web/src/services/deployment/planned.rs index 0dfe87e3b..c47679e1e 100644 --- a/app/crowdb-web/src/services/deployment/planned.rs +++ b/app/crowdb-web/src/services/deployment/planned.rs @@ -148,9 +148,6 @@ async fn build( let name = service_name(selected.kind); let id = format!("{name}-{}", node.id); let (configuration, environment) = if selected.kind == SetupServiceKind::Diskdb { - if selected.listeners.listen_port.is_some() { - choices["rpc_listen_port"] = json!(selected.listeners.rpc_port); - } let body: crate::diskdb::DeployDiskdbBody = serde_json::from_value(choices).map_err(|error| err_400(error.to_string()))?; ( diff --git a/app/crowdb-web/src/services/observation.rs b/app/crowdb-web/src/services/observation.rs index f3fe88a34..6a38cb820 100644 --- a/app/crowdb-web/src/services/observation.rs +++ b/app/crowdb-web/src/services/observation.rs @@ -27,7 +27,7 @@ use axum::{extract::State, Json}; use crowdb_console_shared::{cluster::NodeHealth, config::ServiceType}; use serde::Serialize; -/// One row of `GET /api/servers`: a deployed `crowdb-kv-server` projected +/// One row of `GET /api/servers`: a deployed `crowdb-paxos-kv-server` projected /// from the persisted config plus the live monitor cache. #[derive(Debug, Serialize)] pub struct ServerSummary { diff --git a/app/crowdb-web/src/services/plans.rs b/app/crowdb-web/src/services/plans.rs index 947461e24..e9572b6d6 100644 --- a/app/crowdb-web/src/services/plans.rs +++ b/app/crowdb-web/src/services/plans.rs @@ -89,16 +89,11 @@ fn valid_overrides(overrides: &BTreeMap<String, ListenerOverrides>) -> bool { && (kind == "access-server" || (value.s3.is_none() && value.health.is_none() && value.dataset.is_none())) && (kind != "access-server" || value.rpc.is_none()) - && (!matches!(kind.as_str(), "diskdb" | "diskio") || value.http.is_none()) - && (kind != "diskdb" || value.rpc.map_or(true, |port| port <= 65533)) + && (kind != "diskio" || value.http.is_none()) && [value.http, value.rpc, value.s3, value.health, value.dataset] .into_iter() .flatten() .all(|port| port != 0 && ports.insert(port)) - && (kind != "diskdb" - || value - .rpc - .map_or(true, |port| ports.insert(port + 1) && ports.insert(port + 2))) }) } @@ -253,11 +248,6 @@ pub(super) fn reserved_ports(state: &AppState) -> Result<Vec<u16>, Failure> { .into_iter() .flatten(), ); - if kind == "diskdb" { - if let Some(port) = value.rpc.filter(|port| *port <= 65533) { - ports.extend([port + 1, port + 2]); - } - } } } } diff --git a/app/crowdb-web/src/standalone/readiness.rs b/app/crowdb-web/src/standalone/readiness.rs index b06273cd1..7778ab204 100644 --- a/app/crowdb-web/src/standalone/readiness.rs +++ b/app/crowdb-web/src/standalone/readiness.rs @@ -8,7 +8,7 @@ use std::time::Duration; use crowdb_console_shared::config::ServiceType; use crowdb_console_shared::error::{Error, Result}; -use crowdb_kv_client::ServiceRegistryClient; +use crowdb_paxos_kv_client::ServiceRegistryClient; use crate::state::AppState; diff --git a/app/crowdb-web/src/standalone/recovery.rs b/app/crowdb-web/src/standalone/recovery.rs index c3fda27ab..95da8270a 100644 --- a/app/crowdb-web/src/standalone/recovery.rs +++ b/app/crowdb-web/src/standalone/recovery.rs @@ -127,7 +127,7 @@ impl AppState { ..Default::default() }; let deployed = if node.ssh_enabled() { - let binary = server.binary.as_deref().unwrap_or("crowdb-kv-server"); + let binary = server.binary.as_deref().unwrap_or("crowdb-paxos-kv-server"); crowdb_console_shared::ssh::deploy_via_ssh(&request, node, binary).await? } else { let workspace = self.prepare_node_workspace(node.id)?; diff --git a/app/crowdb-web/src/state.rs b/app/crowdb-web/src/state.rs index 2120606b8..c422fccd2 100644 --- a/app/crowdb-web/src/state.rs +++ b/app/crowdb-web/src/state.rs @@ -32,19 +32,20 @@ pub struct AppState { pub diskdb_client: Arc<tokio::sync::RwLock<Option<crowdb_diskdb_client::DiskdbClient>>>, /// Cached crowdb-rpc transport reused across KV requests to avoid /// spawning 6+ threads per request. Shared by the cached `kv_client`. - pub kv_rpc_transport: Arc<tokio::sync::RwLock<Option<Arc<crowdb_kv_client::KvRpcTransport>>>>, + pub kv_rpc_transport: Arc<tokio::sync::RwLock<Option<Arc<crowdb_paxos_kv_client::PaxosKvRpcTransport>>>>, /// Immutable transport shared by bounded Chunk diagnostics requests. pub(crate) rpc_health: Arc<crate::services::rpc_health::RpcHealth>, pub(crate) chunk_rpc_transport: Arc<crowdb_chunkdb_client::ChunkdbRpcTransport>, - /// Cached `CrowdbKvClient` reused across KV requests so the topology + /// Cached `PaxosKvClient` reused across KV requests so the topology /// cache persists — avoids re-discovering the leader from seeds on /// every put/get/delete. Invalidated on `/internal/reset`. - pub kv_client: Arc<tokio::sync::RwLock<Option<Arc<crowdb_kv_client::CrowdbKvClient>>>>, + pub kv_client: Arc<tokio::sync::RwLock<Option<Arc<crowdb_paxos_kv_client::PaxosKvClient>>>>, /// Cached `ServiceDiscoveryClient` reused across requests so the /// discovery cache (service registry entries) persists. Built from - /// the same `CrowdbKvClient` as `kv_client` so topology + connection + /// the same `PaxosKvClient` as `kv_client` so topology + connection /// pool are shared. - pub discovery_client: Arc<tokio::sync::RwLock<Option<Arc<crowdb_kv_client::ServiceDiscoveryClient>>>>, + pub discovery_client: + Arc<tokio::sync::RwLock<Option<Arc<crowdb_paxos_kv_client::ServiceDiscoveryClient>>>>, /// Rate-limiter for repeated crowdb-rpc failure warnings: maps /// `endpoint` → last-warned timestamp. Prevents flooding the /// console with identical "instance query failed" warnings every @@ -383,11 +384,11 @@ impl AppState { std::fs::canonicalize(base).map_err(Error::Io) } - /// Get or create a cached `KvRpcTransport`. The transport (and its + /// Get or create a cached `PaxosKvRpcTransport`. The transport (and its /// crowdb-rpc server/client + I/O thread) is reused across KV requests /// to avoid spawning threads per request. Shared by the cached /// `kv_client`. - pub async fn kv_rpc_transport(&self) -> Arc<crowdb_kv_client::KvRpcTransport> { + pub async fn kv_rpc_transport(&self) -> Arc<crowdb_paxos_kv_client::PaxosKvRpcTransport> { if let Some(t) = self.kv_rpc_transport.read().await.as_ref() { return Arc::clone(t); } @@ -395,16 +396,16 @@ impl AppState { if let Some(t) = guard.as_ref() { return Arc::clone(t); } - let t = Arc::new(crowdb_kv_client::KvRpcTransport::new()); + let t = Arc::new(crowdb_paxos_kv_client::PaxosKvRpcTransport::new()); *guard = Some(Arc::clone(&t)); t } - /// Get or create a cached `CrowdbKvClient` sharing the cached RPC + /// Get or create a cached `PaxosKvClient` sharing the cached RPC /// transport. The topology cache persists across requests so the /// leader endpoint is not re-discovered from seeds on every call. /// Call `seed_leader` before each use to keep the hint fresh. - pub async fn kv_client(&self) -> Arc<crowdb_kv_client::CrowdbKvClient> { + pub async fn kv_client(&self) -> Arc<crowdb_paxos_kv_client::PaxosKvClient> { if let Some(c) = self.kv_client.read().await.as_ref() { return Arc::clone(c); } @@ -413,8 +414,8 @@ impl AppState { return Arc::clone(c); } let transport = self.kv_rpc_transport().await; - let c = Arc::new(crowdb_kv_client::CrowdbKvClient::new_with_rpc_transport( - crowdb_kv_client::ClientConfig::new(self.authority_seeds.as_ref().clone()), + let c = Arc::new(crowdb_paxos_kv_client::PaxosKvClient::new_with_rpc_transport( + crowdb_paxos_kv_client::ClientConfig::new(self.authority_seeds.as_ref().clone()), transport, )); *guard = Some(Arc::clone(&c)); @@ -423,7 +424,7 @@ impl AppState { /// Drop every client that retains cluster topology so the next /// request rebuilds from the post-reset cluster. The discovery and - /// `DiskDB` clients both retain the shared `CrowdbKvClient`, so + /// `DiskDB` clients both retain the shared `PaxosKvClient`, so /// clearing only `kv_client` would leave its old group-0 leader hint /// reachable through those wrappers. pub async fn clear_cluster_clients(&self) { @@ -436,10 +437,10 @@ impl AppState { } /// Get or lazily build the cached `ServiceDiscoveryClient`. Shares - /// the same `CrowdbKvClient` as `kv_client()` so topology + connection + /// the same `PaxosKvClient` as `kv_client()` so topology + connection /// pool are shared. Used by web handlers that need to discover living /// service instances (diskdb, chunkdb) from the group-0 registry. - pub async fn discovery_client(&self) -> Arc<crowdb_kv_client::ServiceDiscoveryClient> { + pub async fn discovery_client(&self) -> Arc<crowdb_paxos_kv_client::ServiceDiscoveryClient> { if let Some(d) = self.discovery_client.read().await.as_ref() { return Arc::clone(d); } @@ -448,12 +449,12 @@ impl AppState { return Arc::clone(d); } let kv = self.kv_client().await; - let d = Arc::new(crowdb_kv_client::ServiceDiscoveryClient::from_shared_kv(kv)); + let d = Arc::new(crowdb_paxos_kv_client::ServiceDiscoveryClient::from_shared_kv(kv)); *guard = Some(Arc::clone(&d)); d } - /// Re-seed the cached `CrowdbKvClient`'s mgmt seed list from the + /// Re-seed the cached `PaxosKvClient`'s mgmt seed list from the /// current config. Called after cluster init (or server deploy) so /// the shared client can discover leaders for the new topology. /// No-op if no client is cached (it will be created with config @@ -481,7 +482,7 @@ impl AppState { } /// Build an [`OpContext`] for a single request, sharing the cached - /// `CrowdbKvClient` (topology cache + connection pool) and a + /// `PaxosKvClient` (topology cache + connection pool) and a /// snapshot of the persisted [`ConsoleConfig`]. /// /// The group-0 endpoint is resolved from the config's store-0 @@ -737,11 +738,11 @@ mod tests { let _ = config.add_server(ServerEntry { access_origins: std::collections::BTreeMap::new(), id: "s1".into(), - url: "http://127.0.0.1:10000".into(), + url: "http://127.0.0.1:10135".into(), node_id: Some(1), rpc_url: None, - rest_port: Some(10000), - rpc_port: Some(10100), + rest_port: Some(crowdb_protocol::PAXOS_KV_SERVER_MGMT_BASE), + rpc_port: Some(crowdb_protocol::PAXOS_KV_SERVER_RPC_BASE), auto_start: true, binary: None, election_profile: None, @@ -752,7 +753,7 @@ mod tests { }); let (endpoint, seeds) = resolve_group0_endpoint(&config); assert!(endpoint.is_none(), "no store 0 → no RPC endpoint to seed"); - assert_eq!(seeds, vec!["http://127.0.0.1:10000"]); + assert_eq!(seeds, vec!["http://127.0.0.1:10135"]); } #[test] @@ -761,11 +762,11 @@ mod tests { let _ = config.add_server(ServerEntry { access_origins: std::collections::BTreeMap::new(), id: "s1".into(), - url: "http://127.0.0.1:10000".into(), + url: "http://127.0.0.1:10135".into(), node_id: Some(1), - rpc_url: Some("http://127.0.0.1:10100".into()), - rest_port: Some(10000), - rpc_port: Some(10100), + rpc_url: Some("http://127.0.0.1:11135".into()), + rest_port: Some(crowdb_protocol::PAXOS_KV_SERVER_MGMT_BASE), + rpc_port: Some(crowdb_protocol::PAXOS_KV_SERVER_RPC_BASE), auto_start: true, binary: None, election_profile: None, @@ -776,8 +777,8 @@ mod tests { }); config.record_store(0, vec![1]); let (endpoint, seeds) = resolve_group0_endpoint(&config); - assert_eq!(endpoint.as_deref(), Some("http://127.0.0.1:10100")); - assert_eq!(seeds, vec!["http://127.0.0.1:10000"]); + assert_eq!(endpoint.as_deref(), Some("http://127.0.0.1:11135")); + assert_eq!(seeds, vec!["http://127.0.0.1:10135"]); } #[tokio::test] @@ -796,11 +797,11 @@ mod tests { let _ = config.add_server(ServerEntry { access_origins: std::collections::BTreeMap::new(), id: "s1".into(), - url: "http://127.0.0.1:10000".into(), + url: "http://127.0.0.1:10135".into(), node_id: Some(1), - rpc_url: Some("http://127.0.0.1:10100".into()), - rest_port: Some(10000), - rpc_port: Some(10100), + rpc_url: Some("http://127.0.0.1:11135".into()), + rest_port: Some(crowdb_protocol::PAXOS_KV_SERVER_MGMT_BASE), + rpc_port: Some(crowdb_protocol::PAXOS_KV_SERVER_RPC_BASE), auto_start: true, binary: None, election_profile: None, @@ -819,7 +820,7 @@ mod tests { let cached = state.kv_client().await; assert!( Arc::ptr_eq(ctx.kv_arc(), &cached), - "OpContext must share the cached CrowdbKvClient" + "OpContext must share the cached PaxosKvClient" ); } @@ -852,11 +853,11 @@ mod tests { let _ = config.add_server(ServerEntry { access_origins: std::collections::BTreeMap::new(), id: "s1".into(), - url: "http://127.0.0.1:10000".into(), + url: "http://127.0.0.1:10135".into(), node_id: Some(1), - rpc_url: Some("http://127.0.0.1:10100".into()), - rest_port: Some(10000), - rpc_port: Some(10100), + rpc_url: Some("http://127.0.0.1:11135".into()), + rest_port: Some(crowdb_protocol::PAXOS_KV_SERVER_MGMT_BASE), + rpc_port: Some(crowdb_protocol::PAXOS_KV_SERVER_RPC_BASE), auto_start: true, binary: None, election_profile: None, diff --git a/app/crowdb-web/tests/bound_client_test.rs b/app/crowdb-web/tests/bound_client_test.rs index 1d2433ad3..de06cfd3a 100644 --- a/app/crowdb-web/tests/bound_client_test.rs +++ b/app/crowdb-web/tests/bound_client_test.rs @@ -5,14 +5,14 @@ mod memory_kv; use crowdb_console_shared::ConsoleConfig; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{ClientConfig, GetOutcome, PaxosKvClient, ReadMode}; use crowdb_protocol::mgmt::{node::NodeBinding, SystemBootstrapIdentity}; use crowdb_web::AppState; #[tokio::test] async fn bound_operations_discover_group0_without_a_warmed_console_projection() { let cluster = memory_kv::TestKvCluster::start().await; - let kv = CrowdbKvClient::new(ClientConfig::new(cluster.mgmt_endpoints.clone())); + let kv = PaxosKvClient::new(ClientConfig::new(cluster.mgmt_endpoints.clone())); kv.seed_leader(0, 0, cluster.group0_leader_endpoint.clone()); kv.put(0, 0, b"binding-probe", b"committed", None).await.unwrap(); let root = crowdb_test_harness::test_dirs::tempdir_in_test_data("bound-client"); diff --git a/app/crowdb-web/tests/chunk_kv_catalog_test.rs b/app/crowdb-web/tests/chunk_kv_catalog_test.rs index 0c66f7dbf..6dda00054 100644 --- a/app/crowdb-web/tests/chunk_kv_catalog_test.rs +++ b/app/crowdb-web/tests/chunk_kv_catalog_test.rs @@ -13,7 +13,7 @@ use axum::{ body::Body, http::{Request, StatusCode}, }; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient}; use crowdb_protocol::chunk_kv::{ ChunkKvRangeCatalogEntry, ChunkKvRangeCatalogHead, ChunkKvRangeCatalogPage, ChunkKvRangeCatalogPageRef, ChunkKvRangeCatalogPartitionState, Id128, KeyRange, OwnerDescriptor, PartitionArtifact, @@ -44,7 +44,7 @@ async fn get(app: axum::Router, query: &str) -> (StatusCode, Value) { #[tokio::test] async fn catalog_windows_pin_generation_validate_checksums_and_keep_exact_ids() { let cluster = memory_kv::TestKvCluster::start().await; - let kv = Arc::new(CrowdbKvClient::new(ClientConfig::new( + let kv = Arc::new(PaxosKvClient::new(ClientConfig::new( cluster.mgmt_endpoints.clone(), ))); kv.seed_leader(0, 0, cluster.group0_leader_endpoint.clone()); @@ -303,7 +303,7 @@ fn catalog_fixture() -> (ChunkKvRangeCatalogHead, ChunkKvRangeCatalogPage) { (head, page) } -async fn verify_balance(kv: &CrowdbKvClient, app: &axum::Router, page: &ChunkKvRangeCatalogPage) { +async fn verify_balance(kv: &PaxosKvClient, app: &axum::Router, page: &ChunkKvRangeCatalogPage) { use crowdb_protocol::chunk_kv::balance::OBSERVATION_KEY; catalog_balance::seed(kv, page).await; let (status, first) = get(app.clone(), "").await; @@ -319,12 +319,12 @@ async fn verify_balance(kv: &CrowdbKvClient, app: &axum::Router, page: &ChunkKvR first["entries"][0]["balance"]["partition"]["weight"]["byte_units"], "7619047" ); - let crowdb_kv_client::GetOutcome::Found { value, .. } = kv + let crowdb_paxos_kv_client::GetOutcome::Found { value, .. } = kv .get( 0, 0, OBSERVATION_KEY.as_bytes(), - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ) .await diff --git a/app/crowdb-web/tests/cluster_target_test.rs b/app/crowdb-web/tests/cluster_target_test.rs index 17fdcefe0..16c6a1183 100644 --- a/app/crowdb-web/tests/cluster_target_test.rs +++ b/app/crowdb-web/tests/cluster_target_test.rs @@ -12,7 +12,7 @@ use axum::{ }; use cluster_target::TestClusterTarget; use crowdb_console_shared::deployment::CLUSTER_KEY; -use crowdb_kv_client::{GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{GetOutcome, ReadMode}; use serde_json::Value; use tower::ServiceExt; diff --git a/app/crowdb-web/tests/common/catalog_balance.rs b/app/crowdb-web/tests/common/catalog_balance.rs index 41c3bbfe1..4a644c7fe 100644 --- a/app/crowdb-web/tests/common/catalog_balance.rs +++ b/app/crowdb-web/tests/common/catalog_balance.rs @@ -1,7 +1,7 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -use crowdb_kv_client::CrowdbKvClient; +use crowdb_paxos_kv_client::PaxosKvClient; use crowdb_protocol::chunk_kv::{ balance::{ BalanceObservation, BalanceOwnerObservation, BalancePartitionObservation, BalanceWeights, @@ -10,7 +10,7 @@ use crowdb_protocol::chunk_kv::{ ChunkKvRangeBalancePolicy, ChunkKvRangeCatalogPage, }; -pub async fn seed(kv: &CrowdbKvClient, page: &ChunkKvRangeCatalogPage) { +pub async fn seed(kv: &PaxosKvClient, page: &ChunkKvRangeCatalogPage) { let weights = BalanceWeights::new(&[(105, 1_050)], 80).unwrap(); let first = &page.entries[0]; let now = u64::try_from( diff --git a/app/crowdb-web/tests/common/cluster_target.rs b/app/crowdb-web/tests/common/cluster_target.rs index 5ffdee81e..0c44dca3f 100644 --- a/app/crowdb-web/tests/common/cluster_target.rs +++ b/app/crowdb-web/tests/common/cluster_target.rs @@ -7,7 +7,7 @@ use crowdb_console_shared::{ deployment::{PreparedBootstrap, CLUSTER_KEY}, ConsoleConfig, }; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient}; use crowdb_protocol::mgmt::node::NodeBinding; use crowdb_web::AppState; use std::sync::Arc; @@ -16,7 +16,7 @@ pub struct TestClusterTarget { pub state: AppState, pub prepared: PreparedBootstrap, pub binding: NodeBinding, - pub kv: Arc<CrowdbKvClient>, + pub kv: Arc<PaxosKvClient>, pub root: crowdb_test_harness::test_dirs::TestDir, _cluster: TestKvCluster, } @@ -24,7 +24,7 @@ pub struct TestClusterTarget { impl TestClusterTarget { pub async fn start() -> Self { let cluster = TestKvCluster::start().await; - let kv = Arc::new(CrowdbKvClient::new(ClientConfig::new( + let kv = Arc::new(PaxosKvClient::new(ClientConfig::new( cluster.mgmt_endpoints.clone(), ))); kv.seed_leader(0, 0, cluster.group0_leader_endpoint.clone()); diff --git a/app/crowdb-web/tests/common/memory_kv.rs b/app/crowdb-web/tests/common/memory_kv.rs index 3e49a78f9..b3f680e57 100644 --- a/app/crowdb-web/tests/common/memory_kv.rs +++ b/app/crowdb-web/tests/common/memory_kv.rs @@ -7,8 +7,8 @@ #![allow(dead_code)] use axum::{extract::State, routing::get, Json, Router}; -use crowdb_kv::cluster::group::PxGroup; -use crowdb_kv::cluster::{KvServer, PxKvStore, PxLocalReplica, PxLocalReplicaRole}; +use crowdb_paxos_kv::cluster::group::PxGroup; +use crowdb_paxos_kv::cluster::{PaxosKvServer, PxKvStore, PxLocalReplica, PxLocalReplicaRole}; use std::sync::Arc; pub(crate) struct TestKvCluster { diff --git a/app/crowdb-web/tests/common/storage_lifecycle.rs b/app/crowdb-web/tests/common/storage_lifecycle.rs index e8b5b79c4..41b7fb4f1 100644 --- a/app/crowdb-web/tests/common/storage_lifecycle.rs +++ b/app/crowdb-web/tests/common/storage_lifecycle.rs @@ -10,7 +10,7 @@ use crowdb_console_shared::{ }, ConsoleConfig, }; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, HardwareClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{ClientConfig, HardwareClient, PaxosKvClient, ServiceRegistryClient}; use crowdb_protocol::{ common::{DiskId, DiskioHardwareScope, HwStatus, NodeValue, RackValue}, disk_storage::DiskStorageRecord, @@ -32,7 +32,7 @@ pub struct TestStorageApi { impl TestStorageApi { pub async fn start() -> Self { let cluster = TestKvCluster::start().await; - let kv = Arc::new(CrowdbKvClient::new(ClientConfig::new( + let kv = Arc::new(PaxosKvClient::new(ClientConfig::new( cluster.mgmt_endpoints.clone(), ))); kv.seed_leader(0, 0, cluster.group0_leader_endpoint.clone()); diff --git a/app/crowdb-web/tests/common/storage_readiness.rs b/app/crowdb-web/tests/common/storage_readiness.rs index 2c3d490ca..4c503aefb 100644 --- a/app/crowdb-web/tests/common/storage_readiness.rs +++ b/app/crowdb-web/tests/common/storage_readiness.rs @@ -1,7 +1,7 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -use crowdb_kv_client::HardwareClient; +use crowdb_paxos_kv_client::HardwareClient; use crowdb_protocol::{ common::{DiskId, HwStatus, NodeValue, RackValue}, diskdb::rpc::DiskValue, diff --git a/app/crowdb-web/tests/corr_id_propagation_test.rs b/app/crowdb-web/tests/corr_id_propagation_test.rs index 9074a7953..f91ed9035 100644 --- a/app/crowdb-web/tests/corr_id_propagation_test.rs +++ b/app/crowdb-web/tests/corr_id_propagation_test.rs @@ -1,7 +1,7 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -//! Verify `x-crowdb-kv-corr-id` propagates inbound → handler → response. +//! Verify `x-crowdb-paxos-kv-corr-id` propagates inbound → handler → response. //! //! Each test spins up the real `crowdb-web` router. The middleware //! should: diff --git a/app/crowdb-web/tests/deployment_cancellation_test.rs b/app/crowdb-web/tests/deployment_cancellation_test.rs index 2735ce68d..447b2d0dd 100644 --- a/app/crowdb-web/tests/deployment_cancellation_test.rs +++ b/app/crowdb-web/tests/deployment_cancellation_test.rs @@ -133,20 +133,19 @@ async fn cancel_and_reset(kind: &str, restart: bool) { TestChild::new(root.path()) }; let (mut path, mut body, service) = if kind == "diskdb" { - let port = crowdb_protocol::port::alloc::alloc_test_port_range( - crowdb_protocol::ServicePort::DiskdbListen, - 3, - )[0]; + let port = + crowdb_protocol::port::alloc::alloc_test_port_range(crowdb_protocol::ServicePort::DiskdbRpc, 3) + [0]; ( "/api/nodes/1/diskdb/deploy", - json!({"rpc_port": port}), + json!({"rpc_port": port, "http_port": free_port()}), "crowdb-diskdb", ) } else { ( "/api/nodes/1/server/deploy", json!({"rest_port": free_port(), "rpc_port": free_port(), "binary": stub.binary, "election_profile": "test"}), - "crowdb-kv-server", + "crowdb-paxos-kv-server", ) }; if restart { @@ -203,11 +202,11 @@ async fn cancel_and_reset(kind: &str, restart: bool) { async fn prepare_diskdb_authority(state: &AppState) -> memory_kv::TestKvCluster { let metadata = memory_kv::TestKvCluster::start().await; - let kv = std::sync::Arc::new(crowdb_kv_client::CrowdbKvClient::new( - crowdb_kv_client::ClientConfig::new(metadata.mgmt_endpoints.clone()), + let kv = std::sync::Arc::new(crowdb_paxos_kv_client::PaxosKvClient::new( + crowdb_paxos_kv_client::ClientConfig::new(metadata.mgmt_endpoints.clone()), )); kv.seed_leader(0, 0, metadata.group0_leader_endpoint.clone()); - crowdb_kv_client::CrowdbSysmdClient::from_shared(kv.clone()) + crowdb_paxos_kv_client::CrowdbSysmdClient::from_shared(kv.clone()) .add_group(0, 0) .await .unwrap(); diff --git a/app/crowdb-web/tests/diskdb_binding_test.rs b/app/crowdb-web/tests/diskdb_binding_test.rs index c9e179224..100dcd874 100644 --- a/app/crowdb-web/tests/diskdb_binding_test.rs +++ b/app/crowdb-web/tests/diskdb_binding_test.rs @@ -21,7 +21,7 @@ async fn disk_drilldown_uses_persisted_binding_without_local_process_observation 0, 0, b"/deployment/cluster", - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None ) .await diff --git a/app/crowdb-web/tests/diskdb_deployment_configuration_test.rs b/app/crowdb-web/tests/diskdb_deployment_configuration_test.rs index 335a7c1a2..37bf6f125 100644 --- a/app/crowdb-web/tests/diskdb_deployment_configuration_test.rs +++ b/app/crowdb-web/tests/diskdb_deployment_configuration_test.rs @@ -10,7 +10,7 @@ use serde_json::{json, Value}; use tower::ServiceExt; #[test] -fn explicit_diskdb_ranges_keep_adjacent_host_instances_disjoint_and_preserve_legacy_inputs() { +fn explicit_diskdb_ranges_keep_adjacent_host_instances_disjoint() { use crowdb_console_shared::config::NodeEntry; use crowdb_web::diskdb::{configuration_for_tests, DeployDiskdbBody}; let node: NodeEntry = serde_json::from_value(json!({ @@ -20,17 +20,16 @@ fn explicit_diskdb_ranges_keep_adjacent_host_instances_disjoint_and_preserve_leg let mut used = std::collections::BTreeSet::new(); for offset in 0..3 { let body: DeployDiskdbBody = serde_json::from_value(json!({ - "listen_port":11000+offset,"http_port":11100+offset, - "rpc_port":11200+offset,"rpc_listen_port":11200+offset + "http_port":crowdb_protocol::DISKDB_HTTP_BASE+offset, + "rpc_port":crowdb_protocol::DISKDB_RPC_BASE+offset })) .unwrap(); let config = configuration_for_tests(&node, &["http://127.0.0.1:10000".into()], &body) .unwrap_or_else(|(_, body)| panic!("{}", body.error)); let config: toml::Value = toml::from_str(&config).unwrap(); for (field, base) in [ - ("listen_addr", 11000), - ("http_listen_addr", 11100), - ("rpc_listen_addr", 11200), + ("http_listen_addr", crowdb_protocol::DISKDB_HTTP_BASE), + ("rpc_listen_addr", crowdb_protocol::DISKDB_RPC_BASE), ] { let address = config["server"][field].as_str().unwrap(); assert_eq!(address, format!("127.0.0.1:{}", base + offset)); @@ -43,7 +42,7 @@ fn explicit_diskdb_ranges_keep_adjacent_host_instances_disjoint_and_preserve_leg let config: toml::Value = toml::from_str(&config).unwrap(); assert_eq!( config["server"]["rpc_listen_addr"].as_str(), - Some("127.0.0.1:7102") + Some("127.0.0.1:7100") ); } diff --git a/app/crowdb-web/tests/diskdb_routes_test.rs b/app/crowdb-web/tests/diskdb_routes_test.rs index c733c7df8..81896bca6 100644 --- a/app/crowdb-web/tests/diskdb_routes_test.rs +++ b/app/crowdb-web/tests/diskdb_routes_test.rs @@ -18,7 +18,7 @@ use std::time::Duration; mod memory_kv; use crowdb_console_shared::{ConsoleConfig, DiskGroupEntry, NodeEntry, RackEntry, ServerEntry}; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, HardwareClient}; +use crowdb_paxos_kv_client::{ClientConfig, HardwareClient, PaxosKvClient}; use crowdb_protocol::{ common::{NodeValue, RackValue}, diskdb::rpc::DiskGroupValue, @@ -70,7 +70,7 @@ async fn spawn_web_with_config(mut cfg: ConsoleConfig, with_group0: bool) -> Soc let kv = if let Some(cluster) = &cluster { cfg.servers .push(ServerEntry::new("metadata", &cluster.mgmt_endpoints[0])); - let kv = Arc::new(CrowdbKvClient::new(ClientConfig::new( + let kv = Arc::new(PaxosKvClient::new(ClientConfig::new( cluster.mgmt_endpoints.clone(), ))); kv.seed_leader(0, 0, cluster.group0_leader_endpoint.clone()); diff --git a/app/crowdb-web/tests/lifecycle_routes_test.rs b/app/crowdb-web/tests/lifecycle_routes_test.rs index 5e08d4874..934654a51 100644 --- a/app/crowdb-web/tests/lifecycle_routes_test.rs +++ b/app/crowdb-web/tests/lifecycle_routes_test.rs @@ -304,11 +304,11 @@ async fn multiple_racks_and_nodes_create_expected_workspaces() { assert!(std::fs::read_dir(dir.join("N-1/bin")).unwrap().next().is_some()); assert!(dir - .join(format!("N-1/log/crowdb-kv-server-{pid1}.out.log")) + .join(format!("N-1/log/crowdb-paxos-kv-server-{pid1}.out.log")) .exists()); assert!(std::fs::read_dir(dir.join("N-10/bin")).unwrap().next().is_some()); assert!(dir - .join(format!("N-10/log/crowdb-kv-server-{pid10}.out.log")) + .join(format!("N-10/log/crowdb-paxos-kv-server-{pid10}.out.log")) .exists()); assert!(std::fs::read_dir(dir.join("N-2/bin")).unwrap().next().is_none()); diff --git a/app/crowdb-web/tests/managed_mode_test.rs b/app/crowdb-web/tests/managed_mode_test.rs index 150446462..e30537d99 100644 --- a/app/crowdb-web/tests/managed_mode_test.rs +++ b/app/crowdb-web/tests/managed_mode_test.rs @@ -6,8 +6,8 @@ mod memory_kv; use axum::body::Body; use axum::http::{Method, Request, StatusCode}; use crowdb_console_shared::config::web::{WebMode, WebProcessConfig}; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, CrowdbSysmdClient}; use crowdb_monitor::{MonitorPhase, MonitorStatus, ServiceStatus, StatusStore}; +use crowdb_paxos_kv_client::{ClientConfig, CrowdbSysmdClient, PaxosKvClient}; use crowdb_protocol::common::{HwStatus, KvServerIdentity, NodeValue, RackValue, ServiceExtra}; use crowdb_web::{router, AppState}; use tower::ServiceExt; @@ -195,7 +195,7 @@ async fn managed_readonly_mode_does_not_expose_local_topology_or_mutations() { #[tokio::test] async fn managed_snapshot_uses_group0_and_monitor_without_local_fallback() { let cluster = memory_kv::TestKvCluster::start().await; - let kv = CrowdbKvClient::new(ClientConfig::new(cluster.mgmt_endpoints.clone())); + let kv = PaxosKvClient::new(ClientConfig::new(cluster.mgmt_endpoints.clone())); kv.seed_leader(0, 0, cluster.group0_leader_endpoint.clone()); let sysmd = CrowdbSysmdClient::new(kv); sysmd diff --git a/app/crowdb-web/tests/node_storage_authority_test.rs b/app/crowdb-web/tests/node_storage_authority_test.rs index 293deb394..57a56a2d9 100644 --- a/app/crowdb-web/tests/node_storage_authority_test.rs +++ b/app/crowdb-web/tests/node_storage_authority_test.rs @@ -9,7 +9,7 @@ use axum::{ http::{Request, StatusCode}, }; use crowdb_console_shared::ConsoleConfig; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, HardwareClient}; +use crowdb_paxos_kv_client::{ClientConfig, HardwareClient, PaxosKvClient}; use crowdb_protocol::{ common::{DiskId, NodeValue, RackValue}, diskdb::rpc::{DiskGroupValue, DiskValue}, @@ -32,7 +32,7 @@ async fn read(app: &axum::Router, path: &str) -> serde_json::Value { #[tokio::test] async fn bound_storage_reads_observe_monitor_publication_without_console_cache_refresh() { let cluster = memory_kv::TestKvCluster::start().await; - let kv = Arc::new(CrowdbKvClient::new(ClientConfig::new( + let kv = Arc::new(PaxosKvClient::new(ClientConfig::new( cluster.mgmt_endpoints.clone(), ))); kv.seed_leader(0, 0, cluster.group0_leader_endpoint.clone()); diff --git a/app/crowdb-web/tests/remote_kv_projection_test.rs b/app/crowdb-web/tests/remote_kv_projection_test.rs index 864cec16e..bdac9ae4e 100644 --- a/app/crowdb-web/tests/remote_kv_projection_test.rs +++ b/app/crowdb-web/tests/remote_kv_projection_test.rs @@ -6,7 +6,7 @@ mod memory_kv; use axum::{body::Body, http::Request}; use crowdb_console_shared::config::{ConsoleConfig, ServerEntry}; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient}; use crowdb_protocol::mgmt::{ node::{NodeBinding, NodeServiceAction, NodeServiceIntent}, SystemBootstrapIdentity, @@ -19,7 +19,7 @@ use tower::ServiceExt; #[tokio::test] async fn diskdb_restart_claims_the_remote_node_once_before_target_validation() { let cluster = memory_kv::TestKvCluster::start().await; - let kv = Arc::new(CrowdbKvClient::new(ClientConfig::new( + let kv = Arc::new(PaxosKvClient::new(ClientConfig::new( cluster.mgmt_endpoints.clone(), ))); kv.seed_leader(0, 0, cluster.group0_leader_endpoint.clone()); @@ -71,7 +71,7 @@ async fn diskdb_restart_claims_the_remote_node_once_before_target_validation() { #[tokio::test] async fn kv_lifecycle_intents_preserve_admitted_endpoints_while_auxiliary_services_refresh() { let cluster = memory_kv::TestKvCluster::start().await; - let kv = Arc::new(CrowdbKvClient::new(ClientConfig::new( + let kv = Arc::new(PaxosKvClient::new(ClientConfig::new( cluster.mgmt_endpoints.clone(), ))); kv.seed_leader(0, 0, cluster.group0_leader_endpoint.clone()); @@ -135,7 +135,7 @@ async fn kv_lifecycle_intents_preserve_admitted_endpoints_while_auxiliary_servic .any(|row| row["id"] == "chunkdb-1")); } -async fn publish_intents(kv: &CrowdbKvClient, cluster_id: &str) { +async fn publish_intents(kv: &PaxosKvClient, cluster_id: &str) { for (id, kind, configuration) in [ ("kv-1", "kv", "node_id = 1\n"), ( diff --git a/app/crowdb-web/tests/service_lifecycle_test.rs b/app/crowdb-web/tests/service_lifecycle_test.rs index d8d561515..bf1582b52 100644 --- a/app/crowdb-web/tests/service_lifecycle_test.rs +++ b/app/crowdb-web/tests/service_lifecycle_test.rs @@ -290,7 +290,7 @@ async fn reset_removes_auxiliary_launches_before_nodes_and_racks() { #[tokio::test] async fn deployment_defaults_avoid_cross_service_and_live_listener_conflicts() { let mut config = config(); - let occupied = std::net::TcpListener::bind("0.0.0.0:12010").ok(); + let occupied = std::net::TcpListener::bind(("0.0.0.0", crowdb_protocol::CHUNKDB_HTTP_BASE)).ok(); let mut access = ServerEntry::new("access-server-7", "http://127.0.0.1:9092"); access.service_type = ServiceType::AccessServer; config.servers.push(access); @@ -306,18 +306,14 @@ async fn deployment_defaults_avoid_cross_service_and_live_listener_conflicts() { assert_eq!(status, StatusCode::OK, "{body}"); assert_eq!(body["access-server"]["instance_id"], "8"); let mut ports = std::collections::HashSet::new(); - for (kind, values) in body.as_object().unwrap() { + for values in body.as_object().unwrap().values() { for name in ["http_port", "rpc_port", "s3_port", "health_port"] { if let Some(port) = values[name].as_u64() { assert!(![9091, 9092, 19191].contains(&port)); if occupied.is_some() { - assert_ne!(port, 12010); + assert_ne!(port, u64::from(crowdb_protocol::CHUNKDB_HTTP_BASE)); } assert!(ports.insert(port)); - if kind == "diskdb" { - assert!(ports.insert(port + 1)); - assert!(ports.insert(port + 2)); - } } } } diff --git a/app/crowdb-web/tests/service_plans_test.rs b/app/crowdb-web/tests/service_plans_test.rs index c33e4b252..4c416cd5c 100644 --- a/app/crowdb-web/tests/service_plans_test.rs +++ b/app/crowdb-web/tests/service_plans_test.rs @@ -137,8 +137,8 @@ async fn plan_rejects_legacy_kinds_and_invalid_listener_configuration() { json!({"access-server":{"dataset_port":9093,"health_port":9093}}), json!({"chunkdb":{"dataset_port":9093}}), json!({"diskio":{"http_port":13010}}), - json!({"diskdb":{"rpc_port":65534}}), - json!({"diskdb":{"rpc_port":12000},"diskio":{"rpc_port":12001}}), + json!({"diskdb":{"rpc_port":0}}), + json!({"diskdb":{"rpc_port":12000},"diskio":{"rpc_port":12000}}), json!({"chunkdb":{"health_port":9093}}), json!({"kv":{"rpc_port":19920}}), json!({"access-server":{"instance_id":"1"}}), diff --git a/app/crowdb-web/tests/storage_readiness_test.rs b/app/crowdb-web/tests/storage_readiness_test.rs index 5832be703..bce27ec16 100644 --- a/app/crowdb-web/tests/storage_readiness_test.rs +++ b/app/crowdb-web/tests/storage_readiness_test.rs @@ -10,7 +10,7 @@ use crowdb_console_shared::{ config::{ServerEntry, ServiceType}, ConsoleConfig, }; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, HardwareClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{ClientConfig, HardwareClient, PaxosKvClient, ServiceRegistryClient}; use crowdb_protocol::diskdb::rpc::DiskGroupValue; use crowdb_web::{router, AppState}; use serde_json::Value; @@ -35,7 +35,7 @@ async fn readiness(app: &axum::Router) -> Value { #[tokio::test] async fn chunk_storage_requires_live_complete_ownership_even_when_diskio_pid_is_alive() { let cluster = memory_kv::TestKvCluster::start().await; - let kv = Arc::new(CrowdbKvClient::new(ClientConfig::new( + let kv = Arc::new(PaxosKvClient::new(ClientConfig::new( cluster.mgmt_endpoints.clone(), ))); kv.seed_leader(0, 0, cluster.group0_leader_endpoint.clone()); diff --git a/app/crowdb-web/ui/src/App.tsx b/app/crowdb-web/ui/src/App.tsx index cf02db29b..3257782a6 100644 --- a/app/crowdb-web/ui/src/App.tsx +++ b/app/crowdb-web/ui/src/App.tsx @@ -1,5 +1,6 @@ import { PanelDivider } from './components/PanelDivider'; import { ClusterBreadcrumb } from './shell/ClusterBreadcrumb'; +import { PAXOS_KV_SERVER_MGMT_BASE, PAXOS_KV_SERVER_RPC_BASE, DISKDB_RPC_BASE } from './ports'; import { useClusterLayout } from './shell/useClusterLayout'; // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. @@ -367,13 +368,13 @@ function AppContent({ apiPrefix = '/api', readonly = false, modules, onEvent, ma const deployDialogDefaults = useMemo(() => { if (!dialog.deployServer?.nodeId) { - return { defaultRestPort: '19910', defaultRpcPort: '19920' }; + return { defaultRestPort: String(PAXOS_KV_SERVER_MGMT_BASE), defaultRpcPort: String(PAXOS_KV_SERVER_RPC_BASE) }; } return deployPortDefaultsForNode( servers, dialog.deployServer.nodeId, - 19910, - 19920, + PAXOS_KV_SERVER_MGMT_BASE, + PAXOS_KV_SERVER_RPC_BASE, rememberedDeployPorts.mgmt, rememberedDeployPorts.rpc, ); @@ -386,8 +387,8 @@ function AppContent({ apiPrefix = '/api', readonly = false, modules, onEvent, ma ...deployPortDefaultsForNode( servers, nextNodeId, - 19910, - 19920, + PAXOS_KV_SERVER_MGMT_BASE, + PAXOS_KV_SERVER_RPC_BASE, rememberedDeployPorts.mgmt, rememberedDeployPorts.rpc, ), @@ -403,7 +404,7 @@ function AppContent({ apiPrefix = '/api', readonly = false, modules, onEvent, ma ); const deployDiskdbDefaults = useMemo(() => { - if (!dialog.deployDiskdb?.nodeId) return '29920'; + if (!dialog.deployDiskdb?.nodeId) return String(DISKDB_RPC_BASE); return diskdbPortDefaultsForNode( diskdbInstances, dialog.deployDiskdb.nodeId, diff --git a/app/crowdb-web/ui/src/api.ts b/app/crowdb-web/ui/src/api.ts index e6bf57ee5..7decc19c9 100644 --- a/app/crowdb-web/ui/src/api.ts +++ b/app/crowdb-web/ui/src/api.ts @@ -433,7 +433,7 @@ export async function pingNode(nodeId: number, options?: RequestOptions): Promis } /** - * Deploy a crowdb-kv-server instance to a node + * Deploy a crowdb-paxos-kv-server instance to a node */ export async function deployServer( nodeId: number, @@ -512,7 +512,7 @@ export async function removeServer(nodeId: number, options?: RequestOptions): Pr } /** - * Get the OpenAPI spec for a node's crowdb-kv-server instance + * Get the OpenAPI spec for a node's crowdb-paxos-kv-server instance */ export async function getNodeOpenApi(nodeId: number, options?: RequestOptions): Promise<Record<string, any>> { const url = `/api/nodes/${encodeURIComponent(nodeId)}/openapi.json`; @@ -592,7 +592,7 @@ export interface AddStoreRequest { /** * Create a new cluster-wide store. The target nodes must already have a - * running `crowdb-kv-server` (deployed via `deployServer`). + * running `crowdb-paxos-kv-server` (deployed via `deployServer`). */ export async function addStore( req: AddStoreRequest, @@ -901,6 +901,7 @@ export interface AddDisksBatchResult { export interface DeployDiskdbRequest { rpc_port: number; + http_port?: number; keepalive_interval_secs?: number; } diff --git a/app/crowdb-web/ui/src/components/dialogs/AddNodeDialog.tsx b/app/crowdb-web/ui/src/components/dialogs/AddNodeDialog.tsx index efca22f0c..71e7539a1 100644 --- a/app/crowdb-web/ui/src/components/dialogs/AddNodeDialog.tsx +++ b/app/crowdb-web/ui/src/components/dialogs/AddNodeDialog.tsx @@ -69,11 +69,10 @@ export function AddNodeDialog({ isOpen, onClose, racks, defaultRackId, }))); }, [defaults.values]); - const allPorts = selected.flatMap(kind => [...Object.values(ports[kind] ?? {}).filter((value): value is number => typeof value === 'number'), ...(kind === 'diskdb' && ports[kind]?.rpc_port ? [ports[kind]!.rpc_port! + 1, ports[kind]!.rpc_port! + 2] : [])]); + const allPorts = selected.flatMap(kind => Object.values(ports[kind] ?? {}).filter((value): value is number => typeof value === 'number')); const validPorts = selected.every(kind => listenerFields(kind).every(([field]) => ports[kind]?.[field] != null)) && allPorts.every(port => Number.isInteger(port) && port > 0 && port <= 65535) - && new Set(allPorts).size === allPorts.length - && (!selected.includes('diskdb') || (ports.diskdb?.rpc_port ?? 0) <= 65533); + && new Set(allPorts).size === allPorts.length; const valid = /^\d+$/.test(nodeId) && Number.isSafeInteger(Number(nodeId)) && Number(nodeId) > 0 && !!rackId && !!host.trim() && validPorts && !!defaults.values; diff --git a/app/crowdb-web/ui/src/components/dialogs/DeployDiskdbDialog.tsx b/app/crowdb-web/ui/src/components/dialogs/DeployDiskdbDialog.tsx index 39ee6c8ed..206db773c 100644 --- a/app/crowdb-web/ui/src/components/dialogs/DeployDiskdbDialog.tsx +++ b/app/crowdb-web/ui/src/components/dialogs/DeployDiskdbDialog.tsx @@ -22,11 +22,12 @@ export function DeployDiskdbDialog({ onClose, nodes, defaultNodeId, - defaultRpcPort = '29920', + defaultRpcPort = '', onSuccess, }: DeployDiskdbDialogProps) { const [nodeId, setNodeId] = useState(''); const [rpcPort, setRpcPort] = useState(defaultRpcPort); + const [httpPort, setHttpPort] = useState(''); const [isLoading, setIsLoading] = useState(false); const wasOpenRef = useRef(false); const { success, error } = useToast(); @@ -46,10 +47,11 @@ export function DeployDiskdbDialog({ useEffect(() => { if (!defaults.values) return; setRpcPort(String(defaults.values.diskdb.rpc_port)); + setHttpPort(String(defaults.values.diskdb.http_port)); }, [defaults.values]); const isPort = (v: string) => /^\d+$/.test(v) && Number(v) > 0 && Number(v) < 65536; - const valid = nodeId !== '' && isPort(rpcPort); + const valid = nodeId !== '' && isPort(rpcPort) && isPort(httpPort) && rpcPort !== httpPort; const handleSubmit = async () => { if (!valid) return; @@ -57,6 +59,7 @@ export function DeployDiskdbDialog({ try { await deployDiskdb(Number(nodeId), { rpc_port: Number(rpcPort), + http_port: Number(httpPort), }); success(`DiskDB deployed on node ${nodeId}`); onClose(); @@ -99,6 +102,7 @@ export function DeployDiskdbDialog({ ))} </select> </div> + <Input label="HTTP Port (management)" inputMode="numeric" value={httpPort} onChange={e => setHttpPort(e.target.value)} /> <Input label="RPC Port (crowdb-rpc)" inputMode="numeric" diff --git a/app/crowdb-web/ui/src/components/dialogs/DeployServerDialog.tsx b/app/crowdb-web/ui/src/components/dialogs/DeployServerDialog.tsx index c693cf412..73291677b 100644 --- a/app/crowdb-web/ui/src/components/dialogs/DeployServerDialog.tsx +++ b/app/crowdb-web/ui/src/components/dialogs/DeployServerDialog.tsx @@ -18,7 +18,7 @@ export interface DeployServerDialogProps { } /** - * Deploy a `crowdb-kv-server` instance on a node. Required before that + * Deploy a `crowdb-paxos-kv-server` instance on a node. Required before that * node can host any store / group / replica. Backend contract: * `crowdb-console/web/src/lifecycle.rs::DeployNodeServerBody`. */ @@ -26,8 +26,8 @@ export function DeployServerDialog({ isOpen, onClose, nodeId, - defaultRestPort = '19910', - defaultRpcPort = '19920', + defaultRestPort = '', + defaultRpcPort = '', onSuccess, }: DeployServerDialogProps) { const [restPort, setRestPort] = useState(defaultRestPort); diff --git a/app/crowdb-web/ui/src/components/dialogs/createFlows.test.tsx b/app/crowdb-web/ui/src/components/dialogs/createFlows.test.tsx index 15a076567..dafbe0096 100644 --- a/app/crowdb-web/ui/src/components/dialogs/createFlows.test.tsx +++ b/app/crowdb-web/ui/src/components/dialogs/createFlows.test.tsx @@ -10,6 +10,7 @@ import { AddNodeDialog } from './AddNodeDialog'; import { AddStoreDialog } from './AddStoreDialog'; import { AddGroupDialog } from './AddGroupDialog'; import { AddReplicaDialog } from './AddReplicaDialog'; +import { DeployDiskdbDialog } from './DeployDiskdbDialog'; import { DeployServerDialog } from './DeployServerDialog'; import { serviceOrder, serviceLabels } from '../../services/useNodeServicePlans'; import { NodeHealth, ProcState } from '../../types'; @@ -36,7 +37,7 @@ interface CapturedRequest { let captured: CapturedRequest[] = []; const deploymentDefaults = { 'paxos-kv': { instance_id: '1', http_port: 19911, rpc_port: 19921 }, - diskdb: { instance_id: '1', rpc_port: 29921 }, + diskdb: { instance_id: '1', rpc_port: 14135, http_port: 13135 }, diskio: { instance_id: '1', rpc_port: 13010 }, chunkdb: { instance_id: '1', http_port: 12010, rpc_port: 12110 }, 'chunk-kv': { instance_id: '1', http_port: 15010, rpc_port: 15110 }, @@ -166,7 +167,7 @@ describe('Add Node dialog', () => { const cards = screen.getByRole('list', { name: 'Default services' }); expect(within(cards).getAllByRole('checkbox').map(input => input.getAttribute('aria-label'))).toEqual(serviceOrder.map(kind => serviceLabels[kind])); for (const kind of serviceOrder.filter(kind => kind !== 'access-server')) { - expect(within(screen.getByTestId(`service-card-${kind}`)).getAllByRole('textbox')).toHaveLength(1); + expect(within(screen.getByTestId(`service-card-${kind}`)).getAllByRole('textbox')).toHaveLength(kind === 'diskio' ? 1 : 2); expect(screen.getByLabelText(`${serviceLabels[kind]} RPC port`)).toBeEnabled(); } fireEvent.click(screen.getByLabelText('crowdb-access-server')); @@ -282,17 +283,17 @@ describe('Deploy Server dialog', () => { [ { id: 1, - server: { mgmt_url: 'http://127.0.0.1:19910', rpc_url: 'http://127.0.0.1:19920' }, + server: { mgmt_url: 'http://127.0.0.1:10135', rpc_url: 'http://127.0.0.1:11135' }, }, { id: 2, - server: { mgmt_url: 'http://127.0.0.1:19910', rpc_url: 'http://127.0.0.1:19920' }, + server: { mgmt_url: 'http://127.0.0.1:10135', rpc_url: 'http://127.0.0.1:11135' }, }, ], 1, ); - expect(defaults).toEqual({ defaultRestPort: '19911', defaultRpcPort: '19921' }); + expect(defaults).toEqual({ defaultRestPort: '10136', defaultRpcPort: '11136' }); }); it('increments globally when a different node already uses the base ports', () => { @@ -300,77 +301,38 @@ describe('Deploy Server dialog', () => { [ { id: 2, - server: { mgmt_url: 'http://127.0.0.1:19910', rpc_url: 'http://127.0.0.1:19920' }, + server: { mgmt_url: 'http://127.0.0.1:10135', rpc_url: 'http://127.0.0.1:11135' }, }, ], 1, ); - expect(defaults).toEqual({ defaultRestPort: '19911', defaultRpcPort: '19921' }); + expect(defaults).toEqual({ defaultRestPort: '10136', defaultRpcPort: '11136' }); }); it('derives defaults from the node id suffix before checking collisions', () => { - const defaults = deployPortDefaultsForNode([], 2, 19910, 19920); + const defaults = deployPortDefaultsForNode([], 2, 10135, 11135); - expect(defaults).toEqual({ defaultRestPort: '19912', defaultRpcPort: '19922' }); + expect(defaults).toEqual({ defaultRestPort: '10136', defaultRpcPort: '11136' }); }); it('can increment from remembered same-node ports even after the server is gone', () => { - const defaults = deployPortDefaultsForNode([], 1, 19910, 19920, [19910], [19920]); + const defaults = deployPortDefaultsForNode([], 1, 10135, 11135, [10135], [11135]); - expect(defaults).toEqual({ defaultRestPort: '19911', defaultRpcPort: '19921' }); + expect(defaults).toEqual({ defaultRestPort: '10136', defaultRpcPort: '11136' }); }); }); describe('diskdbPortDefaultsForNode', () => { - it('returns the base port when no instances or remembered ports collide', () => { - expect(diskdbPortDefaultsForNode([], 1)).toBe('29921'); + it('uses node offsets zero, one and two', () => { + expect([1, 2, 3].map(id => diskdbPortDefaultsForNode([], id))).toEqual(['14135', '14136', '14137']); }); - - it('derives defaults from the node id suffix before checking collisions', () => { - expect(diskdbPortDefaultsForNode([], 2)).toBe('29922'); - }); - - it('increments past ports already assigned to other diskdb instances', () => { - const instances = [{ rpc_endpoint: 'http://127.0.0.1:29921' }]; - expect(diskdbPortDefaultsForNode(instances, 1)).toBe('29922'); - }); - - it('skips the complete range for remembered ports even when no instances exist', () => { - expect(diskdbPortDefaultsForNode([], 1, undefined, [29921])).toBe('29924'); - }); - - it('allocates disjoint 3-port ranges across nodes 1, 2, and 3', () => { - // Simulate sequential allocation: each node's base is remembered so - // the next node skips past the full derived range (base, base+1, base+2). - const node1 = Number(diskdbPortDefaultsForNode([], 1)); - const node2 = Number(diskdbPortDefaultsForNode([], 2, undefined, [node1])); - const node3 = Number(diskdbPortDefaultsForNode([], 3, undefined, [node1, node2])); - - const ranges = [node1, node2, node3].map((b) => [b, b + 1, b + 2]); - const all = ranges.flat(); - expect(new Set(all).size).toBe(all.length); - // Node 1 starts at its suffix offset (29921). - expect(node1).toBe(29921); - // Node 2's suffix offset (29922) falls inside node 1's range, so it - // skips to the first free 3-port window. - expect(node2).toBe(29924); - // Node 3's suffix offset (29923) falls inside node 1's range, and - // 29924-29926 are node 2's, so it skips further. - expect(node3).toBe(29927); - }); - - it('detects collisions on each derived listener port (main, HTTP, crowdb-rpc)', () => { - // An existing instance with crowdb-rpc endpoint 29923 means its main - // listener is 29921 and HTTP is 29922 — all three collide with node 1's - // default window [29921, 29922, 29923]. - expect(diskdbPortDefaultsForNode([{ rpc_endpoint: 'http://127.0.0.1:29923' }], 1)).toBe('29924'); - // Endpoint 29922 → main 29920, HTTP 29921, rpc 29922. Node 1's window - // [29921, 29922, 29923] collides on HTTP and rpc. - expect(diskdbPortDefaultsForNode([{ rpc_endpoint: 'http://127.0.0.1:29922' }], 1)).toBe('29923'); - // Endpoint 29921 → main 29919, HTTP 29920, rpc 29921. Node 1's window - // collides only on the main port (29921). - expect(diskdbPortDefaultsForNode([{ rpc_endpoint: 'http://127.0.0.1:29921' }], 1)).toBe('29922'); + it('skips active RPC endpoints and remembered ports', () => { + expect(diskdbPortDefaultsForNode([{ rpc_endpoint: 'http://127.0.0.1:14135' }], 1)).toBe('14136'); + expect(diskdbPortDefaultsForNode([], 1, undefined, [14135, 14136])).toBe('14137'); + }); + it('does not reserve imaginary main or HTTP listeners adjacent to RPC', () => { + expect(diskdbPortDefaultsForNode([{ rpc_endpoint: 'http://127.0.0.1:14137' }], 1)).toBe('14135'); }); }); @@ -888,3 +850,19 @@ describe('end-to-end create flow', () => { expect(captured[5].body).toEqual({ node_id: 2, replica_id: 701 }); }); }); + + +describe('Deploy DiskDB independent listeners', () => { + it('submits HTTP and the exact RPC port returned by the server', async () => { + installFetchMock({ node_id: 1, endpoint: 'http://127.0.0.1:14135', pid: 1 }); + render(<DeployDiskdbDialog isOpen nodes={[{ id: 1 }]} onClose={() => {}} />, { wrapper }); + await waitFor(() => expect(screen.getByLabelText('RPC Port (crowdb-rpc)')).toHaveValue('14135')); + expect(screen.getByLabelText('HTTP Port (management)')).toHaveValue('13135'); + fireEvent.change(screen.getByLabelText('HTTP Port (management)'), { target: { value: '14135' } }); + expect(screen.getByRole('button', { name: /^Deploy$/ })).toBeDisabled(); + fireEvent.change(screen.getByLabelText('HTTP Port (management)'), { target: { value: '13135' } }); + fireEvent.click(screen.getByRole('button', { name: /^Deploy$/ })); + await waitFor(() => expect(captured).toHaveLength(1)); + expect(captured[0]).toMatchObject({ url: '/api/nodes/1/diskdb/deploy', body: { rpc_port: 14135, http_port: 13135 } }); + }); +}); diff --git a/app/crowdb-web/ui/src/components/dialogs/defaults.ts b/app/crowdb-web/ui/src/components/dialogs/defaults.ts index 9bb324a1f..346fb23ae 100644 --- a/app/crowdb-web/ui/src/components/dialogs/defaults.ts +++ b/app/crowdb-web/ui/src/components/dialogs/defaults.ts @@ -1,13 +1,10 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -// Per-service port bases. Ports are generated dynamically from these -// bases (offset by the node-id suffix, then incremented past collisions -// against ports already assigned in the console config). The bases are -// dev defaults; production should override via a deployment config story. -export const KV_SERVER_REST_PORT_BASE = 19910; -export const KV_SERVER_RPC_PORT_BASE = 19920; -export const DISKDB_RPC_PORT_BASE = 29920; +import { PAXOS_KV_SERVER_MGMT_BASE, PAXOS_KV_SERVER_RPC_BASE, DISKDB_RPC_BASE } from '../../ports'; +export const KV_SERVER_REST_PORT_BASE = PAXOS_KV_SERVER_MGMT_BASE; +export const KV_SERVER_RPC_PORT_BASE = PAXOS_KV_SERVER_RPC_BASE; +export const DISKDB_RPC_PORT_BASE = DISKDB_RPC_BASE; const DIGIT_SUFFIX = /(\d+)$/; @@ -125,7 +122,7 @@ function preferredPortStart(base: number, nodeId: number): number { const suffix = Number(match[1]); if (!Number.isFinite(suffix) || suffix < 0) return base; - return base + suffix; + return base + Math.max(0, suffix - 1); } export function deployPortDefaultsForNode( @@ -178,26 +175,6 @@ export function diskdbPortDefaultsForNode( rpcStart = DISKDB_RPC_PORT_BASE, extraUsedRpcPorts: number[] = [], ): string { - // The public input is the base listener. DiskDB also binds base + 1 for - // HTTP readiness and base + 2 for crowdb-rpc, so reserve the whole range. - const usedPorts = new Set<number>(); - for (const base of extraUsedRpcPorts) { - usedPorts.add(base); - usedPorts.add(base + 1); - usedPorts.add(base + 2); - } - for (const inst of instances) { - const endpoint = extractPort(inst.rpc_endpoint); - if (endpoint) { - usedPorts.add(endpoint - 2); - usedPorts.add(endpoint - 1); - usedPorts.add(endpoint); - } - } - - let candidate = preferredPortStart(rpcStart, nodeId); - while (candidate < 65534 && [candidate, candidate + 1, candidate + 2].some((port) => usedPorts.has(port))) { - candidate += 1; - } - return String(candidate); + const usedPorts = [...extraUsedRpcPorts, ...instances.map(instance => extractPort(instance.rpc_endpoint)).filter((port): port is number => port !== null)]; + return nextAvailablePort(usedPorts, preferredPortStart(rpcStart, nodeId)); } diff --git a/app/crowdb-web/ui/src/embed.ts b/app/crowdb-web/ui/src/embed.ts index 4ea968798..ebce9eb0a 100644 --- a/app/crowdb-web/ui/src/embed.ts +++ b/app/crowdb-web/ui/src/embed.ts @@ -6,7 +6,7 @@ * * Usage from a host React app: * import { CrowdbConsole } from 'crowdb-console'; - * <CrowdbConsole apiPrefix="/storage/crowdb-kv/api" readonly /> + * <CrowdbConsole apiPrefix="/storage/crowdb-paxos-kv/api" readonly /> */ export { default as CrowdbConsole, type CrowdbConsoleProps } from './App'; export { Domain } from './types'; diff --git a/app/crowdb-web/ui/src/ports.ts b/app/crowdb-web/ui/src/ports.ts new file mode 100644 index 000000000..cf7362374 --- /dev/null +++ b/app/crowdb-web/ui/src/ports.ts @@ -0,0 +1,21 @@ +// Copyright 2026-present Gian <crow.db@outlook.com> +// Licensed under the Apache License, Version 2.0. + +// Generated from ports.rs by tools/ports/generate.py; do not edit. +export const PAXOS_KV_SERVER_MGMT_BASE = 10135; +export const PAXOS_KV_SERVER_RPC_BASE = 11135; +export const DISKDB_HTTP_BASE = 13135; +export const DISKDB_RPC_BASE = 14135; +export const CHUNKDB_HTTP_BASE = 16135; +export const CHUNKDB_RPC_BASE = 17135; +export const DISKIO_RPC_BASE = 18135; +export const WEB_BASE = 6135; +export const CHUNK_KV_HTTP_BASE = 20135; +export const CHUNK_KV_RPC_BASE = 21135; +export const ACCESS_SERVER_HTTP_BASE = 7135; +export const ACCESS_SERVER_ICEBERG_HTTP_BASE = 8135; +export const ACCESS_SERVER_DATASET_HTTP_BASE = 9135; +export const ACCESS_SERVER_HEALTH_HTTP_BASE = 25135; +export const MONITOR_HTTP_BASE = 26135; +export const SSH_PORT = 2222; +export const HOST_SSH_PORT = 22; diff --git a/app/crowdb-web/ui/src/services/DeployServiceDialog.tsx b/app/crowdb-web/ui/src/services/DeployServiceDialog.tsx index 6cedf86f8..274fd025b 100644 --- a/app/crowdb-web/ui/src/services/DeployServiceDialog.tsx +++ b/app/crowdb-web/ui/src/services/DeployServiceDialog.tsx @@ -19,12 +19,11 @@ export function DeployServiceDialog({ nodeId, kind, servers, stores, diskGroups, const value = /^\d+$/.test(suffix) ? BigInt(suffix) : 0n; return value > maximum ? value : maximum; }, 0n) + 1n)); - const base = { chunkdb: 43200, diskio: 44000, 'chunk-kv': 45100, 'access-server': 9092 }[kind]; - const [httpPort, setHttpPort] = useState(String(base)); - const [rpcPort, setRpcPort] = useState(String(kind === 'diskio' ? base : base + 100)); - const [s3Port, setS3Port] = useState('9091'); - const [datasetPort, setDatasetPort] = useState('9093'); - const [healthPort, setHealthPort] = useState('9094'); + const [httpPort, setHttpPort] = useState(''); + const [rpcPort, setRpcPort] = useState(''); + const [s3Port, setS3Port] = useState(''); + const [datasetPort, setDatasetPort] = useState(''); + const [healthPort, setHealthPort] = useState(''); const [diskGroup, setDiskGroup] = useState(String(diskGroups[0]?.id ?? '')); const [store, setStore] = useState(String(stores.find(store => String(store.store_id) !== '0')?.store_id ?? stores[0]?.store_id ?? '')); const firstGroup = stores.find(entry => String(entry.store_id) === store)?.groups.find(entry => String(entry.group_id) !== '0'); diff --git a/app/crowdb-web/ui/src/services/ServicePlanCard.tsx b/app/crowdb-web/ui/src/services/ServicePlanCard.tsx index d28ac86bf..d756d7112 100644 --- a/app/crowdb-web/ui/src/services/ServicePlanCard.tsx +++ b/app/crowdb-web/ui/src/services/ServicePlanCard.tsx @@ -7,7 +7,7 @@ type Ports = Omit<NonNullable<ServiceOverrides[ServiceKind]>, 'dynamic_ownership export function listenerFields(kind: ServiceKind): [keyof Ports, string][] { return kind === 'access-server' ? [['http_port', 'Iceberg'], ['s3_port', 'S3'], ['dataset_port', 'Dataset'], ['health_port', 'Health']] - : [['rpc_port', 'RPC']]; + : kind === 'diskio' ? [['rpc_port', 'RPC']] : [['http_port', 'HTTP'], ['rpc_port', 'RPC']]; } export function ServicePlanCard({ kind, enabled, ports, onToggle, onChange }: { kind: ServiceKind; enabled: boolean; ports: Ports; onToggle: () => void; diff --git a/app/crowdb-web/ui/src/services/useNodeServicePlans.ts b/app/crowdb-web/ui/src/services/useNodeServicePlans.ts index 079ea5fc8..f557526d1 100644 --- a/app/crowdb-web/ui/src/services/useNodeServicePlans.ts +++ b/app/crowdb-web/ui/src/services/useNodeServicePlans.ts @@ -242,7 +242,7 @@ export function useNodeServicePlans(stores: EnrichedStoreView[], groups: Record< if (disposed || stopped.current) break; const value = { ...defaults[kind], ...(overrides.current[id]?.[kind] ?? {}) }; if (kind === 'paxos-kv') await deployServer(id, { rest_port: value.http_port!, rpc_port: value.rpc_port! }); - else if (kind === 'diskdb') await deployDiskdb(id, { rpc_port: value.rpc_port!, ...(value.keepalive_interval_secs === undefined ? {} : { keepalive_interval_secs: value.keepalive_interval_secs }) }); + else if (kind === 'diskdb') await deployDiskdb(id, { rpc_port: value.rpc_port!, ...(value.http_port === undefined ? {} : { http_port: value.http_port }), ...(value.keepalive_interval_secs === undefined ? {} : { keepalive_interval_secs: value.keepalive_interval_secs }) }); else await serviceRequest(`/nodes/${id}/services/deploy`, 'POST', { kind, ...value, ...(kind === 'diskio' ? { disk_group_id: diskGroup?.id ?? 0 } : {}), diff --git a/app/crowdb-web/ui/src/shell/NodeSetupOptions.tsx b/app/crowdb-web/ui/src/shell/NodeSetupOptions.tsx index 6e68d0c83..02607f34f 100644 --- a/app/crowdb-web/ui/src/shell/NodeSetupOptions.tsx +++ b/app/crowdb-web/ui/src/shell/NodeSetupOptions.tsx @@ -1,6 +1,7 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. +import { PAXOS_KV_SERVER_MGMT_BASE, PAXOS_KV_SERVER_RPC_BASE } from '../ports'; import { Input, Select } from '../components/ui/Input'; import { serviceLabels } from '../services/useNodeServicePlans'; import type { DiskInventory } from '../storage/types'; @@ -23,7 +24,7 @@ export function NodeSetupOptions({ choices, inventory, disabled, onChange }: Pro return <fieldset disabled={disabled} className="tw-space-y-4"> <section aria-label="Node service choices" className="tw-space-y-2"> <h3 className="tw-font-semibold">Services</h3> - <p className="tw-text-xs tw-text-muted">Selected services start on the node after their prerequisites are ready. Blank listeners use available defaults. KV bootstrap retains ports 10000 / 10100.</p> + <p className="tw-text-xs tw-text-muted">Selected services start on the node after their prerequisites are ready. Blank listeners use available defaults. KV bootstrap uses ports {PAXOS_KV_SERVER_MGMT_BASE} / {PAXOS_KV_SERVER_RPC_BASE}.</p> <p className="tw-text-xs tw-text-muted">Unselecting a service retains any existing deployment, including the admitted KV process.</p> <div className="tw-grid tw-grid-cols-1 sm:tw-grid-cols-2 tw-gap-2">{choices.services.map((service, index) => <div key={service.kind} className="tw-rounded tw-border tw-border-border tw-p-2 tw-space-y-2"> <label className="tw-flex tw-gap-2 tw-text-sm"><input type="checkbox" checked={service.enabled} onChange={event => updateService(index, { enabled: event.target.checked })} />{serviceLabels[service.kind]}</label> diff --git a/app/crowdb-web/ui/src/shell/nodeSetup.ts b/app/crowdb-web/ui/src/shell/nodeSetup.ts index 631816fa7..220549af6 100644 --- a/app/crowdb-web/ui/src/shell/nodeSetup.ts +++ b/app/crowdb-web/ui/src/shell/nodeSetup.ts @@ -46,11 +46,7 @@ export function validNodeSetup(choices: NodeSetupChoices): boolean { ports.add(Number(value)); } } - if (service.kind === 'diskdb' && service.listeners.rpc_port != null) { - const base = service.listeners.rpc_port; - if (base > 65533 || ports.has(base + 1) || ports.has(base + 2)) return false; - ports.add(base + 1); ports.add(base + 2); - } + } const storage = choices.storage; if (storage.mode === 'diskless') return true; diff --git a/app/crowdb-web/ui/src/types/index.ts b/app/crowdb-web/ui/src/types/index.ts index ee72c3b5d..24fa3b770 100644 --- a/app/crowdb-web/ui/src/types/index.ts +++ b/app/crowdb-web/ui/src/types/index.ts @@ -86,7 +86,7 @@ export interface NodeStore { } // Batched crowdb-tree engine diagnostics (doc/todo-sm.md Step 6); mirrors -// crowdb-kv's `CrowdbTreeStatsView`. Present only when the group's engine is +// crowdb-paxos-kv's `CrowdbTreeStatsView`. Present only when the group's engine is // `CrowdbTreeEngine` (absent/undefined for `InMemKV`). export interface CrowdbTreeStats { last_applied_slot: number; @@ -257,7 +257,7 @@ export interface ApiError { details?: any; } -// Election/lease state (mirrors crowdb-kv's ElectionStateView) +// Election/lease state (mirrors crowdb-paxos-kv's ElectionStateView) export interface ElectionState { election_count?: number; current_term: number; @@ -269,7 +269,7 @@ export interface ElectionState { step_downs_admin?: number; } -// Read-path state gauges (mirrors crowdb-kv's ReadStateView) +// Read-path state gauges (mirrors crowdb-paxos-kv's ReadStateView) export interface ReadState { lease_valid: number; contiguous_applied: number; diff --git a/app/crowdb-web/ui/vite.config.ts b/app/crowdb-web/ui/vite.config.ts index 9f2386542..dc7e2a272 100644 --- a/app/crowdb-web/ui/vite.config.ts +++ b/app/crowdb-web/ui/vite.config.ts @@ -1,27 +1,22 @@ /// <reference types="vitest" /> import { defineConfig } from "vite"; import react from "@vitejs/plugin-react"; +import { WEB_BASE } from "./src/ports"; // During `npm run dev`, Vite serves the SPA on port 5173 and proxies -// `/api/*` and `/healthz` to the local Axum backend on 9090 (the -// crowdb-web default). For production, `npm run build` +// `/api/*` and `/healthz` to the default local Axum backend. +// For production, `npm run build` // emits to `dist/`, which Axum serves directly via ServeDir. +const backend = `http://127.0.0.1:${WEB_BASE}`; +const proxy = { "/api": backend, "/healthz": backend, "/internal": backend }; export default defineConfig({ plugins: [react()], server: { port: 5173, - proxy: { - "/api": "http://127.0.0.1:9090", - "/healthz": "http://127.0.0.1:9090", - "/internal": "http://127.0.0.1:9090", - }, + proxy, }, preview: { - proxy: { - "/api": "http://127.0.0.1:9090", - "/healthz": "http://127.0.0.1:9090", - "/internal": "http://127.0.0.1:9090", - }, + proxy, }, build: { outDir: "dist", diff --git a/container/crowdb-monitor/Cargo.toml b/container/crowdb-monitor/Cargo.toml index d2b6b6e53..71c56da5e 100644 --- a/container/crowdb-monitor/Cargo.toml +++ b/container/crowdb-monitor/Cargo.toml @@ -18,7 +18,7 @@ test-util = [] crowdb-console-shared = { path = "../../lib/crowdb-console-shared" } clap = { version = "4", features = ["derive", "env"] } crowdb-chunk-kv-client = { path = "../../lib/crowdb-chunk-kv-client" } -crowdb-kv-client = { path = "../../lib/crowdb-kv-client" } +crowdb-paxos-kv-client = { path = "../../lib/crowdb-paxos-kv-client" } crowdb-diskio-client = { path = "../../lib/crowdb-diskio-client" } crowdb-protocol = { path = "../../lib/crowdb-protocol" } crowdb-rpc-ffi = { path = "../../lib/crowdb-rpc/ffi" } @@ -42,7 +42,7 @@ if-addrs = "0.13" [dev-dependencies] ssh-key = { version = "0.6", features = ["ed25519"] } -crowdb-kv = { path = "../../lib/crowdb-kv" } +crowdb-paxos-kv = { path = "../../lib/crowdb-paxos-kv" } crowdb-monitor = { path = ".", features = ["test-util"] } crowdb-rpc-ffi = { path = "../../lib/crowdb-rpc/ffi", features = ["test-util"] } crowdb-test-harness = { path = "../../lib/crowdb-test-harness" } diff --git a/container/crowdb-monitor/src/bootstrap/chunk.rs b/container/crowdb-monitor/src/bootstrap/chunk.rs index a909d0030..aa1e6274c 100644 --- a/container/crowdb-monitor/src/bootstrap/chunk.rs +++ b/container/crowdb-monitor/src/bootstrap/chunk.rs @@ -1,7 +1,7 @@ use std::fs; use std::time::Duration; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient, ServiceRegistryClient}; use crowdb_protocol::chunk_kv::Id128; use serde::Deserialize; use thiserror::Error; @@ -17,7 +17,7 @@ pub enum ChunkBootstrapError { #[error("chunk service configuration failed: {0}")] Io(#[from] std::io::Error), #[error("chunk service registry lookup failed: {0}")] - Registry(#[from] crowdb_kv_client::Error), + Registry(#[from] crowdb_paxos_kv_client::Error), #[error("chunk service configuration is invalid: {0}")] Invalid(&'static str), } @@ -106,7 +106,7 @@ pub async fn verify_chunk_services( if expected.is_empty() { return Ok(()); } - let registry = ServiceRegistryClient::new(CrowdbKvClient::new(ClientConfig::new(vec![ + let registry = ServiceRegistryClient::new(PaxosKvClient::new(ClientConfig::new(vec![ management_seed.to_owned() ]))); registry.kv().refresh_topology().await?; diff --git a/container/crowdb-monitor/src/bootstrap/hardware.rs b/container/crowdb-monitor/src/bootstrap/hardware.rs index 77f7a3d28..bc4680d15 100644 --- a/container/crowdb-monitor/src/bootstrap/hardware.rs +++ b/container/crowdb-monitor/src/bootstrap/hardware.rs @@ -1,6 +1,6 @@ use std::collections::BTreeSet; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, HardwareClient}; +use crowdb_paxos_kv_client::{ClientConfig, HardwareClient, PaxosKvClient}; use crowdb_protocol::common::{DiskId, HwStatus, NodeValue, RackValue}; use crowdb_protocol::diskdb::rpc::{DiskGroupValue, DiskType, DiskValue}; use thiserror::Error; @@ -16,7 +16,7 @@ const UNIT_BYTES: u64 = 1024 * 1024; #[derive(Debug, Error)] pub enum HardwareBootstrapError { #[error("Group 0 hardware request failed: {0}")] - Client(#[from] crowdb_kv_client::Error), + Client(#[from] crowdb_paxos_kv_client::Error), #[error("bootstrap manifest failed: {0}")] Manifest(#[from] ManifestError), #[error("monitor lifecycle log failed: {0}")] @@ -69,7 +69,7 @@ pub struct HardwareBootstrap { impl HardwareBootstrap { #[must_use] pub fn new(management_seed: String) -> Self { - let kv = CrowdbKvClient::new(ClientConfig::new(vec![management_seed])); + let kv = PaxosKvClient::new(ClientConfig::new(vec![management_seed])); Self { client: HardwareClient::new(kv), } @@ -444,7 +444,7 @@ fn disk_matches(actual: &DiskValue, expected: &DiskValue) -> bool { } fn verify_written( - write: Result<(), crowdb_kv_client::Error>, + write: Result<(), crowdb_paxos_kv_client::Error>, visible: bool, ) -> Result<(), HardwareBootstrapError> { if visible { diff --git a/container/crowdb-monitor/src/bootstrap/iceberg.rs b/container/crowdb-monitor/src/bootstrap/iceberg.rs index 0a16f4b60..dd13d2e95 100644 --- a/container/crowdb-monitor/src/bootstrap/iceberg.rs +++ b/container/crowdb-monitor/src/bootstrap/iceberg.rs @@ -40,7 +40,7 @@ pub enum IcebergBootstrapError { #[error("Iceberg bootstrap manifest failed: {0}")] Manifest(#[from] ManifestError), #[error("Iceberg bootstrap authority failed: {0}")] - Authority(#[from] crowdb_kv_client::Error), + Authority(#[from] crowdb_paxos_kv_client::Error), #[error("monitor lifecycle log failed: {0}")] MonitorLog(#[from] MonitorLogError), } @@ -95,7 +95,7 @@ impl IcebergBootstrap { program: &std::path::Path, seeds: &str, credentials: &ServerCredentials, - kv: &crowdb_kv_client::CrowdbKvClient, + kv: &crowdb_paxos_kv_client::PaxosKvClient, ) -> Result<(), IcebergBootstrapError> { let operations = shared_console::operations(root, kv).await?; console::ensure_with_operations( @@ -115,7 +115,7 @@ impl IcebergBootstrap { /// Propagates the production journal and Group-0 reservation errors. pub async fn shared_console_operations_for_tests( root: &std::path::Path, - kv: &crowdb_kv_client::CrowdbKvClient, + kv: &crowdb_paxos_kv_client::PaxosKvClient, ) -> Result<(Uuid, Uuid), IcebergBootstrapError> { shared_console::operations(root, kv).await } diff --git a/container/crowdb-monitor/src/bootstrap/iceberg/shared_console.rs b/container/crowdb-monitor/src/bootstrap/iceberg/shared_console.rs index 47c676442..82cf0d421 100644 --- a/container/crowdb-monitor/src/bootstrap/iceberg/shared_console.rs +++ b/container/crowdb-monitor/src/bootstrap/iceberg/shared_console.rs @@ -4,7 +4,7 @@ //! One cluster-scoped Console catalog operation, without per-node races. use super::{console, IcebergBootstrapError}; -use crowdb_kv_client::{CrowdbKvClient, Error, GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{Error, GetOutcome, PaxosKvClient, ReadMode}; use serde::{Deserialize, Serialize}; use std::{path::Path, time::Duration}; use uuid::Uuid; @@ -21,14 +21,14 @@ struct Operations { pub(super) async fn operations( root: &Path, - kv: &CrowdbKvClient, + kv: &PaxosKvClient, ) -> Result<(Uuid, Uuid), IcebergBootstrapError> { tokio::time::timeout(Duration::from_secs(3), reserve(root, kv)) .await .map_err(|_| IcebergBootstrapError::Command("Console catalog authority reservation timed out"))? } -async fn reserve(root: &Path, kv: &CrowdbKvClient) -> Result<(Uuid, Uuid), IcebergBootstrapError> { +async fn reserve(root: &Path, kv: &PaxosKvClient) -> Result<(Uuid, Uuid), IcebergBootstrapError> { let local = console::load(&root.join("console-iceberg-bootstrap.json"))?; let candidate = Operations { version: 1, diff --git a/container/crowdb-monitor/src/bootstrap/logical.rs b/container/crowdb-monitor/src/bootstrap/logical.rs index ac6b595e9..c049daa61 100644 --- a/container/crowdb-monitor/src/bootstrap/logical.rs +++ b/container/crowdb-monitor/src/bootstrap/logical.rs @@ -1,6 +1,6 @@ use std::collections::{BTreeMap, BTreeSet}; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, CrowdbSysmdClient}; +use crowdb_paxos_kv_client::{ClientConfig, CrowdbSysmdClient, PaxosKvClient}; use crowdb_protocol::common::{GroupValue, ReplicaValue, StoreValue}; use thiserror::Error; @@ -14,7 +14,7 @@ const STEP: &str = "logical-topology"; #[derive(Debug, Error)] pub enum LogicalBootstrapError { #[error("Group 0 logical topology request failed: {0}")] - Client(#[from] crowdb_kv_client::Error), + Client(#[from] crowdb_paxos_kv_client::Error), #[error("bootstrap manifest failed: {0}")] Manifest(#[from] ManifestError), #[error("monitor lifecycle log failed: {0}")] @@ -40,7 +40,7 @@ impl LogicalBootstrap { #[must_use] pub fn new(management_seed: String) -> Self { Self { - client: CrowdbSysmdClient::new(CrowdbKvClient::new(ClientConfig::new(vec![management_seed]))), + client: CrowdbSysmdClient::new(PaxosKvClient::new(ClientConfig::new(vec![management_seed]))), } } @@ -221,7 +221,7 @@ fn expected(profile: &DeploymentProfile) -> Expected { } fn verify_write( - write: Result<(), crowdb_kv_client::Error>, + write: Result<(), crowdb_paxos_kv_client::Error>, matches: bool, ) -> Result<(), LogicalBootstrapError> { if matches { diff --git a/container/crowdb-monitor/src/bootstrap/storage_probe.rs b/container/crowdb-monitor/src/bootstrap/storage_probe.rs index eae7d7e26..8bb4e9d92 100644 --- a/container/crowdb-monitor/src/bootstrap/storage_probe.rs +++ b/container/crowdb-monitor/src/bootstrap/storage_probe.rs @@ -1,7 +1,7 @@ use std::time::{Duration, SystemTime, UNIX_EPOCH}; use crowdb_diskio_client::{DiskId, DiskioClient, DiskioClientConfig, DiskioError, OperationOptions}; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, ServiceRegistryClient}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient, ServiceRegistryClient}; use thiserror::Error; use tokio::time::{sleep, Instant}; @@ -15,7 +15,7 @@ pub enum StorageProbeError { #[error("DiskIO probe failed: {0}")] Diskio(#[from] DiskioError), #[error("DiskIO registration lookup failed: {0}")] - Registry(#[from] crowdb_kv_client::Error), + Registry(#[from] crowdb_paxos_kv_client::Error), #[error("DiskIO profile is invalid: {0}")] Invalid(&'static str), #[error("DiskIO registration did not become ready")] @@ -69,7 +69,7 @@ async fn wait_for_registration( .first() .ok_or(StorageProbeError::Invalid("disk is absent"))? .disk_group_id; - let registry = ServiceRegistryClient::new(CrowdbKvClient::new(ClientConfig::new(vec![ + let registry = ServiceRegistryClient::new(PaxosKvClient::new(ClientConfig::new(vec![ management_seed.to_owned() ]))); registry.kv().refresh_topology().await?; diff --git a/container/crowdb-monitor/src/main.rs b/container/crowdb-monitor/src/main.rs index ca27814eb..c6aae9359 100644 --- a/container/crowdb-monitor/src/main.rs +++ b/container/crowdb-monitor/src/main.rs @@ -24,7 +24,7 @@ enum Command { seed: Vec<String>, #[arg(long)] data_root: PathBuf, - #[arg(long, default_value = "0.0.0.0:9095")] + #[arg(long, default_value_t = std::net::SocketAddr::from(([0, 0, 0, 0], crowdb_protocol::MONITOR_HTTP_BASE)))] bind: std::net::SocketAddr, #[arg(long, required = true)] interface: Vec<String>, diff --git a/container/crowdb-monitor/src/node/runtime.rs b/container/crowdb-monitor/src/node/runtime.rs index ea54bcfe8..b9f87ab7f 100644 --- a/container/crowdb-monitor/src/node/runtime.rs +++ b/container/crowdb-monitor/src/node/runtime.rs @@ -251,7 +251,7 @@ async fn verify_binding(binding: &NodeBinding) -> Result<()> { if binding.management_seeds.is_empty() || binding.management_seeds.len() > 16 { return Err("invalid authority seeds".into()); } - let client = crowdb_kv_client::CrowdbKvClient::new(crowdb_kv_client::ClientConfig::new( + let client = crowdb_paxos_kv_client::PaxosKvClient::new(crowdb_paxos_kv_client::ClientConfig::new( binding.management_seeds.clone(), )); let found = client @@ -259,11 +259,11 @@ async fn verify_binding(binding: &NodeBinding) -> Result<()> { 0, 0, b"/deployment/cluster", - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ) .await?; - let crowdb_kv_client::GetOutcome::Found { value, .. } = found else { + let crowdb_paxos_kv_client::GetOutcome::Found { value, .. } = found else { return Err("cluster publication not complete".into()); }; let publication: Value = serde_json::from_slice(&value)?; @@ -275,11 +275,11 @@ async fn verify_binding(binding: &NodeBinding) -> Result<()> { 0, 0, b"/deployment/nodes", - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ) .await?; - let crowdb_kv_client::GetOutcome::Found { value, .. } = registry else { + let crowdb_paxos_kv_client::GetOutcome::Found { value, .. } = registry else { return Err("node registry publication incomplete".into()); }; let registry: Value = serde_json::from_slice(&value)?; @@ -334,7 +334,7 @@ fn web_service(profile: &DeploymentProfile, monitor_port: u16) -> Result<Service profile.paths.log_root.join("web").to_string_lossy().into_owned(), ]; web.env.insert( - "CROWDB_KV_KNOWN_HOSTS".into(), + "CROWDB_PAXOS_KV_KNOWN_HOSTS".into(), root.join("ssh/console-known_hosts") .to_string_lossy() .into_owned(), diff --git a/container/crowdb-monitor/src/node/runtime/admission.rs b/container/crowdb-monitor/src/node/runtime/admission.rs index 2699e4e23..0000adaeb 100644 --- a/container/crowdb-monitor/src/node/runtime/admission.rs +++ b/container/crowdb-monitor/src/node/runtime/admission.rs @@ -25,7 +25,7 @@ pub(super) async fn validate( return Err("admission operation cancelled".into()); } let discovery_id = NodeIdentity::load_or_create(root)?.uuid().to_string(); - let client = crowdb_kv_client::CrowdbKvClient::new(crowdb_kv_client::ClientConfig::new( + let client = crowdb_paxos_kv_client::PaxosKvClient::new(crowdb_paxos_kv_client::ClientConfig::new( grant.management_seeds.clone(), )); let publication = tokio::time::timeout( @@ -34,12 +34,12 @@ pub(super) async fn validate( 0, 0, crowdb_console_shared::deployment::CLUSTER_KEY, - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ), ) .await??; - let crowdb_kv_client::GetOutcome::Found { value, .. } = publication else { + let crowdb_paxos_kv_client::GetOutcome::Found { value, .. } = publication else { return Err("cluster publication unavailable".into()); }; let publication: crowdb_console_shared::deployment::PreparedBootstrap = serde_json::from_slice(&value)?; diff --git a/container/crowdb-monitor/src/node/runtime/file_admission.rs b/container/crowdb-monitor/src/node/runtime/file_admission.rs index e0359c61a..817a2a2b2 100644 --- a/container/crowdb-monitor/src/node/runtime/file_admission.rs +++ b/container/crowdb-monitor/src/node/runtime/file_admission.rs @@ -5,7 +5,7 @@ use std::path::Path; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, HardwareClient}; +use crowdb_paxos_kv_client::{ClientConfig, HardwareClient, PaxosKvClient}; use crowdb_protocol::{ common::{DiskId, HwStatus}, disk_storage::{DiskStorageRecord, PhysicalBacking}, @@ -25,7 +25,7 @@ pub(super) fn authority(root: &Path, request: &PrepareDiskFile) -> Result<Hardwa if binding.node_id != request.node_id || binding.bootstrap.cluster_id != request.cluster_id { return Err("file admission belongs to another node or cluster".into()); } - Ok(HardwareClient::new(CrowdbKvClient::new(ClientConfig::new( + Ok(HardwareClient::new(PaxosKvClient::new(ClientConfig::new( binding.management_seeds, )))) } diff --git a/container/crowdb-monitor/src/node/runtime/file_preparation.rs b/container/crowdb-monitor/src/node/runtime/file_preparation.rs index 0f62eef96..2168e31bc 100644 --- a/container/crowdb-monitor/src/node/runtime/file_preparation.rs +++ b/container/crowdb-monitor/src/node/runtime/file_preparation.rs @@ -119,8 +119,8 @@ impl FileAllocator { FilePreparationPhase::WaitingForDiskIO } else if current == Some(FilePreparationPhase::Unknown) || matches!( - error.downcast_ref::<crowdb_kv_client::Error>(), - Some(crowdb_kv_client::Error::OutcomeUnknown) + error.downcast_ref::<crowdb_paxos_kv_client::Error>(), + Some(crowdb_paxos_kv_client::Error::OutcomeUnknown) ) { FilePreparationPhase::Unknown diff --git a/container/crowdb-monitor/src/node/runtime/kv.rs b/container/crowdb-monitor/src/node/runtime/kv.rs index 8af9ed7de..716ec45de 100644 --- a/container/crowdb-monitor/src/node/runtime/kv.rs +++ b/container/crowdb-monitor/src/node/runtime/kv.rs @@ -51,7 +51,10 @@ pub(super) fn node_profile(config: &super::NodeRuntimeConfig) -> Result<Deployme .find(|service| service.id == "kv") .ok_or("node profile requires KV service")? .env - .insert("CROWDB_KV_MANAGEMENT_ADVERTISE_ADDR".into(), address.to_string()); + .insert( + "CROWDB_PAXOS_KV_MANAGEMENT_ADVERTISE_ADDR".into(), + address.to_string(), + ); Ok(profile) } @@ -173,7 +176,7 @@ async fn spawn( } let mut kv = service(profile, "kv")?; kv.env.insert( - "CROWDB_KV_CLUSTER_ID".into(), + "CROWDB_PAXOS_KV_CLUSTER_ID".into(), accepted.bootstrap.cluster_id.clone(), ); let root = profile diff --git a/container/crowdb-monitor/src/node/runtime/recovery/confirmed.rs b/container/crowdb-monitor/src/node/runtime/recovery/confirmed.rs index 19c060f23..74ac55315 100644 --- a/container/crowdb-monitor/src/node/runtime/recovery/confirmed.rs +++ b/container/crowdb-monitor/src/node/runtime/recovery/confirmed.rs @@ -11,7 +11,7 @@ use crowdb_console_shared::{ }, ConsoleConfig, }; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{ClientConfig, GetOutcome, PaxosKvClient, ReadMode}; use crowdb_protocol::mgmt::node::NodeBinding; use ssh_key::PublicKey; use std::{path::Path, time::Duration}; @@ -31,7 +31,7 @@ async fn authority( }) .ok_or("confirmed recovery requires authority contacts")?; tokio::time::timeout(Duration::from_secs(3), async { - let client = CrowdbKvClient::new(ClientConfig::new(seeds)); + let client = PaxosKvClient::new(ClientConfig::new(seeds)); let GetOutcome::Found { value, .. } = client .get( 0, diff --git a/container/crowdb-monitor/src/node/runtime/recovery/publication.rs b/container/crowdb-monitor/src/node/runtime/recovery/publication.rs index a72a043fc..e49d2104d 100644 --- a/container/crowdb-monitor/src/node/runtime/recovery/publication.rs +++ b/container/crowdb-monitor/src/node/runtime/recovery/publication.rs @@ -9,7 +9,7 @@ use crowdb_console_shared::{ ops::OpContext, ConsoleConfig, }; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, GetOutcome, ReadMode}; +use crowdb_paxos_kv_client::{ClientConfig, GetOutcome, PaxosKvClient, ReadMode}; use crowdb_protocol::mgmt::node::{recovery::MemberEndpoints, NodeServiceAction}; use std::{path::Path, sync::Arc}; @@ -23,7 +23,7 @@ pub(super) async fn reconcile( let local = super::identity(root)?.endpoint; let mut config = topology.clone(); let seeds = apply_member_endpoints(&mut config, endpoints)?; - let client = Arc::new(CrowdbKvClient::new(ClientConfig::new(seeds.clone()))); + let client = Arc::new(PaxosKvClient::new(ClientConfig::new(seeds.clone()))); let GetOutcome::Found { value, .. } = client .get( 0, @@ -135,7 +135,7 @@ fn apply_member_endpoints(config: &mut ConsoleConfig, endpoints: &MemberEndpoint async fn rebind( root: &Path, - client: &CrowdbKvClient, + client: &PaxosKvClient, operation: &PreparedBootstrap, node: u64, current: &str, @@ -240,6 +240,6 @@ pub async fn rebind_recovery_services_for_tests( seeds: &[String], complete: bool, ) -> Result<bool> { - let client = CrowdbKvClient::new(ClientConfig::new(vec!["http://127.0.0.1:9".into()])); + let client = PaxosKvClient::new(ClientConfig::new(vec!["http://127.0.0.1:9".into()])); rebind(root, &client, operation, node, current, seeds, complete).await } diff --git a/container/crowdb-monitor/src/node/runtime/retained_storage.rs b/container/crowdb-monitor/src/node/runtime/retained_storage.rs index 532933987..a8935a7a8 100644 --- a/container/crowdb-monitor/src/node/runtime/retained_storage.rs +++ b/container/crowdb-monitor/src/node/runtime/retained_storage.rs @@ -3,7 +3,7 @@ //! Verify bound authority before inspecting an explicitly retained disk. use super::{durable, Result}; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, HardwareClient}; +use crowdb_paxos_kv_client::{ClientConfig, HardwareClient, PaxosKvClient}; use crowdb_protocol::{ common::HwStatus, disk_storage::DiskStorageRecord, @@ -35,7 +35,7 @@ pub(super) async fn check(root: &Path, request: CheckRetainedDisk) -> Result<Ret disk_group_id: request.disk_group_id, disk_id, }; - let hardware = HardwareClient::new(CrowdbKvClient::new(ClientConfig::new(binding.management_seeds))); + let hardware = HardwareClient::new(PaxosKvClient::new(ClientConfig::new(binding.management_seeds))); let nodes = hardware.list_nodes().await?; let locations: Vec<_> = nodes.iter().filter(|(_, id, _)| *id == request.node_id).collect(); if locations.len() != 1 || locations[0].0 != request.rack_id { diff --git a/container/crowdb-monitor/src/node/runtime/services.rs b/container/crowdb-monitor/src/node/runtime/services.rs index d242000b1..7ccad59f7 100644 --- a/container/crowdb-monitor/src/node/runtime/services.rs +++ b/container/crowdb-monitor/src/node/runtime/services.rs @@ -85,8 +85,9 @@ pub(super) fn kv_paused(profile: &DeploymentProfile) -> Result<bool> { pub(super) async fn desired(profile: &DeploymentProfile) -> Result<Vec<NodeServiceIntent>> { let binding: NodeBinding = durable::read(&profile.paths.data_root.join("node-binding.json"))?.ok_or("node unbound")?; - let client = - crowdb_kv_client::CrowdbKvClient::new(crowdb_kv_client::ClientConfig::new(binding.management_seeds)); + let client = crowdb_paxos_kv_client::PaxosKvClient::new(crowdb_paxos_kv_client::ClientConfig::new( + binding.management_seeds, + )); let records = tokio::time::timeout( Duration::from_secs(3), client.scan_bounded(0, 0, b"/deployment/services/", b"", b"", 4096, false, None), diff --git a/container/crowdb-monitor/src/node/runtime/services/dependencies.rs b/container/crowdb-monitor/src/node/runtime/services/dependencies.rs index 6db53985b..575a02130 100644 --- a/container/crowdb-monitor/src/node/runtime/services/dependencies.rs +++ b/container/crowdb-monitor/src/node/runtime/services/dependencies.rs @@ -13,7 +13,7 @@ pub(super) async fn committed_ready( intent: &NodeServiceIntent, binding: &NodeBinding, ) -> Result<()> { - let kv = crowdb_kv_client::CrowdbKvClient::new(crowdb_kv_client::ClientConfig::new( + let kv = crowdb_paxos_kv_client::PaxosKvClient::new(crowdb_paxos_kv_client::ClientConfig::new( binding.management_seeds.clone(), )); let intents = tokio::time::timeout( @@ -88,8 +88,8 @@ pub(in crate::node::runtime) async fn ready( // has finished WAL replay. Probe the same routed read Access needs. let binding: NodeBinding = super::durable::read(&profile.paths.data_root.join("node-binding.json"))? .ok_or("node is unbound")?; - let kv = Arc::new(crowdb_kv_client::CrowdbKvClient::new( - crowdb_kv_client::ClientConfig::new(binding.management_seeds), + let kv = Arc::new(crowdb_paxos_kv_client::PaxosKvClient::new( + crowdb_paxos_kv_client::ClientConfig::new(binding.management_seeds), )); let catalog = Arc::new(crowdb_chunk_kv_client::Group0ChunkKvRangeCatalogSource::from_shared(kv)); let (_, pages) = catalog.load().await?; @@ -130,8 +130,8 @@ pub(in crate::node::runtime) async fn ready( async fn diskdb_routes_ready(profile: &DeploymentProfile) -> Result<()> { let binding: NodeBinding = super::durable::read(&profile.paths.data_root.join("node-binding.json"))?.ok_or("node is unbound")?; - let kv = Arc::new(crowdb_kv_client::CrowdbKvClient::new( - crowdb_kv_client::ClientConfig::new(binding.management_seeds), + let kv = Arc::new(crowdb_paxos_kv_client::PaxosKvClient::new( + crowdb_paxos_kv_client::ClientConfig::new(binding.management_seeds), )); crowdb_console_shared::deployment::services::diskdb_storage_ready(kv).await?; Ok(()) diff --git a/container/crowdb-monitor/src/node/runtime/services/execution.rs b/container/crowdb-monitor/src/node/runtime/services/execution.rs index 283a46a52..caf9aaef6 100644 --- a/container/crowdb-monitor/src/node/runtime/services/execution.rs +++ b/container/crowdb-monitor/src/node/runtime/services/execution.rs @@ -315,7 +315,7 @@ async fn wait_ready( } async fn verify_intent(binding: &NodeBinding, intent: &NodeServiceIntent) -> Result<()> { - let client = crowdb_kv_client::CrowdbKvClient::new(crowdb_kv_client::ClientConfig::new( + let client = crowdb_paxos_kv_client::PaxosKvClient::new(crowdb_paxos_kv_client::ClientConfig::new( binding.management_seeds.clone(), )); let key = format!("/deployment/services/{}/{}", intent.node_id, intent.service_id); @@ -325,12 +325,12 @@ async fn verify_intent(binding: &NodeBinding, intent: &NodeServiceIntent) -> Res 0, 0, key.as_bytes(), - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ), ) .await??; - let crowdb_kv_client::GetOutcome::Found { value, .. } = outcome else { + let crowdb_paxos_kv_client::GetOutcome::Found { value, .. } = outcome else { return Err("service intent is not committed".into()); }; let current: NodeServiceIntent = serde_json::from_slice(&value)?; diff --git a/container/crowdb-monitor/src/node/runtime/setup/execution.rs b/container/crowdb-monitor/src/node/runtime/setup/execution.rs index d4c42e1d6..cf57862fc 100644 --- a/container/crowdb-monitor/src/node/runtime/setup/execution.rs +++ b/container/crowdb-monitor/src/node/runtime/setup/execution.rs @@ -9,7 +9,7 @@ use super::{ }; use crate::DeploymentProfile; use crowdb_console_shared::deployment::registry; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient}; use crowdb_protocol::mgmt::node::{setup::NodeSetupPhase, NodeBinding}; use std::time::Duration; @@ -55,7 +55,7 @@ async fn advance(profile: &DeploymentProfile, state: &mut AcceptedSetup) -> Resu return Err("node setup admission was retired or rebound".into()); } tokio::time::timeout(Duration::from_secs(3), super::super::verify_binding(&binding)).await??; - let kv = std::sync::Arc::new(CrowdbKvClient::new(ClientConfig::new( + let kv = std::sync::Arc::new(PaxosKvClient::new(ClientConfig::new( binding.management_seeds.clone(), ))); let (registry, _) = tokio::time::timeout(Duration::from_secs(3), registry::read(&kv)) diff --git a/container/crowdb-monitor/src/node/runtime/setup/execution/metadata.rs b/container/crowdb-monitor/src/node/runtime/setup/execution/metadata.rs index 92a24e089..faeea9239 100644 --- a/container/crowdb-monitor/src/node/runtime/setup/execution/metadata.rs +++ b/container/crowdb-monitor/src/node/runtime/setup/execution/metadata.rs @@ -10,7 +10,7 @@ use crowdb_console_shared::{ ops::{kv_logical, OpContext}, ConsoleConfig, }; -use crowdb_kv_client::CrowdbKvClient; +use crowdb_paxos_kv_client::PaxosKvClient; use crowdb_protocol::mgmt::node::NodeBinding; use super::Result; @@ -18,7 +18,7 @@ use super::Result; pub(super) async fn ready( root: &std::path::Path, binding: &NodeBinding, - kv: &Arc<CrowdbKvClient>, + kv: &Arc<PaxosKvClient>, ) -> Result<bool> { let ctx = OpContext::with_shared_client_preserving_hint( Arc::clone(kv), diff --git a/container/crowdb-monitor/src/node/runtime/setup/execution/services.rs b/container/crowdb-monitor/src/node/runtime/setup/execution/services.rs index cb267f82f..ccb502d0d 100644 --- a/container/crowdb-monitor/src/node/runtime/setup/execution/services.rs +++ b/container/crowdb-monitor/src/node/runtime/setup/execution/services.rs @@ -6,7 +6,7 @@ use super::{journal, AcceptedSetup, Result}; use crate::DeploymentProfile; use crowdb_console_shared::deployment::services; -use crowdb_kv_client::CrowdbKvClient; +use crowdb_paxos_kv_client::PaxosKvClient; use crowdb_protocol::mgmt::node::{ setup::{SetupServiceKind, SetupStepPhase}, NodeBinding, NodeControl, NodeServiceAction, @@ -18,7 +18,7 @@ pub(super) async fn advance( profile: &DeploymentProfile, state: &mut AcceptedSetup, binding: &NodeBinding, - kv: &CrowdbKvClient, + kv: &PaxosKvClient, storage_ready: bool, ) -> Result<bool> { if state.status.services.iter().all(|step| { diff --git a/container/crowdb-monitor/src/node/runtime/setup/execution/storage.rs b/container/crowdb-monitor/src/node/runtime/setup/execution/storage.rs index 9e17e18ad..77c9344c1 100644 --- a/container/crowdb-monitor/src/node/runtime/setup/execution/storage.rs +++ b/container/crowdb-monitor/src/node/runtime/setup/execution/storage.rs @@ -5,7 +5,7 @@ use super::{journal, AcceptedSetup, Result}; use crate::DeploymentProfile; -use crowdb_kv_client::{CrowdbKvClient, HardwareClient}; +use crowdb_paxos_kv_client::{HardwareClient, PaxosKvClient}; use crowdb_protocol::mgmt::node::{ setup::{SetupDiskStatus, SetupStepPhase, SetupStorage}, storage::{managed_file_zone_bytes, DiskInput, FileDiskAdmission, FilePreparationPhase, PrepareDiskFile}, @@ -16,7 +16,7 @@ pub(super) async fn advance( profile: &DeploymentProfile, state: &mut AcceptedSetup, binding: &NodeBinding, - kv: &std::sync::Arc<CrowdbKvClient>, + kv: &std::sync::Arc<PaxosKvClient>, ) -> Result<bool> { let (capacity_bytes, reserve_bytes, sparse) = match &state.request.choices.storage { SetupStorage::Automatic { @@ -121,7 +121,7 @@ async fn advance_file( } pub(super) async fn ensure_group( - kv: &std::sync::Arc<CrowdbKvClient>, + kv: &std::sync::Arc<PaxosKvClient>, binding: &NodeBinding, group_id: u64, data_binding: Option<crowdb_protocol::common::BindMapValue>, @@ -170,7 +170,7 @@ pub(super) async fn ensure_group( pub(super) async fn select_group( hardware: &HardwareClient, - kv: &std::sync::Arc<CrowdbKvClient>, + kv: &std::sync::Arc<PaxosKvClient>, rack: u64, binding: &NodeBinding, media_type: Option<i32>, @@ -197,7 +197,7 @@ pub(super) async fn select_group( .await { Ok(()) => {} - Err(crowdb_kv_client::Error::Server(reason)) + Err(crowdb_paxos_kv_client::Error::Server(reason)) if reason == "DiskGroup media class or allocation unit is incompatible with this disk" => { if automatic { diff --git a/container/crowdb-monitor/src/node/runtime/setup/execution/supplied.rs b/container/crowdb-monitor/src/node/runtime/setup/execution/supplied.rs index 08367ef76..b9306f5bb 100644 --- a/container/crowdb-monitor/src/node/runtime/setup/execution/supplied.rs +++ b/container/crowdb-monitor/src/node/runtime/setup/execution/supplied.rs @@ -8,7 +8,7 @@ use super::{ storage, Result, }; use crate::DeploymentProfile; -use crowdb_kv_client::{CrowdbKvClient, HardwareClient}; +use crowdb_paxos_kv_client::{HardwareClient, PaxosKvClient}; use crowdb_protocol::{ common::{DiskId, HwStatus}, disk_storage::{DiskStorageRecord, PhysicalBacking}, @@ -29,7 +29,7 @@ pub(super) async fn advance( profile: &DeploymentProfile, state: &mut AcceptedSetup, binding: &NodeBinding, - kv: &Arc<CrowdbKvClient>, + kv: &Arc<PaxosKvClient>, ) -> Result<bool> { if state.supplied.is_empty() { prepare(profile, state, binding, kv).await?; @@ -102,7 +102,7 @@ async fn prepare( profile: &DeploymentProfile, state: &mut AcceptedSetup, binding: &NodeBinding, - kv: &Arc<CrowdbKvClient>, + kv: &Arc<PaxosKvClient>, ) -> Result<()> { let SetupStorage::Supplied { disks } = &state.request.choices.storage else { return Err("supplied setup choices missing".into()); diff --git a/container/crowdb-monitor/src/node/runtime/storage_health.rs b/container/crowdb-monitor/src/node/runtime/storage_health.rs index db16b261d..e4ab8d9f6 100644 --- a/container/crowdb-monitor/src/node/runtime/storage_health.rs +++ b/container/crowdb-monitor/src/node/runtime/storage_health.rs @@ -9,7 +9,7 @@ use std::{ }; use crowdb_console_shared::deployment::registry; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, HardwareClient}; +use crowdb_paxos_kv_client::{ClientConfig, HardwareClient, PaxosKvClient}; use crowdb_protocol::{common::HwStatus, key::DiskKey, mgmt::node::NodeBinding}; use super::{durable, Result}; @@ -61,7 +61,7 @@ async fn observe(root: PathBuf, after: Option<String>) -> Result<Option<String>> let binding: NodeBinding = durable::read(&root.join("node-binding.json"))?.ok_or("disk health binding is absent")?; tokio::time::timeout(Duration::from_secs(3), super::verify_binding(&binding)).await??; - let client = CrowdbKvClient::new(ClientConfig::new(binding.management_seeds.clone())); + let client = PaxosKvClient::new(ClientConfig::new(binding.management_seeds.clone())); let (registry, _) = registry::read(&client) .await? .ok_or("disk health registry is absent")?; diff --git a/container/crowdb-monitor/src/node/runtime/supplied_storage.rs b/container/crowdb-monitor/src/node/runtime/supplied_storage.rs index 72ddb008f..b1127b47f 100644 --- a/container/crowdb-monitor/src/node/runtime/supplied_storage.rs +++ b/container/crowdb-monitor/src/node/runtime/supplied_storage.rs @@ -112,8 +112,8 @@ impl SuppliedStorage { eprintln!("supplied disk provisioning failed; explicit retry required: {error}"); if let Ok(mut state) = journal::status(&root, &id) { state.phase = if matches!( - error.downcast_ref::<crowdb_kv_client::Error>(), - Some(crowdb_kv_client::Error::OutcomeUnknown) + error.downcast_ref::<crowdb_paxos_kv_client::Error>(), + Some(crowdb_paxos_kv_client::Error::OutcomeUnknown) ) { SuppliedPreparationPhase::Unknown } else { diff --git a/container/crowdb-monitor/src/node/runtime/supplied_storage/execution.rs b/container/crowdb-monitor/src/node/runtime/supplied_storage/execution.rs index d322665b0..94486723e 100644 --- a/container/crowdb-monitor/src/node/runtime/supplied_storage/execution.rs +++ b/container/crowdb-monitor/src/node/runtime/supplied_storage/execution.rs @@ -6,7 +6,7 @@ use super::{authorize, journal, Result}; use crate::node::storage::blank; use crowdb_diskio_client::{DiskId, DiskioClient, DiskioClientConfig, OperationOptions, SegmentTarget}; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, HardwareClient}; +use crowdb_paxos_kv_client::{ClientConfig, HardwareClient, PaxosKvClient}; use crowdb_protocol::{ common::HwStatus, key::DiskKey, @@ -26,7 +26,7 @@ pub(super) async fn run_with_inputs( let mut state = journal::status(root, id)?; let binding = authorize(root, &state.request).await?; let retirement = super::retirement(root, &binding); - let hardware = HardwareClient::new(CrowdbKvClient::new(ClientConfig::new( + let hardware = HardwareClient::new(PaxosKvClient::new(ClientConfig::new( binding.management_seeds.clone(), ))); let record = &state.request.record; diff --git a/container/crowdb-monitor/src/node/startup/policy.rs b/container/crowdb-monitor/src/node/startup/policy.rs index 0148669f7..8571437da 100644 --- a/container/crowdb-monitor/src/node/startup/policy.rs +++ b/container/crowdb-monitor/src/node/startup/policy.rs @@ -108,7 +108,6 @@ pub fn resolve_node_policy_with_port_offset( fn shift_service_ports(services: &mut [SetupService], offset: u16) -> io::Result<()> { for service in services { for port in [ - &mut service.listeners.listen_port, &mut service.listeners.http_port, &mut service.listeners.rpc_port, &mut service.listeners.s3_port, diff --git a/container/crowdb-monitor/tests/access_bootstrap_test.rs b/container/crowdb-monitor/tests/access_bootstrap_test.rs index 2c0fefd0c..50b3c3dc4 100644 --- a/container/crowdb-monitor/tests/access_bootstrap_test.rs +++ b/container/crowdb-monitor/tests/access_bootstrap_test.rs @@ -9,6 +9,7 @@ use crowdb_monitor::{ s3_step_names, show_client_credentials, BootstrapSession, DeploymentProfile, MonitorLog, S3Bootstrap, ServerCredentials, }; +use crowdb_protocol::port::ports::{ACCESS_SERVER_HTTP_BASE, ACCESS_SERVER_ICEBERG_HTTP_BASE}; use uuid::Uuid; struct TestRoot(PathBuf); @@ -69,8 +70,12 @@ async fn s3_bootstrap_reuses_user_and_validates_ready_without_creation() { .unwrap(); assert_eq!(session.manifest().step_complete("s3-user"), Some(true)); let client = show_client_credentials(&data_root).unwrap(); - assert!(client.contains("AWS_ENDPOINT_URL=http://localhost:9091\n")); - assert!(client.contains("ICEBERG_URI=http://localhost:9092\n")); + assert!(client.contains(&format!( + "AWS_ENDPOINT_URL=http://localhost:{ACCESS_SERVER_HTTP_BASE}\n" + ))); + assert!(client.contains(&format!( + "ICEBERG_URI=http://localhost:{ACCESS_SERVER_ICEBERG_HTTP_BASE}\n" + ))); session.mark_ready().unwrap(); let mut restarted = BootstrapSession::open(&data_root, b"profile", b"config", &s3_step_names()).unwrap(); diff --git a/container/crowdb-monitor/tests/blank_medium_test.rs b/container/crowdb-monitor/tests/blank_medium_test.rs index 17c5d9fb8..31469e2cc 100644 --- a/container/crowdb-monitor/tests/blank_medium_test.rs +++ b/container/crowdb-monitor/tests/blank_medium_test.rs @@ -180,8 +180,8 @@ async fn authority(test: &TestMedium) -> node_setup::TestAuthority { }; let authority = node_setup::TestAuthority::start(test.root.path(), &request).await; assert_eq!(authority.registry.nodes[0].physical_host_id, "owned-host"); - let hardware = crowdb_kv_client::HardwareClient::new(crowdb_kv_client::CrowdbKvClient::new( - crowdb_kv_client::ClientConfig::new(authority.binding.management_seeds.clone()), + let hardware = crowdb_paxos_kv_client::HardwareClient::new(crowdb_paxos_kv_client::PaxosKvClient::new( + crowdb_paxos_kv_client::ClientConfig::new(authority.binding.management_seeds.clone()), )); hardware .add_rack( @@ -232,13 +232,13 @@ async fn verified_medium_publishes_init_and_waits_for_real_diskio_without_reinit .await .unwrap(); assert_eq!(first.phase, SuppliedPreparationPhase::WaitingForDiskio); - let crowdb_kv_client::GetOutcome::Found { value, .. } = authority + let crowdb_paxos_kv_client::GetOutcome::Found { value, .. } = authority .kv .get( 0, 0, test.record.key().as_bytes(), - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ) .await @@ -250,8 +250,8 @@ async fn verified_medium_publishes_init_and_waits_for_real_diskio_without_reinit serde_json::from_slice::<DiskStorageRecord>(&value).unwrap(), test.record ); - let hardware = crowdb_kv_client::HardwareClient::new(crowdb_kv_client::CrowdbKvClient::new( - crowdb_kv_client::ClientConfig::new(authority.binding.management_seeds.clone()), + let hardware = crowdb_paxos_kv_client::HardwareClient::new(crowdb_paxos_kv_client::PaxosKvClient::new( + crowdb_paxos_kv_client::ClientConfig::new(authority.binding.management_seeds.clone()), )); let key = DiskKey { rack_id: 1, @@ -292,8 +292,8 @@ async fn nonblank_provisioning_never_publishes_membership_and_changed_retry_cann .await .unwrap_err(); assert!(error.to_string().contains("not blank"), "{error}"); - let hardware = crowdb_kv_client::HardwareClient::new(crowdb_kv_client::CrowdbKvClient::new( - crowdb_kv_client::ClientConfig::new(authority.binding.management_seeds.clone()), + let hardware = crowdb_paxos_kv_client::HardwareClient::new(crowdb_paxos_kv_client::PaxosKvClient::new( + crowdb_paxos_kv_client::ClientConfig::new(authority.binding.management_seeds.clone()), )); let key = DiskKey { rack_id: 1, @@ -391,12 +391,12 @@ async fn missing_target_group_fails_before_blank_scan_and_global_resource_claim( 0, 0, record.key().as_bytes(), - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ) .await .unwrap(); - assert!(matches!(resources, crowdb_kv_client::GetOutcome::NotFound)); + assert!(matches!(resources, crowdb_paxos_kv_client::GetOutcome::NotFound)); let physical = record.physical_backing.as_ref().unwrap(); let encode = |text: &str| { text.as_bytes().iter().fold(String::new(), |mut encoded, byte| { @@ -415,12 +415,12 @@ async fn missing_target_group_fails_before_blank_scan_and_global_resource_claim( 0, 0, path.as_bytes(), - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None, ) .await .unwrap(); - assert!(matches!(claim, crowdb_kv_client::GetOutcome::NotFound)); + assert!(matches!(claim, crowdb_paxos_kv_client::GetOutcome::NotFound)); assert!( fs::metadata(test.root.path().join("disks/owned.img")) .unwrap() diff --git a/container/crowdb-monitor/tests/common/node_setup.rs b/container/crowdb-monitor/tests/common/node_setup.rs index 31f8a27e4..5e5d02a70 100644 --- a/container/crowdb-monitor/tests/common/node_setup.rs +++ b/container/crowdb-monitor/tests/common/node_setup.rs @@ -9,17 +9,17 @@ use crowdb_console_shared::{ ops::OpContext, ConsoleConfig, }; -use crowdb_kv::cluster::{ - group::PxGroup, kv_server::KvServer, PxKvStore, PxLocalReplica, PxLocalReplicaRole, +use crowdb_paxos_kv::cluster::{ + group::PxGroup, paxos_kv_server::PaxosKvServer, PxKvStore, PxLocalReplica, PxLocalReplicaRole, }; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient}; use crowdb_protocol::mgmt::node::{setup::NodeSetupRequest, NodeBinding}; use std::{os::unix::fs::PermissionsExt, path::Path, sync::Arc}; pub struct TestAuthority { store: Arc<PxKvStore>, task: tokio::task::JoinHandle<()>, - pub kv: CrowdbKvClient, + pub kv: PaxosKvClient, pub binding: NodeBinding, pub registry: NodeRegistry, } @@ -43,7 +43,7 @@ impl TestAuthority { let task = tokio::spawn(async move { axum::serve(listener, app).await.unwrap(); }); - let kv = CrowdbKvClient::new(ClientConfig::new(vec![seed.clone()])); + let kv = PaxosKvClient::new(ClientConfig::new(vec![seed.clone()])); kv.seed_leader(0, 0, store.listen_addr().unwrap().to_string()); let bootstrap = bootstrap(root, &kv, &seed, &store.listen_addr().unwrap().to_string()).await; let binding = NodeBinding { @@ -84,7 +84,7 @@ impl TestAuthority { } } -async fn bootstrap(root: &Path, kv: &CrowdbKvClient, management: &str, rpc: &str) -> PreparedBootstrap { +async fn bootstrap(root: &Path, kv: &PaxosKvClient, management: &str, rpc: &str) -> PreparedBootstrap { let mut server = ServerEntry::new("kv-2", management); server.node_id = Some(2); server.rpc_url = Some(rpc.into()); diff --git a/container/crowdb-monitor/tests/common/node_setup_file.rs b/container/crowdb-monitor/tests/common/node_setup_file.rs index 41ca5117a..a778eb51e 100644 --- a/container/crowdb-monitor/tests/common/node_setup_file.rs +++ b/container/crowdb-monitor/tests/common/node_setup_file.rs @@ -53,11 +53,11 @@ pub async fn prepare( } async fn publish_storage(authority: &super::node_setup::TestAuthority) { - use crowdb_kv_client::{ClientConfig, CrowdbKvClient, HardwareClient}; + use crowdb_paxos_kv_client::{ClientConfig, HardwareClient, PaxosKvClient}; use crowdb_protocol::common::{NodeValue, RackValue}; super::node_setup_groups::publish_group(authority, 1).await; let node = &authority.registry.nodes[0]; - let hardware = HardwareClient::new(CrowdbKvClient::new(ClientConfig::new( + let hardware = HardwareClient::new(PaxosKvClient::new(ClientConfig::new( authority.binding.management_seeds.clone(), ))); hardware diff --git a/container/crowdb-monitor/tests/common/service_dependency.rs b/container/crowdb-monitor/tests/common/service_dependency.rs index 6f96d9c93..9b6170e7a 100644 --- a/container/crowdb-monitor/tests/common/service_dependency.rs +++ b/container/crowdb-monitor/tests/common/service_dependency.rs @@ -1,14 +1,14 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -use crowdb_kv_client::{CrowdbKvClient, HardwareClient}; +use crowdb_paxos_kv_client::{HardwareClient, PaxosKvClient}; use crowdb_protocol::{ common::{DiskId, HwStatus, NodeValue, RackValue}, diskdb::rpc::{DiskGroupValue, DiskValue}, }; use std::sync::Arc; -pub async fn publish_online_disk(kv: Arc<CrowdbKvClient>) { +pub async fn publish_online_disk(kv: Arc<PaxosKvClient>) { let hardware = HardwareClient::from_shared(kv.clone()); hardware .add_rack( @@ -74,7 +74,7 @@ pub async fn ready_listener() -> (std::net::SocketAddr, tokio::task::JoinHandle< } pub async fn assert_initial_start_waits( - kv: &CrowdbKvClient, + kv: &PaxosKvClient, profile: &crowdb_monitor::DeploymentProfile, local: &crowdb_protocol::mgmt::node::NodeServiceIntent, consumer: &crowdb_protocol::mgmt::node::NodeServiceIntent, diff --git a/container/crowdb-monitor/tests/console_catalog_shared_test.rs b/container/crowdb-monitor/tests/console_catalog_shared_test.rs index bce87b447..0b0ad6c87 100644 --- a/container/crowdb-monitor/tests/console_catalog_shared_test.rs +++ b/container/crowdb-monitor/tests/console_catalog_shared_test.rs @@ -3,8 +3,8 @@ #[path = "common/node_setup.rs"] mod node_setup; -use crowdb_kv_client::{GetOutcome, ReadMode}; use crowdb_monitor::IcebergBootstrap; +use crowdb_paxos_kv_client::{GetOutcome, ReadMode}; use crowdb_protocol::mgmt::node::setup::{NodeSetupChoices, NodeSetupRequest}; use serde_json::json; use std::{fs, path::Path}; diff --git a/container/crowdb-monitor/tests/file_admission_test.rs b/container/crowdb-monitor/tests/file_admission_test.rs index cb372ce6b..5adb4af91 100644 --- a/container/crowdb-monitor/tests/file_admission_test.rs +++ b/container/crowdb-monitor/tests/file_admission_test.rs @@ -1,8 +1,8 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, GetOutcome, HardwareClient, ReadMode}; use crowdb_monitor::{accept_file_for_tests, prepare_file_for_tests, publish_file_for_tests}; +use crowdb_paxos_kv_client::{ClientConfig, GetOutcome, HardwareClient, PaxosKvClient, ReadMode}; use crowdb_protocol::{ common::{HwStatus, NodeValue, RackValue}, disk_storage::DiskStorageRecord, @@ -93,7 +93,7 @@ async fn fixture(root: &std::path::Path) -> (node_setup::TestAuthority, Hardware }; let authority = node_setup::TestAuthority::start(root, &setup).await; assert_eq!(authority.registry.nodes[0].physical_host_id, "owned-host"); - let hardware = HardwareClient::new(CrowdbKvClient::new(ClientConfig::new( + let hardware = HardwareClient::new(PaxosKvClient::new(ClientConfig::new( authority.binding.management_seeds.clone(), ))); hardware diff --git a/container/crowdb-monitor/tests/node_policy_test.rs b/container/crowdb-monitor/tests/node_policy_test.rs index 0184deb97..8569f23c6 100644 --- a/container/crowdb-monitor/tests/node_policy_test.rs +++ b/container/crowdb-monitor/tests/node_policy_test.rs @@ -44,13 +44,9 @@ fn host_network_policy_shifts_every_diskdb_listener_once_and_keeps_instances_dis crowdb_monitor::accept_node_policy_with_port_offset(root.path(), &omitted, &omitted, offset) .unwrap(); let diskdb = &policy.setup.services[1].listeners; - assert_eq!(diskdb.listen_port, Some(11000 + offset)); - assert_eq!(diskdb.http_port, Some(11100 + offset)); - assert_eq!(diskdb.rpc_port, Some(11200 + offset)); - for port in [diskdb.listen_port, diskdb.http_port, diskdb.rpc_port] - .into_iter() - .flatten() - { + assert_eq!(diskdb.http_port, Some(crowdb_protocol::DISKDB_HTTP_BASE + offset)); + assert_eq!(diskdb.rpc_port, Some(crowdb_protocol::DISKDB_RPC_BASE + offset)); + for port in [diskdb.http_port, diskdb.rpc_port].into_iter().flatten() { assert!(listeners.insert(port)); } assert_eq!( diff --git a/container/crowdb-monitor/tests/node_profile_test.rs b/container/crowdb-monitor/tests/node_profile_test.rs index 2089dd209..2dcfda5a0 100644 --- a/container/crowdb-monitor/tests/node_profile_test.rs +++ b/container/crowdb-monitor/tests/node_profile_test.rs @@ -2,6 +2,9 @@ // Licensed under the Apache License, Version 2.0. use crowdb_monitor::DeploymentProfile; +use crowdb_protocol::port::ports::{ + ACCESS_SERVER_DATASET_HTTP_BASE, ACCESS_SERVER_HTTP_BASE, ACCESS_SERVER_ICEBERG_HTTP_BASE, WEB_BASE, +}; use std::{collections::BTreeMap, path::Path}; fn catalog() -> DeploymentProfile { @@ -34,7 +37,12 @@ fn node_catalog_has_no_prescribed_members_media_or_rendered_service_configuratio .collect(); assert_eq!( endpoints, - BTreeMap::from([("web", 9090), ("s3", 9091), ("iceberg", 9092), ("dataset", 9093)]) + BTreeMap::from([ + ("web", WEB_BASE), + ("s3", ACCESS_SERVER_HTTP_BASE), + ("iceberg", ACCESS_SERVER_ICEBERG_HTTP_BASE), + ("dataset", ACCESS_SERVER_DATASET_HTTP_BASE), + ]) ); } diff --git a/container/crowdb-monitor/tests/node_run_inputs_test.rs b/container/crowdb-monitor/tests/node_run_inputs_test.rs index d3f62fc6b..fe8fb6603 100644 --- a/container/crowdb-monitor/tests/node_run_inputs_test.rs +++ b/container/crowdb-monitor/tests/node_run_inputs_test.rs @@ -41,12 +41,12 @@ fn container_preflight_reuses_explicit_file_ports_with_a_nonzero_offset() { let root = crowdb_test_harness::test_dirs::tempdir_in_test_data("node-cli-file-offset"); let profile = profile(root.path()); let config = root.path().join("node.toml"); + let mut services = crowdb_protocol::mgmt::node::deployment::NodeDeploymentPolicy::default() + .setup + .services; + services[0].listeners.rpc_port = Some(30101); let inputs = NodeDeploymentOverrides { - services: Some( - crowdb_protocol::mgmt::node::deployment::NodeDeploymentPolicy::default() - .setup - .services, - ), + services: Some(services), ..Default::default() }; fs::write(&config, toml::to_string(&inputs).unwrap()).unwrap(); @@ -83,7 +83,7 @@ fn container_preflight_reuses_explicit_file_ports_with_a_nonzero_offset() { let accepted = crowdb_monitor::read_node_policy(&root.path().join("install/data")) .unwrap() .unwrap(); - assert_eq!(accepted.setup.services[0].listeners.rpc_port, Some(10101)); + assert_eq!(accepted.setup.services[0].listeners.rpc_port, Some(30102)); } #[test] diff --git a/container/crowdb-monitor/tests/node_service_dependencies_test.rs b/container/crowdb-monitor/tests/node_service_dependencies_test.rs index a8184d96e..174c92583 100644 --- a/container/crowdb-monitor/tests/node_service_dependencies_test.rs +++ b/container/crowdb-monitor/tests/node_service_dependencies_test.rs @@ -84,7 +84,7 @@ mod service_dependency; #[tokio::test] async fn healthy_diskdb_listener_must_publish_the_registered_disks_ownership_before_consumers_start() { - use crowdb_kv_client::ServiceRegistryClient; + use crowdb_paxos_kv_client::ServiceRegistryClient; use crowdb_protocol::mgmt::node::setup::{NodeSetupChoices, NodeSetupRequest}; let root = crowdb_test_harness::test_dirs::tempdir_in_test_data("diskdb-route-gate"); let request = NodeSetupRequest { @@ -96,8 +96,8 @@ async fn healthy_diskdb_listener_must_publish_the_registered_disks_ownership_bef let authority = node_setup::TestAuthority::start(root.path(), &request).await; assert_eq!(authority.registry.nodes[0].node_id, 2); authority.kv.refresh_topology().await.unwrap(); - let kv = std::sync::Arc::new(crowdb_kv_client::CrowdbKvClient::new( - crowdb_kv_client::ClientConfig::new(authority.binding.management_seeds.clone()), + let kv = std::sync::Arc::new(crowdb_paxos_kv_client::PaxosKvClient::new( + crowdb_paxos_kv_client::ClientConfig::new(authority.binding.management_seeds.clone()), )); service_dependency::publish_online_disk(kv.clone()).await; let (address, server) = service_dependency::ready_listener().await; @@ -283,7 +283,7 @@ fn redirecting_rpc() -> (crowdb_rpc_ffi::RpcServer, Arc<AtomicUsize>) { (rpc, requests) } -async fn publish_catalog(kv: &crowdb_kv_client::CrowdbKvClient, port: i32, second: Option<u16>) { +async fn publish_catalog(kv: &crowdb_paxos_kv_client::PaxosKvClient, port: i32, second: Option<u16>) { let mut page = ChunkKvRangeCatalogPage { generation: 1, page_index: 0, diff --git a/container/crowdb-monitor/tests/node_setup_supplied_test.rs b/container/crowdb-monitor/tests/node_setup_supplied_test.rs index f7fe53ce1..09944f5c7 100644 --- a/container/crowdb-monitor/tests/node_setup_supplied_test.rs +++ b/container/crowdb-monitor/tests/node_setup_supplied_test.rs @@ -1,8 +1,8 @@ // Copyright 2026-present Gian <crow.db@outlook.com> // Licensed under the Apache License, Version 2.0. -use crowdb_kv_client::{ClientConfig, CrowdbKvClient, HardwareClient}; use crowdb_monitor::{DeploymentProfile, NodeIdentity, NodeSetupForTests}; +use crowdb_paxos_kv_client::{ClientConfig, HardwareClient, PaxosKvClient}; use crowdb_protocol::{ common::{BindMapValue, NodeValue, RackValue}, diskdb::rpc::DiskGroupValue, @@ -109,7 +109,7 @@ impl TestSetup { async fn authority(&self) -> node_setup::TestAuthority { let authority = node_setup::TestAuthority::start(self.root.path(), &self.request).await; assert_eq!(authority.registry.nodes[0].physical_host_id, "owned-host"); - let hardware = HardwareClient::new(CrowdbKvClient::new(ClientConfig::new( + let hardware = HardwareClient::new(PaxosKvClient::new(ClientConfig::new( authority.binding.management_seeds.clone(), ))); hardware @@ -249,7 +249,7 @@ async fn media_class_placement_and_disk_operations_survive_repeated_background_p disk_group_id: request.record.disk_group_id, disk_id: request.record.disk_id, }; - let hardware = HardwareClient::new(CrowdbKvClient::new(ClientConfig::new( + let hardware = HardwareClient::new(PaxosKvClient::new(ClientConfig::new( authority.binding.management_seeds.clone(), ))); assert!( diff --git a/container/crowdb-monitor/tests/node_setup_test.rs b/container/crowdb-monitor/tests/node_setup_test.rs index b6dcc339b..91c6a0256 100644 --- a/container/crowdb-monitor/tests/node_setup_test.rs +++ b/container/crowdb-monitor/tests/node_setup_test.rs @@ -227,12 +227,12 @@ async fn assert_diskless_setup(kv_enabled: bool) { 0, 0, preparation_key, - crowdb_kv_client::ReadMode::Linearizable, + crowdb_paxos_kv_client::ReadMode::Linearizable, None ) .await .unwrap(), - crowdb_kv_client::GetOutcome::NotFound, + crowdb_paxos_kv_client::GetOutcome::NotFound, )); assert_eq!(status.node_id, Some(2)); assert_eq!( @@ -300,7 +300,7 @@ async fn explicit_cleanup_retires_only_the_matching_setup_and_never_replays_it() #[tokio::test] async fn default_file_waits_for_metadata_then_reuses_one_durable_request_and_group() { - use crowdb_kv_client::{ClientConfig, CrowdbKvClient, HardwareClient}; + use crowdb_paxos_kv_client::{ClientConfig, HardwareClient, PaxosKvClient}; use crowdb_protocol::common::{NodeValue, RackValue}; let directory = run_root::TestRunRoot::new("setup"); let root = directory.path(); @@ -313,7 +313,7 @@ async fn default_file_waits_for_metadata_then_reuses_one_durable_request_and_gro .accept_for_tests(root, intent.clone()) .unwrap(); let authority = node_setup::TestAuthority::start(root, &intent).await; - let hardware = HardwareClient::new(CrowdbKvClient::new(ClientConfig::new( + let hardware = HardwareClient::new(PaxosKvClient::new(ClientConfig::new( authority.binding.management_seeds.clone(), ))); hardware @@ -370,7 +370,7 @@ async fn default_file_waits_for_metadata_then_reuses_one_durable_request_and_gro #[tokio::test] async fn a_conflicting_reserved_disk_group_binding_cannot_redirect_default_storage() { - use crowdb_kv_client::{ClientConfig, CrowdbKvClient, HardwareClient}; + use crowdb_paxos_kv_client::{ClientConfig, HardwareClient, PaxosKvClient}; use crowdb_protocol::{ common::{NodeValue, RackValue}, diskdb::rpc::DiskGroupValue, @@ -388,7 +388,7 @@ async fn a_conflicting_reserved_disk_group_binding_cannot_redirect_default_stora let authority = node_setup::TestAuthority::start(root, &intent).await; node_setup_groups::publish_group(&authority, 1).await; node_setup_groups::publish_group(&authority, 8).await; - let hardware = HardwareClient::new(CrowdbKvClient::new(ClientConfig::new( + let hardware = HardwareClient::new(PaxosKvClient::new(ClientConfig::new( authority.binding.management_seeds.clone(), ))); hardware diff --git a/container/crowdb-monitor/tests/profile_test.rs b/container/crowdb-monitor/tests/profile_test.rs index 1416a6995..eb8d78863 100644 --- a/container/crowdb-monitor/tests/profile_test.rs +++ b/container/crowdb-monitor/tests/profile_test.rs @@ -60,7 +60,7 @@ port = 14000 [[services]] id = "kv" -program = "/opt/crowdb/bin/crowdb-kv-server" +program = "/opt/crowdb/bin/crowdb-paxos-kv-server" args = [] dependencies = [] config_template = "/opt/crowdb/etc/templates/kv.toml" diff --git a/container/crowdb-monitor/tests/recovery_owners_test.rs b/container/crowdb-monitor/tests/recovery_owners_test.rs index 872532656..0c57c6c76 100644 --- a/container/crowdb-monitor/tests/recovery_owners_test.rs +++ b/container/crowdb-monitor/tests/recovery_owners_test.rs @@ -5,7 +5,7 @@ mod node_setup; use crowdb_console_shared::{ops::OpContext, ConsoleConfig}; -use crowdb_kv_client::{ClientConfig, CrowdbKvClient}; +use crowdb_paxos_kv_client::{ClientConfig, PaxosKvClient}; use crowdb_protocol::{ common::ReplicaValue, kv_membership::GroupMember, @@ -28,7 +28,7 @@ async fn installing_membership_authorizes_catchup_routes_before_replica_promotio let authority = node_setup::TestAuthority::start(root.path(), &request).await; authority.kv.refresh_topology().await.unwrap(); assert_eq!(authority.registry.nodes[0].node_id, 2); - let kv = Arc::new(CrowdbKvClient::new(ClientConfig::new( + let kv = Arc::new(PaxosKvClient::new(ClientConfig::new( authority.binding.management_seeds.clone(), ))); let ctx = OpContext::with_shared_client_preserving_hint( diff --git a/container/crowdb-monitor/tests/single_node_profile_test.rs b/container/crowdb-monitor/tests/single_node_profile_test.rs index 812844cf8..4e8f343f3 100644 --- a/container/crowdb-monitor/tests/single_node_profile_test.rs +++ b/container/crowdb-monitor/tests/single_node_profile_test.rs @@ -37,7 +37,7 @@ fn single_node_preview_has_exact_topology_and_endpoints() { .collect::<BTreeMap<_, _>>(); assert_eq!( endpoints, - BTreeMap::from([("iceberg", 9092), ("s3", 9091), ("web", 9090)]) + BTreeMap::from([("iceberg", 8135), ("s3", 7135), ("web", 6135)]) ); let access_service = profile .services @@ -46,13 +46,13 @@ fn single_node_preview_has_exact_topology_and_endpoints() { .unwrap(); assert_eq!( access_service.env.get("CROWDB_ICEBERG_PUBLIC_URI"), - Some(&"http://localhost:9092".to_owned()) + Some(&"http://localhost:8135".to_owned()) ); - assert_eq!(access_service.probe.target, "http://127.0.0.1:9092/v1/config"); + assert_eq!(access_service.probe.target, "http://127.0.0.1:8135/v1/config"); assert_eq!(access_service.additional_probes.len(), 1); assert_eq!( access_service.additional_probes[0].target, - "http://127.0.0.1:9094/_crowdb/health/ready" + "http://127.0.0.1:25135/_crowdb/health/ready" ); assert_eq!( access_service.additional_probes[0].failure_threshold, @@ -60,16 +60,16 @@ fn single_node_preview_has_exact_topology_and_endpoints() { ); assert_eq!( access_service.env.get("CROWDB_MANAGEMENT_SEEDS"), - Some(&"http://127.0.0.1:10000".to_owned()) + Some(&"http://127.0.0.1:10135".to_owned()) ); assert_eq!(access_service.args[1], "/opt/crowdb/run/config/access.toml"); assert_eq!( access_service.fence_listeners, [ - "127.0.0.1:9092", - "127.0.0.1:9091", - "127.0.0.1:9093", - "127.0.0.1:9094" + "127.0.0.1:8135", + "127.0.0.1:7135", + "127.0.0.1:9135", + "127.0.0.1:25135" ] ); let access = std::fs::read_to_string( @@ -77,25 +77,25 @@ fn single_node_preview_has_exact_topology_and_endpoints() { ) .unwrap(); let access: toml::Value = toml::from_str(&access).unwrap(); - assert_eq!(access["iceberg"]["listen"].as_str(), Some("0.0.0.0:9092")); - assert_eq!(access["s3"]["listen"].as_str(), Some("0.0.0.0:9091")); - assert_eq!(access["dataset"]["listen"].as_str(), Some("0.0.0.0:9093")); + assert_eq!(access["iceberg"]["listen"].as_str(), Some("0.0.0.0:8135")); + assert_eq!(access["s3"]["listen"].as_str(), Some("0.0.0.0:7135")); + assert_eq!(access["dataset"]["listen"].as_str(), Some("0.0.0.0:9135")); assert_eq!( access_service.env.get("CROWDB_DATASET_PUBLIC_URI"), - Some(&"http://localhost:9093".to_owned()) + Some(&"http://localhost:9135".to_owned()) ); let web = profile .services .iter() .find(|service| service.id == "web") .unwrap(); - assert_eq!(web.probe.target, "http://127.0.0.1:9090/healthz"); + assert_eq!(web.probe.target, "http://127.0.0.1:6135/healthz"); let template = std::fs::read_to_string( Path::new(env!("CARGO_MANIFEST_DIR")).join("../single-node-container/templates/crowdb-web.toml"), ) .unwrap(); let config: toml::Value = toml::from_str(&template).unwrap(); - assert_eq!(config["port"].as_integer(), Some(9090)); + assert_eq!(config["port"].as_integer(), Some(6135)); } #[test] diff --git a/container/oci-image/Dockerfile b/container/oci-image/Dockerfile index 47a2e030a..71493b5f7 100644 --- a/container/oci-image/Dockerfile +++ b/container/oci-image/Dockerfile @@ -1,3 +1,4 @@ +# Generated by tools/ports/generate.py from ports.rs; edit the .in template. FROM ubuntu:24.04@sha256:496754492fb28b4d3049432f2ca787449331e23fb14f0dd3fffea86bf5a93eb4 RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates openssh-server util-linux curl tini libibverbs1 librdmacm1 ibverbs-providers ibverbs-utils && rm -rf /var/lib/apt/lists/* \ && groupadd --system --gid 10001 crowdb \ @@ -45,6 +46,6 @@ ENV PATH="/opt/crowdb/bin:${PATH}" \ CROWDB_RUNTIME_ROOT="/opt/crowdb/run" USER root VOLUME ["/opt/crowdb/data"] -EXPOSE 9090 16000 17000 18000 9095 2222 +EXPOSE 10135 11135 13135 14135 16135 17135 18135 6135 20135 21135 7135 8135 9135 25135 26135 2222 STOPSIGNAL SIGTERM ENTRYPOINT ["/usr/bin/tini", "--", "/opt/crowdb/bin/entrypoint"] diff --git a/container/oci-image/Dockerfile.ports.in b/container/oci-image/Dockerfile.ports.in new file mode 100644 index 000000000..a52b79341 --- /dev/null +++ b/container/oci-image/Dockerfile.ports.in @@ -0,0 +1,50 @@ +FROM ubuntu:24.04@sha256:496754492fb28b4d3049432f2ca787449331e23fb14f0dd3fffea86bf5a93eb4 +RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates openssh-server util-linux curl tini libibverbs1 librdmacm1 ibverbs-providers ibverbs-utils && rm -rf /var/lib/apt/lists/* \ + && groupadd --system --gid 10001 crowdb \ + && useradd --system --uid 10001 --gid 10001 --home-dir /opt/crowdb --shell /bin/sh crowdb \ + && echo 'crowdb:crowdb' | chpasswd \ + && mkdir -p /run/sshd +ARG SOURCE_REVISION +ARG SOURCE_TREE +ARG RUNTIME_SHA256 +ARG PREVIEW_VERSION +ARG DEBUG_INFO=stripped +RUN --mount=type=bind,source=.,target=/staged,ro \ + test -n "$SOURCE_REVISION" && test -n "$PREVIEW_VERSION" \ + && test "$(cat /staged/SOURCE_REVISION)" = "$SOURCE_REVISION" \ + && { test -z "$SOURCE_TREE" || test "$(cat /staged/SOURCE_TREE)" = "$SOURCE_TREE"; } \ + && test "$(cat /staged/VERSION)" = "$PREVIEW_VERSION" \ + && mkdir -p /opt/crowdb/etc /opt/crowdb/data /opt/crowdb/run \ + && cp -a /staged/bin /staged/lib /staged/ui /opt/crowdb/ \ + && if test -d /staged/etc; then cp -a /staged/etc /opt/crowdb/; fi \ + && cp /staged/profile.toml /opt/crowdb/etc/profile.toml \ + && cp /staged/entrypoint.sh /opt/crowdb/bin/entrypoint \ + && chown -R crowdb:crowdb /opt/crowdb/data /opt/crowdb/run \ + && chmod 0755 /opt/crowdb/bin/entrypoint \ + && for binary in /opt/crowdb/bin/crowdb-* /opt/crowdb/lib/* /opt/crowdb/lib/libibverbs/*.so; do \ + test -f "$binary" || continue; \ + LD_LIBRARY_PATH=/opt/crowdb/lib ldd -r "$binary" > /tmp/dependencies.txt 2>&1 || { cat /tmp/dependencies.txt; exit 1; }; \ + if grep -Eq 'not found|undefined symbol' /tmp/dependencies.txt; then cat /tmp/dependencies.txt; exit 1; fi; \ + done && rm /tmp/dependencies.txt + +LABEL org.opencontainers.image.title="CROWDB Node" \ + org.opencontainers.image.description="CROWDB node with discovery, management UI and storage services." \ + org.opencontainers.image.url="https://crowdb.dev/" \ + org.opencontainers.image.source="https://github.com/buzzcrow/crowdb" \ + org.opencontainers.image.authors="Gian <crow.db@outlook.com>" \ + org.opencontainers.image.licenses="Apache-2.0" \ + org.opencontainers.image.revision="$SOURCE_REVISION" \ + org.crowdb.source.tree="$SOURCE_TREE" \ + org.opencontainers.image.version="$PREVIEW_VERSION" \ + org.crowdb.runtime.sha256="$RUNTIME_SHA256" \ + org.crowdb.debug-info="$DEBUG_INFO" \ + org.crowdb.startup.default="local" +ENV PATH="/opt/crowdb/bin:${PATH}" \ + LD_LIBRARY_PATH="/opt/crowdb/lib" \ + RDMAV_DRIVERS_DIR="/opt/crowdb/lib/libibverbs" \ + CROWDB_RUNTIME_ROOT="/opt/crowdb/run" +USER root +VOLUME ["/opt/crowdb/data"] +EXPOSE @PAXOS_KV_SERVER_MGMT_BASE@ @PAXOS_KV_SERVER_RPC_BASE@ @DISKDB_HTTP_BASE@ @DISKDB_RPC_BASE@ @CHUNKDB_HTTP_BASE@ @CHUNKDB_RPC_BASE@ @DISKIO_RPC_BASE@ @WEB_BASE@ @CHUNK_KV_HTTP_BASE@ @CHUNK_KV_RPC_BASE@ @ACCESS_SERVER_HTTP_BASE@ @ACCESS_SERVER_ICEBERG_HTTP_BASE@ @ACCESS_SERVER_DATASET_HTTP_BASE@ @ACCESS_SERVER_HEALTH_HTTP_BASE@ @MONITOR_HTTP_BASE@ @SSH_PORT@ +STOPSIGNAL SIGTERM +ENTRYPOINT ["/usr/bin/tini", "--", "/opt/crowdb/bin/entrypoint"] diff --git a/container/oci-image/README.md b/container/oci-image/README.md index 9d634992d..af38c33ce 100644 --- a/container/oci-image/README.md +++ b/container/oci-image/README.md @@ -3,12 +3,7 @@ # Shared OCI image -Build one Linux amd64 image for Docker and containerd. Ubuntu 24.04 is pinned -by digest; the common recipe packages the existing release binaries, UI and SSH. -Construction requires Pixi and Linux kernel privileges, with no Docker or -containerd service. The `oci-build` shortcut also loads the result into local -Docker by default; use `--no-load` for Docker-free archive construction. -Running the image is a separate operation. +Build one Linux amd64 image for Docker and containerd. Ubuntu 24.04 is pinned by digest; the common recipe packages the existing release binaries, UI and SSH. Construction requires Pixi and Linux kernel privileges, with no Docker or containerd service. The `oci-build` shortcut also loads the result into local Docker by default; use `--no-load` for Docker-free archive construction. Running the image is a separate operation. ## Build and import @@ -22,49 +17,19 @@ pixi run oci-build --no-load pixi run --manifest-path container/oci-image/pixi.toml import target/crowdb.oci.tar --image crowdb-node:dev ``` -`oci-build` compiles/stages with the repository environment, then selects the -locked standalone builder environment. It loads `crowdb-node:dev` into Docker -after archive validation, or uses `CROWDB_CONTAINER_IMAGE` / `--load-image NAME` -for a different tag. Docker availability is checked before compilation when -loading is enabled; import failures fail the command and preserve the archive. -`--no-load` skips all Docker access. `--staged` reuses an existing -`target/container-runtime`. Unsupported platforms fail before compilation. -Only Linux amd64 is enabled; macOS will compile inside a Linux VM backend. - -Rootless builds require host-installed setuid `newuidmap`/`newgidmap`, subordinate -UID/GID ranges for the invoking user, and permitted unprivileged user namespaces. -Pixi supplies RootlessKit, BuildKit and runc; it cannot grant kernel privileges. -The builder uses the native snapshotter and host network, so no FUSE or CNI is -required. Startup errors include complete daemon diagnostics. Rootless preflight -validates these requirements by starting the pinned RootlessKit with the same -namespace and mount setup before compiling or starting BuildKit. -Both rootless build/import and the explicit rootful recipe are supported. - -On Linux amd64, `pixi run install-deps` also prepares the OCI builder's host -prerequisites. On hosts that restrict unprivileged user namespaces through -AppArmor, the system RootlessKit profile does not cover the Pixi executable. -When the namespace probe reports that denial, setup generates, validates, -installs and loads a profile for the pinned executable through sudo. Enter the -sudo password in the same terminal when prompted; dependency installation then -continues. A healthy namespace probe requires no sudo or policy changes. - -`oci-build` performs the same check and setup before compiling, so it also works -directly on a host that has not run `install-deps`: +`oci-build` compiles/stages with the repository environment, then selects the locked standalone builder environment. It loads `crowdb-node:dev` into Docker after archive validation, or uses `CROWDB_CONTAINER_IMAGE` / `--load-image NAME` for a different tag. Docker availability is checked before compilation when loading is enabled; import failures fail the command and preserve the archive. `--no-load` skips all Docker access. `--staged` reuses an existing `target/container-runtime`. Unsupported platforms fail before compilation. Only Linux amd64 is enabled; macOS will compile inside a Linux VM backend. + +Rootless builds require host-installed setuid `newuidmap`/`newgidmap`, subordinate UID/GID ranges for the invoking user, and permitted unprivileged user namespaces. Pixi supplies RootlessKit, BuildKit and runc; it cannot grant kernel privileges. The builder uses the native snapshotter and host network, so no FUSE or CNI is required. Startup errors include complete daemon diagnostics. Rootless preflight validates these requirements by starting the pinned RootlessKit with the same namespace and mount setup before compiling or starting BuildKit. Both rootless build/import and the explicit rootful recipe are supported. + +On Linux amd64, `pixi run install-deps` also prepares the OCI builder's host prerequisites. On hosts that restrict unprivileged user namespaces through AppArmor, the system RootlessKit profile does not cover the Pixi executable. When the namespace probe reports that denial, setup generates, validates, installs and loads a profile for the pinned executable through sudo. Enter the sudo password in the same terminal when prompted; dependency installation then continues. A healthy namespace probe requires no sudo or policy changes. + +`oci-build` performs the same check and setup before compiling, so it also works directly on a host that has not run `install-deps`: ```bash pixi run oci-build ``` -Profiles allow user namespaces for one executable path, have distinct names per -checkout/environment path and leave the host-wide restriction unchanged. This -is host configuration, separate from the tools installed in Pixi. A setup -failure stops before compilation. Unattended runs use noninteractive sudo and -fail promptly if administrator provisioning or authentication is needed. Other -kernel or UID/GID mapping errors remain host prerequisite failures. -Administrators can inspect generated policy with `preflight.py --apparmor-profile` -in the OCI Pixi environment. If compilation and staging already completed before -a host prerequisite failure, use `pixi run oci-build --staged` to reuse those -artifacts. +Profiles allow user namespaces for one executable path, have distinct names per checkout/environment path and leave the host-wide restriction unchanged. This is host configuration, separate from the tools installed in Pixi. A setup failure stops before compilation. Unattended runs use noninteractive sudo and fail promptly if administrator provisioning or authentication is needed. Other kernel or UID/GID mapping errors remain host prerequisite failures. Administrators can inspect generated policy with `preflight.py --apparmor-profile` in the OCI Pixi environment. If compilation and staging already completed before a host prerequisite failure, use `pixi run oci-build --staged` to reuse those artifacts. If rootless prerequisites are unavailable, an administrator can explicitly run: @@ -76,26 +41,11 @@ pixi run sudo "$(command -v pixi)" run --as-is \ --context "$PWD/target/container-runtime" --output "$PWD/target/crowdb.oci.tar" ``` -There is no automatic privilege fallback. `--registry-mirror host/path` may be -repeated for Docker Hub mirrors; credential-bearing mirror/proxy settings are -rejected. Builds use a private socket and persistent cache, stop their own -process group on exit/interruption, and never stop a system daemon. Concurrent -builds need separate `--cache` directories. The output archive and JSON receipt -are finalized after descriptor/blob/config validation. A mismatched pair is -rejected by import/publication. The receipt records revision, version, base, -tool versions, staged-content hash, image manifest/config digests and archive checksum. -Local uncommitted builds carry HEAD's revision; the content hash identifies the -actual staging. Release CI checks out the exact source revision. - -Docker 28.3.3's classic store was tested using explicit Skopeo conversion into a -local Docker archive. This changes only the import representation. The published -OCI graph is unchanged; Docker's local config ID differs from its OCI manifest -digest. Never substitute a local config ID for a registry digest. - -The image runs Tini as PID 1 in both runtimes. It reaps orphaned SSH session -processes and forwards stop signals to the monitor; the monitor retains service -shutdown ordering and child exit ownership. No runtime-specific `--init` flag is -needed. Do not replace the image entrypoint with the monitor for normal nodes. +There is no automatic privilege fallback. `--registry-mirror host/path` may be repeated for Docker Hub mirrors; credential-bearing mirror/proxy settings are rejected. Builds use a private socket and persistent cache, stop their own process group on exit/interruption, and never stop a system daemon. Concurrent builds need separate `--cache` directories. The output archive and JSON receipt are finalized after descriptor/blob/config validation. A mismatched pair is rejected by import/publication. The receipt records revision, version, base, tool versions, staged-content hash, image manifest/config digests and archive checksum. Local uncommitted builds carry HEAD's revision; the content hash identifies the actual staging. Release CI checks out the exact source revision. + +Docker 28.3.3's classic store was tested using explicit Skopeo conversion into a local Docker archive. This changes only the import representation. The published OCI graph is unchanged; Docker's local config ID differs from its OCI manifest digest. Never substitute a local config ID for a registry digest. + +The image runs Tini as PID 1 in both runtimes. It reaps orphaned SSH session processes and forwards stop signals to the monitor; the monitor retains service shutdown ordering and child exit ownership. No runtime-specific `--init` flag is needed. Do not replace the image entrypoint with the monitor for normal nodes. ## Local cluster shortcuts @@ -118,61 +68,25 @@ pixi run oci-run-3c --cluster demo-three pixi run oci-clean --cluster demo-three ``` -Run never compiles, downloads or pulls an image. It accepts `--image`, `--cpus`, -`--memory-mib` and `--port-base` on first creation, and retains those inputs. -A different node count or conflicting input requires another cluster name. -Ports for each node are UI/S3/Iceberg/Dataset at base/base+1/base+2/base+3; -subsequent members add 10 and 20. All published ports bind to `127.0.0.1`. -Run and stop default to `local`; stop accepts `--all` for every recorded local -cluster. Default clean skips running recorded local and host-network clusters, -including all resources of a bridge cluster with any running member. It removes -stopped clusters and their node data, then cleans unused resources on the selected -Docker daemon, including containers without CROWDB receipts: +Run never compiles, downloads or pulls an image. It accepts `--image`, `--cpus`, `--memory-mib` and `--port-base` on first creation, and retains those inputs. A different node count or conflicting input requires another cluster name. Ports for each node are UI/S3/Iceberg/Dataset at base/base+1/base+2/base+3; subsequent members add 10 and 20. All published ports bind to `127.0.0.1`. Run and stop default to `local`; stop accepts `--all` for every recorded local cluster. Default clean skips running recorded local and host-network clusters, including all resources of a bridge cluster with any running member. It removes stopped clusters and their node data, then cleans unused resources on the selected Docker daemon, including containers without CROWDB receipts: ```bash pixi run oci-clean ``` -- Remove stopped containers first, then unused networks and both named and - anonymous volumes, followed by unused Docker build cache. Running containers - and referenced resources remain intact. Docker removals never force active - containers or in-use resources; a concurrent new reference makes removal fail. -- Preserve resources belonging to skipped running clusters and their restart - receipts, including detached volumes and stopped members of those clusters. -- Use `--keep-data` to preserve all Docker volumes, node data and restart receipts; - `--delete-data` explicitly selects the default deletion mode. -- `--cluster NAME` explicitly stops and removes one recorded cluster, without - daemon-wide garbage collection. `--all` explicitly stops and removes every - recorded cluster and also cleans unused Docker resources. -- With no cluster receipts, unused Docker resource cleanup still runs if Docker - is installed. An explicitly named cluster without a receipt remains an error. - Containerd-only host cleanup needs no Docker daemon. - -Host receipts live in `.crowdb-runtime/persistent/oci-clusters/`. The shortcuts -verify the Docker daemon/context and exact ownership labels before effects. -Clean removes selected stopped containers, unused owned bridges and owned data -volumes. It also removes the launch receipt and matching optional CLI target -binding. `--keep-data` retains volumes and restart inputs. Image cleanup -only considers dangling CROWDB images, preserving retained cluster references -and images used by any running or stopped container. Attached or relabelled -networks and volumes referenced elsewhere are rejected. Daemon-wide garbage -collection runs only when no `--cluster` is selected. -These host scripts invoke Docker directly and require no CLI binary or application -target binding. A binding with different ownership is rejected before data -deletion. - -One node and three nodes use the same editable monitor lifecycle. Local mode -prepares test storage and all access services. An incomplete three-node start -waits for peers; it never falls back to singleton bootstrap. Each node retains -its identity, SSH keys, cluster binding and accepted setup in its own volume. -Discovery uses mDNS/DNS-SD; no DNS server installation or peer list is required. -Advertisements supply hints; pinned SSH identity verifies new member addresses. +- Remove stopped containers first, then unused networks and both named and anonymous volumes, followed by unused Docker build cache. Running containers and referenced resources remain intact. Docker removals never force active containers or in-use resources; a concurrent new reference makes removal fail. +- Preserve resources belonging to skipped running clusters and their restart receipts, including detached volumes and stopped members of those clusters. +- Use `--keep-data` to preserve all Docker volumes, node data and restart receipts; `--delete-data` explicitly selects the default deletion mode. +- `--cluster NAME` explicitly stops and removes one recorded cluster, without daemon-wide garbage collection. `--all` explicitly stops and removes every recorded cluster and also cleans unused Docker resources. +- With no cluster receipts, unused Docker resource cleanup still runs if Docker is installed. An explicitly named cluster without a receipt remains an error. Containerd-only host cleanup needs no Docker daemon. + +Host receipts live in `.crowdb-runtime/persistent/oci-clusters/`. The shortcuts verify the Docker daemon/context and exact ownership labels before effects. Clean removes selected stopped containers, unused owned bridges and owned data volumes. It also removes the launch receipt and matching optional CLI target binding. `--keep-data` retains volumes and restart inputs. Image cleanup only considers dangling CROWDB images, preserving retained cluster references and images used by any running or stopped container. Attached or relabelled networks and volumes referenced elsewhere are rejected. Daemon-wide garbage collection runs only when no `--cluster` is selected. These host scripts invoke Docker directly and require no CLI binary or application target binding. A binding with different ownership is rejected before data deletion. + +One node and three nodes use the same editable monitor lifecycle. Local mode prepares test storage and all access services. An incomplete three-node start waits for peers; it never falls back to singleton bootstrap. Each node retains its identity, SSH keys, cluster binding and accepted setup in its own volume. Discovery uses mDNS/DNS-SD; no DNS server installation or peer list is required. Advertisements supply hints; pinned SSH identity verifies new member addresses. ## Common CLI lifecycle and application targets -The CLI can create the same local Docker clusters and wait for complete -application readiness. It shares host ownership receipts with the shortcuts; -neither entry point implements a separate bootstrap or storage deployment. +The CLI can create the same local Docker clusters and wait for complete application readiness. It shares host ownership receipts with the shortcuts; neither entry point implements a separate bootstrap or storage deployment. ```bash pixi run crowdb-cli --cluster demo cluster start @@ -186,18 +100,9 @@ pixi run crowdb-cli --management-endpoint http://127.0.0.1:9190 s3 object get ex pixi run oci-clean --cluster demo-three ``` -Use the built `crowdb-cli` executable on PATH, or substitute its absolute path -after `pixi run`. Start defaults to cluster `local`, one node, 1 CPU, 1536 MiB, -and console port 9090. It accepts `--image`, `--cpus`, `--memory-mib` and -`--port-base` before sealing the first launch inputs. Subsequent starts verify -those inputs and resume existing state. `--wait` bounds readiness observation; -an unmet check returns 1 and leaves accepted work resumable. +Use the built `crowdb-cli` executable on PATH, or substitute its absolute path after `pixi run`. Start defaults to cluster `local`, one node, 1 CPU, 1536 MiB, and console port 6135. It accepts `--image`, `--cpus`, `--memory-mib` and `--port-base` before sealing the first launch inputs. Subsequent starts verify those inputs and resume existing state. `--wait` bounds readiness observation; an unmet check returns 1 and leaves accepted work resumable. -For production, `cluster start --network host` runs one node on this physical -host. Select `--runtime docker` or `--runtime containerd`; containerd also accepts -`--runtime-endpoint` and `--snapshotter native|overlayfs`. Python 3.11 or newer -and the selected runtime client must be installed on the host. Load the immutable -node image first; this command never builds or pulls it. +For production, `cluster start --network host` runs one node on this physical host. Select `--runtime docker` or `--runtime containerd`; containerd also accepts `--runtime-endpoint` and `--snapshotter native|overlayfs`. Python 3.11 or newer and the selected runtime client must be installed on the host. Load the immutable node image first; this command never builds or pulls it. ```bash pixi run crowdb-cli --cluster production cluster start --runtime containerd \ @@ -211,45 +116,13 @@ pixi run crowdb-cli --cluster production cluster start pixi run oci-clean --cluster production ``` -The existing data directory must be writable by UID 10001. The node policy -must select production mode, the same cluster and initial voter count, supplied -media and reachable public protocol URIs. `--nodes 3` selects the three-voter -formation policy; run one node on each host. Host networking uses port 9090 and -rejects `--port-base`. The management address must belong to the selected local -interface. Disk files use `--disk-file`; files and devices are explicit existing -inputs and are never created by the CLI. - -The CLI seals a private host receipt before creating a container. Start/resume -and stop verify the recorded daemon, Unix endpoint, context/namespace, owner, -image, resource limits, mounts and startup inputs. Resume preserves the accepted -policy and media identities; foreign same-name containers are rejected. -The same `oci-clean` command works without the CLI binary for Docker bridge, -Docker host-network and containerd host-network clusters. Host cleanup removes -the verified container, recorded node data directory and matching launch and -application receipts. It preserves supplied disk files, raw devices, node policy -and password inputs, including supplied files inside the data directory; their -containing directories remain when needed. A data directory that is itself a -mount point is emptied without removing the mount point. Nonempty data requires -the matching retained storage declaration. Symlinked roots, nested mounts, -overlapping host data directories and data referenced by another container in -the selected runtime are rejected. Run with the host permissions needed to -delete that data; failures retain ownership receipts for retry. -Use `--keep-data` to preserve host data and receipts for recreation on the next -start. Cleanup never wipes supplied media, so reusing those media still requires -their retained cluster identity. - -Status/check and application commands need only management HTTP access. A named -CLI target pins the observed cluster UUID; explicit management endpoints must -agree with that UUID and with one another. An unavailable selected endpoint -does not fall back to localhost. S3 commands retrieve the selected cluster's -published S3 endpoint and credentials, then issue signed native S3 requests. -They require no Docker socket or local deployment directory. HTTP diagnostics -redact authorization and session tokens. The CLI has no S3-specific cluster -lifecycle or container cleanup command. - -Workloads attach to the same selected cluster. They never start, stop, reset or -provision it. Native KV/storage clients additionally need network reachability -to the advertised RPC addresses; HTTP-only port forwarding is insufficient. +The existing data directory must be writable by UID 10001. The node policy must select production mode, the same cluster and initial voter count, supplied media and reachable public protocol URIs. `--nodes 3` selects the three-voter formation policy; run one node on each host. Host networking uses port 6135 and rejects `--port-base`. The management address must belong to the selected local interface. Disk files use `--disk-file`; files and devices are explicit existing inputs and are never created by the CLI. + +The CLI seals a private host receipt before creating a container. Start/resume and stop verify the recorded daemon, Unix endpoint, context/namespace, owner, image, resource limits, mounts and startup inputs. Resume preserves the accepted policy and media identities; foreign same-name containers are rejected. The same `oci-clean` command works without the CLI binary for Docker bridge, Docker host-network and containerd host-network clusters. Host cleanup removes the verified container, recorded node data directory and matching launch and application receipts. It preserves supplied disk files, raw devices, node policy and password inputs, including supplied files inside the data directory; their containing directories remain when needed. A data directory that is itself a mount point is emptied without removing the mount point. Nonempty data requires the matching retained storage declaration. Symlinked roots, nested mounts, overlapping host data directories and data referenced by another container in the selected runtime are rejected. Run with the host permissions needed to delete that data; failures retain ownership receipts for retry. Use `--keep-data` to preserve host data and receipts for recreation on the next start. Cleanup never wipes supplied media, so reusing those media still requires their retained cluster identity. + +Status/check and application commands need only management HTTP access. A named CLI target pins the observed cluster UUID; explicit management endpoints must agree with that UUID and with one another. An unavailable selected endpoint does not fall back to localhost. S3 commands retrieve the selected cluster's published S3 endpoint and credentials, then issue signed native S3 requests. They require no Docker socket or local deployment directory. HTTP diagnostics redact authorization and session tokens. The CLI has no S3-specific cluster lifecycle or container cleanup command. + +Workloads attach to the same selected cluster. They never start, stop, reset or provision it. Native KV/storage clients additionally need network reachability to the advertised RPC addresses; HTTP-only port forwarding is insufficient. ```bash pixi run crowdb-cli --cluster demo-three bench kv prepare --store 0 --group 1 --keys 100 @@ -260,13 +133,7 @@ pixi run crowdb-cli --cluster demo-three bench chunkio write --objects 1 --durat --mirror-copies 1 --block-size 1048576 --chunk-size 16777216 --prefetch-chunks 1 ``` -The local storage policy accepts 1 MiB mirror strips with one or two copies. -Use the deployment's supported layout explicitly; a benchmark never changes -that policy. DiskDB checks current disk-group geometry and accounting units -instead of requiring a fixed group/disk count. `bench kv prepare` writes only -its declared keys. S3 workloads use a fresh bucket per invocation and delete -only that workload's objects/bucket. `--max-dataset-bytes` bounds the prepared -dataset; benchmark output does not claim remote memory usage or backend type. +The local storage policy accepts 1 MiB mirror strips with one or two copies. Use the deployment's supported layout explicitly; a benchmark never changes that policy. DiskDB checks current disk-group geometry and accounting units instead of requiring a fixed group/disk count. `bench kv prepare` writes only its declared keys. S3 workloads use a fresh bucket per invocation and delete only that workload's objects/bucket. `--max-dataset-bytes` bounds the prepared dataset; benchmark output does not claim remote memory usage or backend type. ## Direct Docker and Compose @@ -274,48 +141,46 @@ A direct single-container cluster needs a volume and four published ports: ```bash pixi run docker run -d --name crowdb-one --cpus 1 --memory 1536m \ - -e CROWDB_S3_PUBLIC_URI=http://127.0.0.1:9091 \ - -e CROWDB_ICEBERG_PUBLIC_URI=http://127.0.0.1:9092 \ - -e CROWDB_DATASET_PUBLIC_URI=http://127.0.0.1:9093 \ - -p 127.0.0.1:9090:9090 -p 127.0.0.1:9091:16000 \ - -p 127.0.0.1:9092:17000 -p 127.0.0.1:9093:18000 \ + -e CROWDB_S3_PUBLIC_URI=http://127.0.0.1:7135 \ + -e CROWDB_ICEBERG_PUBLIC_URI=http://127.0.0.1:8135 \ + -e CROWDB_DATASET_PUBLIC_URI=http://127.0.0.1:9135 \ + -p 127.0.0.1:6135:6135 -p 127.0.0.1:7135:7135 \ + -p 127.0.0.1:8135:8135 -p 127.0.0.1:9135:9135 \ --mount type=volume,source=crowdb-one-data,target=/opt/crowdb/data \ crowdb-node:dev ``` -For three nodes, create a dedicated bridge and execute these commands separately -in any order. No node addresses or membership list are supplied: +For three nodes, create a dedicated bridge and execute these commands separately in any order. No node addresses or membership list are supplied: ```bash pixi run docker network create crowdb-demo pixi run docker run -d --name crowdb-demo-1 --network crowdb-demo --cpus 1 --memory 1536m \ -e CROWDB_CLUSTER=demo -e CROWDB_INITIAL_VOTERS=3 \ - -e CROWDB_S3_PUBLIC_URI=http://127.0.0.1:9091 \ - -e CROWDB_ICEBERG_PUBLIC_URI=http://127.0.0.1:9092 \ - -e CROWDB_DATASET_PUBLIC_URI=http://127.0.0.1:9093 \ - -p 127.0.0.1:9090:9090 -p 127.0.0.1:9091:16000 \ - -p 127.0.0.1:9092:17000 -p 127.0.0.1:9093:18000 \ + -e CROWDB_S3_PUBLIC_URI=http://127.0.0.1:7135 \ + -e CROWDB_ICEBERG_PUBLIC_URI=http://127.0.0.1:8135 \ + -e CROWDB_DATASET_PUBLIC_URI=http://127.0.0.1:9135 \ + -p 127.0.0.1:6135:6135 -p 127.0.0.1:7135:7135 \ + -p 127.0.0.1:8135:8135 -p 127.0.0.1:9135:9135 \ --mount type=volume,source=crowdb-demo-1-data,target=/opt/crowdb/data crowdb-node:dev pixi run docker run -d --name crowdb-demo-2 --network crowdb-demo --cpus 1 --memory 1536m \ -e CROWDB_CLUSTER=demo -e CROWDB_INITIAL_VOTERS=3 \ - -e CROWDB_S3_PUBLIC_URI=http://127.0.0.1:9101 \ - -e CROWDB_ICEBERG_PUBLIC_URI=http://127.0.0.1:9102 \ - -e CROWDB_DATASET_PUBLIC_URI=http://127.0.0.1:9103 \ - -p 127.0.0.1:9100:9090 -p 127.0.0.1:9101:16000 \ - -p 127.0.0.1:9102:17000 -p 127.0.0.1:9103:18000 \ + -e CROWDB_S3_PUBLIC_URI=http://127.0.0.1:7136 \ + -e CROWDB_ICEBERG_PUBLIC_URI=http://127.0.0.1:8136 \ + -e CROWDB_DATASET_PUBLIC_URI=http://127.0.0.1:9136 \ + -p 127.0.0.1:6136:6135 -p 127.0.0.1:7136:7135 \ + -p 127.0.0.1:8136:8135 -p 127.0.0.1:9136:9135 \ --mount type=volume,source=crowdb-demo-2-data,target=/opt/crowdb/data crowdb-node:dev pixi run docker run -d --name crowdb-demo-3 --network crowdb-demo --cpus 1 --memory 1536m \ -e CROWDB_CLUSTER=demo -e CROWDB_INITIAL_VOTERS=3 \ - -e CROWDB_S3_PUBLIC_URI=http://127.0.0.1:9111 \ - -e CROWDB_ICEBERG_PUBLIC_URI=http://127.0.0.1:9112 \ - -e CROWDB_DATASET_PUBLIC_URI=http://127.0.0.1:9113 \ - -p 127.0.0.1:9110:9090 -p 127.0.0.1:9111:16000 \ - -p 127.0.0.1:9112:17000 -p 127.0.0.1:9113:18000 \ + -e CROWDB_S3_PUBLIC_URI=http://127.0.0.1:7137 \ + -e CROWDB_ICEBERG_PUBLIC_URI=http://127.0.0.1:8137 \ + -e CROWDB_DATASET_PUBLIC_URI=http://127.0.0.1:9137 \ + -p 127.0.0.1:6137:6135 -p 127.0.0.1:7137:7135 \ + -p 127.0.0.1:8137:8135 -p 127.0.0.1:9137:9135 \ --mount type=volume,source=crowdb-demo-3-data,target=/opt/crowdb/data crowdb-node:dev ``` -Equivalent Compose examples are [compose.yaml](compose.yaml) and -[compose-three.yaml](compose-three.yaml): +Equivalent Compose examples are [compose.yaml](compose.yaml) and [compose-three.yaml](compose-three.yaml): ```bash pixi run docker compose -f container/oci-image/compose.yaml up -d @@ -324,23 +189,11 @@ pixi run docker compose -f container/oci-image/compose-three.yaml up -d pixi run docker compose -f container/oci-image/compose-three.yaml down ``` -Fresh nodes listen on container ports 9090 (Web), 16000 (S3), 17000 (Iceberg) -and 18000 (Dataset). The examples retain host ports 9090–9093. New local Docker -launch receipts use version 2; existing version-1 receipts retain their original -container ports and pinned image when resumed or recreated. +Fresh nodes listen on container ports 9090 (Web), 16000 (S3), 17000 (Iceberg) and 18000 (Dataset). The examples retain host ports 9090–9093. New local Docker launch receipts use version 2; existing version-1 receipts retain their original container ports and pinned image when resumed or recreated. -For Compose volumes with accepted legacy listeners, set -`CROWDB_S3_CONTAINER_PORT=9091`, `CROWDB_ICEBERG_CONTAINER_PORT=9092` and -`CROWDB_DATASET_CONTAINER_PORT=9093` to match that policy. Public-origin variables -still describe the host-side ports. Retained policy is not rewritten on restart. +For Compose volumes with accepted legacy listeners, set `CROWDB_S3_CONTAINER_PORT=9091`, `CROWDB_ICEBERG_CONTAINER_PORT=9092` and `CROWDB_DATASET_CONTAINER_PORT=9093` to match that policy. Public-origin variables still describe the host-side ports. Retained policy is not rewritten on restart. -`down` removes the Compose containers and network, preserving named volumes. -Set `CROWDB_IMAGE` to select the loaded image and `CROWDB_CLUSTER` to isolate -another project. Change the port variables shown in each file before starting -another project concurrently. Keep accepted public origins unchanged on reuse. -Plain Docker and Compose resources have their own ownership; Pixi clean does -not adopt them. Use Docker/Compose stop/remove/down for those resources and -remove each dedicated bridge after its containers are removed. +`down` removes the Compose containers and network, preserving named volumes. Set `CROWDB_IMAGE` to select the loaded image and `CROWDB_CLUSTER` to isolate another project. Change the port variables shown in each file before starting another project concurrently. Keep accepted public origins unchanged on reuse. Plain Docker and Compose resources have their own ownership; Pixi clean does not adopt them. Use Docker/Compose stop/remove/down for those resources and remove each dedicated bridge after its containers are removed. Read progress and credentials explicitly with standard Docker tools: @@ -349,23 +202,11 @@ pixi run docker logs crowdb-one pixi run docker exec crowdb-one crowdb-monitor credentials show --format env ``` -The credential command prints S3, Iceberg and Dataset client values and their -public origins. Keep that output private. Setup completion is historical; -protocol availability requires current services and storage to be ready. -See [local storage and client details](../single-node-container/README.md). +The credential command prints S3, Iceberg and Dataset client values and their public origins. Keep that output private. Setup completion is historical; protocol availability requires current services and storage to be ready. See [local storage and client details](../single-node-container/README.md). ## Linux host deployment -Prepare a persistent directory owned by UID/GID 10001 and a private password file -(mode 0600), or preinstall `ssh/authorized_keys` in that directory. Passwords are -only used for admission; each node persists its own Ed25519 keys. Production -requires an explicit stable physical host ID, management interface and resources. -CPU policy is explicit: `--cpus` sets a hard ceiling; `--cpu-shares 1024` -sets one CPU of relative weight and allows use of idle host CPU. These options -are mutually exclusive. Memory remains a verified hard limit in both modes. -Container E2E and CLI clusters default to the shares policy and one Tokio -worker per service. The host launcher accepts `--worker-threads 1` for this -explicit configuration, inherited by services as `TOKIO_WORKER_THREADS`. +Prepare a persistent directory owned by UID/GID 10001 and a private password file (mode 0600), or preinstall `ssh/authorized_keys` in that directory. Passwords are only used for admission; each node persists its own Ed25519 keys. Production requires an explicit stable physical host ID, management interface and resources. CPU policy is explicit: `--cpus` sets a hard ceiling; `--cpu-shares 1024` sets one CPU of relative weight and allows use of idle host CPU. These options are mutually exclusive. Memory remains a verified hard limit in both modes. Container E2E and CLI clusters default to the shares policy and one Tokio worker per service. The host launcher accepts `--worker-threads 1` for this explicit configuration, inherited by services as `TOKIO_WORKER_THREADS`. ```bash @@ -376,22 +217,9 @@ pixi run --manifest-path container/oci-image/pixi.toml -e runtime run --runtime --password-file /private/crowdb-password ``` -Host networking uses port 9090 directly; no `-p` mapping is needed. Fixed listener -ports mean one node per host. Use `--seed http://host:9095` when multicast is unavailable. -Explicit `--device /dev/...` permits individual block devices; the host must grant -UID/GID 10001 access. The launcher checks device type and effective read/write -permissions, actual cgroup v2 CPU/memory limits and the host network namespace. -A failed check retains the owned container and its data for diagnosis and explicit -cleanup. A startup error does not authorize removing runtime or storage state. -Physical hardware I/O must also be validated for the selected deployment. - -Use `--node-policy-file /absolute/path/node.toml` to pass the normal node's TOML -startup inputs, including explicit supplied-media setup and public protocol URIs. -The launcher rejects symlinked, oversized, unreadable or malformed files before -runtime creation. It mounts the file read-only; the monitor validates its complete -semantics and seals accepted policy before identity, admission or storage setup. -The file must be readable by container UID/GID 10001 and remain available for -recreation with the same accepted inputs. +Host networking uses port 6135 directly; no `-p` mapping is needed. Fixed listener ports mean one node per host. Use `--seed http://host:26135` when multicast is unavailable. Explicit `--device /dev/...` permits individual block devices; the host must grant UID/GID 10001 access. The launcher checks device type and effective read/write permissions, actual cgroup v2 CPU/memory limits and the host network namespace. A failed check retains the owned container and its data for diagnosis and explicit cleanup. A startup error does not authorize removing runtime or storage state. Physical hardware I/O must also be validated for the selected deployment. + +Use `--node-policy-file /absolute/path/node.toml` to pass the normal node's TOML startup inputs, including explicit supplied-media setup and public protocol URIs. The launcher rejects symlinked, oversized, unreadable or malformed files before runtime creation. It mounts the file read-only; the monitor validates its complete semantics and seals accepted policy before identity, admission or storage setup. The file must be readable by container UID/GID 10001 and remain available for recreation with the same accepted inputs. For containerd, install the separate locked runtime environment: @@ -405,97 +233,31 @@ pixi run --manifest-path container/oci-image/pixi.toml -e runtime run --runtime --password-file /private/crowdb-password ``` -The host administrator starts containerd separately. Initial support is rootful -Linux, cgroup v2 and host networking. The cluster name maps to the containerd -namespace `crowdb-<cluster>`; runtime namespace is not a separate cluster selector. -The default snapshotter is overlayfs; use `--snapshotter native` where overlayfs -is unavailable. No Docker daemon, runtime socket mount or CNI plugin is needed. -The tested tool versions are containerd 2.4.1, nerdctl 2.4.1 and runc 1.4.3. +The host administrator starts containerd separately. Initial support is rootful Linux, cgroup v2 and host networking. The cluster name maps to the containerd namespace `crowdb-<cluster>`; runtime namespace is not a separate cluster selector. The default snapshotter is overlayfs; use `--snapshotter native` where overlayfs is unavailable. No Docker daemon, runtime socket mount or CNI plugin is needed. The tested tool versions are containerd 2.4.1, nerdctl 2.4.1 and runc 1.4.3. ## Persistent storage and candidate admission -Keep `/opt/crowdb/data` on a persistent host directory or named runtime volume. -The OCI image contains the programs; the container writable layer is not the -node's durable storage. Node identity, bootstrap receipts and storage journals -must survive container replacement along with the original media. - -Use repeated `--disk-file /host/path/disk.img` for existing files and `--device -/dev/...` for raw devices. Both map into `/opt/crowdb/data/disks/<basename>` in -Docker and containerd. Basenames and canonical underlying resources must be -unique. A file and its loop alias cannot be supplied as independent disks. -Retain all recorded source mappings on replacement; changed inode/device, -missing source or insufficient UID/GID 10001 access fails without substitution. -Never mount a Docker or containerd socket into the node to manage its storage. -Standard startup prepares the common disk directory for UID/GID 10001. It does -not recursively change the ownership or mode of supplied files or devices. - -- With no supplied inputs, Local node setup defaults to one 16-GiB sparse - managed file and a 1-GiB host reserve. Production automatic storage uses - preallocation and rejects sparse allocation. It is created in the retained - data directory. Additional Create File Disk operations use generated paths in - that same directory. Sparse allocation is an explicit advanced choice; - failed preallocation remains a failure. -- Supplied inputs appear in discovery. Review them during node setup or use - Node → Storage → Join Candidates after Group 0 is ready. Select an existing - bound DiskGroup and media class; a raw device needs its own acknowledgement. - Group 0 owns effective membership. Blank verification reads every usable byte - and never wipes data; nonblank media fail. Supported raw lineage is a whole - device or a whole-file loop mapping. Partitions, partitioned whole devices, - stacked devices, ambiguous virtual lineage and loop slices are rejected. -- Physical size can exceed usable whole-zone geometry. The unused tail remains - untouched. Backing properties distinguish host/resource identity, physical - bytes, usable bytes and tail bytes. File allocated bytes and shared filesystem - free space are separate observations, not a sum of virtual file capacities. -- Close/reopen does not cancel accepted per-disk work. Failed rows keep their - original operation for explicit retry. Offline, then Remove, excludes the disk - while retaining its bytes, ID, geometry and allocation metadata. Rejoin uses - that retained identity Offline; explicit validated Online restores participation. - Removal does not evacuate data or establish recovery/redundancy. -- The accepted deployment policy explicitly selects Local or Production and one - or three initial voters. Local permits simulated placement with zero physical - host failure tolerance. Production requires configured credentials and real - placement; a single production voter cannot satisfy protected ChunkDB - placement. Supplied or diskless storage is an explicit policy choice. Retained - policy, identity and bootstrap inputs cannot be replaced by image defaults. +Keep `/opt/crowdb/data` on a persistent host directory or named runtime volume. The OCI image contains the programs; the container writable layer is not the node's durable storage. Node identity, bootstrap receipts and storage journals must survive container replacement along with the original media. + +Use repeated `--disk-file /host/path/disk.img` for existing files and `--device /dev/...` for raw devices. Both map into `/opt/crowdb/data/disks/<basename>` in Docker and containerd. Basenames and canonical underlying resources must be unique. A file and its loop alias cannot be supplied as independent disks. Retain all recorded source mappings on replacement; changed inode/device, missing source or insufficient UID/GID 10001 access fails without substitution. Never mount a Docker or containerd socket into the node to manage its storage. Standard startup prepares the common disk directory for UID/GID 10001. It does not recursively change the ownership or mode of supplied files or devices. + +- With no supplied inputs, Local node setup defaults to one 16-GiB sparse managed file and a 1-GiB host reserve. Production automatic storage uses preallocation and rejects sparse allocation. It is created in the retained data directory. Additional Create File Disk operations use generated paths in that same directory. Sparse allocation is an explicit advanced choice; failed preallocation remains a failure. +- Supplied inputs appear in discovery. Review them during node setup or use Node → Storage → Join Candidates after Group 0 is ready. Select an existing bound DiskGroup and media class; a raw device needs its own acknowledgement. Group 0 owns effective membership. Blank verification reads every usable byte and never wipes data; nonblank media fail. Supported raw lineage is a whole device or a whole-file loop mapping. Partitions, partitioned whole devices, stacked devices, ambiguous virtual lineage and loop slices are rejected. +- Physical size can exceed usable whole-zone geometry. The unused tail remains untouched. Backing properties distinguish host/resource identity, physical bytes, usable bytes and tail bytes. File allocated bytes and shared filesystem free space are separate observations, not a sum of virtual file capacities. +- Close/reopen does not cancel accepted per-disk work. Failed rows keep their original operation for explicit retry. Offline, then Remove, excludes the disk while retaining its bytes, ID, geometry and allocation metadata. Rejoin uses that retained identity Offline; explicit validated Online restores participation. Removal does not evacuate data or establish recovery/redundancy. +- The accepted deployment policy explicitly selects Local or Production and one or three initial voters. Local permits simulated placement with zero physical host failure tolerance. Production requires configured credentials and real placement; a single production voter cannot satisfy protected ChunkDB placement. Supplied or diskless storage is an explicit policy choice. Retained policy, identity and bootstrap inputs cannot be replaced by image defaults. ## Daemon and node lifecycle -Keep the runtime Pixi workspace/environment in an administrator-owned, versioned -location. Install its locked runtime environment there. Do not clean or update -an active daemon's environment. [service/containerd.service.in](service/containerd.service.in) -uses a concrete `@RUNTIME_PREFIX@` (the runtime environment directory); render it -into `/etc/systemd/system/crowdb-containerd.service`. Install -[service/containerd.toml](service/containerd.toml) as `/etc/crowdb-containerd.toml`. -The service has dedicated socket/content/state paths and disables CRI. Never -replace a Kubernetes-owned containerd configuration with this profile. - -Install/start the unit explicitly with administrator permissions through Pixi. -For upgrades, install a new versioned environment, stop/repoint/restart the unit, -and verify probes and running node identities. The systemd service restarts the -daemon; containerd's restart plugin and `unless-stopped` policy recover exited -nodes. Monitor supervises the child services. Readiness failure alone never -restarts or reinitializes the whole node. - -Use standard runtime tools to inspect, read logs, execute probes, stop and remove -nodes. For containerd select its address and namespace on `nerdctl`; for Docker -use its selected context. Removing a stopped container does not delete its -external data. Host networking uses one node per host with the fixed listener -ports; RDMA uses this host-network path. +Keep the runtime Pixi workspace/environment in an administrator-owned, versioned location. Install its locked runtime environment there. Do not clean or update an active daemon's environment. [service/containerd.service.in](service/containerd.service.in) uses a concrete `@RUNTIME_PREFIX@` (the runtime environment directory); render it into `/etc/systemd/system/crowdb-containerd.service`. Install [service/containerd.toml](service/containerd.toml) as `/etc/crowdb-containerd.toml`. The service has dedicated socket/content/state paths and disables CRI. Never replace a Kubernetes-owned containerd configuration with this profile. + +Install/start the unit explicitly with administrator permissions through Pixi. For upgrades, install a new versioned environment, stop/repoint/restart the unit, and verify probes and running node identities. The systemd service restarts the daemon; containerd's restart plugin and `unless-stopped` policy recover exited nodes. Monitor supervises the child services. Readiness failure alone never restarts or reinitializes the whole node. + +Use standard runtime tools to inspect, read logs, execute probes, stop and remove nodes. For containerd select its address and namespace on `nerdctl`; for Docker use its selected context. Removing a stopped container does not delete its external data. Host networking uses one node per host with the fixed listener ports; RDMA uses this host-network path. ## CI and acceptance -The manual release workflow constructs one OCI artifact without Docker installed, -then downloads it into independent Docker and Docker-free containerd jobs. Only -matching successful image digests allow publication. The protected publish job -copies the graph with digest preservation, verifies the remote digest, signs it, -and attaches SPDX SBOM and SLSA provenance before updating `latest`. The canonical repository is `crowdb/crowdb-node`; the -`crowdb/crowdb-iceberg`, `crowdb/crowdb-s3` and `crowdb/crowdb-dataset` repositories -are single-node usage aliases of that image. Image names do not select a different lifecycle or UI permission policy. -The Dataset alias uses the same image. Dataset access runs through the -`server access dataset serve` entry point with the versioned REST API and -Dataset-specific chunk storage. No second Docker image is published. -Preview has the same construction/runtime gates without publication authority. -Workflow execution and live registry signing require a release dispatch. +The manual release workflow constructs one OCI artifact without Docker installed, then downloads it into independent Docker and Docker-free containerd jobs. Only matching successful image digests allow publication. The protected publish job copies the graph with digest preservation, verifies the remote digest, signs it, and attaches SPDX SBOM and SLSA provenance before updating `latest`. The canonical repository is `crowdb/crowdb-node`; the `crowdb/crowdb-iceberg`, `crowdb/crowdb-s3` and `crowdb/crowdb-dataset` repositories are single-node usage aliases of that image. Image names do not select a different lifecycle or UI permission policy. The Dataset alias uses the same image. Dataset access runs through the `server access dataset serve` entry point with the versioned REST API and Dataset-specific chunk storage. No second Docker image is published. Preview has the same construction/runtime gates without publication authority. Workflow execution and live registry signing require a release dispatch. ```bash pixi run test-oci-image @@ -508,22 +270,8 @@ pixi run sudo "$(command -v pixi)" run --as-is test-container-e2e --profile cont --archive "$PWD/target/crowdb.oci.tar" ``` -Developer-only `tests/build-sandbox.py` and `tests/runtime-sandbox.py` use system -Docker to provide disposable privileged Linux hosts. Inside those hosts the -builder/runtime tests have no Docker CLI/socket. They verify the isolated process -contract, not a bare-metal installation. The native host gate exercises production metadata lifecycle separately from -Local full-protocol readiness and retained recreation. Production metadata alone -is not whole-cluster Ready. Enforced resources, host networking and retained -identity/key checks apply to both policies. Three isolated Linux namespace hosts additionally cover -discovery, SSH admission, Group-0 quorum and member recovery. This simulation is -not a physical multi-host network qualification. Existing Docker tests cover -application writes and crash recovery. Local registry transfer retained the exact -OCI digest; Syft generated an SPDX 2.3 SBOM from the tested archive. macOS VM execution, ARM64, Kubernetes orchestration and broader layer -E2E migration remain separate follow-up work. - -Build tools: BuildKit 0.34.0, RootlessKit 3.2.0, runc 1.4.3, Skopeo 1.24.1. -Publishing tools: Syft 1.54.1 and Cosign 3.1.3. Conda-forge supplies runc/Skopeo; -checked-in Pixi source recipes supply missing tools from SHA256-pinned upstream -releases. All six upstream projects use Apache-2.0 licensing; recipes retain -upstream sources/checksums and license metadata. The dedicated Linux workspace -leaves the repository's macOS dependency solve unchanged. +Developer-only `tests/build-sandbox.py` and `tests/runtime-sandbox.py` use system Docker to provide disposable privileged Linux hosts. Inside those hosts the builder/runtime tests have no Docker CLI/socket. They verify the isolated process contract, not a bare-metal installation. The native host gate exercises production metadata lifecycle separately from Local full-protocol readiness and retained recreation. Production metadata alone is not whole-cluster Ready. Enforced resources, host networking and retained identity/key checks apply to both policies. Three isolated Linux namespace hosts additionally cover discovery, SSH admission, Group-0 quorum and member recovery. This simulation is not a physical multi-host network qualification. Existing Docker tests cover application writes and crash recovery. Local registry transfer retained the exact OCI digest; Syft generated an SPDX 2.3 SBOM from the tested archive. macOS VM execution, ARM64, Kubernetes orchestration and broader layer E2E migration remain separate follow-up work. + +Build tools: BuildKit 0.34.0, RootlessKit 3.2.0, runc 1.4.3, Skopeo 1.24.1. Publishing tools: Syft 1.54.1 and Cosign 3.1.3. Conda-forge supplies runc/Skopeo; checked-in Pixi source recipes supply missing tools from SHA256-pinned upstream releases. All six upstream projects use Apache-2.0 licensing; recipes retain upstream sources/checksums and license metadata. The dedicated Linux workspace leaves the repository's macOS dependency solve unchanged. + +Default listener ports are defined in [protocol ports.rs](../../lib/crowdb-protocol/src/port/ports.rs). Generated packaging defaults are maintained with [tools/ports](../../tools/ports/README.md). diff --git a/container/oci-image/collect-libs.sh b/container/oci-image/collect-libs.sh index 5b454781e..76dec9e47 100644 --- a/container/oci-image/collect-libs.sh +++ b/container/oci-image/collect-libs.sh @@ -24,7 +24,7 @@ if [[ -d "$provider_root" && -d "$provider_config" ]]; then fi for binary in \ - crowdb-monitor crowdb-kv-server crowdb-diskdb crowdb-diskio \ + crowdb-monitor crowdb-paxos-kv-server crowdb-diskdb crowdb-diskio \ crowdb-chunkdb crowdb-chunk-kv-server crowdb-access-server \ crowdb-web; do if [[ "$binary" == crowdb-diskio ]]; then @@ -66,7 +66,7 @@ for binary in "$output"/bin/* "$output"/lib/libibverbs/*.so; do done < "$output/dependencies.txt" patchelf --set-rpath '/opt/crowdb/lib' "$binary" done -if [[ ! -f "$output/lib/libcrowdb_kv_client.so" ]]; then +if [[ ! -f "$output/lib/libcrowdb_paxos_kv_client.so" ]]; then echo 'DiskIO FFI library was not collected' >&2 exit 1 fi diff --git a/container/oci-image/compose-three.yaml b/container/oci-image/compose-three.yaml index 01af56fff..aadaf6e0f 100644 --- a/container/oci-image/compose-three.yaml +++ b/container/oci-image/compose-three.yaml @@ -1,3 +1,4 @@ +# Generated by tools/ports/generate.py from ports.rs; edit the .in template. # Copyright 2026-present Gian <crow.db@outlook.com> # Licensed under the Apache License, Version 2.0. name: crowdb-${CROWDB_CLUSTER:-local-three} @@ -12,14 +13,14 @@ services: environment: CROWDB_CLUSTER: ${CROWDB_CLUSTER:-local-three} CROWDB_INITIAL_VOTERS: "3" - CROWDB_S3_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE1_S3_PORT:-9091} - CROWDB_ICEBERG_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE1_ICEBERG_PORT:-9092} - CROWDB_DATASET_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE1_DATASET_PORT:-9093} + CROWDB_S3_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE1_S3_PORT:-7135} + CROWDB_ICEBERG_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE1_ICEBERG_PORT:-8135} + CROWDB_DATASET_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE1_DATASET_PORT:-9135} ports: - - "127.0.0.1:${CROWDB_NODE1_UI_PORT:-9090}:9090" - - "127.0.0.1:${CROWDB_NODE1_S3_PORT:-9091}:${CROWDB_S3_CONTAINER_PORT:-16000}" - - "127.0.0.1:${CROWDB_NODE1_ICEBERG_PORT:-9092}:${CROWDB_ICEBERG_CONTAINER_PORT:-17000}" - - "127.0.0.1:${CROWDB_NODE1_DATASET_PORT:-9093}:${CROWDB_DATASET_CONTAINER_PORT:-18000}" + - "127.0.0.1:${CROWDB_NODE1_UI_PORT:-6135}:6135" + - "127.0.0.1:${CROWDB_NODE1_S3_PORT:-7135}:${CROWDB_S3_CONTAINER_PORT:-7135}" + - "127.0.0.1:${CROWDB_NODE1_ICEBERG_PORT:-8135}:${CROWDB_ICEBERG_CONTAINER_PORT:-8135}" + - "127.0.0.1:${CROWDB_NODE1_DATASET_PORT:-9135}:${CROWDB_DATASET_CONTAINER_PORT:-9135}" volumes: - node1-data:/opt/crowdb/data node2: @@ -27,14 +28,14 @@ services: environment: CROWDB_CLUSTER: ${CROWDB_CLUSTER:-local-three} CROWDB_INITIAL_VOTERS: "3" - CROWDB_S3_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE2_S3_PORT:-9101} - CROWDB_ICEBERG_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE2_ICEBERG_PORT:-9102} - CROWDB_DATASET_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE2_DATASET_PORT:-9103} + CROWDB_S3_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE2_S3_PORT:-7136} + CROWDB_ICEBERG_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE2_ICEBERG_PORT:-8136} + CROWDB_DATASET_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE2_DATASET_PORT:-9136} ports: - - "127.0.0.1:${CROWDB_NODE2_UI_PORT:-9100}:9090" - - "127.0.0.1:${CROWDB_NODE2_S3_PORT:-9101}:${CROWDB_S3_CONTAINER_PORT:-16000}" - - "127.0.0.1:${CROWDB_NODE2_ICEBERG_PORT:-9102}:${CROWDB_ICEBERG_CONTAINER_PORT:-17000}" - - "127.0.0.1:${CROWDB_NODE2_DATASET_PORT:-9103}:${CROWDB_DATASET_CONTAINER_PORT:-18000}" + - "127.0.0.1:${CROWDB_NODE2_UI_PORT:-6136}:6135" + - "127.0.0.1:${CROWDB_NODE2_S3_PORT:-7136}:${CROWDB_S3_CONTAINER_PORT:-7135}" + - "127.0.0.1:${CROWDB_NODE2_ICEBERG_PORT:-8136}:${CROWDB_ICEBERG_CONTAINER_PORT:-8135}" + - "127.0.0.1:${CROWDB_NODE2_DATASET_PORT:-9136}:${CROWDB_DATASET_CONTAINER_PORT:-9135}" volumes: - node2-data:/opt/crowdb/data node3: @@ -42,14 +43,14 @@ services: environment: CROWDB_CLUSTER: ${CROWDB_CLUSTER:-local-three} CROWDB_INITIAL_VOTERS: "3" - CROWDB_S3_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE3_S3_PORT:-9111} - CROWDB_ICEBERG_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE3_ICEBERG_PORT:-9112} - CROWDB_DATASET_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE3_DATASET_PORT:-9113} + CROWDB_S3_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE3_S3_PORT:-7137} + CROWDB_ICEBERG_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE3_ICEBERG_PORT:-8137} + CROWDB_DATASET_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE3_DATASET_PORT:-9137} ports: - - "127.0.0.1:${CROWDB_NODE3_UI_PORT:-9110}:9090" - - "127.0.0.1:${CROWDB_NODE3_S3_PORT:-9111}:${CROWDB_S3_CONTAINER_PORT:-16000}" - - "127.0.0.1:${CROWDB_NODE3_ICEBERG_PORT:-9112}:${CROWDB_ICEBERG_CONTAINER_PORT:-17000}" - - "127.0.0.1:${CROWDB_NODE3_DATASET_PORT:-9113}:${CROWDB_DATASET_CONTAINER_PORT:-18000}" + - "127.0.0.1:${CROWDB_NODE3_UI_PORT:-6137}:6135" + - "127.0.0.1:${CROWDB_NODE3_S3_PORT:-7137}:${CROWDB_S3_CONTAINER_PORT:-7135}" + - "127.0.0.1:${CROWDB_NODE3_ICEBERG_PORT:-8137}:${CROWDB_ICEBERG_CONTAINER_PORT:-8135}" + - "127.0.0.1:${CROWDB_NODE3_DATASET_PORT:-9137}:${CROWDB_DATASET_CONTAINER_PORT:-9135}" volumes: - node3-data:/opt/crowdb/data volumes: diff --git a/container/oci-image/compose-three.yaml.ports.in b/container/oci-image/compose-three.yaml.ports.in new file mode 100644 index 000000000..d8027c831 --- /dev/null +++ b/container/oci-image/compose-three.yaml.ports.in @@ -0,0 +1,58 @@ +# Copyright 2026-present Gian <crow.db@outlook.com> +# Licensed under the Apache License, Version 2.0. +name: crowdb-${CROWDB_CLUSTER:-local-three} +x-node: &node + image: ${CROWDB_IMAGE:-crowdb-node:dev} + pull_policy: never + cpus: 1 + mem_limit: 1536m +services: + node1: + <<: *node + environment: + CROWDB_CLUSTER: ${CROWDB_CLUSTER:-local-three} + CROWDB_INITIAL_VOTERS: "3" + CROWDB_S3_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE1_S3_PORT:-@ACCESS_SERVER_HTTP_BASE@} + CROWDB_ICEBERG_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE1_ICEBERG_PORT:-@ACCESS_SERVER_ICEBERG_HTTP_BASE@} + CROWDB_DATASET_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE1_DATASET_PORT:-@ACCESS_SERVER_DATASET_HTTP_BASE@} + ports: + - "127.0.0.1:${CROWDB_NODE1_UI_PORT:-@WEB_BASE@}:@WEB_BASE@" + - "127.0.0.1:${CROWDB_NODE1_S3_PORT:-@ACCESS_SERVER_HTTP_BASE@}:${CROWDB_S3_CONTAINER_PORT:-@ACCESS_SERVER_HTTP_BASE@}" + - "127.0.0.1:${CROWDB_NODE1_ICEBERG_PORT:-@ACCESS_SERVER_ICEBERG_HTTP_BASE@}:${CROWDB_ICEBERG_CONTAINER_PORT:-@ACCESS_SERVER_ICEBERG_HTTP_BASE@}" + - "127.0.0.1:${CROWDB_NODE1_DATASET_PORT:-@ACCESS_SERVER_DATASET_HTTP_BASE@}:${CROWDB_DATASET_CONTAINER_PORT:-@ACCESS_SERVER_DATASET_HTTP_BASE@}" + volumes: + - node1-data:/opt/crowdb/data + node2: + <<: *node + environment: + CROWDB_CLUSTER: ${CROWDB_CLUSTER:-local-three} + CROWDB_INITIAL_VOTERS: "3" + CROWDB_S3_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE2_S3_PORT:-@ACCESS_SERVER_HTTP_BASE+1@} + CROWDB_ICEBERG_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE2_ICEBERG_PORT:-@ACCESS_SERVER_ICEBERG_HTTP_BASE+1@} + CROWDB_DATASET_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE2_DATASET_PORT:-@ACCESS_SERVER_DATASET_HTTP_BASE+1@} + ports: + - "127.0.0.1:${CROWDB_NODE2_UI_PORT:-@WEB_BASE+1@}:@WEB_BASE@" + - "127.0.0.1:${CROWDB_NODE2_S3_PORT:-@ACCESS_SERVER_HTTP_BASE+1@}:${CROWDB_S3_CONTAINER_PORT:-@ACCESS_SERVER_HTTP_BASE@}" + - "127.0.0.1:${CROWDB_NODE2_ICEBERG_PORT:-@ACCESS_SERVER_ICEBERG_HTTP_BASE+1@}:${CROWDB_ICEBERG_CONTAINER_PORT:-@ACCESS_SERVER_ICEBERG_HTTP_BASE@}" + - "127.0.0.1:${CROWDB_NODE2_DATASET_PORT:-@ACCESS_SERVER_DATASET_HTTP_BASE+1@}:${CROWDB_DATASET_CONTAINER_PORT:-@ACCESS_SERVER_DATASET_HTTP_BASE@}" + volumes: + - node2-data:/opt/crowdb/data + node3: + <<: *node + environment: + CROWDB_CLUSTER: ${CROWDB_CLUSTER:-local-three} + CROWDB_INITIAL_VOTERS: "3" + CROWDB_S3_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE3_S3_PORT:-@ACCESS_SERVER_HTTP_BASE+2@} + CROWDB_ICEBERG_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE3_ICEBERG_PORT:-@ACCESS_SERVER_ICEBERG_HTTP_BASE+2@} + CROWDB_DATASET_PUBLIC_URI: http://127.0.0.1:${CROWDB_NODE3_DATASET_PORT:-@ACCESS_SERVER_DATASET_HTTP_BASE+2@} + ports: + - "127.0.0.1:${CROWDB_NODE3_UI_PORT:-@WEB_BASE+2@}:@WEB_BASE@" + - "127.0.0.1:${CROWDB_NODE3_S3_PORT:-@ACCESS_SERVER_HTTP_BASE+2@}:${CROWDB_S3_CONTAINER_PORT:-@ACCESS_SERVER_HTTP_BASE@}" + - "127.0.0.1:${CROWDB_NODE3_ICEBERG_PORT:-@ACCESS_SERVER_ICEBERG_HTTP_BASE+2@}:${CROWDB_ICEBERG_CONTAINER_PORT:-@ACCESS_SERVER_ICEBERG_HTTP_BASE@}" + - "127.0.0.1:${CROWDB_NODE3_DATASET_PORT:-@ACCESS_SERVER_DATASET_HTTP_BASE+2@}:${CROWDB_DATASET_CONTAINER_PORT:-@ACCESS_SERVER_DATASET_HTTP_BASE@}" + volumes: + - node3-data:/opt/crowdb/data +volumes: + node1-data: + node2-data: + node3-data: diff --git a/container/oci-image/compose.yaml b/container/oci-image/compose.yaml index 96e5e608f..1b3848577 100644 --- a/container/oci-image/compose.yaml +++ b/container/oci-image/compose.yaml @@ -1,3 +1,4 @@ +# Generated by tools/ports/generate.py from ports.rs; edit the .in template. # Copyright 2026-present Gian <crow.db@outlook.com> # Licensed under the Apache License, Version 2.0. name: crowdb-${CROWDB_CLUSTER:-local} @@ -10,14 +11,14 @@ services: environment: CROWDB_CLUSTER: ${CROWDB_CLUSTER:-local} CROWDB_INITIAL_VOTERS: "1" - CROWDB_S3_PUBLIC_URI: http://127.0.0.1:${CROWDB_S3_PORT:-9091} - CROWDB_ICEBERG_PUBLIC_URI: http://127.0.0.1:${CROWDB_ICEBERG_PORT:-9092} - CROWDB_DATASET_PUBLIC_URI: http://127.0.0.1:${CROWDB_DATASET_PORT:-9093} + CROWDB_S3_PUBLIC_URI: http://127.0.0.1:${CROWDB_S3_PORT:-7135} + CROWDB_ICEBERG_PUBLIC_URI: http://127.0.0.1:${CROWDB_ICEBERG_PORT:-8135} + CROWDB_DATASET_PUBLIC_URI: http://127.0.0.1:${CROWDB_DATASET_PORT:-9135} ports: - - "127.0.0.1:${CROWDB_UI_PORT:-9090}:9090" - - "127.0.0.1:${CROWDB_S3_PORT:-9091}:${CROWDB_S3_CONTAINER_PORT:-16000}" - - "127.0.0.1:${CROWDB_ICEBERG_PORT:-9092}:${CROWDB_ICEBERG_CONTAINER_PORT:-17000}" - - "127.0.0.1:${CROWDB_DATASET_PORT:-9093}:${CROWDB_DATASET_CONTAINER_PORT:-18000}" + - "127.0.0.1:${CROWDB_UI_PORT:-6135}:6135" + - "127.0.0.1:${CROWDB_S3_PORT:-7135}:${CROWDB_S3_CONTAINER_PORT:-7135}" + - "127.0.0.1:${CROWDB_ICEBERG_PORT:-8135}:${CROWDB_ICEBERG_CONTAINER_PORT:-8135}" + - "127.0.0.1:${CROWDB_DATASET_PORT:-9135}:${CROWDB_DATASET_CONTAINER_PORT:-9135}" volumes: - node-data:/opt/crowdb/data volumes: diff --git a/container/oci-image/compose.yaml.ports.in b/container/oci-image/compose.yaml.ports.in new file mode 100644 index 000000000..ecb30283f --- /dev/null +++ b/container/oci-image/compose.yaml.ports.in @@ -0,0 +1,24 @@ +# Copyright 2026-present Gian <crow.db@outlook.com> +# Licensed under the Apache License, Version 2.0. +name: crowdb-${CROWDB_CLUSTER:-local} +services: + node: + image: ${CROWDB_IMAGE:-crowdb-node:dev} + pull_policy: never + cpus: 1 + mem_limit: 1536m + environment: + CROWDB_CLUSTER: ${CROWDB_CLUSTER:-local} + CROWDB_INITIAL_VOTERS: "1" + CROWDB_S3_PUBLIC_URI: http://127.0.0.1:${CROWDB_S3_PORT:-@ACCESS_SERVER_HTTP_BASE@} + CROWDB_ICEBERG_PUBLIC_URI: http://127.0.0.1:${CROWDB_ICEBERG_PORT:-@ACCESS_SERVER_ICEBERG_HTTP_BASE@} + CROWDB_DATASET_PUBLIC_URI: http://127.0.0.1:${CROWDB_DATASET_PORT:-@ACCESS_SERVER_DATASET_HTTP_BASE@} + ports: + - "127.0.0.1:${CROWDB_UI_PORT:-@WEB_BASE@}:@WEB_BASE@" + - "127.0.0.1:${CROWDB_S3_PORT:-@ACCESS_SERVER_HTTP_BASE@}:${CROWDB_S3_CONTAINER_PORT:-@ACCESS_SERVER_HTTP_BASE@}" + - "127.0.0.1:${CROWDB_ICEBERG_PORT:-@ACCESS_SERVER_ICEBERG_HTTP_BASE@}:${CROWDB_ICEBERG_CONTAINER_PORT:-@ACCESS_SERVER_ICEBERG_HTTP_BASE@}" + - "127.0.0.1:${CROWDB_DATASET_PORT:-@ACCESS_SERVER_DATASET_HTTP_BASE@}:${CROWDB_DATASET_CONTAINER_PORT:-@ACCESS_SERVER_DATASET_HTTP_BASE@}" + volumes: + - node-data:/opt/crowdb/data +volumes: + node-data: diff --git a/container/oci-image/docker_local_launch.py b/container/oci-image/docker_local_launch.py index 99222ace3..ee0feb2f7 100644 --- a/container/oci-image/docker_local_launch.py +++ b/container/oci-image/docker_local_launch.py @@ -6,15 +6,23 @@ import math import uuid +from ports import WEB_BASE, ACCESS_SERVER_HTTP_BASE, ACCESS_SERVER_ICEBERG_HTTP_BASE, ACCESS_SERVER_DATASET_HTTP_BASE from docker_local_state import CLUSTER, OWNER, check_daemon, load, require_owned, seal CONTAINER_PORTS = (9090, 16000, 17000, 18000) def container_ports(record): - return (9090, 9091, 9092, 9093) if record["version"] == 1 else CONTAINER_PORTS + if record["version"] == 1: + return (9090, 9091, 9092, 9093) + if record["version"] == 2: + return CONTAINER_PORTS + return tuple(port for port, _ in PUBLIC_PORTS) +PUBLIC_PORTS = ((WEB_BASE, None), (ACCESS_SERVER_HTTP_BASE, "S3"), + (ACCESS_SERVER_ICEBERG_HTTP_BASE, "ICEBERG"), (ACCESS_SERVER_DATASET_HTTP_BASE, "DATASET")) + def arguments(record): return ["--label", f"{CLUSTER}={record['cluster']}", "--label", f"{OWNER}={record['owner']}"] @@ -49,11 +57,11 @@ def accepted(docker, args): raise ValueError("local node image must be Linux amd64") if metadata.get("Config", {}).get("Labels", {}).get("org.opencontainers.image.source") != "https://github.com/buzzcrow/crowdb": raise ValueError("image is not a CROWDB node image") - record = {"version": 2, "cluster": args.cluster, "owner": str(uuid.uuid4()), "runtime": docker.identity(), - "nodes": args.nodes if args.nodes is not None else 1, "port_base": args.port_base if args.port_base is not None else 9090, "image": image, + record = {"version": 3, "cluster": args.cluster, "owner": str(uuid.uuid4()), "runtime": docker.identity(), + "nodes": args.nodes if args.nodes is not None else 1, "port_base": args.port_base if args.port_base is not None else WEB_BASE, "image": image, "image_id": metadata["Id"], "cpus": args.cpus if args.cpus is not None else 1, "memory_mib": args.memory_mib if args.memory_mib is not None else 1536} - if record["nodes"] not in (1, 3) or not 1 <= record["port_base"] <= 65512: + if record["nodes"] not in (1, 3) or not 1 <= record["port_base"] <= 65535 - (ACCESS_SERVER_DATASET_HTTP_BASE - WEB_BASE) - 2: raise ValueError("local cluster requires one or three nodes and a valid port range") if not math.isfinite(record["cpus"]) or record["cpus"] <= 0 or record["memory_mib"] < 512: raise ValueError("CPU must be positive and memory at least 512 MiB") @@ -64,12 +72,13 @@ def accepted(docker, args): def verify_container(item, record, index, volume, network): require_owned(item, record, "container") host = item["HostConfig"] - base = record["port_base"] + (index - 1) * 10 - ports = {f"{port}/tcp": [{"HostIp": "127.0.0.1", "HostPort": str(base+offset)}] + base = record["port_base"] + (index - 1) * (10 if record["version"] < 3 else 1) + ports = {f"{port}/tcp": [{"HostIp": "127.0.0.1", "HostPort": str(base + (offset if record["version"] < 3 else port - WEB_BASE))}] for offset, port in enumerate(container_ports(record))} expected_environment = {"CROWDB_CLUSTER": record["cluster"], "CROWDB_INITIAL_VOTERS": str(record["nodes"])} for offset, protocol in enumerate((None, "S3", "ICEBERG", "DATASET")): if protocol: + offset = offset if record["version"] < 3 else container_ports(record)[offset] - WEB_BASE expected_environment[f"CROWDB_{protocol}_PUBLIC_URI"] = f"http://127.0.0.1:{base+offset}" environment = dict(value.split("=", 1) for value in item["Config"].get("Env", []) if "=" in value) mounts = [mount for mount in item["Mounts"] if mount["Destination"] == "/opt/crowdb/data"] @@ -102,13 +111,15 @@ def run(docker, record): check_daemon(docker, record) docker.call("start", item["Id"]) continue - base = record["port_base"] + (index - 1) * 10 + base = record["port_base"] + (index - 1) * (10 if record["version"] < 3 else 1) command = ["run", "-d", "--pull", "never", "--name", name, *arguments(record), "--network", network, "--cpus", record["cpus"], "--memory", f"{record['memory_mib']}m", "--mount", f"type=volume,source={volume},target=/opt/crowdb/data", "-e", "CROWDB_CLUSTER=" + record["cluster"], "-e", f"CROWDB_INITIAL_VOTERS={record['nodes']}"] for offset, protocol in enumerate((None, "S3", "ICEBERG", "DATASET")): - command += ["-p", f"127.0.0.1:{base+offset}:{container_ports(record)[offset]}"] + internal = container_ports(record)[offset] + offset = offset if record["version"] < 3 else internal - WEB_BASE + command += ["-p", f"127.0.0.1:{base+offset}:{internal}"] if protocol: command += ["-e", f"CROWDB_{protocol}_PUBLIC_URI=http://127.0.0.1:{base+offset}"] check_daemon(docker, record) diff --git a/container/oci-image/docker_local_state.py b/container/oci-image/docker_local_state.py index 960492c6f..171a8fcc7 100644 --- a/container/oci-image/docker_local_state.py +++ b/container/oci-image/docker_local_state.py @@ -11,6 +11,8 @@ import stat import uuid +from ports import WEB_BASE, ACCESS_SERVER_DATASET_HTTP_BASE + CLUSTER = "org.crowdb.cluster" OWNER = "org.crowdb.cluster.owner" ROOT = Path(__file__).resolve().parents[2] @@ -71,7 +73,7 @@ def load(cluster, directory=RECORDS): if not stat.S_ISREG(metadata.st_mode) or metadata.st_mode & 0o077 or metadata.st_size > 1024 * 1024: raise ValueError("cluster receipt must be a bounded private regular file") record = json.loads(path.read_text()) - if record.get("version") not in (1, 2) or record.get("cluster") != cluster: + if record.get("version") not in (1, 2, 3) or record.get("cluster") != cluster: raise ValueError("invalid cluster receipt") required = {"version", "cluster", "owner", "runtime", "nodes", "port_base", "image", "image_id", "cpus", "memory_mib"} if set(record) != required or not isinstance(record["runtime"], dict): @@ -79,10 +81,11 @@ def load(cluster, directory=RECORDS): owner = uuid.UUID(record["owner"]) if owner.int == 0 or str(owner) != record["owner"]: raise ValueError("invalid cluster receipt owner") + spread = (ACCESS_SERVER_DATASET_HTTP_BASE - WEB_BASE) + 2 if record["version"] == 3 else 23 if (set(record["runtime"]) != {"daemon", "context", "endpoint"} or not all(isinstance(value, str) and value for value in record["runtime"].values()) or record["nodes"] not in (1, 3) or not isinstance(record["port_base"], int) - or not 1 <= record["port_base"] <= 65512 + or not 1 <= record["port_base"] <= 65535 - spread or not isinstance(record["cpus"], (float, int)) or not math.isfinite(record["cpus"]) or record["cpus"] <= 0 or not isinstance(record["memory_mib"], int) or record["memory_mib"] < 512 or not all(isinstance(record[key], str) and record[key] for key in ("image", "image_id"))): diff --git a/container/oci-image/host_inputs.py b/container/oci-image/host_inputs.py index 117ce7bb4..1b020757a 100644 --- a/container/oci-image/host_inputs.py +++ b/container/oci-image/host_inputs.py @@ -2,6 +2,8 @@ # Licensed under the Apache License, Version 2.0. """Reuse the ordinary host launch validator before accepting runtime ownership.""" +from ports import WEB_BASE + import hashlib import ipaddress import json @@ -81,7 +83,7 @@ def management(options): if str(address) not in {value['local'] for interface in interfaces for value in interface.get('addr_info', [])}: raise ValueError('management address is not assigned to the selected interface') - return 'http://' + (f'[{address}]' if address.version == 6 else str(address)) + ':9090' + return 'http://' + (f'[{address}]' if address.version == 6 else str(address)) + f':{WEB_BASE}' def validate(record, running): diff --git a/container/oci-image/node-profile.toml b/container/oci-image/node-profile.toml index 76ac257b4..ee925762e 100644 --- a/container/oci-image/node-profile.toml +++ b/container/oci-image/node-profile.toml @@ -1,3 +1,4 @@ +# Generated by tools/ports/generate.py from ports.rs; edit the .in template. # Copyright 2026-present Gian <crow.db@outlook.com> # Licensed under the Apache License, Version 2.0. # Programs and supervision defaults only. Accepted node policy and committed @@ -26,31 +27,31 @@ mirror_warnings_to_stderr = true [[public_endpoints]] id = "s3" bind = "0.0.0.0" -port = 9091 +port = 7135 [[public_endpoints]] id = "iceberg" bind = "0.0.0.0" -port = 9092 +port = 8135 [[public_endpoints]] id = "web" bind = "0.0.0.0" -port = 9090 +port = 6135 [[public_endpoints]] id = "dataset" bind = "0.0.0.0" -port = 9093 +port = 9135 [[services]] id = "kv" -program = "/opt/crowdb/bin/crowdb-kv-server" +program = "/opt/crowdb/bin/crowdb-paxos-kv-server" args = [] dependencies = [] [services.probe] kind = "http" -target = "http://127.0.0.1:10000/health" +target = "http://127.0.0.1:10135/health" timeout_ms = 1000 failure_threshold = 5 [services.restart] @@ -65,7 +66,7 @@ args = [] dependencies = ["kv"] [services.probe] kind = "http" -target = "http://127.0.0.1:7101/ready" +target = "http://127.0.0.1:13135/ready" timeout_ms = 1000 failure_threshold = 5 [services.restart] @@ -80,7 +81,7 @@ args = [] dependencies = ["kv", "diskdb"] [services.probe] kind = "rpc-ping" -target = "127.0.0.1:7200" +target = "127.0.0.1:18135" timeout_ms = 1000 failure_threshold = 5 [services.restart] @@ -95,7 +96,7 @@ args = [] dependencies = ["kv", "diskdb", "diskio"] [services.probe] kind = "http" -target = "http://127.0.0.1:7300/ready" +target = "http://127.0.0.1:16135/ready" timeout_ms = 1000 failure_threshold = 5 [services.restart] @@ -110,7 +111,7 @@ args = [] dependencies = ["kv", "chunkdb", "diskio"] [services.probe] kind = "http" -target = "http://127.0.0.1:7400/ready" +target = "http://127.0.0.1:20135/ready" timeout_ms = 1000 failure_threshold = 5 [services.restart] @@ -125,7 +126,7 @@ args = [] dependencies = ["kv", "chunk-kv", "chunkdb", "diskio"] [services.probe] kind = "http" -target = "http://127.0.0.1:9094/_crowdb/health/ready" +target = "http://127.0.0.1:25135/_crowdb/health/ready" timeout_ms = 1000 failure_threshold = 5 [services.restart] @@ -140,7 +141,7 @@ args = [] dependencies = [] [services.probe] kind = "http" -target = "http://127.0.0.1:9090/healthz" +target = "http://127.0.0.1:6135/healthz" timeout_ms = 1000 failure_threshold = 5 [services.restart] diff --git a/container/oci-image/node-profile.toml.ports.in b/container/oci-image/node-profile.toml.ports.in new file mode 100644 index 000000000..dbe23e92b --- /dev/null +++ b/container/oci-image/node-profile.toml.ports.in @@ -0,0 +1,149 @@ +# Copyright 2026-present Gian <crow.db@outlook.com> +# Licensed under the Apache License, Version 2.0. +# Programs and supervision defaults only. Accepted node policy and committed +# service intents supply membership, media, configuration and actual listeners. + +version = 1 +name = "node-container" +display_name = "CROWDB Node" +placement_mode = "node-policy" +s3_tenant = "default" +iceberg_catalog = "default" + +[paths] +install_root = "/opt/crowdb" +bin_root = "/opt/crowdb/bin" +template_root = "/opt/crowdb/etc/templates" +data_root = "/opt/crowdb/data" +run_root = "/opt/crowdb/run" +log_root = "/opt/crowdb/data/log" + +[logs] +max_file_bytes = 31457280 +max_files = 5 +mirror_warnings_to_stderr = true + +[[public_endpoints]] +id = "s3" +bind = "0.0.0.0" +port = @ACCESS_SERVER_HTTP_BASE@ + +[[public_endpoints]] +id = "iceberg" +bind = "0.0.0.0" +port = @ACCESS_SERVER_ICEBERG_HTTP_BASE@ + +[[public_endpoints]] +id = "web" +bind = "0.0.0.0" +port = @WEB_BASE@ + +[[public_endpoints]] +id = "dataset" +bind = "0.0.0.0" +port = @ACCESS_SERVER_DATASET_HTTP_BASE@ + +[[services]] +id = "kv" +program = "/opt/crowdb/bin/crowdb-paxos-kv-server" +args = [] +dependencies = [] +[services.probe] +kind = "http" +target = "http://127.0.0.1:@PAXOS_KV_SERVER_MGMT_BASE@/health" +timeout_ms = 1000 +failure_threshold = 5 +[services.restart] +max_attempts = 5 +backoff_base_ms = 250 +backoff_max_ms = 5000 + +[[services]] +id = "diskdb" +program = "/opt/crowdb/bin/crowdb-diskdb" +args = [] +dependencies = ["kv"] +[services.probe] +kind = "http" +target = "http://127.0.0.1:@DISKDB_HTTP_BASE@/ready" +timeout_ms = 1000 +failure_threshold = 5 +[services.restart] +max_attempts = 5 +backoff_base_ms = 250 +backoff_max_ms = 5000 + +[[services]] +id = "diskio" +program = "/opt/crowdb/bin/crowdb-diskio" +args = [] +dependencies = ["kv", "diskdb"] +[services.probe] +kind = "rpc-ping" +target = "127.0.0.1:@DISKIO_RPC_BASE@" +timeout_ms = 1000 +failure_threshold = 5 +[services.restart] +max_attempts = 5 +backoff_base_ms = 250 +backoff_max_ms = 5000 + +[[services]] +id = "chunkdb" +program = "/opt/crowdb/bin/crowdb-chunkdb" +args = [] +dependencies = ["kv", "diskdb", "diskio"] +[services.probe] +kind = "http" +target = "http://127.0.0.1:@CHUNKDB_HTTP_BASE@/ready" +timeout_ms = 1000 +failure_threshold = 5 +[services.restart] +max_attempts = 5 +backoff_base_ms = 250 +backoff_max_ms = 5000 + +[[services]] +id = "chunk-kv" +program = "/opt/crowdb/bin/crowdb-chunk-kv-server" +args = [] +dependencies = ["kv", "chunkdb", "diskio"] +[services.probe] +kind = "http" +target = "http://127.0.0.1:@CHUNK_KV_HTTP_BASE@/ready" +timeout_ms = 1000 +failure_threshold = 5 +[services.restart] +max_attempts = 5 +backoff_base_ms = 250 +backoff_max_ms = 5000 + +[[services]] +id = "access" +program = "/opt/crowdb/bin/crowdb-access-server" +args = [] +dependencies = ["kv", "chunk-kv", "chunkdb", "diskio"] +[services.probe] +kind = "http" +target = "http://127.0.0.1:@ACCESS_SERVER_HEALTH_HTTP_BASE@/_crowdb/health/ready" +timeout_ms = 1000 +failure_threshold = 5 +[services.restart] +max_attempts = 5 +backoff_base_ms = 250 +backoff_max_ms = 5000 + +[[services]] +id = "web" +program = "/opt/crowdb/bin/crowdb-web" +args = [] +dependencies = [] +[services.probe] +kind = "http" +target = "http://127.0.0.1:@WEB_BASE@/healthz" +timeout_ms = 1000 +failure_threshold = 5 +[services.restart] +max_attempts = 5 +backoff_base_ms = 250 +backoff_max_ms = 5000 diff --git a/container/oci-image/paxos-kv-cluster.py b/container/oci-image/paxos-kv-cluster.py new file mode 100644 index 000000000..92a0a358b --- /dev/null +++ b/container/oci-image/paxos-kv-cluster.py @@ -0,0 +1,212 @@ +#!/usr/bin/env python3 +# Copyright 2026-present Gian <crow.db@outlook.com> +# Licensed under the Apache License, Version 2.0. +"""Explicit standalone Paxos-KV container deployment; never called by bench.""" +from ports import PAXOS_KV_SERVER_MGMT_BASE, PAXOS_KV_SERVER_RPC_BASE + +import argparse +import json +import os +import socket +import subprocess +import sys +import time +import uuid +from pathlib import Path +from urllib.request import build_opener, ProxyHandler, Request + +LABEL = "crowdb.paxos-kv.deployment" +HTTP = build_opener(ProxyHandler({})) + + +def docker(*args): + p = subprocess.run(["docker", *args], text=True, capture_output=True, timeout=60) + if p.returncode: + raise RuntimeError(p.stdout + p.stderr) + return (p.stdout + p.stderr if args[0] == "logs" else p.stdout).strip() + + +def request(origin, path, body=None): + data = None if body is None else json.dumps(body).encode() + with HTTP.open(Request(origin + path, data=data, headers={"Content-Type": "application/json"}), timeout=5) as r: + raw = r.read() + return json.loads(raw) if raw else None + + +def wait_ready(origins): + deadline = time.monotonic() + 30 + last = None + while time.monotonic() < deadline: + try: + result = [request(origin, "/topology") for origin in origins] + if all(t.get("stores") and t["stores"][0].get("groups") for t in result): + return result + except (OSError, ValueError) as error: + last = str(error) + time.sleep(.1) + raise RuntimeError(f"node readiness timeout: {last}") + + +def stop(state, artifacts=None, graceful=False): + # Check every identity before mutating any deployment member. + inspected = {} + for name in state["containers"]: + found = json.loads(docker("inspect", name))[0] + if (found["Config"].get("Labels") or {}).get(LABEL) != state["id"]: + raise RuntimeError(f"ownership mismatch: {name}") + inspected[name] = found + failures = [] + remaining = [] + for name in state["containers"]: + try: + failures.extend(stop_node(name, inspected[name], state, artifacts, graceful)) + except (RuntimeError, OSError, ValueError, subprocess.TimeoutExpired) as error: + # Retain this node for another collection attempt, but still stop + # and collect the other owned nodes. Never report missing evidence + # as a successful sanitizer run. + failures.append(f"{name}: cleanup incomplete: {error}") + remaining.append(name) + state["containers"] = remaining + return failures + + +def stop_node(name, found, state, artifacts, graceful): + failures = [] + if graceful: + docker("stop", "--time", "30", name) + found = json.loads(docker("inspect", name))[0] + if found["State"]["ExitCode"] != 0 or found["State"]["OOMKilled"]: + failures.append(f"{name}: {found['State']}") + if artifacts is not None: + target = artifacts / name + target.mkdir(parents=True, exist_ok=True) + (target / "inspect.json").write_text(json.dumps(found, indent=2)) + (target / "stdout.log").write_text(docker("logs", name)) + docker("cp", f"{name}:/opt/crowdb/data/kv/log/.", str(target / "log")) + if state.get("sanitizer"): + docker("cp", f"{name}:/opt/crowdb/data/asan/.", str(target / "asan")) + for report in (target / "asan").glob("report*"): + failures.append(f"{name}: sanitizer report {report}\n{report.read_text()}") + docker("rm", "-f", "-v", name) + return failures + + +def start(args): + if not args.image: + raise ValueError("up requires --image") + if args.sanitizer: + image = json.loads(docker("image", "inspect", args.image))[0] + if image["Config"].get("Labels", {}).get("crowdb.sanitizer") != "address": + raise ValueError("--sanitizer requires a staged ASan image") + if args.state.exists(): + raise ValueError("state file exists; stop its deployment first") + if len(args.offsets) != 3 or len(set(args.offsets)) != 3 or any(o < 0 or o > 999 for o in args.offsets): + raise ValueError("three distinct offsets in 0..999 required") + if args.cpuset and len(args.cpuset) != 3: + raise ValueError("supply one cpuset per node") + ports = [port + offset for offset in args.offsets for port in [PAXOS_KV_SERVER_MGMT_BASE, PAXOS_KV_SERVER_RPC_BASE]] + if len(set(ports)) != len(ports): + raise ValueError("offset port pools overlap") + for port in ports: + with socket.socket() as s: + s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + s.bind(("0.0.0.0", port)) + state = {"id":uuid.uuid4().hex,"containers":[],"endpoints":[],"network":"host","offsets":args.offsets, + "sanitizer":args.sanitizer} + args.state.parent.mkdir(parents=True, exist_ok=True) + try: + for i, offset in enumerate(args.offsets): + name = f"paxos-kv-{state['id'][:12]}-{i+1}" + command = ["run", "-d", "--name", name, "--network", "host", "--label", f"{LABEL}={state['id']}", + "--pids-limit", "256", "-e", f"TOKIO_WORKER_THREADS={len(os.sched_getaffinity(0))}"] + if args.memory: + command += ["--memory", args.memory, "--memory-swap", args.memory] + if args.cpus is not None: + command += ["--cpus", str(args.cpus)] + if args.cpuset: + command += ["--cpuset-cpus", args.cpuset[i]] + command += [args.image] + if not args.sanitizer: + command += ["server", "paxos-kv"] + command += ["--root","/opt/crowdb/data/kv", + "--management-port",str(PAXOS_KV_SERVER_MGMT_BASE+offset),"--management-addr","0.0.0.0", + "--management-advertise-addr","127.0.0.1","--ports",str(PAXOS_KV_SERVER_RPC_BASE+offset), + "--stores","0","--groups","1","--replica",str(i+1), + "--kv-backend","mem-block","--wal-backend","mem-block","--event-write", + "--election-profile", args.election_profile, + "--max-inflight",str(args.window),"--coalesce-max-keys",str(args.coalesce), + "--peer-pool-size","4","--rpc-workers",str(args.workers),"--log"] + docker(*command) + state["containers"].append(name) + state["endpoints"].append(f"http://127.0.0.1:{PAXOS_KV_SERVER_MGMT_BASE+offset}") + args.state.write_text(json.dumps(state, indent=2)) + topologies = wait_ready(state["endpoints"]) + stores = [] + for topology, offset in zip(topologies, args.offsets, strict=True): + store = topology["stores"][0] + store["listen_addr"] = f"127.0.0.1:{PAXOS_KV_SERVER_RPC_BASE+offset}" + stores.append(store) + for origin in state["endpoints"]: + request(origin,"/stores/0/groups/1/remotes/batch",{"stores":stores}) + deadline = time.monotonic() + 30 + while time.monotonic() < deadline: + groups = [request(o,"/topology")["stores"][0]["groups"][0] for o in state["endpoints"]] + leaders = {g["leader_id"] for g in groups} + if len(leaders) == 1 and next(iter(leaders)) in (1,2,3): + print(json.dumps(state, indent=2)) + return + time.sleep(.1) + raise RuntimeError("leader convergence timeout") + except BaseException: + for name in state["containers"]: + try: + print(docker("logs", name), file=sys.stderr) + except RuntimeError as error: + print(f"Cannot collect logs for {name}: {error}", file=sys.stderr) + failures = stop(state, args.state.parent / "startup-failure" if args.sanitizer else None, args.sanitizer) + for failure in failures: + print(failure, file=sys.stderr) + if state["containers"]: + args.state.write_text(json.dumps(state, indent=2)) + else: + args.state.unlink(missing_ok=True) + raise + + +def main(): + p = argparse.ArgumentParser(description=__doc__) + p.add_argument("action",choices=("up","down")) + p.add_argument("--state",type=Path,required=True) + p.add_argument("--image") + p.add_argument("--sanitizer", action="store_true", help="direct-server ASan image, staged by sanitizer tooling") + p.add_argument("--graceful", action="store_true", help="SIGTERM shutdown; fail on nonzero exit or OOM") + p.add_argument("--artifacts", type=Path, help="retain shutdown inspect, logs and sanitizer reports") + p.add_argument("--offsets",type=int,nargs=3,default=[0,1,2]) + p.add_argument("--memory", help="optional per-node memory limit; omitted for unrestricted performance baseline") + p.add_argument("--cpus",type=float,help="optional per-node CPU quota; omitted for unrestricted performance baseline") + p.add_argument("--cpuset",action="append",default=[]) + p.add_argument("--workers",type=int,default=2) + p.add_argument("--window",type=int,default=32) + p.add_argument("--coalesce",type=int,default=16) + p.add_argument("--election-profile", choices=("default", "e2e"), default="default") + args = p.parse_args() + if args.action == "up": + start(args) + else: + state = json.loads(args.state.read_text()) + if state.get("sanitizer") and (not args.graceful or args.artifacts is None): + raise ValueError("sanitizer shutdown requires --graceful --artifacts") + original = dict(state) + failures = stop(state, args.artifacts, args.graceful) + args.state.with_suffix(".stopped.json").write_text(json.dumps( + {"state":original, "failures":failures, "remaining_containers":state["containers"]}, indent=2)) + if state["containers"]: + args.state.write_text(json.dumps(state, indent=2)) + else: + args.state.unlink() + if failures: + raise RuntimeError("\n".join(failures)) + + +if __name__ == "__main__": + main() diff --git a/container/oci-image/ports.py b/container/oci-image/ports.py new file mode 100644 index 000000000..a8a2f56e5 --- /dev/null +++ b/container/oci-image/ports.py @@ -0,0 +1,22 @@ +# Copyright 2026-present Gian <crow.db@outlook.com> +# Licensed under the Apache License, Version 2.0. + +"""Generated from ports.rs by tools/ports/generate.py; do not edit.""" + +PAXOS_KV_SERVER_MGMT_BASE = 10135 +PAXOS_KV_SERVER_RPC_BASE = 11135 +DISKDB_HTTP_BASE = 13135 +DISKDB_RPC_BASE = 14135 +CHUNKDB_HTTP_BASE = 16135 +CHUNKDB_RPC_BASE = 17135 +DISKIO_RPC_BASE = 18135 +WEB_BASE = 6135 +CHUNK_KV_HTTP_BASE = 20135 +CHUNK_KV_RPC_BASE = 21135 +ACCESS_SERVER_HTTP_BASE = 7135 +ACCESS_SERVER_ICEBERG_HTTP_BASE = 8135 +ACCESS_SERVER_DATASET_HTTP_BASE = 9135 +ACCESS_SERVER_HEALTH_HTTP_BASE = 25135 +MONITOR_HTTP_BASE = 26135 +SSH_PORT = 2222 +HOST_SSH_PORT = 22 diff --git a/container/oci-image/run-node.py b/container/oci-image/run-node.py index fe5a94d39..6618874eb 100755 --- a/container/oci-image/run-node.py +++ b/container/oci-image/run-node.py @@ -11,6 +11,8 @@ import stat from pathlib import Path +import ports + from runtime import Runtime, node_policy_file from storage import inputs @@ -20,13 +22,16 @@ def digest_reference(reference): return digest.startswith("sha256:") and len(digest) == 71 and all(c in "0123456789abcdef" for c in digest[7:]) -# Base listeners used by one node. Host-network instances add -# ``CROWDB_PORT_OFFSET`` to every listener before the policy is sealed. These -# values mirror the canonical service blocks in -# ``lib/crowdb-protocol/src/port/ports.rs``. -HOST_PORTS = (2222, 9090, 9095, 10000, 10100, 10101, 11000, 11100, 11200, - 12000, 12100, 12200, 13000, 14000, 15100, 15200, 16000, 17000, - 18000, 19000) +# Default node-policy listeners. Explicitly configured service endpoints remain authoritative. +HOST_PORTS = (ports.SSH_PORT, ports.PAXOS_KV_SERVER_MGMT_BASE, + ports.PAXOS_KV_SERVER_RPC_BASE, ports.PAXOS_KV_SERVER_RPC_BASE + 1, + ports.DISKDB_HTTP_BASE, ports.DISKDB_RPC_BASE, + ports.CHUNKDB_HTTP_BASE, ports.CHUNKDB_RPC_BASE, + ports.DISKIO_RPC_BASE, ports.WEB_BASE, + ports.CHUNK_KV_HTTP_BASE, ports.CHUNK_KV_RPC_BASE, + ports.ACCESS_SERVER_HTTP_BASE, ports.ACCESS_SERVER_ICEBERG_HTTP_BASE, + ports.ACCESS_SERVER_DATASET_HTTP_BASE, ports.ACCESS_SERVER_HEALTH_HTTP_BASE, + ports.MONITOR_HTTP_BASE) def check_host_ports(ports=HOST_PORTS): @@ -96,7 +101,7 @@ def validate(args, parser, check_ports=True): parser.error("management interface and physical host identity are required") if args.network == "host": if args.ui_port is not None: - parser.error("host networking uses the node's port 9090 directly") + parser.error(f"host networking uses the node's port {ports.WEB_BASE} directly") try: socket.if_nametoindex(args.interface) if check_ports: diff --git a/container/oci-image/runtime.py b/container/oci-image/runtime.py index db627149d..43317c553 100644 --- a/container/oci-image/runtime.py +++ b/container/oci-image/runtime.py @@ -2,6 +2,8 @@ # Licensed under the Apache License, Version 2.0. """Host Docker/containerd commands and verified container resource boundaries.""" +from ports import WEB_BASE + import json import math import os @@ -134,13 +136,13 @@ def create(self, args): if args.seed: command += ["-e", "CROWDB_DISCOVERY_SEEDS=" + ",".join(args.seed)] if args.ui_port: - command += ["-p", f"127.0.0.1:{args.ui_port}:9090"] + command += ["-p", f"127.0.0.1:{args.ui_port}:{WEB_BASE}"] try: container = self.invoke(*command, args.image).strip() self.verify(container, args.cpus, args.memory_mib, args.network == "host", shares) for disk in disks: kind = "-f" if disk["backend"] == "file" else "-b" - self.invoke("exec", "--user", "10001:10001", container, "sh", "-c", + self.invoke("exec", "--user", "10001:10136", container, "sh", "-c", 'test "$1" "$2" && test -r "$2" && test -w "$2"', "sh", kind, disk["path"]) record_inputs(args.data_root, disks) return container diff --git a/container/oci-image/stage.sh b/container/oci-image/stage.sh index d15bd84be..794698fb9 100644 --- a/container/oci-image/stage.sh +++ b/container/oci-image/stage.sh @@ -4,6 +4,7 @@ set -euo pipefail [[ $(uname -sm) == "Linux x86_64" ]] || { echo "Linux amd64 build host required; macOS VM backend is reserved" >&2; exit 1; } cd "$(git rev-parse --show-toplevel)" +python tools/ports/generate.py --check source_tree=$(python container/oci-image/source_tree.py) case "${CROWDB_KEEP_DEBUG_INFO:-0}" in 0) cmake_debug_args=() ;; @@ -14,11 +15,11 @@ case "${CROWDB_KEEP_DEBUG_INFO:-0}" in *) echo 'CROWDB_KEEP_DEBUG_INFO must be 0 or 1' >&2; exit 1 ;; esac # Build on the host, reusing the existing Cargo, CMake and npm artifacts. -cargo build --locked --release -p crowdb-kv-client --features ffi +cargo build --locked --release -p crowdb-paxos-kv-client --features ffi cmake -S app/crowdb-diskio -B app/crowdb-diskio/build -DCMAKE_BUILD_TYPE=Release "${cmake_debug_args[@]}" cmake --build app/crowdb-diskio/build --parallel "$(getconf _NPROCESSORS_ONLN)" --target crowdb-diskio -cargo build --locked --release --features crowdb-kv-client/ffi \ - -p crowdb-kv-client -p crowdb-monitor -p crowdb-kv-server -p crowdb-diskdb \ +cargo build --locked --release --features crowdb-paxos-kv-client/ffi \ + -p crowdb-paxos-kv-client -p crowdb-monitor -p crowdb-paxos-kv-server -p crowdb-diskdb \ -p crowdb-chunkdb -p crowdb-chunk-kv-server \ -p crowdb-access-server -p crowdb-web (cd app/crowdb-web/ui && npm ci --prefer-offline && npm run build) diff --git a/container/oci-image/tests/docker_local_test.py b/container/oci-image/tests/docker_local_test.py index 858062fcd..b6a36491a 100644 --- a/container/oci-image/tests/docker_local_test.py +++ b/container/oci-image/tests/docker_local_test.py @@ -15,15 +15,16 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1])) from docker_local_cleanup import clean, image_garbage, selected from docker_local_launch import ensure, verify_container +from ports import WEB_BASE, ACCESS_SERVER_HTTP_BASE, ACCESS_SERVER_ICEBERG_HTTP_BASE, ACCESS_SERVER_DATASET_HTTP_BASE from docker_local_state import CLUSTER, OWNER, Docker, load, seal class DockerLocalTest(unittest.TestCase): def setUp(self): - self.record = {"version": 2, "cluster": "first", "owner": str(uuid.uuid4()), + self.record = {"version": 3, "cluster": "first", "owner": str(uuid.uuid4()), "runtime": {"daemon": "owned", "context": "default", "endpoint": "unix:///owned"}, "image_id": "sha256:retained", "image": "crowdb-node:dev", "nodes": 1, - "port_base": 9090, "cpus": 1, "memory_mib": 1536} + "port_base": WEB_BASE, "cpus": 1, "memory_mib": 1536} self.labels = {CLUSTER: "first", OWNER: self.record["owner"]} self.docker = MagicMock() self.docker.identity.return_value = self.record["runtime"] @@ -120,35 +121,58 @@ def test_create_cannot_adopt_a_foreign_named_network(self): self.docker.call.assert_not_called() def test_changed_container_inputs_are_rejected_before_resume(self): - record = dict(self.record, nodes=1, port_base=9090, cpus=1, memory_mib=1536) + record = dict(self.record, nodes=1, port_base=WEB_BASE, cpus=1, memory_mib=1536) environment = ["CROWDB_CLUSTER=first", "CROWDB_INITIAL_VOTERS=1"] - for offset, name in enumerate(("S3", "ICEBERG", "DATASET"), start=1): - environment.append(f"CROWDB_{name}_PUBLIC_URI=http://127.0.0.1:{9090+offset}") + for port, name in ((ACCESS_SERVER_HTTP_BASE, "S3"), (ACCESS_SERVER_ICEBERG_HTTP_BASE, "ICEBERG"), (ACCESS_SERVER_DATASET_HTTP_BASE, "DATASET")): + environment.append(f"CROWDB_{name}_PUBLIC_URI=http://127.0.0.1:{port}") item = {"Name": "/crowdb-first-1", "Image": record["image_id"], "Config": {"Labels": self.labels, "Env": environment}, "HostConfig": {"NanoCpus": 1_000_000_000, "Memory": 1536 * 1024 * 1024, "NetworkMode": "crowdb-first", "PortBindings": { - f"{port}/tcp": [{"HostIp": "127.0.0.1", "HostPort": str(9090+offset)}] - for offset, port in enumerate((9090, 16000, 17000, 18000))}}, + f"{port}/tcp": [{"HostIp": "127.0.0.1", "HostPort": str(port)}] + for port in (WEB_BASE, ACCESS_SERVER_HTTP_BASE, ACCESS_SERVER_ICEBERG_HTTP_BASE, ACCESS_SERVER_DATASET_HTTP_BASE)}}, "Mounts": [{"Destination": "/opt/crowdb/data", "Name": "owned-volume", "Type": "volume", "RW": True}]} verify_container(item, record, 1, "owned-volume", "crowdb-first") - legacy = dict(item, HostConfig=dict(item["HostConfig"], PortBindings={ - f"{port}/tcp": [{"HostIp": "127.0.0.1", "HostPort": str(port)}] - for port in range(9090, 9094)})) - verify_container(legacy, dict(record, version=1), 1, "owned-volume", "crowdb-first") - with self.assertRaisesRegex(ValueError, "endpoints"): - verify_container(legacy, record, 1, "owned-volume", "crowdb-first") for field, value in (("Image", "foreign"), ("Mounts", []), ("HostConfig", {})): with self.subTest(field=field), self.assertRaises((ValueError, KeyError)): verify_container(dict(item, **{field: value}), record, 1, "owned-volume", "crowdb-first") + def test_retained_versions_preserve_exact_ports_and_public_endpoints(self): + for version, internal in ((1, (9090, 9091, 9092, 9093)), + (2, (9090, 16000, 17000, 18000)), + (3, (WEB_BASE, ACCESS_SERVER_HTTP_BASE, + ACCESS_SERVER_ICEBERG_HTTP_BASE, ACCESS_SERVER_DATASET_HTTP_BASE))): + with self.subTest(version=version), tempfile.TemporaryDirectory() as temporary: + record = dict(self.record, version=version, nodes=3, + port_base=65000 if version < 3 else WEB_BASE+37) + seal(record, Path(temporary)) + self.assertEqual(load("first", Path(temporary)), record) + base = record["port_base"] + (20 if version < 3 else 2) + offsets = range(4) if version < 3 else [port-WEB_BASE for port in internal] + published = [base+offset for offset in offsets] + environment = ["CROWDB_CLUSTER=first", "CROWDB_INITIAL_VOTERS=3"] + for name, port in zip(("S3", "ICEBERG", "DATASET"), published[1:]): + environment.append(f"CROWDB_{name}_PUBLIC_URI=http://127.0.0.1:{port}") + item = {"Name": "/crowdb-first-3", "Image": record["image_id"], + "Config": {"Labels": self.labels, "Env": environment}, + "HostConfig": {"NanoCpus": 1_000_000_000, "Memory": 1536*1024*1024, + "NetworkMode": "crowdb-first", "PortBindings": { + f"{internal_port}/tcp": [{"HostIp": "127.0.0.1", "HostPort": str(port)}] + for internal_port, port in zip(internal, published)}}, + "Mounts": [{"Destination": "/opt/crowdb/data", "Name": "owned-volume", + "Type": "volume", "RW": True}]} + verify_container(item, record, 3, "owned-volume", "crowdb-first") + item["Config"]["Env"][-1] = "CROWDB_DATASET_PUBLIC_URI=http://127.0.0.1:1" + with self.assertRaisesRegex(ValueError, "endpoints"): + verify_container(item, record, 3, "owned-volume", "crowdb-first") + def test_compose_examples_publish_the_standard_node_listeners(self): for filename in ("compose.yaml", "compose-three.yaml"): document = yaml.safe_load((Path(__file__).resolve().parents[1] / filename).read_text()) for service in document["services"].values(): self.assertEqual([port.split("}:", 1)[1] for port in service["ports"]], - ["9090", "${CROWDB_S3_CONTAINER_PORT:-16000}", - "${CROWDB_ICEBERG_CONTAINER_PORT:-17000}", - "${CROWDB_DATASET_CONTAINER_PORT:-18000}"]) + [str(WEB_BASE), f"${{CROWDB_S3_CONTAINER_PORT:-{ACCESS_SERVER_HTTP_BASE}}}", + f"${{CROWDB_ICEBERG_CONTAINER_PORT:-{ACCESS_SERVER_ICEBERG_HTTP_BASE}}}", + f"${{CROWDB_DATASET_CONTAINER_PORT:-{ACCESS_SERVER_DATASET_HTTP_BASE}}}"]) def test_receipts_are_immutable_scoped_and_reject_symlinks(self): with tempfile.TemporaryDirectory() as temporary: diff --git a/container/oci-image/tests/host_inputs_test.py b/container/oci-image/tests/host_inputs_test.py index 77205ebbd..db01c20da 100644 --- a/container/oci-image/tests/host_inputs_test.py +++ b/container/oci-image/tests/host_inputs_test.py @@ -19,8 +19,8 @@ class HostInputsTest(unittest.TestCase): def test_port_offset_shifts_every_host_listener_for_additional_instances(self): shifted = run_node.shifted_ports(1) self.assertEqual(shifted[0], 2223) - self.assertIn(10001, shifted) - self.assertIn(10101, shifted) + self.assertIn(run_node.ports.PAXOS_KV_SERVER_MGMT_BASE + 1, shifted) + self.assertIn(run_node.ports.PAXOS_KV_SERVER_RPC_BASE + 1, shifted) with self.assertRaisesRegex(ValueError, "beyond port 65535"): run_node.shifted_ports(65535) diff --git a/container/oci-image/tests/ports_test.py b/container/oci-image/tests/ports_test.py new file mode 100644 index 000000000..58a9ce926 --- /dev/null +++ b/container/oci-image/tests/ports_test.py @@ -0,0 +1,49 @@ +# Copyright 2026-present Gian <crow.db@outlook.com> +# Licensed under the Apache License, Version 2.0. +"""Port defaults and deployment arguments agree with the protocol source.""" + +from pathlib import Path +import subprocess +import sys +import unittest + +ROOT = Path(__file__).resolve().parents[3] +sys.path.insert(0, str(ROOT / "container/oci-image")) +import ports +from docker_local_launch import PUBLIC_PORTS, verify_container + + +class PortsTest(unittest.TestCase): + def test_generated_defaults_are_current(self): + subprocess.run([sys.executable, str(ROOT / "tools/ports/generate.py"), "--check"], check=True) + + def test_default_bridge_ports_use_node_offsets_zero_one_two(self): + all_ports = set() + for offset in range(3): + for base, _ in PUBLIC_PORTS: + self.assertNotIn(base + offset, all_ports) + all_ports.add(base + offset) + self.assertEqual([port for port, _ in PUBLIC_PORTS], [6135, 7135, 8135, 9135]) + + def test_second_container_verification_uses_service_spacing_and_node_offset(self): + record = dict(version=3, cluster="ports", owner="owner", nodes=3, image_id="sha256:fixture", + cpus=1, memory_mib=1536, port_base=ports.WEB_BASE + 100) + env = ["CROWDB_CLUSTER=ports", "CROWDB_INITIAL_VOTERS=3"] + for base, protocol in PUBLIC_PORTS: + if protocol: + env.append(f"CROWDB_{protocol}_PUBLIC_URI=http://127.0.0.1:{base + 101}") + item = { + "Name": "/crowdb-ports-2", "Image": record["image_id"], + "Config": {"Labels": {"org.crowdb.local.cluster": "ports", "org.crowdb.local.owner": "owner"}, "Env": env}, + "HostConfig": {"NanoCpus": 1_000_000_000, "Memory": 1536 * 1024 * 1024, + "NetworkMode": "crowdb-ports", "PortBindings": { + f"{base}/tcp": [{"HostIp": "127.0.0.1", "HostPort": str(base + 101)}] + for base, _ in PUBLIC_PORTS}}, + "Mounts": [{"Destination": "/opt/crowdb/data", "Type": "volume", "Name": "data", "RW": True}], + } + from docker_local_state import CLUSTER, OWNER + item["Config"]["Labels"] = {CLUSTER: record["cluster"], OWNER: record["owner"]} + verify_container(item, record, 2, "data", "crowdb-ports") + item["HostConfig"]["PortBindings"][f"{ports.WEB_BASE}/tcp"][0]["HostPort"] = str(ports.WEB_BASE + 110) + with self.assertRaises(ValueError): + verify_container(item, record, 2, "data", "crowdb-ports") diff --git a/container/oci-image/tests/staging_test.py b/container/oci-image/tests/staging_test.py index 74500f739..194aedfec 100644 --- a/container/oci-image/tests/staging_test.py +++ b/container/oci-image/tests/staging_test.py @@ -10,7 +10,7 @@ ROOT = Path(__file__).resolve().parents[3] PIXI = ["pixi", "run", "--as-is", "--manifest-path", str(ROOT / "pixi.toml")] -BINARIES = ("crowdb-monitor", "crowdb-kv-server", "crowdb-diskdb", "crowdb-diskio", +BINARIES = ("crowdb-monitor", "crowdb-paxos-kv-server", "crowdb-diskdb", "crowdb-diskio", "crowdb-chunkdb", "crowdb-chunk-kv-server", "crowdb-access-server", "crowdb-web") @@ -50,8 +50,8 @@ def stage_fixture(self, directory, aliased): for name, function in definitions.items(): self.compile(f"int {function}(void) {{ return 0; }}", libraries / name, "-shared", "-fPIC", "-Wl,-soname," + name) - self.compile("int ffi(void) { return 0; }", release / "libcrowdb_kv_client.so", - "-shared", "-fPIC", "-Wl,-soname,libcrowdb_kv_client.so") + self.compile("int ffi(void) { return 0; }", release / "libcrowdb_paxos_kv_client.so", + "-shared", "-fPIC", "-Wl,-soname,libcrowdb_paxos_kv_client.so") self.compile("int provider_dependency(void); int provider(void) { return provider_dependency(); }", libraries / "libfixture.so.1", "-shared", "-fPIC", "-L" + str(libraries), "-lprovider-dependency", "-Wl,-rpath," + str(libraries)) @@ -61,7 +61,7 @@ def stage_fixture(self, directory, aliased): executable = directory / "node" self.compile(source, executable, "-L" + str(libraries), "-L" + str(release), "-Wl,-l:libibverbs.so.1", "-Wl,-l:librdmacm.so.1", "-Wl,-l:libcrypto.so.3", - "-lcrowdb_kv_client") + "-lcrowdb_paxos_kv_client") for name in BINARIES: target = root / "app/crowdb-diskio/build" if name == "crowdb-diskio" else release target.mkdir(parents=True, exist_ok=True) @@ -76,7 +76,7 @@ def stage_fixture(self, directory, aliased): command = [*PIXI, "bash", "-c", 'PATH="$1:$PATH"; exec bash "$2" "$3"', "_", str(tools), str(ROOT / "container/oci-image/collect-libs.sh"), str(staged)] subprocess.run(command, check=True, env=dict(os.environ, CROWDB_STAGING_TEST_ROOT=str(root))) - for name in (*definitions, "libcrowdb_kv_client.so"): + for name in (*definitions, "libcrowdb_paxos_kv_client.so"): self.assertTrue((staged / "lib" / name).is_file(), name) provider = staged / "lib/libibverbs/libfixture-rdmav99.so" self.assertTrue(provider.is_file()) diff --git a/container/oci-image/tests/task_entrypoints_test.py b/container/oci-image/tests/task_entrypoints_test.py index 7e1cd3b6e..3e6ac69ba 100644 --- a/container/oci-image/tests/task_entrypoints_test.py +++ b/container/oci-image/tests/task_entrypoints_test.py @@ -45,8 +45,8 @@ def test_build_compiles_cpp_release_frontend_and_tests_without_running_tests(sel result, commands = self.execute('build-release.sh') self.assertEqual(result.returncode, 0, result.stdout + result.stderr) self.assertTrue(commands[0].startswith('cmake -S lib/crowdb-tree')) - self.assertIn('cargo build --release --workspace --exclude crowdb-kv-client', commands) - self.assertIn('cargo build --release -p crowdb-kv-client --features ffi', commands) + self.assertIn('cargo build --release --workspace --exclude crowdb-paxos-kv-client', commands) + self.assertIn('cargo build --release -p crowdb-paxos-kv-client --features ffi', commands) self.assertIn('npm run build', commands) self.assertEqual(commands[-1], 'cargo test --workspace --all-targets --no-run') diff --git a/container/single-node-container/README.md b/container/single-node-container/README.md index b0ba77b41..3936e3b8f 100644 --- a/container/single-node-container/README.md +++ b/container/single-node-container/README.md @@ -3,37 +3,19 @@ # Local node storage and client usage -One node is a topology case of the normal editable cluster lifecycle. For image -builds, named Docker shortcuts, independent Docker commands and Compose, use the -[shared OCI guide](../oci-image/README.md). The old manual/single startup mode -selector is removed; use the accepted deployment mode and initial voter policy. +One node is a topology case of the normal editable cluster lifecycle. For image builds, named Docker shortcuts, independent Docker commands and Compose, use the [shared OCI guide](../oci-image/README.md). The old manual/single startup mode selector is removed; use the accepted deployment mode and initial voter policy. ## Node-first local storage policy -The node-first local policy accepts one or three initial voters. Each member -prepares one sparse 16-GiB file with a 16-GiB zone, 1-MiB allocation units and -a 1-GiB free-space reserve. These are test media; sparse capacity does not -reserve physical host space. Production does not fall back to this policy. +The node-first local policy accepts one or three initial voters. Each member prepares one sparse 16-GiB file with a 16-GiB zone, 1-MiB allocation units and a 1-GiB free-space reserve. These are test media; sparse capacity does not reserve physical host space. Production does not fall back to this policy. - One-node writes use one mirror copy; three-node writes use two copies. -- Tree, stream and Access payload allocation uses 1-MiB mirror strips. Tree - chunk capacity remains independent of strip capacity. +- Tree, stream and Access payload allocation uses 1-MiB mirror strips. Tree chunk capacity remains independent of strip capacity. - Mirror-to-EC conversion is disabled, including Dataset payload writes. -- Local placement declares zero physical-node failure tolerance. Colocated - containers retain the same physical-host identity. The explicit local - placement allowance can put two copies on separate extents of the same - disk; copy count does not promise disk, node or rack protection. Inspect - each strip's placement assessment for its actual protection. -- Accepted media and operation identities survive restart. Failed preparation - is retained for diagnosis; restarting is not permission to replace its data. - -The node-first local client origins default to `http://127.0.0.1` on ports -9091 (S3), 9092 (Iceberg) and 9093 (Dataset). When publishing different host -ports or addresses, supply `CROWDB_S3_PUBLIC_URI`, `CROWDB_ICEBERG_PUBLIC_URI` -and `CROWDB_DATASET_PUBLIC_URI` to each node. These HTTP origins are sealed in -the node policy before startup; omitted restart values retain the accepted -origins. Internal discovery, service listeners and Console requests continue -to use node addresses. The node does not inspect the Docker socket for NAT. +- Local placement declares zero physical-node failure tolerance. Colocated containers retain the same physical-host identity. The explicit local placement allowance can put two copies on separate extents of the same disk; copy count does not promise disk, node or rack protection. Inspect each strip's placement assessment for its actual protection. +- Accepted media and operation identities survive restart. Failed preparation is retained for diagnosis; restarting is not permission to replace its data. + +The node-first local client origins default to `http://127.0.0.1` on ports 7135 (S3), 8135 (Iceberg) and 9135 (Dataset). When publishing different host ports or addresses, supply `CROWDB_S3_PUBLIC_URI`, `CROWDB_ICEBERG_PUBLIC_URI` and `CROWDB_DATASET_PUBLIC_URI` to each node. These HTTP origins are sealed in the node policy before startup; omitted restart values retain the accepted origins. Internal discovery, service listeners and Console requests continue to use node addresses. The node does not inspect the Docker socket for NAT. Read client credentials explicitly with: @@ -41,50 +23,21 @@ Read client credentials explicitly with: pixi run docker exec NODE crowdb-monitor credentials show --format env ``` -This exports S3 credentials, the Iceberg writer token and the Dataset bearer -with their public origins. The private client file is -`/opt/crowdb/data/console/secrets/client.env`; it has mode `0600`. Server master -keys stay in the protected server files. Do not put client credential output -into logs or cluster status. +This exports S3 credentials, the Iceberg writer token and the Dataset bearer with their public origins. The private client file is `/opt/crowdb/data/console/secrets/client.env`; it has mode `0600`. Server master keys stay in the protected server files. Do not put client credential output into logs or cluster status. ## Client listeners and readiness -The same Access service exposes S3 at 9091, Iceberg at 9092 and Dataset at 9093. -The Web UI listens at 9090. Publish these ports when using a bridge. A listener -or an old completed setup record is not proof of current storage availability. - -Use the client origins returned by the credential command. Iceberg metadata and -Parquet URLs and Dataset payload references must be reachable from the client; -changing Docker port mappings without changing the accepted public origins -before first startup produces unreachable references. Internal service addresses -remain independent of public origins. - -All members of one local cluster share the accepted client credential set. -Retain each member's own data volume across stop and image replacement. -Default `oci-clean` preserves running clusters and removes stopped clusters, -their owned data, and unused Docker containers, networks, volumes and build -cache. Use `--cluster NAME` to explicitly stop and remove one cluster, or -`--all` to stop and remove all recorded clusters. Use `--keep-data` to retain -volumes and restart receipts. See the [cleanup contract](../oci-image/README.md#local-cluster-shortcuts). -Ordinary `docker compose down` also preserves its named volumes. +The same Access service exposes S3 at 7135, Iceberg at 8135 and Dataset at 9135. The Web UI listens at 6135. Publish these ports when using a bridge. A listener or an old completed setup record is not proof of current storage availability. + +Use the client origins returned by the credential command. Iceberg metadata and Parquet URLs and Dataset payload references must be reachable from the client; changing Docker port mappings without changing the accepted public origins before first startup produces unreachable references. Internal service addresses remain independent of public origins. + +All members of one local cluster share the accepted client credential set. Retain each member's own data volume across stop and image replacement. Default `oci-clean` preserves running clusters and removes stopped clusters, their owned data, and unused Docker containers, networks, volumes and build cache. Use `--cluster NAME` to explicitly stop and remove one cluster, or `--all` to stop and remove all recorded clusters. Use `--keep-data` to retain volumes and restart receipts. See the [cleanup contract](../oci-image/README.md#local-cluster-shortcuts). Ordinary `docker compose down` also preserves its named volumes. ## Tested S3 client recipes -- The locked `s3-e2e` environment pins boto3/botocore 1.43.92, AWS CLI 2.36.47, - and rclone 1.75.1 (the packaged binary identifies its build as `1.75.1-DEV`). Run repository acceptance commands through Pixi. -- boto3 uses its default checksum calculation and validation settings. Only - path-style addressing, SigV4 and the endpoint are configured. Ordinary and multipart - uploads verify request CRC32 checksums and exact downloaded bytes. Presigned - URLs and AWS chunked trailers have separate integrity tests. Supported request - checksums do not imply arbitrary response checksum negotiation. -- AWS CLI uses the configured recipe below for discovery, ordinary and multipart - transfers, prefix listing, server copy, sync with deletion, and recursive - cleanup. One concurrent request matches the bounded development fixture. - The canonical `pixi run test-e2e-s3 --case s3.protocol::test_aws_cli_workflow` case verifies ten - concurrent requests with the same classic transfer and path-style setup on - the real-storage fixture, including exact bytes and cleanup. The container - recipe retains one worker. Copy explicitly requests COPY metadata; the CLI's broader - property/annotation discovery is unsupported. +- The locked `s3-e2e` environment pins boto3/botocore 1.43.92, AWS CLI 2.36.47, and rclone 1.75.1 (the packaged binary identifies its build as `1.75.1-DEV`). Run repository acceptance commands through Pixi. +- boto3 uses its default checksum calculation and validation settings. Only path-style addressing, SigV4 and the endpoint are configured. Ordinary and multipart uploads verify request CRC32 checksums and exact downloaded bytes. Presigned URLs and AWS chunked trailers have separate integrity tests. Supported request checksums do not imply arbitrary response checksum negotiation. +- AWS CLI uses the configured recipe below for discovery, ordinary and multipart transfers, prefix listing, server copy, sync with deletion, and recursive cleanup. One concurrent request matches the bounded development fixture. The canonical `pixi run test-e2e-s3 --case s3.protocol::test_aws_cli_workflow` case verifies ten concurrent requests with the same classic transfer and path-style setup on the real-storage fixture, including exact bytes and cleanup. The container recipe retains one worker. Copy explicitly requests COPY metadata; the CLI's broader property/annotation discovery is unsupported. ```ini [default] @@ -97,43 +50,18 @@ s3 = multipart_chunksize = 5MB ``` -Use that profile in `AWS_CONFIG_FILE`, export the generated credentials, then -run `aws --endpoint-url http://127.0.0.1:9091 ...`. For server copy, supply -`--metadata-directive COPY` to `aws s3 cp`. The repeatable repository gate is -`pixi run test-e2e-s3 --case s3.protocol::test_aws_cli_workflow`; container acceptance runs the same recipe -and checks persisted bytes after recovery before publication credentials become -available. - -- The configured rclone recipe covers discovery, ordinary/multipart transfer, - prefix listing, server copy, sync/delete and exact downloads. It uses the - Other provider, path-style addressing, ListObjectsV2, no system metadata, - one transfer/checker/upload worker, an 8 MiB multipart cutoff and 5 MiB parts. - User metadata preserves file modification times; checksum metadata remains - enabled. Run `pixi run test-e2e-s3 --case s3.protocol::test_rclone_workflow`. Container acceptance runs - this recipe alongside AWS CLI and verifies both after recovery/restart. +Use that profile in `AWS_CONFIG_FILE`, export the generated credentials, then run `aws --endpoint-url http://127.0.0.1:7135 ...`. For server copy, supply `--metadata-directive COPY` to `aws s3 cp`. The repeatable repository gate is `pixi run test-e2e-s3 --case s3.protocol::test_aws_cli_workflow`; container acceptance runs the same recipe and checks persisted bytes after recovery before publication credentials become available. + +- The configured rclone recipe covers discovery, ordinary/multipart transfer, prefix listing, server copy, sync/delete and exact downloads. It uses the Other provider, path-style addressing, ListObjectsV2, no system metadata, one transfer/checker/upload worker, an 8 MiB multipart cutoff and 5 MiB parts. User metadata preserves file modification times; checksum metadata remains enabled. Run `pixi run test-e2e-s3 --case s3.protocol::test_rclone_workflow`. Container acceptance runs this recipe alongside AWS CLI and verifies both after recovery/restart. - Mounting the endpoint as a filesystem with s3fs-fuse is unsupported. -- Optional [Java, JavaScript and Go SDK recipes](../../e2e-test/clients/S3-SDK.md) - run independently through manual Pixi tasks and workflow dispatch. Their - isolated language toolchains and checks are outside container/release gates. -- User metadata supports lowercase keys and printable ASCII values, with a - combined key/value size up to 2 KiB. PUT, multipart initiation and COPY/REPLACE - persist the attributes; HEAD and GET return them. Duplicate names and invalid - values are rejected. Older multipart session schemas are unsupported; - restart recovery applies to the current format, with no migration decoder. -- The endpoint has one configured namespace shared by accepted credentials; - per-user ACLs, versioning and annotations are unsupported. - ListBuckets CreationDate is a stable Unix-epoch placeholder. ListObjectsV2 - supports `encoding-type=url` for keys requiring XML-safe encoding. +- Optional [Java, JavaScript and Go SDK recipes](../../e2e-test/clients/S3-SDK.md) run independently through manual Pixi tasks and workflow dispatch. Their isolated language toolchains and checks are outside container/release gates. +- User metadata supports lowercase keys and printable ASCII values, with a combined key/value size up to 2 KiB. PUT, multipart initiation and COPY/REPLACE persist the attributes; HEAD and GET return them. Duplicate names and invalid values are rejected. Older multipart session schemas are unsupported; restart recovery applies to the current format, with no migration decoder. +- The endpoint has one configured namespace shared by accepted credentials; per-user ACLs, versioning and annotations are unsupported. ListBuckets CreationDate is a stable Unix-epoch placeholder. ListObjectsV2 supports `encoding-type=url` for keys requiring XML-safe encoding. ## Iceberg and Dataset clients -The [Iceberg ecosystem guide](../../e2e-test/clients/ecosystem/README.md) describes -locked PyIceberg and external ecosystem clients. PyIceberg tests write real -metadata/Parquet objects and read them through each node's configured public -origin. Dataset has its own bearer token and REST namespace; use its returned -payload references rather than constructing storage URLs. +The [Iceberg ecosystem guide](../../e2e-test/clients/ecosystem/README.md) describes locked PyIceberg and external ecosystem clients. PyIceberg tests write real metadata/Parquet objects and read them through each node's configured public origin. Dataset has its own bearer token and REST namespace; use its returned payload references rather than constructing storage URLs. + +The OCI image carries the same server binaries for all protocols. It neither mounts the Docker socket nor delegates bootstrap or storage preparation to the host CLI. Monitor-owned accepted operations continue after the launching shell or browser exits. Changing an image does not reset node identity or storage. -The OCI image carries the same server binaries for all protocols. It neither -mounts the Docker socket nor delegates bootstrap or storage preparation to the -host CLI. Monitor-owned accepted operations continue after the launching shell -or browser exits. Changing an image does not reset node identity or storage. +Default listener ports are defined in [protocol ports.rs](../../lib/crowdb-protocol/src/port/ports.rs). Generated packaging defaults are maintained with [tools/ports](../../tools/ports/README.md). diff --git a/container/single-node-container/dev-container.sh b/container/single-node-container/dev-container.sh index 85e82f4d5..e1bdcdab8 100644 --- a/container/single-node-container/dev-container.sh +++ b/container/single-node-container/dev-container.sh @@ -1,4 +1,5 @@ #!/usr/bin/env bash +# Generated by tools/ports/generate.py from ports.rs; edit the .in template. # Copyright 2026-present Gian <crow.db@outlook.com> # Licensed under the Apache License, Version 2.0. set -euo pipefail @@ -49,7 +50,7 @@ start_container() { if ! docker run --detach --name "$name" -e CROWDB_STARTUP_MODE=single \ --mount "type=volume,source=$data_volume,target=/opt/crowdb/data" \ - --expose 9090 --expose 9091 --expose 9092 --expose 9093 --expose 9094 \ + --expose 6135 --expose 7135 --expose 8135 --expose 9135 --expose 25135 \ "$image"; then if docker container inspect "$name" >/dev/null 2>&1; then docker rm --force "$name" >/dev/null @@ -128,10 +129,10 @@ inject_container() { fi echo "Injecting into CROWDB container $name at $container_ip" pixi run -e iceberg-e2e python tools/load-tpch.py \ - --console-url "http://$container_ip:9090" \ + --console-url "http://$container_ip:6135" \ --credentials-file "$credentials_file" \ - --catalog-uri "http://$container_ip:9092" \ - --s3-endpoint "http://$container_ip:9091" + --catalog-uri "http://$container_ip:8135" \ + --s3-endpoint "http://$container_ip:7135" } clean_container() { diff --git a/container/single-node-container/dev-container.sh.ports.in b/container/single-node-container/dev-container.sh.ports.in new file mode 100644 index 000000000..5e06316e8 --- /dev/null +++ b/container/single-node-container/dev-container.sh.ports.in @@ -0,0 +1,153 @@ +#!/usr/bin/env bash +# Copyright 2026-present Gian <crow.db@outlook.com> +# Licensed under the Apache License, Version 2.0. +set -euo pipefail + +root=$(git rev-parse --show-toplevel) +cd "$root" + +image=${CROWDB_CONTAINER_IMAGE:-crowdb-node:dev} +name=${CROWDB_CONTAINER_NAME:-crowdb-single-node} +data_volume=${CROWDB_CONTAINER_DATA_VOLUME:-${name}-data} + +usage() { + echo 'Usage: dev-container.sh {start|clean|inject [container IP|ID|name]}' >&2 + exit 2 +} + +wait_until_healthy() { + local status + for _ in $(seq 1 240); do + status=$(docker inspect --format '{{.State.Status}}' "$name" 2>/dev/null || true) + if [[ "$status" != running ]]; then + docker logs "$name" >&2 || true + echo "Container $name stopped before becoming healthy" >&2 + return 1 + fi + if docker exec "$name" crowdb-monitor readiness; then + echo "Container $name is healthy" + return 0 + fi + sleep 1 + done + docker logs "$name" >&2 || true + echo "Timed out waiting for container $name to become healthy" >&2 + return 1 +} + +start_container() { + if docker container inspect "$name" >/dev/null 2>&1; then + if [[ $(docker inspect --format '{{.State.Status}}' "$name") == running ]]; then + echo "Container $name is already running" + wait_until_healthy + return + fi + docker start "$name" + wait_until_healthy + return + fi + + if ! docker run --detach --name "$name" -e CROWDB_STARTUP_MODE=single \ + --mount "type=volume,source=$data_volume,target=/opt/crowdb/data" \ + --expose @WEB_BASE@ --expose @ACCESS_SERVER_HTTP_BASE@ --expose @ACCESS_SERVER_ICEBERG_HTTP_BASE@ --expose @ACCESS_SERVER_DATASET_HTTP_BASE@ --expose @ACCESS_SERVER_HEALTH_HTTP_BASE@ \ + "$image"; then + if docker container inspect "$name" >/dev/null 2>&1; then + docker rm --force "$name" >/dev/null + fi + return 1 + fi + if ! wait_until_healthy; then + docker rm --force "$name" >/dev/null 2>&1 || true + return 1 + fi + docker exec "$name" crowdb-monitor credentials show --format env +} + +select_inject_container() { + local target=${1:-${CROWDB_CONTAINER_NAME:-}} + local rows id candidate_name candidate_image source ips ip + local -a candidates=() matches=() + rows=$(docker ps --no-trunc --format '{{.ID}}\t{{.Names}}\t{{.Image}}\t{{.Label "org.opencontainers.image.source"}}') || return + while IFS=$'\t' read -r id candidate_name candidate_image source; do + [[ -n "$id" ]] || continue + if [[ "$source" != https://github.com/buzzcrow/crowdb && + "$candidate_image" != "$image" && "$candidate_image" != *crowdb* ]]; then + continue + fi + ips=$(docker inspect --format '{{range .NetworkSettings.Networks}}{{if .IPAddress}}{{.IPAddress}} {{end}}{{end}}' "$id") || return + candidates+=("$id $candidate_name $candidate_image ${ips:-<no IP>}") + if [[ -z "$target" || "$target" == "$candidate_name" || "$id" == "$target"* ]]; then + matches+=("$id") + else + for ip in $ips; do + if [[ "$target" == "$ip" ]]; then + matches+=("$id") + break + fi + done + fi + done <<< "$rows" + if ((${#matches[@]} != 1)); then + if [[ -n "$target" ]]; then + echo "Cannot uniquely select a running CROWDB container for '$target'." >&2 + else + echo 'Expected one running CROWDB container; specify its IP, ID, or name.' >&2 + fi + if ((${#candidates[@]})); then + echo 'Candidates (ID NAME IMAGE IP):' >&2 + printf '%s\n' "${candidates[@]}" >&2 + else + echo "No running CROWDB containers found; use 'pixi run oci-run-1c'." >&2 + fi + echo 'Usage: pixi run bash container/single-node-container/dev-container.sh inject <container IP|ID|name>' >&2 + return 1 + fi + name=${matches[0]} +} + +inject_container() { + select_inject_container "${1:-}" || return + local credentials_file + credentials_file=$(mktemp "${TMPDIR:-/tmp}/crowdb-container-credentials.XXXXXX") + trap 'rm -f "$credentials_file"' RETURN + docker exec "$name" crowdb-monitor credentials show --format env > "$credentials_file" + local container_ip ip addresses + local -a container_ips + addresses=$(docker inspect --format '{{range .NetworkSettings.Networks}}{{if .IPAddress}}{{.IPAddress}} {{end}}{{end}}' "$name") || return + read -r -a container_ips <<< "$addresses" + container_ip=${container_ips[0]:-} + for ip in "${container_ips[@]}"; do + if [[ "$ip" == "${1:-${CROWDB_CONTAINER_NAME:-}}" ]]; then + container_ip=$ip + break + fi + done + if [[ -z "$container_ip" ]]; then + echo "Container $name has no Docker network address" >&2 + return 1 + fi + echo "Injecting into CROWDB container $name at $container_ip" + pixi run -e iceberg-e2e python tools/load-tpch.py \ + --console-url "http://$container_ip:@WEB_BASE@" \ + --credentials-file "$credentials_file" \ + --catalog-uri "http://$container_ip:@ACCESS_SERVER_ICEBERG_HTTP_BASE@" \ + --s3-endpoint "http://$container_ip:@ACCESS_SERVER_HTTP_BASE@" +} + +clean_container() { + if docker container inspect "$name" >/dev/null 2>&1; then + docker rm --force "$name" + fi + + echo "Preserved data volume $data_volume; images and other containers were not removed." +} + +case "${1:-}" in + start) start_container ;; + inject) + (($# <= 2)) || usage + inject_container "${2:-}" + ;; + clean) clean_container ;; + *) usage ;; +esac diff --git a/container/single-node-container/entrypoint.sh b/container/single-node-container/entrypoint.sh index 6c65fa3fc..e20bb7adc 100644 --- a/container/single-node-container/entrypoint.sh +++ b/container/single-node-container/entrypoint.sh @@ -1,4 +1,5 @@ #!/bin/sh +# Generated by tools/ports/generate.py from ports.rs; edit the .in template. set -eu if ! grep -q ' /opt/crowdb/data ' /proc/self/mountinfo; then @@ -30,7 +31,7 @@ if test "$#" -gt 0; then shift service=${1:-} case "$service" in - kv) program=crowdb-kv-server ;; + paxos-kv) program=crowdb-paxos-kv-server ;; diskdb) program=crowdb-diskdb ;; diskio) program=crowdb-diskio ;; chunkdb) program=crowdb-chunkdb ;; @@ -69,15 +70,14 @@ if test "$port_offset" -gt 0; then # The preview profile is intentionally self-contained. Shift every # declared listener and endpoint together so host-network containers can # coexist while retaining the same relative service layout. - for base in 19000 18000 17000 16000 15200 15100 14000 13000 12200 12100 12000 11200 11100 11000 10101 10100 10000 9095 9094 9093 9092 9091 9090 7401 7400 7301 7300 7200 7102 7101 7100; do + for base in 10135 11135 11136 13135 14135 16135 17135 18135 6135 20135 21135 7135 8135 9135 25135 26135; do + sed -i -E "s/(^|[^0-9])${base}([^0-9]|$)/\\1CROWDB_PORT_${base}_END\\2/g" \ + "$offset_root/profile.toml" "$offset_templates"/*.toml + done + for base in 10135 11135 11136 13135 14135 16135 17135 18135 6135 20135 21135 7135 8135 9135 25135 26135; do next=$((base + port_offset)) - sed -i -E "s/(^|[^0-9])${base}([^0-9]|$)/\\1${next}\\2/g" "$offset_root/profile.toml" - if test -d "$offset_templates"; then - for template in "$offset_templates"/*.toml; do - test -f "$template" || continue - sed -i -E "s/(^|[^0-9])${base}([^0-9]|$)/\\1${next}\\2/g" "$template" - done - fi + sed -i "s/CROWDB_PORT_${base}_END/${next}/g" \ + "$offset_root/profile.toml" "$offset_templates"/*.toml done sed -i "s#/opt/crowdb/etc/templates#$offset_templates#g" "$offset_root/profile.toml" profile="$offset_root/profile.toml" diff --git a/container/single-node-container/entrypoint.sh.ports.in b/container/single-node-container/entrypoint.sh.ports.in new file mode 100644 index 000000000..bef144af1 --- /dev/null +++ b/container/single-node-container/entrypoint.sh.ports.in @@ -0,0 +1,175 @@ +#!/bin/sh +set -eu + +if ! grep -q ' /opt/crowdb/data ' /proc/self/mountinfo; then + echo 'CROWDB preview requires one volume mounted at /opt/crowdb/data' >&2 + exit 1 +fi + +echo 'CROWDB preview data: Docker creates an anonymous volume when none is specified. For data you want to keep across container recreation, use --mount type=volume,source=crowdb-data,target=/opt/crowdb/data.' + +CROWDB_CORE_DIR=/opt/crowdb/data/crash +export CROWDB_CORE_DIR + +case "$(cat /proc/sys/kernel/core_pattern)" in + core|core.*) + if [ "$(ulimit -c)" = 0 ]; then + echo 'CROWDB core files require a nonzero Docker --ulimit core setting.' >&2 + fi + ;; + '|'*) + echo 'CROWDB core dumps use the Docker host collector; inspect the host for dumps.' >&2 + ;; + *) + echo 'CROWDB core_pattern does not use a relative core filename; inspect the Docker host for dumps.' >&2 + ;; +esac + +if test "$#" -gt 0; then + test "$1" = server || { echo 'Expected server SERVICE [arguments]' >&2; exit 1; } + shift + service=${1:-} + case "$service" in + paxos-kv) program=crowdb-paxos-kv-server ;; + diskdb) program=crowdb-diskdb ;; + diskio) program=crowdb-diskio ;; + chunkdb) program=crowdb-chunkdb ;; + chunk-kv) program=crowdb-chunk-kv-server ;; + access) program=crowdb-access-server ;; + web) program=crowdb-web ;; + *) echo 'Unknown server service' >&2; exit 1 ;; + esac + shift + if test -e /opt/crowdb/data/accepted-node.json || test -e /opt/crowdb/data/bootstrap/manifest.json; then + echo 'Standalone server mode requires a separate data root from a managed node' >&2 + exit 1 + fi + mkdir -p "$CROWDB_CORE_DIR" "/opt/crowdb/data/log/$service" + chown crowdb:crowdb /opt/crowdb/data /opt/crowdb/data/log "$CROWDB_CORE_DIR" "/opt/crowdb/data/log/$service" + cd "$CROWDB_CORE_DIR" + exec setpriv --reuid=crowdb --regid=crowdb --init-groups "/opt/crowdb/bin/$program" "$@" +fi + +profile=${CROWDB_PROFILE_PATH:-/opt/crowdb/etc/profile.toml} +port_offset=${CROWDB_PORT_OFFSET:-0} +case "$port_offset" in + ''|*[!0-9]*) echo 'CROWDB_PORT_OFFSET must be a non-negative integer' >&2; exit 1 ;; +esac +if test "$port_offset" -gt 0; then + offset_root=/opt/crowdb/run/port-offset + offset_templates=$offset_root/templates + mkdir -p "$offset_templates" + cp "$profile" "$offset_root/profile.toml" + if test -d /opt/crowdb/etc/templates; then + for template in /opt/crowdb/etc/templates/*.toml; do + test -f "$template" || continue + cp "$template" "$offset_templates/" + done + fi + # The preview profile is intentionally self-contained. Shift every + # declared listener and endpoint together so host-network containers can + # coexist while retaining the same relative service layout. + for base in @PAXOS_KV_SERVER_MGMT_BASE@ @PAXOS_KV_SERVER_RPC_BASE@ @PAXOS_KV_SERVER_RPC_BASE+1@ @DISKDB_HTTP_BASE@ @DISKDB_RPC_BASE@ @CHUNKDB_HTTP_BASE@ @CHUNKDB_RPC_BASE@ @DISKIO_RPC_BASE@ @WEB_BASE@ @CHUNK_KV_HTTP_BASE@ @CHUNK_KV_RPC_BASE@ @ACCESS_SERVER_HTTP_BASE@ @ACCESS_SERVER_ICEBERG_HTTP_BASE@ @ACCESS_SERVER_DATASET_HTTP_BASE@ @ACCESS_SERVER_HEALTH_HTTP_BASE@ @MONITOR_HTTP_BASE@; do + sed -i -E "s/(^|[^0-9])${base}([^0-9]|$)/\\1CROWDB_PORT_${base}_END\\2/g" \ + "$offset_root/profile.toml" "$offset_templates"/*.toml + done + for base in @PAXOS_KV_SERVER_MGMT_BASE@ @PAXOS_KV_SERVER_RPC_BASE@ @PAXOS_KV_SERVER_RPC_BASE+1@ @DISKDB_HTTP_BASE@ @DISKDB_RPC_BASE@ @CHUNKDB_HTTP_BASE@ @CHUNKDB_RPC_BASE@ @DISKIO_RPC_BASE@ @WEB_BASE@ @CHUNK_KV_HTTP_BASE@ @CHUNK_KV_RPC_BASE@ @ACCESS_SERVER_HTTP_BASE@ @ACCESS_SERVER_ICEBERG_HTTP_BASE@ @ACCESS_SERVER_DATASET_HTTP_BASE@ @ACCESS_SERVER_HEALTH_HTTP_BASE@ @MONITOR_HTTP_BASE@; do + next=$((base + port_offset)) + sed -i "s/CROWDB_PORT_${base}_END/${next}/g" \ + "$offset_root/profile.toml" "$offset_templates"/*.toml + done + sed -i "s#/opt/crowdb/etc/templates#$offset_templates#g" "$offset_root/profile.toml" + profile="$offset_root/profile.toml" +fi +ssh_port=$((@SSH_PORT@ + port_offset)) +/opt/crowdb/bin/crowdb-monitor validate "$profile" + +if test -n "${CROWDB_STARTUP_MODE:-}"; then + echo 'CROWDB_STARTUP_MODE was removed; use CROWDB_DEPLOYMENT_MODE and CROWDB_INITIAL_VOTERS.' >&2 + exit 1 +fi +accepted_mode=$(setpriv --reuid=crowdb --regid=crowdb --init-groups \ + /opt/crowdb/bin/crowdb-monitor accept-policy --profile "$profile") +CROWDB_DEPLOYMENT_MODE=$accepted_mode +export CROWDB_DEPLOYMENT_MODE +disks=/opt/crowdb/data/disks +if test -L "$disks" || { test -e "$disks" && ! test -d "$disks"; }; then + echo 'Disk directory must be a real directory, not a symlink or file.' >&2 + exit 1 +fi +mkdir -p "$disks" +chown -h crowdb:crowdb "$disks" +if test -L /opt/crowdb/data/ssh || { test -e /opt/crowdb/data/ssh && ! test -d /opt/crowdb/data/ssh; }; then + echo 'SSH identity directory must be a real directory.' >&2 + exit 1 +fi +mkdir -p /opt/crowdb/data/ssh +chmod 700 /opt/crowdb/data/ssh +ssh_password_auth=yes +if test -n "${CROWDB_SSH_PASSWORD_FILE:-}"; then + test -f "$CROWDB_SSH_PASSWORD_FILE" || { echo 'SSH password file missing' >&2; exit 1; } + CROWDB_SSH_PASSWORD=$(cat "$CROWDB_SSH_PASSWORD_FILE") +fi +if test "$accepted_mode" = local; then + local_password=/opt/crowdb/data/ssh/local-admission-password + if test -L "$local_password" || { test -e "$local_password" && ! test -f "$local_password"; }; then + echo 'Local admission credential must be a regular private file.' >&2 + exit 1 + fi + if test -f "$local_password"; then + test -s "$local_password" && test "$(stat -c %s "$local_password")" -le 4096 && + test "$(stat -c %a "$local_password")" = 600 || { + echo 'Retained local admission credential is empty, oversized or not private.' >&2 + exit 1 + } + fi + if test -z "${CROWDB_SSH_PASSWORD:-}" && test -f "$local_password"; then + CROWDB_SSH_PASSWORD=$(cat "$local_password") + test -n "$CROWDB_SSH_PASSWORD" || { echo 'Retained local admission credential is invalid.' >&2; exit 1; } + fi + CROWDB_SSH_PASSWORD=${CROWDB_SSH_PASSWORD:-crowdb} + password_tmp=$(mktemp /opt/crowdb/data/ssh/.local-admission.XXXXXX) + printf '%s' "$CROWDB_SSH_PASSWORD" > "$password_tmp" + chmod 600 "$password_tmp" + chown crowdb:crowdb "$password_tmp" + mv "$password_tmp" "$local_password" +fi +if test -n "${CROWDB_SSH_PASSWORD:-}"; then + printf 'crowdb:%s\n' "$CROWDB_SSH_PASSWORD" | chpasswd + unset CROWDB_SSH_PASSWORD +elif test "$accepted_mode" = production; then + if ! test -s /opt/crowdb/data/ssh/authorized_keys; then + echo 'Production requires explicit SSH credentials or preinstalled authorized_keys' >&2 + exit 1 + fi + passwd -d crowdb + ssh_password_auth=no +fi +mkdir -p /opt/crowdb/data/ssh /opt/crowdb/run +chmod 700 /opt/crowdb/data/ssh +for name in id_ed25519 ssh_host_ed25519_key; do + if ! test -f "/opt/crowdb/data/ssh/$name"; then + ssh-keygen -q -t ed25519 -N '' -f "/opt/crowdb/data/ssh/$name" + fi +done +if ! test -f /opt/crowdb/data/ssh/authorized_keys; then + cp /opt/crowdb/data/ssh/id_ed25519.pub /opt/crowdb/data/ssh/authorized_keys +fi +chmod 600 /opt/crowdb/data/ssh/authorized_keys +chown -R crowdb:crowdb /opt/crowdb/data/ssh /opt/crowdb/run +if ! test -e /opt/crowdb/.ssh; then ln -s /opt/crowdb/data/ssh /opt/crowdb/.ssh; fi +cat > /opt/crowdb/run/sshd_config <<EOF +Port $ssh_port +HostKey /opt/crowdb/data/ssh/ssh_host_ed25519_key +AuthorizedKeysFile /opt/crowdb/data/ssh/authorized_keys +PermitRootLogin no +AllowUsers crowdb +PasswordAuthentication $ssh_password_auth +KbdInteractiveAuthentication no +UsePAM no +PidFile /opt/crowdb/run/sshd.pid +Subsystem sftp internal-sftp +EOF +/usr/sbin/sshd -f /opt/crowdb/run/sshd_config +exec setpriv --reuid=crowdb --regid=crowdb --init-groups /opt/crowdb/bin/crowdb-monitor run \ + --profile "$profile" --interface "${CROWDB_MANAGEMENT_INTERFACE:-eth0}" diff --git a/container/single-node-container/profile.toml b/container/single-node-container/profile.toml index 0b5cbcba0..707852fb4 100644 --- a/container/single-node-container/profile.toml +++ b/container/single-node-container/profile.toml @@ -1,3 +1,4 @@ +# Generated by tools/ports/generate.py from ports.rs; edit the .in template. version = 1 name = "single-node-container" display_name = "CROWDB Single-Node Container" @@ -27,7 +28,7 @@ store_id = 0 group_id = 0 replica_id = 1 node_id = 1 -rpc_endpoint = "127.0.0.1:10100" +rpc_endpoint = "127.0.0.1:11135" role = "system" [[groups]] @@ -35,7 +36,7 @@ store_id = 0 group_id = 1 replica_id = 2 node_id = 1 -rpc_endpoint = "127.0.0.1:10100" +rpc_endpoint = "127.0.0.1:11135" role = "data" [[disks]] @@ -73,29 +74,29 @@ zone_size_bytes = 17179869184 [[public_endpoints]] id = "s3" bind = "0.0.0.0" -port = 9091 +port = 7135 [[public_endpoints]] id = "iceberg" bind = "0.0.0.0" -port = 9092 +port = 8135 [[public_endpoints]] id = "web" bind = "0.0.0.0" -port = 9090 +port = 6135 [[services]] id = "kv" env = { CROWDB_RDMA_SERVICE_SLOT = "1" } -program = "/opt/crowdb/bin/crowdb-kv-server" -args = ["--root", "/opt/crowdb/data/kv/node-1", "--config", "/opt/crowdb/run/config/kv.toml", "--management-port", "10000", "--ports", "10100,10101", "--instance-id", "1", "--node-id", "1", "--binding-monitor-interval", "1", "--log-dir", "/opt/crowdb/data/log/kv", "--log"] +program = "/opt/crowdb/bin/crowdb-paxos-kv-server" +args = ["--root", "/opt/crowdb/data/kv/node-1", "--config", "/opt/crowdb/run/config/kv.toml", "--management-port", "10135", "--ports", "11135,11136", "--instance-id", "1", "--node-id", "1", "--binding-monitor-interval", "1", "--log-dir", "/opt/crowdb/data/log/kv", "--log"] dependencies = [] -fence_listeners = ["127.0.0.1:10000", "127.0.0.1:10100", "127.0.0.1:10101"] +fence_listeners = ["127.0.0.1:10135", "127.0.0.1:11135", "127.0.0.1:11136"] config_template = "/opt/crowdb/etc/templates/kv.toml" [services.probe] kind = "http" -target = "http://127.0.0.1:10000/health" +target = "http://127.0.0.1:10135/health" timeout_ms = 1000 failure_threshold = 5 [services.restart] @@ -109,11 +110,11 @@ env = { CROWDB_RDMA_SERVICE_SLOT = "2" } program = "/opt/crowdb/bin/crowdb-diskdb" args = ["--config", "/opt/crowdb/run/config/diskdb.toml", "--log-dir", "/opt/crowdb/data/log/diskdb", "--log"] dependencies = ["kv"] -fence_listeners = ["127.0.0.1:11000", "127.0.0.1:11100", "127.0.0.1:11200"] +fence_listeners = ["127.0.0.1:13135", "127.0.0.1:14135"] config_template = "/opt/crowdb/etc/templates/diskdb.toml" [services.probe] kind = "http" -target = "http://127.0.0.1:11100/health" +target = "http://127.0.0.1:13135/health" timeout_ms = 1000 failure_threshold = 5 [services.restart] @@ -127,11 +128,11 @@ env = { CROWDB_RDMA_SERVICE_SLOT = "3" } program = "/opt/crowdb/bin/crowdb-diskio" args = ["--config", "/opt/crowdb/run/config/diskio.toml"] dependencies = ["kv", "diskdb"] -fence_listeners = ["127.0.0.1:13000"] +fence_listeners = ["127.0.0.1:18135"] config_template = "/opt/crowdb/etc/templates/diskio.toml" [services.probe] kind = "rpc-ping" -target = "127.0.0.1:13000" +target = "127.0.0.1:18135" timeout_ms = 1000 failure_threshold = 5 [services.restart] @@ -145,11 +146,11 @@ env = { CROWDB_RDMA_SERVICE_SLOT = "4" } program = "/opt/crowdb/bin/crowdb-chunkdb" args = ["--config", "/opt/crowdb/run/config/chunkdb.toml", "--log-dir", "/opt/crowdb/data/log/chunkdb", "--log"] dependencies = ["kv", "diskdb", "diskio"] -fence_listeners = ["127.0.0.1:12100", "127.0.0.1:12200"] +fence_listeners = ["127.0.0.1:16135", "127.0.0.1:17135"] config_template = "/opt/crowdb/etc/templates/chunkdb.toml" [services.probe] kind = "http" -target = "http://127.0.0.1:12100/ready" +target = "http://127.0.0.1:16135/ready" timeout_ms = 1000 failure_threshold = 5 [services.restart] @@ -163,11 +164,11 @@ env = { CROWDB_RDMA_SERVICE_SLOT = "5" } program = "/opt/crowdb/bin/crowdb-chunk-kv-server" args = ["--config", "/opt/crowdb/run/config/chunk-kv.toml", "--log-dir", "/opt/crowdb/data/log/chunk-kv", "--log"] dependencies = ["kv", "chunkdb", "diskio"] -fence_listeners = ["127.0.0.1:15100", "127.0.0.1:15200"] +fence_listeners = ["127.0.0.1:20135", "127.0.0.1:21135"] config_template = "/opt/crowdb/etc/templates/chunk-kv.toml" [services.probe] kind = "http" -target = "http://127.0.0.1:15100/health" +target = "http://127.0.0.1:20135/health" timeout_ms = 1000 failure_threshold = 5 [services.restart] @@ -179,19 +180,19 @@ backoff_max_ms = 5000 id = "access" program = "/opt/crowdb/bin/crowdb-access-server" args = ["--config", "/opt/crowdb/run/config/access.toml"] -env = { CROWDB_S3_PUBLIC_URI = "http://localhost:9091", CROWDB_S3_REGION = "us-east-1", CROWDB_ICEBERG_PUBLIC_URI = "http://localhost:9092", CROWDB_DATASET_PUBLIC_URI = "http://localhost:9093", CROWDB_ICEBERG_GC_ENABLED = "0", CROWDB_MANAGEMENT_SEEDS = "http://127.0.0.1:10000", CROWDB_ACCESS_HEALTH_LISTEN = "127.0.0.1:9094", CROWDB_ACCESS_LOG_DIR = "/opt/crowdb/data/log/access", CROWDB_RDMA_SERVICE_SLOT = "6" } +env = { CROWDB_S3_PUBLIC_URI = "http://localhost:7135", CROWDB_S3_REGION = "us-east-1", CROWDB_ICEBERG_PUBLIC_URI = "http://localhost:8135", CROWDB_DATASET_PUBLIC_URI = "http://localhost:9135", CROWDB_ICEBERG_GC_ENABLED = "0", CROWDB_MANAGEMENT_SEEDS = "http://127.0.0.1:10135", CROWDB_ACCESS_HEALTH_LISTEN = "127.0.0.1:25135", CROWDB_ACCESS_LOG_DIR = "/opt/crowdb/data/log/access", CROWDB_RDMA_SERVICE_SLOT = "6" } dependencies = ["kv", "chunk-kv", "chunkdb", "diskio"] -fence_listeners = ["127.0.0.1:9092", "127.0.0.1:9091", "127.0.0.1:9093", "127.0.0.1:9094"] +fence_listeners = ["127.0.0.1:8135", "127.0.0.1:7135", "127.0.0.1:9135", "127.0.0.1:25135"] config_template = "/opt/crowdb/etc/templates/access.toml" [services.probe] kind = "http" -target = "http://127.0.0.1:9092/v1/config" +target = "http://127.0.0.1:8135/v1/config" bearer_env = "CROWDB_ICEBERG_READ_TOKEN" timeout_ms = 1000 failure_threshold = 5 [[services.additional_probes]] kind = "http" -target = "http://127.0.0.1:9094/_crowdb/health/ready" +target = "http://127.0.0.1:25135/_crowdb/health/ready" timeout_ms = 1000 failure_threshold = 5 [services.restart] @@ -203,13 +204,13 @@ backoff_max_ms = 5000 id = "web" env = { CROWDB_RDMA_SERVICE_SLOT = "7" } program = "/opt/crowdb/bin/crowdb-web" -args = ["--config", "/opt/crowdb/run/config/crowdb-web.toml", "--node-monitor", "http://127.0.0.1:9095"] +args = ["--config", "/opt/crowdb/run/config/crowdb-web.toml", "--node-monitor", "http://127.0.0.1:26135"] dependencies = ["access"] -fence_listeners = ["127.0.0.1:9090"] +fence_listeners = ["127.0.0.1:6135"] config_template = "/opt/crowdb/etc/templates/crowdb-web.toml" [services.probe] kind = "http" -target = "http://127.0.0.1:9090/healthz" +target = "http://127.0.0.1:6135/healthz" timeout_ms = 1000 failure_threshold = 5 [services.restart] diff --git a/container/single-node-container/profile.toml.ports.in b/container/single-node-container/profile.toml.ports.in new file mode 100644 index 000000000..7e62f04e9 --- /dev/null +++ b/container/single-node-container/profile.toml.ports.in @@ -0,0 +1,218 @@ +version = 1 +name = "single-node-container" +display_name = "CROWDB Single-Node Container" +placement_mode = "unsafe-colocated" +s3_tenant = "preview" +iceberg_catalog = "preview" + +[paths] +install_root = "/opt/crowdb" +bin_root = "/opt/crowdb/bin" +template_root = "/opt/crowdb/etc/templates" +data_root = "/opt/crowdb/data" +run_root = "/opt/crowdb/run" +log_root = "/opt/crowdb/data/log" + +[logs] +max_file_bytes = 31457280 +max_files = 5 +mirror_warnings_to_stderr = true + +[[nodes]] +node_id = 1 +rack_id = 1 + +[[groups]] +store_id = 0 +group_id = 0 +replica_id = 1 +node_id = 1 +rpc_endpoint = "127.0.0.1:@PAXOS_KV_SERVER_RPC_BASE@" +role = "system" + +[[groups]] +store_id = 0 +group_id = 1 +replica_id = 2 +node_id = 1 +rpc_endpoint = "127.0.0.1:@PAXOS_KV_SERVER_RPC_BASE@" +role = "data" + +[[disks]] +disk_id = "00000000000000010000000000000001" +disk_group_id = 101 +node_id = 1 +path = "/opt/crowdb/data/disks/disk-0001.img" +capacity_bytes = 17179869184 +zone_size_bytes = 17179869184 + +[[disks]] +disk_id = "00000000000000010000000000000002" +disk_group_id = 101 +node_id = 1 +path = "/opt/crowdb/data/disks/disk-0002.img" +capacity_bytes = 17179869184 +zone_size_bytes = 17179869184 + +[[disks]] +disk_id = "00000000000000010000000000000003" +disk_group_id = 101 +node_id = 1 +path = "/opt/crowdb/data/disks/disk-0003.img" +capacity_bytes = 17179869184 +zone_size_bytes = 17179869184 + +[[disks]] +disk_id = "00000000000000010000000000000004" +disk_group_id = 101 +node_id = 1 +path = "/opt/crowdb/data/disks/disk-0004.img" +capacity_bytes = 17179869184 +zone_size_bytes = 17179869184 + +[[public_endpoints]] +id = "s3" +bind = "0.0.0.0" +port = @ACCESS_SERVER_HTTP_BASE@ + +[[public_endpoints]] +id = "iceberg" +bind = "0.0.0.0" +port = @ACCESS_SERVER_ICEBERG_HTTP_BASE@ + +[[public_endpoints]] +id = "web" +bind = "0.0.0.0" +port = @WEB_BASE@ + +[[services]] +id = "kv" +env = { CROWDB_RDMA_SERVICE_SLOT = "1" } +program = "/opt/crowdb/bin/crowdb-paxos-kv-server" +args = ["--root", "/opt/crowdb/data/kv/node-1", "--config", "/opt/crowdb/run/config/kv.toml", "--management-port", "@PAXOS_KV_SERVER_MGMT_BASE@", "--ports", "@PAXOS_KV_SERVER_RPC_BASE@,@PAXOS_KV_SERVER_RPC_BASE+1@", "--instance-id", "1", "--node-id", "1", "--binding-monitor-interval", "1", "--log-dir", "/opt/crowdb/data/log/kv", "--log"] +dependencies = [] +fence_listeners = ["127.0.0.1:@PAXOS_KV_SERVER_MGMT_BASE@", "127.0.0.1:@PAXOS_KV_SERVER_RPC_BASE@", "127.0.0.1:@PAXOS_KV_SERVER_RPC_BASE+1@"] +config_template = "/opt/crowdb/etc/templates/kv.toml" +[services.probe] +kind = "http" +target = "http://127.0.0.1:@PAXOS_KV_SERVER_MGMT_BASE@/health" +timeout_ms = 1000 +failure_threshold = 5 +[services.restart] +max_attempts = 5 +backoff_base_ms = 250 +backoff_max_ms = 5000 + +[[services]] +id = "diskdb" +env = { CROWDB_RDMA_SERVICE_SLOT = "2" } +program = "/opt/crowdb/bin/crowdb-diskdb" +args = ["--config", "/opt/crowdb/run/config/diskdb.toml", "--log-dir", "/opt/crowdb/data/log/diskdb", "--log"] +dependencies = ["kv"] +fence_listeners = ["127.0.0.1:@DISKDB_HTTP_BASE@", "127.0.0.1:@DISKDB_RPC_BASE@"] +config_template = "/opt/crowdb/etc/templates/diskdb.toml" +[services.probe] +kind = "http" +target = "http://127.0.0.1:@DISKDB_HTTP_BASE@/health" +timeout_ms = 1000 +failure_threshold = 5 +[services.restart] +max_attempts = 5 +backoff_base_ms = 250 +backoff_max_ms = 5000 + +[[services]] +id = "diskio" +env = { CROWDB_RDMA_SERVICE_SLOT = "3" } +program = "/opt/crowdb/bin/crowdb-diskio" +args = ["--config", "/opt/crowdb/run/config/diskio.toml"] +dependencies = ["kv", "diskdb"] +fence_listeners = ["127.0.0.1:@DISKIO_RPC_BASE@"] +config_template = "/opt/crowdb/etc/templates/diskio.toml" +[services.probe] +kind = "rpc-ping" +target = "127.0.0.1:@DISKIO_RPC_BASE@" +timeout_ms = 1000 +failure_threshold = 5 +[services.restart] +max_attempts = 5 +backoff_base_ms = 250 +backoff_max_ms = 5000 + +[[services]] +id = "chunkdb" +env = { CROWDB_RDMA_SERVICE_SLOT = "4" } +program = "/opt/crowdb/bin/crowdb-chunkdb" +args = ["--config", "/opt/crowdb/run/config/chunkdb.toml", "--log-dir", "/opt/crowdb/data/log/chunkdb", "--log"] +dependencies = ["kv", "diskdb", "diskio"] +fence_listeners = ["127.0.0.1:@CHUNKDB_HTTP_BASE@", "127.0.0.1:@CHUNKDB_RPC_BASE@"] +config_template = "/opt/crowdb/etc/templates/chunkdb.toml" +[services.probe] +kind = "http" +target = "http://127.0.0.1:@CHUNKDB_HTTP_BASE@/ready" +timeout_ms = 1000 +failure_threshold = 5 +[services.restart] +max_attempts = 5 +backoff_base_ms = 250 +backoff_max_ms = 5000 + +[[services]] +id = "chunk-kv" +env = { CROWDB_RDMA_SERVICE_SLOT = "5" } +program = "/opt/crowdb/bin/crowdb-chunk-kv-server" +args = ["--config", "/opt/crowdb/run/config/chunk-kv.toml", "--log-dir", "/opt/crowdb/data/log/chunk-kv", "--log"] +dependencies = ["kv", "chunkdb", "diskio"] +fence_listeners = ["127.0.0.1:@CHUNK_KV_HTTP_BASE@", "127.0.0.1:@CHUNK_KV_RPC_BASE@"] +config_template = "/opt/crowdb/etc/templates/chunk-kv.toml" +[services.probe] +kind = "http" +target = "http://127.0.0.1:@CHUNK_KV_HTTP_BASE@/health" +timeout_ms = 1000 +failure_threshold = 5 +[services.restart] +max_attempts = 5 +backoff_base_ms = 250 +backoff_max_ms = 5000 + +[[services]] +id = "access" +program = "/opt/crowdb/bin/crowdb-access-server" +args = ["--config", "/opt/crowdb/run/config/access.toml"] +env = { CROWDB_S3_PUBLIC_URI = "http://localhost:@ACCESS_SERVER_HTTP_BASE@", CROWDB_S3_REGION = "us-east-1", CROWDB_ICEBERG_PUBLIC_URI = "http://localhost:@ACCESS_SERVER_ICEBERG_HTTP_BASE@", CROWDB_DATASET_PUBLIC_URI = "http://localhost:@ACCESS_SERVER_DATASET_HTTP_BASE@", CROWDB_ICEBERG_GC_ENABLED = "0", CROWDB_MANAGEMENT_SEEDS = "http://127.0.0.1:@PAXOS_KV_SERVER_MGMT_BASE@", CROWDB_ACCESS_HEALTH_LISTEN = "127.0.0.1:@ACCESS_SERVER_HEALTH_HTTP_BASE@", CROWDB_ACCESS_LOG_DIR = "/opt/crowdb/data/log/access", CROWDB_RDMA_SERVICE_SLOT = "6" } +dependencies = ["kv", "chunk-kv", "chunkdb", "diskio"] +fence_listeners = ["127.0.0.1:@ACCESS_SERVER_ICEBERG_HTTP_BASE@", "127.0.0.1:@ACCESS_SERVER_HTTP_BASE@", "127.0.0.1:@ACCESS_SERVER_DATASET_HTTP_BASE@", "127.0.0.1:@ACCESS_SERVER_HEALTH_HTTP_BASE@"] +config_template = "/opt/crowdb/etc/templates/access.toml" +[services.probe] +kind = "http" +target = "http://127.0.0.1:@ACCESS_SERVER_ICEBERG_HTTP_BASE@/v1/config" +bearer_env = "CROWDB_ICEBERG_READ_TOKEN" +timeout_ms = 1000 +failure_threshold = 5 +[[services.additional_probes]] +kind = "http" +target = "http://127.0.0.1:@ACCESS_SERVER_HEALTH_HTTP_BASE@/_crowdb/health/ready" +timeout_ms = 1000 +failure_threshold = 5 +[services.restart] +max_attempts = 5 +backoff_base_ms = 250 +backoff_max_ms = 5000 + +[[services]] +id = "web" +env = { CROWDB_RDMA_SERVICE_SLOT = "7" } +program = "/opt/crowdb/bin/crowdb-web" +args = ["--config", "/opt/crowdb/run/config/crowdb-web.toml", "--node-monitor", "http://127.0.0.1:@MONITOR_HTTP_BASE@"] +dependencies = ["access"] +fence_listeners = ["127.0.0.1:@WEB_BASE@"] +config_template = "/opt/crowdb/etc/templates/crowdb-web.toml" +[services.probe] +kind = "http" +target = "http://127.0.0.1:@WEB_BASE@/healthz" +timeout_ms = 1000 +failure_threshold = 5 +[services.restart] +max_attempts = 5 +backoff_base_ms = 250 +backoff_max_ms = 5000 diff --git a/container/single-node-container/templates/access.toml b/container/single-node-container/templates/access.toml index 961a58459..c7b0b2bd7 100644 --- a/container/single-node-container/templates/access.toml +++ b/container/single-node-container/templates/access.toml @@ -1,3 +1,4 @@ +# Generated by tools/ports/generate.py from ports.rs; edit the .in template. # Access configuration shared by the S3 and Iceberg listeners. # Secrets stay in /opt/crowdb/data/secrets/server.env. @@ -6,7 +7,7 @@ mode = "local" max_node_failures = 0 [common] -management_seeds = ["http://127.0.0.1:10000"] +management_seeds = ["http://127.0.0.1:10135"] diskio_connections_per_endpoint = 1 diskio_rpc_workers = 1 @@ -32,7 +33,7 @@ chunk_capacity_bytes = 67108864 [s3] large_mirror_copies = 1 -listen = "0.0.0.0:9091" +listen = "0.0.0.0:7135" tenant = "preview" region = "us-east-1" small_object_limit = 8388608 @@ -55,7 +56,7 @@ chunk_capacity_bytes = 67108864 [iceberg] large_mirror_copies = 1 -listen = "0.0.0.0:9092" +listen = "0.0.0.0:8135" native_budget_bytes = 268435456 max_chunk_size = 67108864 @@ -86,7 +87,7 @@ minimum_retention_ms = 604800000 catalogs = [] [dataset] -listen = "0.0.0.0:9093" +listen = "0.0.0.0:9135" max_in_flight = 8 request_timeout_seconds = 60 max_batch_size = 10000 diff --git a/container/single-node-container/templates/access.toml.ports.in b/container/single-node-container/templates/access.toml.ports.in new file mode 100644 index 000000000..69007bc6d --- /dev/null +++ b/container/single-node-container/templates/access.toml.ports.in @@ -0,0 +1,92 @@ +# Access configuration shared by the S3 and Iceberg listeners. +# Secrets stay in /opt/crowdb/data/secrets/server.env. + +[deployment] +mode = "local" +max_node_failures = 0 + +[common] +management_seeds = ["http://127.0.0.1:@PAXOS_KV_SERVER_MGMT_BASE@"] +diskio_connections_per_endpoint = 1 +diskio_rpc_workers = 1 + +[read] +stream_window_bytes = 1048576 +stream_slots = 3 +global_stream_bytes = 268435456 +recovery_memory_bytes = 268435456 + +[small_write] +threshold_ratio = 0.9 +disk_block_bytes = 1048576 +conversion_enabled = false +mirror_copies = 1 +ec_data = 2 +ec_code = 1 +memory_budget_bytes = 1342177280 +queue_capacity = 1024 +min_pipelines = 1 +max_pipelines = 32 +max_batch_bytes = 1048576 +chunk_capacity_bytes = 67108864 + +[s3] +large_mirror_copies = 1 +listen = "0.0.0.0:@ACCESS_SERVER_HTTP_BASE@" +tenant = "preview" +region = "us-east-1" +small_object_limit = 8388608 +list_scan_items = 1001 +list_scan_bytes = 4194304 +continuation_ttl_seconds = 900 +native_budget_bytes = 268435456 +cleanup_backlog_limit = 10000 +ec_data = 2 +ec_code = 1 +max_chunk_size = 67108864 + +[s3.small_write] +conversion_enabled = false +mirror_copies = 1 +disk_block_bytes = 1048576 +ec_data = 2 +ec_code = 1 +chunk_capacity_bytes = 67108864 + +[iceberg] +large_mirror_copies = 1 +listen = "0.0.0.0:@ACCESS_SERVER_ICEBERG_HTTP_BASE@" +native_budget_bytes = 268435456 +max_chunk_size = 67108864 + +[iceberg.small_write] +conversion_enabled = false +mirror_copies = 1 +disk_block_bytes = 1048576 +ec_data = 2 +ec_code = 1 +chunk_capacity_bytes = 33554432 + +[iceberg.gc] +enabled = false +interval_ms = 60000 +step_bytes = 8388608 +step_ms = 1000 +page_items = 64 +page_bytes = 49152 +concurrency = 1 +kv_bytes = 67108864 +kv_requests = 128 +chunk_bytes = 8388608 +chunk_requests = 128 +retry_base_ms = 1000 +retry_max_ms = 60000 +corruption_attempts = 3 +minimum_retention_ms = 604800000 +catalogs = [] + +[dataset] +listen = "0.0.0.0:@ACCESS_SERVER_DATASET_HTTP_BASE@" +max_in_flight = 8 +request_timeout_seconds = 60 +max_batch_size = 10000 diff --git a/container/single-node-container/templates/chunk-kv.toml b/container/single-node-container/templates/chunk-kv.toml index 6c94e9c27..035a58e29 100644 --- a/container/single-node-container/templates/chunk-kv.toml +++ b/container/single-node-container/templates/chunk-kv.toml @@ -1,10 +1,11 @@ +# Generated by tools/ports/generate.py from ports.rs; edit the .in template. instance_id = 1 node_id = {{node.0.id}} -rpc_listen_addr = "127.0.0.1:15200" -rpc_advertise_addr = "127.0.0.1:15200" -http_listen_addr = "127.0.0.1:15100" -http_advertise_addr = "127.0.0.1:15100" -group0_mgmt_seeds = ["http://127.0.0.1:10000"] +rpc_listen_addr = "127.0.0.1:21135" +rpc_advertise_addr = "127.0.0.1:21135" +http_listen_addr = "127.0.0.1:20135" +http_advertise_addr = "127.0.0.1:20135" +group0_mgmt_seeds = ["http://127.0.0.1:10135"] catalog_refresh_interval_ms = 200 rpc_workers = 1 diff --git a/container/single-node-container/templates/chunk-kv.toml.ports.in b/container/single-node-container/templates/chunk-kv.toml.ports.in new file mode 100644 index 000000000..7849d209f --- /dev/null +++ b/container/single-node-container/templates/chunk-kv.toml.ports.in @@ -0,0 +1,32 @@ +instance_id = 1 +node_id = {{node.0.id}} +rpc_listen_addr = "127.0.0.1:@CHUNK_KV_RPC_BASE@" +rpc_advertise_addr = "127.0.0.1:@CHUNK_KV_RPC_BASE@" +http_listen_addr = "127.0.0.1:@CHUNK_KV_HTTP_BASE@" +http_advertise_addr = "127.0.0.1:@CHUNK_KV_HTTP_BASE@" +group0_mgmt_seeds = ["http://127.0.0.1:@PAXOS_KV_SERVER_MGMT_BASE@"] +catalog_refresh_interval_ms = 200 +rpc_workers = 1 + +[balance] +enabled = false +target_partitions_per_owner = 1 +target_partition_bytes = 9223372036854775807 +minimum_weighted_improvement_percent = 100 +cooldown_ms = 9223372036854775807 +max_owner_request_rate = 0 + +[storage] +metadata_store_id = 0 +stream_mirror_copies = 1 +tree_chunk_capacity_bytes = 16777216 +stream_chunk_capacity_bytes = 16777216 +diskio_connections_per_endpoint = 1 +diskio_rpc_workers = 1 + +[bootstrap_partition] +partition_id = { high = 1, low = 1 } +tree_id = 1 +stream_name = { high = 2, low = 1 } +owner_epoch = 1 +metadata_group_id = {{group.1.id}} diff --git a/container/single-node-container/templates/chunkdb.toml b/container/single-node-container/templates/chunkdb.toml index 96ca1a04a..836fb26cf 100644 --- a/container/single-node-container/templates/chunkdb.toml +++ b/container/single-node-container/templates/chunkdb.toml @@ -1,3 +1,4 @@ +# Generated by tools/ports/generate.py from ports.rs; edit the .in template. [slot_bootstrap] service_instances = [1] storage_groups = [{ store_id = 0, group_id = 1 }] @@ -8,10 +9,10 @@ max_node_failures = 0 [server] rpc_workers = 1 -http_listen_addr = "127.0.0.1:12100" -rpc_listen_addr = "127.0.0.1:12200" +http_listen_addr = "127.0.0.1:16135" +rpc_listen_addr = "127.0.0.1:17135" instance_id = "1" -kv_server_mgmt_seeds = ["http://127.0.0.1:10000"] +kv_server_mgmt_seeds = ["http://127.0.0.1:10135"] keepalive_interval_secs = 1 kv_pool_size = 1 kv_rpc_workers = 1 diff --git a/container/single-node-container/templates/chunkdb.toml.ports.in b/container/single-node-container/templates/chunkdb.toml.ports.in new file mode 100644 index 000000000..e3961096d --- /dev/null +++ b/container/single-node-container/templates/chunkdb.toml.ports.in @@ -0,0 +1,38 @@ +[slot_bootstrap] +service_instances = [1] +storage_groups = [{ store_id = 0, group_id = 1 }] + +[deployment] +mode = "local" +max_node_failures = 0 + +[server] +rpc_workers = 1 +http_listen_addr = "127.0.0.1:@CHUNKDB_HTTP_BASE@" +rpc_listen_addr = "127.0.0.1:@CHUNKDB_RPC_BASE@" +instance_id = "1" +kv_server_mgmt_seeds = ["http://127.0.0.1:@PAXOS_KV_SERVER_MGMT_BASE@"] +keepalive_interval_secs = 1 +kv_pool_size = 1 +kv_rpc_workers = 1 +diskdb_pool_size = 1 +diskdb_rpc_workers = 1 + +[conversion_io] +normal_connections_per_endpoint = 1 +priority_connections_per_endpoint = 1 +rpc_workers = 1 + +[topology] +refresh_interval_secs = 1 + +[range_guard] +allow_all_when_empty = false + +[lifecycle] +cache_capacity = 10000 +sweep_chunk_lock_interval_secs = 60 +lock_hold_warn_threshold_ms = 1000 + +[placement] +mode = "unsafe_colocated" diff --git a/container/single-node-container/templates/crowdb-web.toml b/container/single-node-container/templates/crowdb-web.toml index e5b500aaa..fa812c214 100644 --- a/container/single-node-container/templates/crowdb-web.toml +++ b/container/single-node-container/templates/crowdb-web.toml @@ -1,8 +1,9 @@ +# Generated by tools/ports/generate.py from ports.rs; edit the .in template. version = 1 mode = "docker" bind = "0.0.0.0" -port = 9090 -group0_management_seeds = ["http://127.0.0.1:10000"] +port = 6135 +group0_management_seeds = ["http://127.0.0.1:10135"] ui_root = "{{install_root}}/ui" monitor_status = "{{run_root}}/status/monitor.json" log_dir = "{{log_root}}/web" diff --git a/container/single-node-container/templates/crowdb-web.toml.ports.in b/container/single-node-container/templates/crowdb-web.toml.ports.in new file mode 100644 index 000000000..9e1bd69d9 --- /dev/null +++ b/container/single-node-container/templates/crowdb-web.toml.ports.in @@ -0,0 +1,10 @@ +version = 1 +mode = "docker" +bind = "0.0.0.0" +port = @WEB_BASE@ +group0_management_seeds = ["http://127.0.0.1:@PAXOS_KV_SERVER_MGMT_BASE@"] +ui_root = "{{install_root}}/ui" +monitor_status = "{{run_root}}/status/monitor.json" +log_dir = "{{log_root}}/web" +log_max_file_mb = 30 +log_max_files = 5 diff --git a/container/single-node-container/templates/diskdb.toml b/container/single-node-container/templates/diskdb.toml index 039ab6e49..f1dfddf00 100644 --- a/container/single-node-container/templates/diskdb.toml +++ b/container/single-node-container/templates/diskdb.toml @@ -1,12 +1,12 @@ +# Generated by tools/ports/generate.py from ports.rs; edit the .in template. [server] rpc_workers = 1 kv_pool_size = 1 kv_rpc_workers = 1 -listen_addr = "127.0.0.1:11000" -http_listen_addr = "127.0.0.1:11100" -rpc_listen_addr = "127.0.0.1:11200" +http_listen_addr = "127.0.0.1:13135" +rpc_listen_addr = "127.0.0.1:14135" instance_id = "{{node.0.id}}" -kv_server_mgmt_seeds = ["http://127.0.0.1:10000"] +kv_server_mgmt_seeds = ["http://127.0.0.1:10135"] [storage] zone_size_bytes = {{disk.0.capacity_bytes}} diff --git a/container/single-node-container/templates/diskdb.toml.ports.in b/container/single-node-container/templates/diskdb.toml.ports.in new file mode 100644 index 000000000..ccea9e6ec --- /dev/null +++ b/container/single-node-container/templates/diskdb.toml.ports.in @@ -0,0 +1,64 @@ +[server] +rpc_workers = 1 +kv_pool_size = 1 +kv_rpc_workers = 1 +http_listen_addr = "127.0.0.1:@DISKDB_HTTP_BASE@" +rpc_listen_addr = "127.0.0.1:@DISKDB_RPC_BASE@" +instance_id = "{{node.0.id}}" +kv_server_mgmt_seeds = ["http://127.0.0.1:@PAXOS_KV_SERVER_MGMT_BASE@"] + +[storage] +zone_size_bytes = {{disk.0.capacity_bytes}} +block_size_bytes = 1048576 +allocate_granularity = 1048576 +zone_rotate_count = 4 +cas_retry_limit = 100 + +[heartbeat] +interval_secs = 10 +miss_threshold = 3 +temp_failure_timeout_secs = 900 + +[persistence] +free_batch_enabled = false +free_flush_max_batch = 256 +compaction_cadence_secs = 300 +snapshot_compaction_threshold = 4096 +load_concurrency = 16 + +[scanner] +scan_interval_secs = 600 +tentative_owner_scan_interval_secs = 600 +tentative_owner_zone_delay_secs = 3 +tentative_owner_grace_secs = 86400 +reverify_delay_ms = 1000 + +[scanner.ghost] +detect = true +auto_correct = false + +[scanner.integrity] +verify = true +detect_owner_mismatch = false + +[rebalance] +enabled = true +plan_interval_secs = 300 +imbalance_threshold_pct = 20 +max_jobs_per_cycle = 1 +zone_delay_secs = 3 + +[allocator] +load_aware = true +load_aware_weight = "free_bytes" + +[sync] +group0_store_id = 0 +group0_group_id = 0 +sync_interval_secs = 10 + +[reporting] +interval_secs = 10 + +[notify] +notify_enabled = false diff --git a/container/single-node-container/templates/diskio.toml b/container/single-node-container/templates/diskio.toml index 854290858..cc62eda7b 100644 --- a/container/single-node-container/templates/diskio.toml +++ b/container/single-node-container/templates/diskio.toml @@ -1,3 +1,4 @@ +# Generated by tools/ports/generate.py from ports.rs; edit the .in template. [server] bind_address = "127.0.0.1" listen_port = 13000 @@ -11,7 +12,7 @@ thread_pool_size = 1 sq_entries = 256 [group0] -kv_seeds = ["http://127.0.0.1:10000"] +kv_seeds = ["http://127.0.0.1:10135"] instance_id = {{node.0.id}} rack_id = {{node.0.rack_id}} disk_group_id = 101 diff --git a/container/single-node-container/templates/diskio.toml.ports.in b/container/single-node-container/templates/diskio.toml.ports.in new file mode 100644 index 000000000..fdf18b4ac --- /dev/null +++ b/container/single-node-container/templates/diskio.toml.ports.in @@ -0,0 +1,43 @@ +[server] +bind_address = "127.0.0.1" +listen_port = 13000 +rpc_workers = 1 +node_id = {{node.0.id}} +dummy_disk_type = "null" +o_direct = true + +[engine] +thread_pool_size = 1 +sq_entries = 256 + +[group0] +kv_seeds = ["http://127.0.0.1:@PAXOS_KV_SERVER_MGMT_BASE@"] +instance_id = {{node.0.id}} +rack_id = {{node.0.rack_id}} +disk_group_id = 101 +sync_interval_ms = 5000 +auto_discover_disks = false + +[metrics] +log_dir = "{{log_root}}/diskio" +interval_secs = 5 + +[[disk]] +id = "1:1" +path = "{{disk.0.path}}" +zone_capacity = {{disk.0.capacity_bytes}} + +[[disk]] +id = "1:2" +path = "{{disk.1.path}}" +zone_capacity = {{disk.1.capacity_bytes}} + +[[disk]] +id = "1:3" +path = "{{disk.2.path}}" +zone_capacity = {{disk.2.capacity_bytes}} + +[[disk]] +id = "1:4" +path = "{{disk.3.path}}" +zone_capacity = {{disk.3.capacity_bytes}} diff --git a/container/single-node-container/templates/kv.toml b/container/single-node-container/templates/kv.toml index 54006f518..3485509ab 100644 --- a/container/single-node-container/templates/kv.toml +++ b/container/single-node-container/templates/kv.toml @@ -1,3 +1,4 @@ +# Generated by tools/ports/generate.py from ports.rs; edit the .in template. wal_early_ack = true async_engine_apply = true diff --git a/container/single-node-container/templates/kv.toml.ports.in b/container/single-node-container/templates/kv.toml.ports.in new file mode 100644 index 000000000..54006f518 --- /dev/null +++ b/container/single-node-container/templates/kv.toml.ports.in @@ -0,0 +1,6 @@ +wal_early_ack = true +async_engine_apply = true + +[server] +rpc_workers = 1 +peer_pool_size = 1 diff --git a/container/single-node-container/tests/image-smoke.sh b/container/single-node-container/tests/image-smoke.sh index cd10a3fa9..7c862233f 100644 --- a/container/single-node-container/tests/image-smoke.sh +++ b/container/single-node-container/tests/image-smoke.sh @@ -17,12 +17,17 @@ test "$(docker image inspect --format '{{index .Config.Labels "org.opencontainer volumes=$(docker image inspect --format '{{json .Config.Volumes}}' "$image") jq -e 'has("/opt/crowdb/data")' <<<"$volumes" >/dev/null exposed=$(docker image inspect --format '{{json .Config.ExposedPorts}}' "$image") -for port in 9092 9091 9090 9095 2222; do - jq -e --arg port "$port/tcp" 'has($port)' <<<"$exposed" >/dev/null -done -for port in 10000 13000 15200; do - jq -e --arg port "$port/tcp" 'has($port) | not' <<<"$exposed" >/dev/null -done +PYTHONPATH=tools/ports python - "$exposed" <<'PY' +import json +import sys +import default_ports + +expected = {f"{port}/tcp" for name, port in vars(default_ports).items() + if name.endswith("_BASE") or name == "SSH_PORT"} +actual = set(json.loads(sys.argv[1])) +if actual != expected: + raise SystemExit(f"image ports differ: missing={expected - actual}, extra={actual - expected}") +PY docker run --rm --network none --entrypoint /opt/crowdb/bin/crowdb-monitor "$image" validate /opt/crowdb/etc/profile.toml docker run --rm --network none --entrypoint /bin/sh "$image" -ec ' diff --git a/doc/assets/chunk-io.svg b/doc/assets/chunk-io.svg new file mode 100644 index 000000000..bec78a03d --- /dev/null +++ b/doc/assets/chunk-io.svg @@ -0,0 +1,156 @@ +<svg xmlns="http://www.w3.org/2000/svg" width="1400" height="1500" viewBox="0 0 1400 1500" role="img" aria-labelledby="title desc"> +<!-- Copyright 2026-present Gian; Licensed under the Apache License, Version 2.0. --> +<title id="title">Chunk write pathsLarge write: one dedicated pipeline and active chunk per object. Small write: N objects across M pipelines and shared chunks. Direct EC and mirror-to-EC conversion have different completion paths. + + + + + + + + + + +Chunk write paths +Large writes own their pipeline and chunks; small writes share a bounded pool. + +Large write · 1:1 · dedicated pipeline and active chunk + +Large object A +Streaming input + +Dedicated pipeline A +Bounded framing and buffers + +Chunk A +Object A only · mirror or EC + +Seal → location(s) +Complete this object + + + + +Large object B +Streaming input + +Dedicated pipeline B +Bounded framing and buffers + +Chunk B +Object B only · mirror or EC + +Seal → location(s) +Complete this object + + + +A larger object seals and rotates through additional dedicated chunks; it never shares them with another object. + +Small write · N:M · M pipelines, M active shared chunks +N objects + +Object 1 + + +Object 2 + + +… + + +Object N + + +Shared writer pool +Bounded admission +Choose a pipeline + + +Pipeline 1 +Batch → frame → mirror writes + + +Shared chunk 1 + +obj 1 + +obj 4 + +… + + +Pipeline 2 +Batch → frame → mirror writes + + +Shared chunk 2 + +obj 2 + +obj 5 + +… + + +Pipeline M +Batch → frame → mirror writes + + +Shared chunk M + +obj 3 + +… + +obj N +Each pipeline owns one active chunk. Objects keep independent locations and completion results. + +Large write: direct EC +k data shards + m parity shards + +Data shards +Arrive incrementally + +DiskIO data writes +Submit each data shard + +EC parity + DiskIO +Accumulate parity as data arrives +Write parity at strip end + + + + + +Join data + parity → seal chunk +Next-strip data can overlap the preceding strip’s parity I/O. +Partial tails retain their exact valid byte length. + +Small write: mirror → EC +Optional conversion · 8 data + 4 parity + +Mirror strips +Write every replica + +Parity accumulators +Fold each +closed strip + +Parity DiskIO +Write + fsync + + + + + +ChunkDB publishes the 8+4 layout +One healthy replica per mirror set becomes a data shard +Data shards reuse the stored mirror bytes. +Mirrors remain authoritative until EC publication. +Incomplete or failed conversion stays mirrored. + +Payload completion → per-object locations → access-library publication +Iceberg, Dataset, and S3 own visibility. Shared writers confirm the durable cursor for immediately readable publication. +CROWDB + diff --git a/doc/assets/chunk-kv.svg b/doc/assets/chunk-kv.svg new file mode 100644 index 000000000..70df36b45 --- /dev/null +++ b/doc/assets/chunk-kv.svg @@ -0,0 +1,115 @@ + + +A range-based tree distributed across nodesDynamic splits create partitions; balancing places those partitions on different serving nodes. + + + + + + + + + + +A range-based tree distributed across nodes +Dynamic splits create partitions; balancing places those partitions on different serving nodes. + +Range-based tree +Ordered key space [a, z) + +[a, m) + +[m, z) + + +split at m + + + +split at t; then balance + +Node 1 + +Partition 1 · [a, m) +Own range, Tree, and WAL + + +WAL +Ordered mutations + + +ChunkStream + + + +crowdb-tree +Ordered key/value state + + +Chunk backend + + +Node 2 + +Partition 2 · [m, t) +Own range, Tree, and WAL + + +WAL +Ordered mutations + + +ChunkStream + + + +crowdb-tree +Ordered key/value state + + +Chunk backend + + +Node 3 + +Partition 3 · [t, z) +Own range, Tree, and WAL + + +WAL +Ordered mutations + + +ChunkStream + + + +crowdb-tree +Ordered key/value state + + +Chunk backend + + +Shared Chunk layer + +WAL chunks + +Tree page chunks + +WAL chunks + +Tree page chunks + +WAL chunks + +Tree page chunks + + + + + + +Split changes range boundaries; balance changes serving placement. Both storage paths retain their shared backing. +CROWDB + diff --git a/doc/assets/chunk-placement.svg b/doc/assets/chunk-placement.svg new file mode 100644 index 000000000..7ca6c78f3 --- /dev/null +++ b/doc/assets/chunk-placement.svg @@ -0,0 +1,103 @@ + + +Chunk placement across nodes and disksThe client obtains a persisted layout, then writes payload directly through DiskIO. + + + + + + + + + + +Chunk placement across nodes and disks +The client obtains a persisted layout, then writes payload directly through DiskIO. + +Chunk client +Allocate or query a chunk; use its locations + +How a chunk is allocated + +ChunkDB +Choose replica / EC placement +Build the chunk layout + +DiskDB +Allocate physical blocks + +Paxos-KV +Persisted Chunk metadata +Nonzero store groups + +1. Allocate / query + +4. Return Chunk ID, strips, and segment locations + +2. Allocate blocks + + +3. Persist the complete layout + +Payload · TCP / RDMA +One logical chunk maps to physical segments across nodes and disks. + + +Node 1 + +DiskIO +Local byte I/O + + +Disk 1 +Segment + + +Disk 2 +Blocks + + +Disk 3 +Blocks + + + +Node 2 + +DiskIO +Local byte I/O + + +Disk 1 +Segment + + +Disk 2 +Blocks + + +Disk 3 +Blocks + + + +Node 3 + +DiskIO +Local byte I/O + + +Disk 1 +Segment + + +Disk 2 +Blocks + + +Disk 3 +Blocks + +ChunkDB owns the layout. DiskDB owns block allocation. DiskIO reads and writes the disks. +CROWDB + diff --git a/doc/assets/chunk-read.svg b/doc/assets/chunk-read.svg new file mode 100644 index 000000000..84f231775 --- /dev/null +++ b/doc/assets/chunk-read.svg @@ -0,0 +1,64 @@ + + +Chunk read flowStored locations resolve to the current layout; reads follow mirror or EC geometry. + + + + + + + + + + +Chunk read flow +Stored locations resolve to the current layout; reads follow mirror or EC geometry. + +Object locations + range +Logical offset and length + +Resolve ChunkDB metadata +Current strips, segments, and valid bytes + +Map requested intervals +Preserve logical byte order + + + +Mirror layout + +EC layout + + + +Read a usable replica +Fetch only the requested interval + +Try another replica +Keep the same logical range + +on read failure + +success + +Read the needed data shards +Healthy data needs no parity decode + +Recover a missing interval +Read enough surviving shards → bounded decode + +if required data is unavailable + +success + +Verify frames → assemble the requested logical range +Checksums, exact lengths, and current-layout validity + + + +Deliver bounded buffers +Consumer backpressure and cancellation + +Repair is a separate ChunkDB-owned operation; reading a replacement does not publish a new layout. +CROWDB + diff --git a/doc/assets/chunk-stream.svg b/doc/assets/chunk-stream.svg new file mode 100644 index 000000000..508a11a84 --- /dev/null +++ b/doc/assets/chunk-stream.svg @@ -0,0 +1,48 @@ + + +One continuous logical range across chunksChunk rollover changes physical storage, while logical offsets keep increasing. + + + + + + + + + + +One continuous logical range across chunks +Chunk rollover changes physical storage, while logical offsets keep increasing. +LOGICAL BYTE RANGE + +[0, L1) + +[L1, L2) + +[L2, tail) + +0 + +L1 + +L2 + +tail + +append + + +Chunk A +Sealed + + +Chunk B +Sealed + + +Chunk C +Active tail +end(A) = start(B) = L1 · end(B) = start(C) = L2 +No logical gaps at rollover. The journal owner decides when an old prefix can be trimmed. +CROWDB + diff --git a/doc/assets/crowdb-architecture.svg b/doc/assets/crowdb-architecture.svg new file mode 100644 index 000000000..c6496c090 --- /dev/null +++ b/doc/assets/crowdb-architecture.svg @@ -0,0 +1,97 @@ + + +CROWDB architecture +Iceberg, Dataset, and S3 form one access layer. Range metadata sits above the shared Chunk client; access libraries also reach the Chunk client directly. Paxos-KV forms the metadata foundation with Group 0, the system group, and Groups 1 through X. + + + +CROWDB architecture +Access models, shared storage, and the replicated metadata foundation. + +Access libraries + +Range metadata + +crowdb-paxos-kv + + + + + + + + + + + + + + + + +HTTP applications +Iceberg, Dataset, and S3 clients + +Trusted applications +Dataset / S3 native clients + +crowdb-access-server +Listeners, authentication, and runtime + +crowdb-sdk-native +Shared topology, budgets, and buffer leases + +Iceberg +crowdb-access-iceberg + +Dataset +crowdb-access-dataset + +S3 +crowdb-access-s3 + +crowdb-chunk-kv +Partition state, catalog, and serving authority + +crowdb-tree +Ordered page state + +crowdb-chunk-stream +Ordered journal bytes +metadata +Shared services + +crowdb-rpc +Internal transport: TCP / RDMA + +crowdb-monitor +Discovery and process supervision + +crowdb-chunk-client +Payload pipelines for access libraries, pages, and journals +page I/O +journal I/O +Direct payload I/O + +crowdb-chunkdb +Layouts and protection + +crowdb-diskdb +Physical allocation + +crowdb-diskio +Physical byte I/O +placement +byte I/O +Replicated metadata + +Group 0 (system group) +System metadata and service registry + +Group 1 +Replicated KV state +... + +Group X +Replicated KV state +metadataCROWDB diff --git a/doc/assets/crowdb-chunk.png b/doc/assets/crowdb-chunk.png new file mode 100644 index 000000000..050b6c3a0 Binary files /dev/null and b/doc/assets/crowdb-chunk.png differ diff --git a/doc/assets/crowdb-chunkkv.png b/doc/assets/crowdb-chunkkv.png new file mode 100644 index 000000000..9aa8dabfc Binary files /dev/null and b/doc/assets/crowdb-chunkkv.png differ diff --git a/doc/assets/crowdb-cluster.png b/doc/assets/crowdb-cluster.png new file mode 100644 index 000000000..5dfcec0b2 Binary files /dev/null and b/doc/assets/crowdb-cluster.png differ diff --git a/doc/assets/crowdb-dataset.png b/doc/assets/crowdb-dataset.png new file mode 100644 index 000000000..d250502c4 Binary files /dev/null and b/doc/assets/crowdb-dataset.png differ diff --git a/doc/assets/crowdb-iceberg.png b/doc/assets/crowdb-iceberg.png new file mode 100644 index 000000000..af5c1a25a Binary files /dev/null and b/doc/assets/crowdb-iceberg.png differ diff --git a/doc/assets/crowdb-overview.svg b/doc/assets/crowdb-overview.svg new file mode 100644 index 000000000..36c9b30eb --- /dev/null +++ b/doc/assets/crowdb-overview.svg @@ -0,0 +1,54 @@ + + +One storage core, independent data modelsIceberg, Dataset, and S3 share three storage layers: distributed metadata on top, chunk storage with chunk and disk management in the middle, and a Paxos cluster underneath. + + + + + + + + + + +One storage core, independent data models +Application semantics stay with their access model; storage work is shared. + +Iceberg +Table · Snapshot · Manifest · File + + +Dataset +Dataset · Snapshot · Split · Sample + + +S3 +Bucket · Object + + + + +Shared storage + +Distributed metadata +Chunk-KV · range-based trees · dynamic partitions + + +Chunk storage + +Chunk management +ChunkDB · placement and lifecycle +Chunk client · replication, EC, and streaming + +Disk management +DiskDB · disk and block allocation +DiskIO · asynchronous I/O over TCP / RDMA + + +Paxos cluster + +Group 0 · System metadata + +Groups 1–X · Replicated KV +CROWDB + diff --git a/doc/assets/crowdb-paxoskv.png b/doc/assets/crowdb-paxoskv.png new file mode 100644 index 000000000..49ef001d8 Binary files /dev/null and b/doc/assets/crowdb-paxoskv.png differ diff --git a/doc/assets/crowdb-s3.png b/doc/assets/crowdb-s3.png new file mode 100644 index 000000000..43d36df1b Binary files /dev/null and b/doc/assets/crowdb-s3.png differ diff --git a/doc/assets/dataset-read.svg b/doc/assets/dataset-read.svg new file mode 100644 index 000000000..54e93f689 --- /dev/null +++ b/doc/assets/dataset-read.svg @@ -0,0 +1,78 @@ + + +Dataset read: selection, storage, and deliveryResolve immutable input, select metadata first, and fetch only the projected field payloads. + + + + + + + + + + +Dataset read: selection, storage, and delivery +Resolve immutable input, select metadata first, and fetch only the projected field payloads. + +Dataset + Snapshot or latest/stable + optional Split + fields + read policy +HTTP Access Server or trusted native API + +Bind immutable input +Snapshot ID + optional Split revision + +Read plan + continuation +Selection, projection, order / shuffle +Worker partition and position + +Chunk-KV metadata +Snapshot, Manifest, Split membership +Sample fields and payload locators + + + + +Select a bounded batch + +Inspect metadata in bounded windows +Schema, membership, and selection predicates +Split filter → shuffle / order → worker partition + + +Project the requested fields + +Resolve field locators +Inline values or completed uploaded fields +Immutable upload description + part receipts + + + +Inline field +Value is already in metadata + +Uploaded field + +Ordered part receipts +Lengths, digests, and locations + +Chunk client range / stream reads +DiskIO · TCP / RDMA · mirror or EC + + + + +Verify and deliver the batch +Frame / field integrity and exact lengths +HTTP: header → payload → terminators → EOF +Native: direct RDMA path → retained host buffers + + + +Application processes the batch +Save the next continuation only after success + + +Next request uses the saved continuation +Each request checks access and deletion state. A continuation replays a batch without a persistent read session. +Fetching, EOF, and releasing buffers do not acknowledge application progress. +CROWDB + diff --git a/doc/assets/disk-allocation.svg b/doc/assets/disk-allocation.svg new file mode 100644 index 000000000..810c60809 --- /dev/null +++ b/doc/assets/disk-allocation.svg @@ -0,0 +1,109 @@ + + +Disk allocation across multiple nodesDisk groups route allocation; durable records track ownership of disk zones and blocks. + + + + + + + + + + +Disk allocation across multiple nodes +Disk groups route allocation; durable records track ownership of disk zones and blocks. + +ChunkDB +Requests space for a chunk + +DiskDB client +Route by disk group and current owner + +Group 0 · System metadata +Nodes, disks, and disk-group bindings + + + + + +Node 1 + +DiskDB owner 1 +Allocation, free, scanner, and recovery + + +Disk group 1 + + +Disk 1 +Zones +Busy / free +blocks + +Disk 2 +Zones +Busy / free +blocks + +Disk 3 +Zones +Busy / free +blocks + + +Node 2 + +DiskDB owner 2 +Allocation, free, scanner, and recovery + + +Disk group 2 + + +Disk 1 +Zones +Busy / free +blocks + +Disk 2 +Zones +Busy / free +blocks + +Disk 3 +Zones +Busy / free +blocks + + +Node 3 + +DiskDB owner 3 +Allocation, free, scanner, and recovery + + +Disk group 3 + + +Disk 1 +Zones +Busy / free +blocks + +Disk 2 +Zones +Busy / free +blocks + +Disk 3 +Zones +Busy / free +blocks + + +Paxos-KV data groups +Durable allocation records and ownership · each disk group has an explicit data-group binding +Working bitmaps are rebuilt from records. Payload transfers use the separate DiskIO path. +CROWDB + diff --git a/doc/assets/disk-io.svg b/doc/assets/disk-io.svg new file mode 100644 index 000000000..46076f76e --- /dev/null +++ b/doc/assets/disk-io.svg @@ -0,0 +1,74 @@ + + +Disk I/O over RDMA or TCPTransport buffers stay owned through network, device I/O, and response completion. + + + + + + + + + + +Disk I/O over RDMA or TCP +Transport buffers stay owned through network, device I/O, and response completion. + +Client + +Chunk client +Payload operations + +DiskIO client +Route Segment to its storage node + +RPC request +Read / write / fsync +Owned payload buffers + + + +RDMA +Registered memory +SEND / RECV + completion queues + +TCP +Framed control + payload + + + +Storage node + +crowdb-rpc → DiskIO +Receive request and retain its buffer +Validate Segment, range, and payload length + + + +Retained I/O buffers +Eligible RDMA paths reuse registered receive storage +Alignment edges use explicit staging buffers + + +I/O engine +io_uring or platform fallback + + +Local disk 1 + + +Local disk 2 + + +Local disk 3 + +Same DiskIO operations +on either transport + +Read payload / write or fsync result → selected transport → caller +Synchronized writes respond after the write and disk flush complete. +Read buffers remain retained through response SEND and caller release. + + +CROWDB + diff --git a/doc/assets/iceberg-commit.svg b/doc/assets/iceberg-commit.svg new file mode 100644 index 000000000..93c57cf58 --- /dev/null +++ b/doc/assets/iceberg-commit.svg @@ -0,0 +1,54 @@ + + +Iceberg write: native files to an atomic table commitThe native upload path returns a storage-bearing URI before the engine builds its manifests. + + + + + + + + + + +Iceberg write: native files to an atomic table commit +The native upload path returns a storage-bearing URI before the engine builds its manifests. + +1. Native Parquet upload +Engine / loader → Iceberg access +Bounded streaming + checksum verification + +2. Durable Chunk storage +Chunk client → DiskIO → disks +Complete file bytes and an owner-bound root + +3. Publish immutable file +Sign the root descriptor into its URI +Create the file record in Chunk-KV + + + + +Returned URI carries the Chunk location +s3://iceberg/<table>/f/v1/<name>.<locator>.parquet +Signed locator: file identity + length + integrity + Chunk directory root + + +4. Build standard manifests +Use the returned URI as file_path +Store manifests and the manifest list + +5. REST catalog commit +Validate requirements against table state +Prepare immutable metadata JSON + +6. Publish the table head +One compare-and-swap in Chunk-KV +The new snapshot becomes visible + + + +Iceberg files live in CROWDB Chunk storage +S3-compatible FileIO is a protocol surface; this path needs no separate S3 storage backend. +CROWDB + diff --git a/doc/assets/iceberg-read.svg b/doc/assets/iceberg-read.svg new file mode 100644 index 000000000..7de358660 --- /dev/null +++ b/doc/assets/iceberg-read.svg @@ -0,0 +1,55 @@ + + +Iceberg read: the URI already carries the storage locationNative direct URIs remove the Iceberg file-record lookup from each data-file read. + + + + + + + + + + +Iceberg read: the URI already carries the storage location +Native direct URIs remove the Iceberg file-record lookup from each data-file read. + +Load table + select snapshot +REST catalog and canonical table metadata + +Read manifests +Manifest list → manifests → selected files + +Select a Parquet file URI +The same URI published by native upload + + + + +URI → authenticated native descriptor +Chunk ID + root offset + lengths + integrity + file identity +A stable storage reference, independent of current replica addresses + + +Direct file read · no Iceberg file metadata lookup + +Validate grant + locator +Check scope and signature +Reconstruct the descriptor from the URI + +Read the verified Chunk root +Follow only the requested file range +Bounded directory traversal + +Chunk client → DiskIO +Resolve mirror / EC layout +Read the selected bytes from disks + + + + +Stream verified bytes to the analytical engine +GET / Range GET · bounded buffers · consumer backpressure +Catalog planning and Chunk layout routing remain; the per-file Iceberg KV lookup is removed. +CROWDB + diff --git a/doc/assets/paxos-kv.svg b/doc/assets/paxos-kv.svg new file mode 100644 index 000000000..84be71224 --- /dev/null +++ b/doc/assets/paxos-kv.svg @@ -0,0 +1,136 @@ + + +Paxos-KV: store, groups, and replicasA store contains multiple groups; each group replicates its own state across nodes. + + + + + + + + + + +Paxos-KV: store, groups, and replicas +A store contains multiple groups; each group replicates its own state across nodes. + +Store S + +Group 0 +System metadata +Paxos replication within this group + +Replica · Node 1 · leader + +WAL + +crowdb-tree + +Local disk + + + +Replica · Node 2 + +WAL + +crowdb-tree + +Local disk + + + +Replica · Node 3 + +WAL + +crowdb-tree + +Local disk + + + + + + + +Group 1 +KV data +Paxos replication within this group + +Replica · Node 1 + +WAL + +crowdb-tree + +Local disk + + + +Replica · Node 2 · leader + +WAL + +crowdb-tree + +Local disk + + + +Replica · Node 3 + +WAL + +crowdb-tree + +Local disk + + + + + + + +Group X +KV data +Paxos replication within this group + +Replica · Node 1 + +WAL + +crowdb-tree + +Local disk + + + +Replica · Node 2 + +WAL + +crowdb-tree + +Local disk + + + +Replica · Node 3 · leader + +WAL + +crowdb-tree + +Local disk + + + + + + + +Client routing: store → group → replica +Groups have independent consensus, WAL, and Tree state. Their leaders can be on different nodes. +CROWDB + diff --git a/doc/assets/s3-publication.svg b/doc/assets/s3-publication.svg new file mode 100644 index 000000000..c9e9517d8 --- /dev/null +++ b/doc/assets/s3-publication.svg @@ -0,0 +1,62 @@ + + +S3 I/O: bucket and key to Chunk storagePUT stores bytes before publishing the object; GET uses object metadata to locate those bytes. + + + + + + + + + + +S3 I/O: bucket and key to Chunk storage +PUT stores bytes before publishing the object; GET uses object metadata to locate those bytes. + +PUT / upload + +S3 client → S3 access +PUT bucket / key +Validate request and body + + +Chunk client → DiskIO +Stream to shared Chunk storage +Large or small writer + + +Publish object metadata +Chunk-KV: bucket + key +Locations, length, ETag, attributes + + +Return success +Object is visible by key +Payload completion precedes publication +Multipart uses the same storage path; Complete publishes the selected parts as one object. + +GET / Range GET + +S3 client → S3 access +GET bucket / key +Optional byte range + + +Read object metadata +Chunk-KV: resolve bucket + key +Select locations and requested range + + +Chunk client → DiskIO +Read mirror / EC layout +Verify the requested bytes + + +Stream the response +Return bytes as the client consumes +Bounded buffers and backpressure +HEAD and listing use metadata; reading an object payload follows the Chunk storage path. +The S3 access library connects directly to CROWDB metadata and payload services. +CROWDB + diff --git a/doc/assets/tree-flush.svg b/doc/assets/tree-flush.svg new file mode 100644 index 000000000..e28547970 --- /dev/null +++ b/doc/assets/tree-flush.svg @@ -0,0 +1,83 @@ + + +Tree flush: fork down the tree, join toward the rootIndependent subtrees advance concurrently; snapshot persistence follows the same dependencies. + + + + + + + + + + +Tree flush: fork down the tree, join toward the root +Independent subtrees advance concurrently; snapshot persistence follows the same dependencies. + +One round · fixed MemTables and version frontier +Select memory-only flush or flush with snapshot at the start. + +Root +Fork by index separators + + + +Index [a, m) +Fork child range jobs + + +Index [m, z) +Fork child range jobs +FORK + + +Leaf [a, f) +Receive this range’s inputs +Consolidate / split → final pages +Persistent: write final pages + + + +Leaf [f, m) +Receive this range’s inputs +Consolidate / split → final pages +Persistent: write final pages + + + +Leaf [m, t) +Receive this range’s inputs +Consolidate / split → final pages +Persistent: write final pages + + + +Leaf [t, z) +Receive this range’s inputs +Consolidate / split → final pages +Persistent: write final pages + +JOIN + +Join + consolidate index +Wait only for this parent’s children +Merge siblings and update separators + + +Join + consolidate index +Wait only for this parent’s children +Merge siblings and update separators + + +Join at root +Complete page, mapping, and coverage results + + + +flush +Publish the complete memory tree + +flush with snapshot +Changed mapping segments → barriers → durable root +CROWDB + diff --git a/doc/assets/tree.svg b/doc/assets/tree.svg new file mode 100644 index 000000000..e63849233 --- /dev/null +++ b/doc/assets/tree.svg @@ -0,0 +1,93 @@ + + +crowdb-tree: concurrent ingestion, ordered pages, two backendsCAS-based skiplist tables feed a Bw-tree with versioned page mapping and retained read views. + + + + + + + + + + +crowdb-tree: concurrent ingestion, ordered pages, two backends +CAS-based skiplist tables feed a Bw-tree with versioned page mapping and retained read views. + +Versioned put / delete · point reads · ordered scans +Consensus slots preserve logical order while independent updates complete concurrently. + +Concurrent skiplist tables · L0 + +a + + +g + +a + + +e + + +g + +a + + +c + + +e + + +g +CAS publishes nodes and immutable version sets + +Bw-tree · immutable pages and overlays + +Mapping table · PID → page version +A retained mapping keeps every page lookup consistent + +Root / index +Ordered separators + + + +Leaf [a, m) +Retained base + ordered overlays + + +Leaf [m, z) +Retained base + ordered overlays + +flush + +Readers retain their view +Consistent sources + mapping + page owners +New page publication preserves existing readers + + +Two outputs from one flush design + +flush +Publish memory pages and mapping + +flush with snapshot +Persist final pages, changed mapping, and durable root + +PageStore +Persistent pages, mapping segments, and recoverable roots + + + +Local disk backend +Pages on local disks +Paxos-KV replicas + + +Chunk backend +Pages in shared Chunks +Chunk-KV partitions +CROWDB + diff --git a/doc/backlog/R102-diskdb-dynamic-binding-migration.md b/doc/backlog/R102-diskdb-dynamic-binding-migration.md deleted file mode 100644 index 1c4991137..000000000 --- a/doc/backlog/R102-diskdb-dynamic-binding-migration.md +++ /dev/null @@ -1,412 +0,0 @@ - - - -### R102: diskdb — Dynamic Disk-Group Binding Migration - -## Problem - -### Current behavior + impact - -diskdb disk-group → KV-group binding is operator-manual today: the -operator writes `BindMapValue` (disk-group → `(store_id, group_id)`) -and `OwnerMapValue` (disk-group → diskdb instance) via the console -through `HardwareClient` (design §5 "Map semantics"). Three gaps -remain for cluster-expansion scenarios: - -1. **No safe migration flow on bind change.** When an existing - disk-group is rebound to a different KV-group, the operator - writes a new `BindMapValue` directly — but the disk-group's zone - records (busy/free/snapshot) are physically stored on the old - KV-group, keyed by `disk_id` (no `disk_group_id`/`group_id` in - the key). The diskdb sync loop silently overwrites `dg.bind`, so - recovery/compaction/scanner immediately scan the new KV-group and - conclude old busy blocks are free (data corruption), and frees of - old blocks target the wrong group (leaks). The existing disk-move - code (`http_move_disk` + `copy_disk_records`) has the right shape - (Maintenance quiescence + copy + flip) but uses a fixed 10 s sleep - with a latent free/compaction race during the copy, and is - disk-scope only — there is no disk-group-scope bind-change flow. -2. **No binding client or enforcement.** Clients route zone-record - writes to a KV-group with no check that the disk-group is actually - bound there; a stale route silently corrupts. diskdb instances - accept RPCs for any disk-group with no `NotMyBinding` reject — a - misrouted write goes to the wrong KV-group undetected. -3. **No imbalance visibility.** The operator has no warning when the - disk-group → KV-group distribution is significantly imbalanced - (e.g. one KV-group hosts far more disk-groups than others after - uneven growth). Rebalancing is left to operator vigilance. - -R99 built the common binding framework (binding client, binding -monitor, reject-and-retry protocol) for chunkdb instance sharding. -GAP-R99-5 decided that diskdb disk-group → KV-group rebinding should -be a separate follow-up requirement. This is R102: diskdb reuses the -binding client + enforcement + reject-and-retry protocol (work items -1-3) and the binding monitor (work item 4, warning-only — diskdb bind -changes are rare; auto-rebalancing is appropriate for chunkdb but -not for diskdb), and adds the safe migration flow (work item 5) -shared between disk-group bind change and disk move. - -### Design pointers - -- `doc/design/diskdb/design-crowdb-diskdb.md` §3.2 (disk-group → paxos - group binding via a table, not hash — dynamic scaling without - rehashing), §5 ("Map semantics" — `OwnerMapValue`, - `BindMapValue` are operator-manual today). -- `doc/design/kv/design-crowdb-kv-group0.md` §2.1 (`crowdb-kv-client` is - the single sysdata API surface), §2.6 (two monitoring models: push - + pull). -- `doc/backlog/R99-kv-dynamic-range-binding-framework.md` — common - binding framework (`RangeBindingClient`, binding monitor, - reject-and-retry protocol). - -### Use scenarios - -diskdb bind changes are rare in practice — the common capacity- -expansion patterns do not require rebinding: - -- **New node + new KV-group (common, no migration)** — operator adds - a new node, creates a new KV-group, adds the node's disk-groups, - and binds them to the new KV-group at creation time. The disk-groups - are new (no records); the bind is set before any writes. No - migration needed. R102's binding client + enforcement (work items - 1-3) ensure clients route to the correct KV-group from the start. -- **New node, KV-group replica moved (common, no diskdb migration)** - — KV-groups are pre-created (replicas on existing nodes); adding a - node means moving a KV-group replica to the new node (Paxos - reconfiguration — journal replication to the new replica). The - disk-group → KV-group binding does not change; diskdb records stay - on the same KV-group. This is a KV-layer migration, transparent to - diskdb. Not R102's scope. -- **Rebind existing disk-group to a different KV-group (rare, - requires migration)** — an existing disk-group with live records is - rebound from KV-group A to KV-group B (e.g. to evacuate a KV-group - that is being retired, or to rebalance after uneven growth). The - operator triggers the rebind via the console; the console drives - the five-step migration flow (work item 5). This is the case that - requires copying records. -- **diskdb instance crashes (rare, ownership reassignment)** — a - diskdb instance crashes (service registry expiry); the monitor - warns about the lost instance. The operator reassigns its - disk-groups to a surviving instance (updating `OwnerMapValue`). If - the surviving instance is on the same KV-group, no record migration - is needed (just an ownership change). If on a different KV-group, - the operator triggers the five-step migration flow. -- **Monitor warns about imbalance** — the binding monitor observes - the disk-group → KV-group distribution and emits a warning when it - is significantly imbalanced (e.g. one KV-group hosts far more - disk-groups than others). The operator decides whether to act - (trigger rebinds via the console) or ignore. The monitor does NOT - auto-trigger rebinding for diskdb — unlike chunkdb (R99), where - auto-rebalancing is appropriate. - -## Solution - -Reuse R99's common binding framework for the binding client + -enforcement + reject-and-retry protocol (work items 1-3). The binding -monitor (work item 4) observes the disk-group → KV-group distribution -and emits imbalance warnings, but does NOT auto-trigger rebinding for -diskdb — rebinding is operator-triggered via the console (diskdb bind -changes are rare; auto-rebalancing is appropriate for chunkdb but not -for diskdb). Bind change and disk move share one five-step migration -flow (Maintenance → copy → switch → cleanup → Up) at different scopes -(work item 5), reusing the existing `http_move_disk` + -`copy_disk_records` pattern. - -### Work items - -1. **diskdb binding schema** — `lib/crowdb-protocol/src/fbs/common_type.fbs` - (extend): add `DiskdbBindingValue` (disk-group-id → paxos-group-id - `(store_id, group_id)` + diskdb instance endpoint); stored in - group-0 with key pattern `/diskdb/dg_bind/`. Add a - `MigrationIntentValue` carrying the migration scope (disk-group or - single disk), `old_bind` `(store_id, group_id)`, `new_bind` - `(store_id, group_id)`, and optional `disk_id` (present for - disk-level moves, absent for disk-group bind changes); stored at - `/diskdb/migrate/` (disk-group scope) or - `/diskdb/migrate//` (disk scope). The - intent serves double duty: a write-quiescence signal (the diskdb - instance reads it and suspends frees + compaction for the affected - disk/disk-group during the copy, before the bind flip) and a - crash-recovery marker (a new monitor leader reads it and resumes). - No `Copying`/`Cutover`/`Complete` state enum — progress is derived - from comparing the current `BindMapValue` to `new_bind` (copy done - iff bind already points to `new_bind`). Add - `ERROR_CODE_NOT_MY_BINDING` to `ret_code.fbs` for the - reject-and-retry protocol. -2. **diskdb binding client** — `lib/crowdb-kv-client/src/` (extend): - add `DiskdbBindingClient` (fetch + cache + watch/notify), - following R99's `RangeBindingClient` pattern. Caches - `DashMap`, - subscribes to watch/notify for real-time updates. Provides - `route(disk_group_id) -> (store_id, group_id, endpoint)` for - clients to route zone-record writes to the correct paxos group. - Reject-and-retry: on `NotMyBinding` error, refresh the binding - cache, re-route, retry — follows the `NotLeaderHint` retry pattern - in `crowdb-kv-client/src/config.rs`. -3. **diskdb server binding enforcement** — `app/crowdb-diskdb/src/` - (extend): add a binding guard (following `range_guard.rs` pattern - from chunkdb). On every RPC, check the disk-group is bound to this - instance's paxos group; reject with `NotMyBinding` error if not. - Binding is read from group-0 at startup + updated via - watch/notify. The existing `DdbDiskGroup.bind` field - (`model/disk_group.rs:32`, `RwLock<(u64, u64)>`) is populated from - the binding client instead of operator config. -4. **Monitor integration (warning-only)** — the binding monitor - (relocated to `crowdb-kv-server` per R99 rework, - `app/crowdb-kv-server/src/`) observes the disk-group → KV-group - distribution alongside chunkdb range binding; same leader-gated - background task. For diskdb, the monitor computes the current - distribution and emits an imbalance warning (log + metric) when - one KV-group hosts significantly more disk-groups than others, or - when a diskdb instance is absent from the service registry (crash - suspicion). The monitor does NOT auto-trigger rebinding or - migration for diskdb — unlike chunkdb (R99) where auto-rebalancing - is appropriate. Rebinding is operator-triggered via the console - (work item 5's flow). The monitor's warning gives the operator the - signal to act; the console provides the safe migration path. -5. **Migration flow** — disk-group bind change and disk move share - one flow at different scopes (whole disk-group vs one disk), reusing - the existing `http_move_disk` + `copy_disk_records` pattern - (`app/crowdb-web/src/lifecycle.rs`). Both are operator-triggered via - the console (the monitor only warns; it does not drive migration). - The flow is five steps: - (a) **Quiesce** — the console sets the disk-group (or disk) to - `Maintenance` in group 0 and writes a `MigrationIntentValue`; - allocates block (`permits(Maintenance, Allocate) = false`); the - diskdb instance reads the intent on the next sync tick (or via - watch/notify) and suspends frees + compaction for the affected - disk/disk-group; reads still serve from the old bind (`dg.bind` - unchanged). - (b) **Copy** — `copy_disk_records(old_bind, new_bind, disk_id)` - per disk (one disk for disk move; every disk in the group for - bind change): prefix-scan `ZoneKey`/`BusyBlockKey`/`FreeBlockKey` - by `DiskId` from the old bind, batch-write to the new bind. - Idempotent — re-running overwrites the same keys, so crash-resume - is safe. - (c) **Switch** — write the new `BindMapValue` to group 0 (bind - change) or add the disk to its new disk-group placement (disk - move, whose new disk-group already has its own bind). The next - sync tick updates `dg.bind` → new bind; reads + writes now target - the new bind. The diskdb instance clears its in-memory migrating - flag (frees + compaction resume on the new bind). - (d) **Cleanup** — delete the migrated records from the old bind - (per-disk prefix scan + `batch_write` of `Delete` ops). For disk - move, only the moved disk's records (other disks still use the - old bind); for bind change, all disks in the group. - (e) **Resume** — set the disk-group (or disk) to `Up` and delete - the `MigrationIntentValue`; allocates resume on the new bind. - No `Copying`/`Cutover`/`Complete` state machine, no dual-reads — - `Maintenance` is the quiescence signal, the bind write is the - cutover, the intent deletion is the completion marker. - -### Flow diagram - -``` - driver: console (crowdb-web), operator-triggered - (monitor only warns — does not drive migration) - │ - (a) QUIESCE - ├── set disk-group (or disk) to Maintenance in group 0 - ├── write MigrationIntentValue { old_bind, new_bind, disk_id? } to group 0 - ▼ - watch/notify / next sync tick → diskdb instance - ├── allocates blocked (Maintenance) - ├── suspends frees + compaction for affected disk/disk-group (intent) - └── reads still serve from old bind (dg.bind unchanged) - │ - (b) COPY — copy_disk_records(old_bind, new_bind, disk_id) per disk - ├── prefix-scan ZoneKey / BusyBlockKey / FreeBlockKey by DiskId from old bind - └── batch_write Put to new bind (idempotent — safe to re-run on resume) - │ - (c) SWITCH - ├── bind change: write new BindMapValue to group 0 - ├── disk move: add disk to new placement (new disk-group's bind already set) - ▼ - next sync tick → diskdb updates dg.bind → new bind - ├── reads + writes now target new bind - └── diskdb clears migrating flag (frees + compaction resume on new bind) - │ - (d) CLEANUP — delete migrated records from old bind (per-disk prefix scan + batch_delete) - │ - (e) RESUME - ├── set disk-group (or disk) to Up - └── delete MigrationIntentValue → allocates resume on new bind -``` - -### Edge cases at a glance - -- **disk-group with no data** — fast flip: copy is a no-op (no zone - records to scan), switch + cleanup + resume proceed immediately. -- **Console crash mid-migration** — the `MigrationIntentValue` - persists in group-0; the operator retries (or a new console - session reads the intent) and resumes from the appropriate step - (re-run copy — idempotent; then switch; then cleanup; then - resume). The diskdb instance keeps the migrating flag set (frees - + compaction stay suspended) until the intent is deleted. -- **Free or compaction during copy** — the diskdb instance's - migrating flag (from the intent) suspends frees + compaction for - the affected disk/disk-group before the bind flip, so no - `Delete BusyBlockKey + Put FreeBlockKey` or compaction - `batch_write` races with the copy. After the flip, frees + - compaction resume on the new bind and are correct. -- **Concurrent rebinding requests** — rebinding is operator-triggered - via the console; the console serializes bind changes and disk moves - on the same disk-group via the `MigrationIntentValue` (one intent - per scope key; the console rejects a second intent while one - exists). The monitor does not trigger concurrent migrations (it - only warns). -- **Disk move to a disk-group on the same kv-group** — copy is a - no-op (old_bind == new_bind); the flow still quiesces, flips the - placement, and resumes. Detected by comparing `old_bind` and - `new_bind` in the intent. - -## Dependencies - -- Depends on **R99** — common binding framework (`RangeBindingClient` - pattern, binding monitor, reject-and-retry protocol). -- **R99 rework** (Phase 4-5 of `doc/working/plan-gaps-r99-r100.md`) - must land first — the binding monitor must be relocated to - `crowdb-kv-server` (leader-gated background task on group-0 - replicas) before diskdb rebinding can be added to the same - monitor. -- **R103** (chunkdb range migration) is a sibling requirement. R102 - and R103 no longer share a migration state machine — chunkdb holds - user chunk data and may need online dual-read migration (R103), - while diskdb is a metadata allocator whose brief write pause with - retry is tolerable, so R102 uses the simpler quiesce + copy + flip - flow. They can still proceed in parallel. -- **Disk move** (`app/crowdb-web/src/lifecycle.rs:1630` - `http_move_disk` + `copy_disk_records`) is the existing - disk-scope migration; R102 generalizes the same flow to - disk-group scope and adds the `MigrationIntentValue` for - write-quiescence + crash safety (the current disk-move code uses - a fixed 10 s sleep and has a latent free/compaction race during - the copy — R102 fixes both by sharing the intent-based flow). - -## Acceptance - -### diskdb binding schema - -- `DiskdbBindingValue` encodes `disk_group_id` → `(store_id, - group_id)` + instance endpoint → round-trip serialize/deserialize - preserves all fields. `[Unit test]` -- `MigrationIntentValue` encodes scope (disk-group or disk) + - `old_bind` + `new_bind` + optional `disk_id` → round-trip - preserves all fields. `[Unit test]` -- Binding value stored at `/diskdb/dg_bind/` in - group-0 → `DiskdbBindingClient` fetches and decodes it correctly. - `[Integration test]` -- Migration intent stored at `/diskdb/migrate/` (or - `.../`) → console + diskdb instance fetch and decode - it correctly. `[Integration test]` - -### diskdb binding client - -- `DiskdbBindingClient::route(disk_group_id)` with a cached binding - → returns the correct `(store_id, group_id, endpoint)`. `[Unit test]` -- Binding table updated in group-0 → watch/notify fires → - `DiskdbBindingClient` cache updates within 1s → next `route` - returns the new binding. `[Integration test]` -- Client receives `NotMyBinding` → refreshes binding cache → - re-routes → retries against the correct paxos group → succeeds. - `[Integration test]` - -### diskdb server binding enforcement - -- diskdb instance receives an RPC for a disk-group bound to its - paxos group → binding guard allows → RPC processes normally. - `[Unit test]` -- diskdb instance receives an RPC for a disk-group NOT bound to its - paxos group → binding guard rejects with `NotMyBinding` including - the current binding (new paxos group + endpoint). `[Integration test]` -- `DdbDiskGroup.bind` is populated from the binding client at startup - (not operator config) → assert `bind` matches group-0 value. - `[Integration test]` - -### Monitor integration (warning-only) - -- The disk-group → KV-group distribution is significantly imbalanced - (one KV-group hosts far more disk-groups than others) → monitor - emits an imbalance warning (log + metric) within the polling - interval → assert no `BindMapValue` or `OwnerMapValue` writes from - the monitor. `[Integration test]` -- A diskdb instance is absent from the service registry (crash - suspicion) → monitor emits a warning → assert no auto-rebind of - its disk-groups (the operator triggers reassignment via the - console). `[Integration test]` -- Monitor follower (non-leader) does not emit warnings or perform - writes → assert no binding writes + no warning metrics from - followers. `[Integration test]` - -### Migration flow - -- disk-group bind change (A → B): console sets disk-group to - `Maintenance` + writes `MigrationIntentValue` → diskdb suspends - frees + compaction → `copy_disk_records` copies all disks' - records from A to B → console writes new `BindMapValue` → diskdb - updates `dg.bind` → B → console deletes old records from A → - console sets disk-group to `Up` + deletes intent → allocates - resume on B. `[Integration test]` -- disk move (old disk-group → new disk-group on a different - kv-group): same five-step flow at disk scope — disk set to - `Maintenance`, one disk's records copied, disk added to new - placement, old records deleted, disk set to `Up`. Reuses - `copy_disk_records`. `[Integration test]` -- disk-group with no data: copy is a no-op (no records to scan) → - switch + cleanup + resume proceed immediately → new bind serves - allocates. `[Integration test]` -- disk move to a disk-group on the same kv-group (`old_bind == - new_bind`): copy is skipped → placement flip + resume proceed. - `[Integration test]` -- E2E test: full bind-change lifecycle driven by the console with - live clients → assert no permanent request failures beyond - transient `NotMyBinding` redirects, and no busy-block records lost - (a block allocated before the migration is still busy on the new - bind after resume). `[E2E test]` - -### Edge cases - -- Console crashes mid-migration → `MigrationIntentValue` persists → - operator retries (or a new console session reads the intent) → - re-runs copy (idempotent) → completes switch + cleanup + resume. - `[Integration test]` -- Free or compaction attempted during copy (before bind flip) → - diskdb migrating flag suspends it → no `Delete BusyBlockKey` or - compaction `batch_write` races with the copy → records on new - bind match old bind exactly. `[Integration test]` -- Concurrent rebinding requests for different disk-groups → console - serializes via per-scope intents → both complete without conflict. - `[Integration test]` -- Second migration intent for the same scope while one exists → - console rejects (one intent per scope key) → no conflicting - concurrent migrations. `[Integration test]` -- Client with stale binding cache sends write to old paxos group - after the flip → `NotMyBinding` → client refreshes cache + - retries on new paxos group successfully. `[Integration test]` - -### Build & style - -- `pixi run test-diskdb` -- `pixi run test-kv-server` -- `pixi run cargo fmt --all -- --check` -- `pixi run cargo clippy --all-targets -- -D warnings` - -## Open Questions - -- **Copy data or just rebind?** Resolved: **copy is required.** Zone - records are physically stored on the disk-group's bound kv-group - (`dg.bind` routes every `persist_busy`/`persist_free`/`put_zone`/ - recovery/compaction/scanner read+write). Record keys are - `disk_id`-keyed (no `disk_group_id`/`store_id`/`group_id`), so a - bare bind change to a different kv-group leaves the records - stranded on the old kv-group — the sync loop silently overwrites - `dg.bind`, after which recovery/compaction/scanner scan the new - kv-group and conclude old busy blocks are free (data corruption) - and frees of old blocks target the wrong group (leaks). The - five-step flow copies records before the flip. No cross-group - replication is assumed in CROWDB v1. -- **Imbalance warning threshold** — what ratio triggers the - monitor's imbalance warning (e.g. max/min disk-groups per KV-group - > 2×, or absolute deviation from the mean)? Needs a tunable - default + a design decision on whether to warn on disk-group count - alone or also factor in per-disk-group capacity/usage. diff --git a/doc/backlog/R103-chunkdb-range-migration.md b/doc/backlog/R103-chunkdb-range-migration.md deleted file mode 100644 index 90cc9cafa..000000000 --- a/doc/backlog/R103-chunkdb-range-migration.md +++ /dev/null @@ -1,287 +0,0 @@ - - - -### R103: chunkdb — Dynamic Slot Ownership and KV-Group Expansion/Shrink - -Status: Deferred by user decision on 2026-10-03. The implemented fixed topology -uses three nodes and three selected nonzero Chunk-storage KV groups, plus -control-plane group 0. Start this follow-up when dynamic ownership or storage -capacity changes are requested. - -#### Problem - -[fixed slot routing](../design/chunkdb/design-crowdb-chunkdb-range-binding.md) -establishes two independent maps over 1024 fixed chunk-ID hash slots: - -- An ordinary client resolves slot to ChunkDB server. Group 0 stores one bitmap - record per server, including servers owning zero slots. -- ChunkDB resolves slot to a selected nonzero Paxos KV group (pxgroup). Group 0 - stores one bitmap record per group. All records belonging to a chunk follow - its chunk ID into that group. - -The initial R202 delivery freezes these assignments. It supports fixed-owner -restart and recovery, but does not make changing either bitmap safe. ChunkDB -servers perform chunk lifecycle and heavy task/IO orchestration without an -authoritative local chunk database. Changing a server owner therefore requires -execution-authority handoff, while changing a storage group requires moving -durable state. These operations have different costs and failure modes. - -The current [slot routing design](../design/chunkdb/design-crowdb-chunkdb-range-binding.md) -rejects ownership changes and describes the fixed service/storage maps. -Service-only handoff avoids metadata copying only when storage placement -remains unchanged. Existing ChunkDB routing, range guards and task -claims are useful foundations, but neither a routing refresh nor a grace-period -timer proves exclusive authority or complete storage transfer. - -Concrete scenarios are adding a ChunkDB server for more processing capacity, -draining a server, taking over slots after a failed owner, adding a KV group for -capacity, evacuating a group before removal, and resuming an interrupted move. -A source group may hold slots served by several servers; a server may own slots -stored in several groups. A one-to-one server/group assumption breaks both -handoff and migration. - -#### Solution - -##### Goal and scope - -Make the fixed R202 layout safely changeable without changing chunk identity or -losing acknowledged chunk/task state. Deliver two independently usable operations: - -1. **Service slot handoff.** Transfer selected slots between ChunkDB server - bitmaps. Keep storage-group bindings and persistent records in place. - Revoke old execution authority and recover tasks on the new server. -2. **Storage slot migration.** Transfer selected slots between pxgroup bitmaps - when adding, draining, removing or rebalancing storage groups. Move the - complete associated record set before activating the new destination. - Keep service assignments unchanged unless a separate handoff is requested. - -A supported combined change must coordinate these operations explicitly; it -must never infer one mapping from the other. Initial implementation may serialize -overlapping changes rather than execute both concurrently. - -In scope: - -- Explicit move requests, destination validation, durable progress, independent - authority generations, retry/recovery and scoped cleanup. -- Service join/leave/failure handoff and stale client/worker rejection. -- KV-group expansion/shrink and slot redistribution within the direct-KV backend. -- Migration of canonical Chunk records, all associated tasks and task indexes, - reservations and other per-chunk transaction participants. -- Safe refusal when source completeness or exclusive authority cannot be proven. -- Observable move status and an unambiguous completion condition for operators. - -Out of scope: - -- R202's initial fixed layout, chunk-type cleanup and PageIndex implementation. -- Changing the 1024-slot space, rehashing IDs, or hash-by-current-group-count. -- Moving DiskIO payload or reallocating DiskDB blocks as a side effect. -- Moving stream catalogs, tree roots or unrelated records merely because they - share a KV group; ownership is determined by the per-chunk record contract. -- Direct-KV-to-chunk-kv conversion, which belongs to - [R207](R207-chunkdb-repo-metadata-chunk-kv.md). -- Automatic cluster-wide capacity planning, creating/removing Paxos replicas, - or deleting a shared KV group. Removing a group from Chunk storage is distinct - from destroying the group. -- Mandatory conversion of pre-R202 layouts or legacy Repo/Stream/PageIndex data. - R202 uses fresh test state. A legacy group-0 source may be supported later only - with an explicit compatible record/layout conversion contract; it is not a - prerequisite for this requirement's R202-layout migration. - -##### Fixed granularity and publication - -- The move unit is one or more existing logical slots, selected by the owning - chunk ID. Related task IDs and encoded key order do not choose the destination. - Unmoved slots retain their hash meaning and bindings. -- Keep one steady-state bitmap record per server and per pxgroup. Do not create - a binding record, KV group or worker for each of the 1024 slots. Durable - per-operation progress is separate from these owner records. -- Publish each changed map as one complete committed generation. Readers cannot - observe independently cleared source bits and set target bits as active - authority. Service and storage maps have separate generations. -- Use group-0 atomic publication for routing/control state, but do not mistake it - for an atomic transaction with copied records in other groups. Preparation, - authority revocation, validation and activation form a recoverable protocol. -- Fence mutations at their authoritative acceptance/publication boundary. A - cached ownership check before asynchronous work is insufficient. Cover late - completions, task claims, reservation updates and operations already admitted - before revocation; authority cannot depend only on a process-local lock. -- Preserve group-local conditional atomic chunk operations. Source and target - revisions belong to their own groups; transferred values cannot make an old - source revision a valid target CAS token. - -##### Service handoff - -1. Validate source/target servers and requested slots against the current map; - persist operation identity and expected service generation. -2. Stop or revoke old admission and task authority for those slots. Resolve - admitted work and persisted claims using a durable fence/recovery contract. - Failure detection alone is not proof that an old server stopped running. -3. Publish source/target server bitmaps as one committed service generation. - Activate the new owner only under the new authority. -4. Refresh ordinary clients and recover task execution from the unchanged KV - groups. Preserve claim generations and prevent stale completion publication. -5. Complete when the new owner serves the slots and obsolete execution grants - cannot commit. Do not copy a local ChunkDB database or change storage bitmaps. - -An empty source bitmap means no remaining partition work, not allow-all. -An unreachable owner may delay handoff until fencing is proven; it must not -force a second active owner for availability. - -##### Storage migration and group drain - -1. **Prepare.** Validate a provisioned nonzero eligible target and persist the - slot set, source/target groups, expected storage generation and progress. - A newly added group has no authority simply because it is eligible. -2. **Copy.** Enumerate complete per-chunk state for the selected slots, including - indexes ordered by time/priority rather than chunk ID. Choose a bounded - slot-write pause or a consistent snapshot plus reliable change capture. - Handle creations, updates, deletes and claims without omissions or resurrected - records. A partial target copy is never serving authority. -3. **Fence and reconcile.** Coordinate every server that owns affected slots; - prevent old-group writes and delayed task publication. Bring the target to - the final acknowledged source state and validate all transaction participants. -4. **Cut over.** Publish source/target pxgroup bitmaps as one storage generation - only after completeness and fencing are proven. Enable operations against - the target; leave service bitmaps unchanged for a storage-only move. -5. **Clean up.** Retire old reads/claims safely and remove only moved source - records. Cleanup and retries are idempotent. Lost replies are reconciled - from durable authority/progress rather than interpreted as failure or success. - -For shrink, mark a group draining, stop new assignments to it, move all its slots -and finish relevant cleanup. Remove it from the Chunk-storage eligible set only -when no ownership or migration obligation needs it. Unrelated group contents -remain intact. Group expansion/shrink never changes the fixed hash function. - -##### Failure and cancellation outcomes - -- Before source revocation, a failed target leaves the source authoritative. - Retry or abandon a partial copy without exposing it to clients. -- During the fenced interval, operations may fail or retry; never acknowledge - writes that have no durable authoritative destination. -- If the source is unavailable before target completeness is proven, wait for - source recovery or a proven complete recovery source. Do not promote a partial - copy to finish a drain. -- After cutover, target unavailability does not reactivate the old source. - Restore the target or perform a new fenced move. -- Controller restart or an uncertain cutover reply resumes from persisted - operation identity and mapping generation before granting authority or cleanup. -- Cancellation before cutover must reconcile fences and partial target data; - cancellation after cutover cannot roll back by restoring old bitmap bits. -- Serialize or explicitly reject conflicting requests on the same slots. - Independent slots may progress within bounded copy/scan/task resources. -- Unavailable owner metadata is uncertainty, not evidence that payload is unused. - Preserve system/repo task separation throughout failures and recovery. - -##### Invariants - -- **M1 — Exclusive authority.** Only the current service and storage authorities - can commit chunk mutations or task outcomes, including delayed work. -- **M2 — Complete movement.** All acknowledged per-chunk state and maintenance - discoverability survive; atomic-operation participants share one active group. -- **M3 — Recoverable progress.** Retries, restart and uncertain replies cannot - create authority from an incomplete copy or reactivate an obsolete source. -- **M4 — Safe retirement.** Cleanup is scoped and idempotent; drain completion - requires zero remaining ownership and migration obligations. -- **M5 — Independent layers.** Service-only changes copy no metadata; - storage-only changes preserve service assignments. Neither moves payload, - changes slot identity nor merges system/repo task authority. -- **M6 — Coherent maps.** Each active slot has one owner in each committed map; - intermediate records and partial publications grant no authority. - -##### Work items - -1. Extend the KV domain monitor and ChunkdbRangeStrategy beyond fixed-layout - initialization with explicit service/storage move admission, conflict checks, - durable progress and separately published bitmap generations. -2. Extend RangeGuard, lifecycle handlers and task manager/scanner/executor with - service revocation and durable stale-publication rejection; integrate the - native tree and Rust client routing paths completed by R202. -3. Extend ChunkStore, TaskStore and reservation persistence with complete - slot-state enumeration/copy/validation under the selected change-capture - strategy. Preserve destination-local CAS semantics. -4. Implement storage fencing, recovery, cancellation and idempotent cleanup; - expose move/drain progress and actionable failure state. -5. Verify independent and combined changes, resource bounds and failure cases; - update permanent ChunkDB routing/migration documentation. - -#### Dependencies - -- [fixed slot routing](../design/chunkdb/design-crowdb-chunkdb-range-binding.md) supplies tested fixed slots, - independent bitmap maps, typed operations, task isolation, all client paths - and group-local atomicity. Start from its completed fixed-layout contract. - Until R103 is implemented, keep initialized assignments unchanged; do not - enable the old automatic rebalance path as a substitute. -- Existing direct-KV scan/snapshot/transfer facilities are candidate primitives. - Confirm snapshot consistency, delete capture and hash-slot enumeration before - choosing a copy strategy. Ordered key-range transfer alone is insufficient. -- [R207](R207-chunkdb-repo-metadata-chunk-kv.md) may reuse this authority/recovery - model later but needs its own backend transaction and placement contract. - R207 does not block either operation in R103. - -#### Acceptance - -- Given the completed R202 fixed topology, request a server-only slot move; - assert new-owner service, unchanged group bindings/data, no payload copy and - zero-slot old-owner rejection (M1, M5, M6). Integration test. -- Given a failed or partitioned old server, request takeover and reconnect its - delayed workers; assert takeover waits for proven fencing and stale writes, - reservations and task completions fail after activation (M1, M3). Integration test. -- Given an eligible additional group, move selected slots while handling allowed - foreground/task traffic; assert complete target state, preserved acknowledgements, - unchanged service map and unmoved slots (M1, M2, M5). E2E test. -- Given a group serving slots owned by several servers, drain it; assert all - affected owners follow storage cutover and removal waits for cleanup, while - unrelated data and DiskIO payload remain intact (M2, M4, M5). E2E test. -- Given non-contiguous chunk/task/index/reservation keys, copy with creates, - deletes and claims; assert complete enumeration, no resurrected deletes, - discoverable tasks and valid destination-local CAS (M2). Integration test. -- Given interrupted source/target bitmap publication, reload both maps; assert - readers use complete generations with one authority per slot and no partial - target activation (M3, M6). Integration test. -- Given source/target/controller failure or lost replies at every move stage, - restart and retry; assert acknowledged data survives, incomplete targets are - not promoted and obsolete sources never reactivate (M1–M3). Integration test. -- Given cancellation before and after cutover, reconcile the operation; assert - safe pre-cutover termination or forward recovery after cutover, with no bitmap - rollback that restores obsolete authority (M1, M3, M4). Integration test. -- Given conflicting service/storage changes for the same slots, submit both; - assert explicit serialization/rejection or a proven coordinated transition, - never inferred mapping changes or two writers (M1, M5, M6). Integration test. -- Given interrupted/repeated cleanup alongside unrelated slots, resume cleanup; - assert no duplicate release, foreign deletion or premature group retirement - (M3, M4). Integration test. -- Given invalid group 0 targets, incompatible layouts or attempted slot resizing, - submit a move; assert rejection without modifying current authority (M5, M6). - Unit test. -- Given both operation domains and unavailable ownership metadata during a move, - scan tasks and query payload ownership; assert no cross-domain claims and - uncertainty does not authorize reclamation (M1, M2, M5). Integration test. -- Given bounded migration concurrency under foreground load, run independent - moves and inspect status; assert configured resource limits, observable progress - and eventual completion after recoverable faults clear (M3, M4). E2E test. - -#### Open Questions - -- Prefer a bounded write pause per moved slot, or online snapshot plus reliable - catch-up? The former reduces protocol complexity but pauses affected writes; - the latter needs proven change capture and increases implementation cost. -- Should group-add select slots automatically from measured load, or initially - require explicit slot selection? Manual selection keeps the first migration - contract smaller; automatic policy needs agreed load/capacity objectives. -- What pause/latency and migration resource budgets are acceptable? Establish - measured limits before claiming online migration performance. - -Implementation selects the durable fence and state schema against R202's actual -interfaces; these are required technical proofs, not permission to weaken the -invariants. This rewrite ran no runtime tests. Extend the named suites with the -acceptance cases before treating these commands as migration coverage: - -```sh -pixi run cargo test -p crowdb-chunkdb --test routing_test --test lifecycle_test --test owner_metadata_test --test conversion_test -pixi run cargo test -p crowdb-chunkdb-client --test client_test -pixi run cargo test -p crowdb-kv-client --test chunkdb_partition_test -pixi run cargo test -p crowdb-kv-server --test domain_monitor_test -pixi run test-tree-ct -pixi run rs-fmt-check -pixi run cargo clippy -p crowdb-chunkdb -p crowdb-chunkdb-client -p crowdb-kv-client -p crowdb-kv-server --all-targets -- -D warnings -``` diff --git a/doc/backlog/R139-group0-service-config.md b/doc/backlog/R139-group0-service-config.md deleted file mode 100644 index d7b568b3e..000000000 --- a/doc/backlog/R139-group0-service-config.md +++ /dev/null @@ -1,147 +0,0 @@ - - - -### R139: config — Deployment profiles and Group-0 service configuration - -## Status - -Deferred until the file-backed service configuration contract is established. - -## Problem - -Service configuration is local to each node. Changing a cluster-wide policy -requires editing files or launch arguments on every host, and operators cannot -see which revision each process has accepted. Blind live reload is unsafe -because some fields can be applied dynamically while others define listener, -thread, storage, or durability state and require a restart. - -Single-container deployment, multi-node clusters and test fixtures currently -select tuning through different paths: service TOML files, command-line presets -such as KV's `e2e` election profile, and deployment request overrides such as -DiskDB's keepalive interval. A single container and a cluster require different -settings; clusters also need low- and high-performance profiles. These choices -must be explicit and reproducible without building separate images or scattering -special timing values through test clients. - -The group-0 architecture in -`doc/design/kv/design-crowdb-kv-group0.md` already provides durable replicated -sysdata and service registration, but it has no versioned configuration model, -target selection, acknowledgement state, or restart contract. - -## Solution - -Provide named, versioned configuration profiles through the normal deployment -and service configuration contract. Resolve the distributed record and rollout -details below before implementation. - -Profile invariants: - -- **I1 — Explicit selection:** distinguish single-container and cluster profiles; - cluster profiles include low- and high-performance settings. Selection is - recorded and remains stable across restart and cluster expansion. -- **I2 — One runtime:** profiles configure the same release image and binaries. - Tests may select a documented test tuning profile or explicit timing overrides - using the same schema, validation and startup mechanism as operators. -- **I3 — Complete validation:** validate the resolved configuration and its - cross-field constraints before starting or applying changes. Performance - settings never relax placement, quorum, lease safety or durability guarantees. -- **I4 — Observable configuration:** expose the profile name/version, resolved - values and accepted/applied revision for each service; errors identify the - conflicting setting and target. - -Numbered work items: - -1. Define single-container and cluster profiles, with low- and high-performance - cluster choices. Document the intended resource envelope and service settings - for each rather than treating a higher number as universally faster. -2. Unify profile selection at container/cluster creation and service startup; - support typed caller overrides and persist the resolved choice. Cover KV - election/heartbeat/lease settings, DiskDB keepalive/synchronization intervals, - and applicable worker, concurrency and batching limits. Test startup must be - able to request, for example, a 3-second DiskDB cadence through this contract. -3. Define deterministic precedence between profile defaults, cluster defaults, - service/node/instance overrides and explicitly supplied startup values. - Existing CLI presets and deployment fields must resolve through this contract - rather than silently overwrite one another. Distinguish heartbeat cadence - from metadata synchronization cadence and specify their actual consumers. -4. Keep bootstrap selection available before Group 0 exists. Persist profile - identity and effective local configuration, then reconcile with the validated - distributed configuration when authority is available. Joining nodes inherit - the cluster selection unless an explicit validated override applies. -5. Define the distributed configuration and rollout model: - -- Define a versioned group-0 record for cluster defaults plus service-, - instance-, and node-scoped overrides, with deterministic precedence. -- Reuse the file-backed schemas as typed payloads or define an evolvable - envelope that preserves unknown fields during mixed-version rollout. -- Classify every field as dynamic or restart-required and make each service - validate a complete candidate before atomically applying any dynamic subset. -- Publish per-instance observed, accepted, applied, and restart-required - revisions through the service registry so operators can detect drift. -- Define bootstrap and outage behavior: local file remains the seed and - fallback until group 0 is reachable; the last validated group-0 snapshot is - cached locally for restart. -- Add an authenticated administrative write path with compare-and-set revision - fencing, validation, staged rollout, and rollback semantics. - -Edge cases requiring design include an older binary receiving unknown fields, -group-0 loss during rollout, a process restart between acceptance and apply, -conflicting scope overrides, invalid cross-field combinations, and a static -field changed while a service is live. - -## Dependencies - -- Depends on the unified file-backed server configuration contract. -- Depends on group-0 service registry and watch/notify support. -- A first version may poll if watch delivery is unavailable; correctness must - not depend on notifications. - -## Acceptance - -- Given the same release image, starting with a single-container profile and - with each cluster performance profile produces the documented effective - settings and records the selected version. Invariants: I1, I2, I4. E2E test. -- Given profile defaults plus scoped and explicit overrides, resolution produces - one deterministic validated configuration; an unknown profile, zero interval - or unsafe election/lease combination fails before partial startup. Invariants: - I1, I3. Unit test. -- Given a test tuning selection and a 3-second DiskDB cadence override, normal - deployment creates the effective configuration and the actual timer consumes - that value; no test server build or hidden fixture mutation is used. - Invariants: I2, I4. Integration test. -- Given a selected cluster profile, service restart and admission of a new node - preserve or inherit the profile and report effective overrides accurately. - Invariants: I1, I4. E2E test. -- Given heartbeat and metadata synchronization settings differ, each runtime - timer uses its documented setting; updates follow their dynamic or - restart-required classification. Invariants: I3, I4. Integration test. - -- Given cluster, service, node, and instance records, resolving a candidate is - deterministic and produces the documented precedence. Invariant: every - instance resolves the same revision from the same records. Unit test. -- Given a candidate containing unknown fields during a mixed-version rollout, - the old service reports unsupported configuration without corrupting or - partially applying it. Invariant: candidate validation is atomic. E2E test. -- Given dynamic and restart-required changes in one revision, the service - applies only the validated dynamic subset and reports the static subset as - pending restart. Invariant: runtime state and reported revision agree. - Integration test. -- Given group 0 becomes unavailable, a running service retains its last applied - config and a restarting service uses its last validated cache or local file - according to the bootstrap contract. Invariant: control-plane loss does not - silently reset tunables to compiled defaults. E2E test. -- Given concurrent publishers, compare-and-set revision fencing permits one - winner and produces an auditable rejected update. Invariant: published - revisions form one ordered history. Integration test. -- Given a staged rollout and an unhealthy acknowledgement, rollback restores a - prior valid revision and all targeted instances converge or report a precise - blocker. Invariant: rollout status exposes all configuration drift. E2E test. - -Required gates: - -- `pixi run -- cargo fmt --all -- --check` -- `pixi run rs-lint` -- `pixi run test-core` -- `pixi run test-storage` -- `pixi run clean --runtime-only && pixi run test-container-e2e --scope console.operations` -- `pixi run test-suite` diff --git a/doc/backlog/R144-chunk-kv-partition-merge.md b/doc/backlog/R144-chunk-kv-partition-merge.md deleted file mode 100644 index d2c570682..000000000 --- a/doc/backlog/R144-chunk-kv-partition-merge.md +++ /dev/null @@ -1,261 +0,0 @@ - - - -### R144: chunk-kv — Adjacent partition merge - -**Status: Deferred.** Implement after R140, R142, R143, and R145 establish -stable manifest reuse, split/transfer fencing, group-0 transition recovery, and -routed retry semantics. - -## Problem - -R142 can split a partition and transfer ownership, but it intentionally exposes -no merge operation. Without merge, many formerly hot ranges can remain as -permanent small partitions after their load or data volume falls, increasing -routing, checkpoint, stream, and per-partition memory overhead. - -Merge is not the inverse of split at the control-plane level. Two adjacent -parents can have different owners, ownership epochs, tree manifests, WAL -streams, mutation sequences, checkpoints, and in-flight operations. Publishing -a merged child after fencing only one parent can lose acknowledged writes or -leave an overlapping writer. Reusing both trees also requires a valid combined -B+tree root and a leaf sibling chain that crosses the old boundary without -escaping the merged range. - -The existing crowdb-tree snapshot APIs rebuild or export one tree and R140 -rebuilds one manifest into a subrange; neither composes two roots. Copying every -key through Rust would lose page-reference reuse and make time and memory -proportional to the combined data set. The merged partition also cannot reset -its logical mutation sequence below a resolved sequence already stored in a -reused page. - -The root design links are R140's chunk page-store design, R142's partition -lifecycle design, and R143's group-0 range-map and transition design. - -## Solution - -Add an explicit two-parent merge transition that prepares one combined child -from adjacent immutable manifests, briefly fences both parents for catch-up, -and activates the child through one epoch-fenced group-0 range-map revision. - -1. Extend the permanent tree and chunk-KV designs with adjacent-manifest - composition, dual-parent lifecycle fencing, destination sequence seeding, - publication, rollback, recovery, retention, and failure semantics. Only two - exactly adjacent partitions may merge in one transition; repeated or - multi-way merge is composed from successful pairwise transitions. -2. Add a native crowdb-tree operation that builds a destination manifest for - `[left.start, right.end)` from the left and right pinned manifests. Reuse - verified interior pages from both sources, build a combined root, and - rewrite boundary ancestors plus the last left leaf so its sibling reference - leads to the first right leaf. It must not materialize all keys in Rust. -3. Accept an idempotent merge plan containing `transition_id`, both parent - identities, exact adjacent bounds, epochs and manifests, and the destination - identity/range/epoch. Reject gaps, overlaps, reversed order, stale inputs, - changed plans under one transition ID, or a parent already participating in - another lifecycle transition. -4. Enter `MergePreparing` on both parents, publish and pin independent base - checkpoints at sequences `b_left` and `b_right`, and build the destination - base while both parents continue serving. Seed its destination-local - sequence at `max(b_left, b_right)`, then replay both source deltas while the - parents remain writable. Preserve order within each source, assign each - replayed mutation a new contiguous destination sequence, and checkpoint the - two source cursors with that sequence so restart can resume without relying - on colliding source-local numbers. Preserve source WAL prefixes and every - reused page-pack reference until commit or proven abort. -5. Close mutation admission on both parents only after destination replay lag - is below configured record, byte, and estimated-time limits. If either - source cannot catch up, keep both serving and cancel or retry preparation. - Once fenced, drain all operations admitted on either side, reconcile their - durable streams through `c_left` and `c_right`, and assign the bounded final - delta contiguous destination sequences. Sequence overflow fails before the - dual fence. -6. Persist and validate one destination manifest at its final destination - sequence `d`, and create a distinct empty destination stream whose next - mutation sequence is `d + 1`. Return one immutable merge artifact binding - both exact parents, source cutover cursors, recorded conditional outcomes, - and retained request-result windows to that destination. Coalesce an exact - duplicate `(client_instance_id, client_sequence, operation_digest, result)`; - a repeated request identity with a different digest or result rejects merge - before fencing as `RequestConflict`. Reads may continue from the authoritative - parents during preparation and fencing; new mutations receive a retryable - merge error after the dual fence. -7. Extend R143's catalog transition so one new immutable range-map generation - replaces both still-current parents with the exact destination and publishes - one head. Current group-0 sysdata exposes blind puts rather than general - compare-and-swap, so publication must not depend on coordinated updates to - two live parent keys. Commit proof retires both parents and permits - destination activation. Abort proof resumes both unchanged parents. An - ambiguous head outcome keeps both parents mutation-fenced until an - authoritative head read resolves it. -8. Make preparation, fencing, and resolution restart-idempotent. A replacement - group-0 domain monitor reconstructs the transition, validates manifests and - stream tails, and either reuses the exact artifact or restarts an unpublished - build. If either parent owner dies, it waits for the old serving lease fence - before recovering that parent or aborting; it never infers commit from local - files, memory, or a service heartbeat. -9. Select a healthy destination owner and open the merged artifact as - `Prepared`. Publish its serving lease only after the catalog head commits and - both parent grants are fenced. A balance or split planner cannot start a - second lifecycle transition on either parent or the destination until merge - commit or abort is proven. -10. Reclaim parent manifests, streams, request-result records, and abandoned - destination objects only after catalog references and transition pins allow - it. Shared page packs survive as long as the destination or another manifest - references them. -11. Add merge metrics for preparation, dual-fence drain, delta replay, reused - and rewritten pages, publication/abort, ambiguous resolution, source pins, - orphan bytes, owner failures, and write-fence duration. - -Edge outcomes are explicit: non-adjacent partitions are rejected; either stale -parent epoch fails the whole transition; failure before publication leaves both -parents authoritative; partial parent replacement is invalid; sequence -overflow fails before fencing; and the destination never serves concurrently -with either writable parent. - -Operation outcomes are explicit during merge: - -- Gets, ordered seeks, and bounded scans already admitted to a parent complete - on that authoritative parent view. A new request after catalog publication is - routed to the destination; a stale request receives `NotMyRange`. -- Successful and condition-failed mutations admitted before the fence remain in - source order, including their recorded CAS result and request ID. Mutations - after the fence receive a retryable lifecycle result without WAL append. -- A lost pre-fence mutation response can be retried on the destination after - commit and returns the retained original result rather than executing again. -- Parent scan tokens are invalid after commit and return refresh-required. The - client cannot resume a parent token against the wider destination range. -- A journal, tree, owner, or group-0 failure follows R142/R143 containment and - lease rules; it never makes the unfenced parent or uncommitted destination - authoritative by inference. - -## Dependencies - -- Depends on R140's generation-addressed manifests, verified page fences, - immutable reference retention, and native async chunk backend. R144 extends - its range rebuild machinery with two-manifest adjacent composition. -- Depends on R141's independent stream replay, fencing, stable positions, and - prefix pins for both parents and the destination stream. -- Depends on R142's partition manager, logical mutation sequences, lock-free - admission fence, lifecycle proof boundary, checkpoint tuples, and fault - behavior. R142 remains complete without R144. -- Depends on R143's authoritative paged range map, monotonic epochs, persisted - transitions, domain monitor, owner serving grants, and atomic catalog-head - publication. R143 must add a two-parent merge plan and exact commit/abort - proof before R144 can ship. -- Depends on R145 for routed retry, request-identity persistence, scan replanning, - and real-process merge E2E coverage. - -## Acceptance - -- Given adjacent parents `[a, m)` and `[m, z)` with verified manifests, when - native composition completes, assert the destination scan equals their - ordered union, its root covers `[a, z)`, and the rewritten sibling chain does - not escape that range. Invariant: tree composition preserves exact key and - navigation bounds. Integration test. -- Given fully contained interior pages and pages touching `m`, when composition - runs, assert eligible interiors retain their page references while boundary - ancestors and the final left leaf are rewritten. Invariant: merge reuses only - pages whose references remain structurally valid. Unit test. -- Given parents with a gap, overlap, reversed order, stale epoch, or changed - plan under the same transition ID, when merge preparation is requested, - assert it fails before either mutation fence. Invariant: only one exact, - adjacent, current parent pair can enter a merge. Unit test. -- Given writes continue on both parents during destination base construction, - when preparation runs, assert the writes remain acknowledged and both base - manifests and WAL prefixes stay pinned. Invariant: base construction does not - impose the merge write outage. Integration test. -- Given both parents contain colliding source-local sequence numbers and later - mutations, when replay and the dual fence drain through `c_left` and - `c_right`, assert every acknowledged mutation appears once, destination - sequences are contiguous above `max(b_left, b_right)`, and the next sequence - is greater than every reused-page sequence. Invariant: independent parent - histories converge without sequence collision or rollback. E2E test. -- Given retained request windows contain an exact duplicate identity/result or - the same `(client_instance_id, client_sequence)` with conflicting digests, - when preparation combines them, assert the exact duplicate is coalesced and - the conflict aborts before either parent fence. Invariant: merge cannot make - request retry identity ambiguous. Unit test. -- Given either source write rate keeps destination replay lag above the fence - limits, when merge preparation reaches its catch-up deadline, assert both - parents remain serving and no dual fence begins. Invariant: merge never - begins a predictably unbounded final outage. Integration test. -- Given one parent faults or fails to drain, when finalization runs, assert no - destination artifact becomes publishable and the other parent does not - resume until group 0 proves abort. Invariant: dual-parent fencing is one - transition, not two independent decisions. Integration test. -- Given one parent owner dies during preparation or after the dual fence, when - the group-0 monitor resumes the transition, assert it waits for the old lease - fence, recovers the exact durable parent suffix, and commits or aborts without - granting overlapping authority. Invariant: owner death cannot shortcut a - dual-parent fence. E2E test. -- Given a valid merge artifact, when group 0 publishes its epoch-fenced - range-map activation record, assert the visible revision replaces both - parents or neither and only an exact commit proof activates the destination. - Invariant: the range map never exposes a gap, overlap, or partial merge. E2E - test. -- Given a timeout after publication is submitted, when its result is unknown, - assert both parents remain mutation-fenced until an authoritative read - returns commit or abort proof. Invariant: ambiguity cannot create concurrent - writable ownership. E2E test. -- Given the destination is prepared on a healthy server, when the catalog head - has not committed or either parent grant remains valid, assert it rejects all - data operations; after commit and both fences, assert one destination serving - lease is issued. Invariant: preparation and storage readiness do not grant - ownership. E2E test. -- Given a crash during preparation, fencing, commit, or abort, when a new - coordinator resumes the persisted transition, assert the final range map and - manifests match one valid all-parent or one valid destination outcome. - Invariant: merge is restart-idempotent. E2E test. -- Given source manifests, streams, and page packs referenced by a live or - unresolved transition, when GC runs, assert they remain; after committed or - aborted references and pins clear, assert only unreachable objects become - reclaimable. Invariant: reclamation never precedes merge resolution. - Integration test. -- Given concurrent reads during preparation and fencing, when the merge commits - or aborts, assert every read is served by an authoritative parent snapshot or - the published destination and never observes keys outside its routed range. - Invariant: merge preserves read range isolation. Integration test. -- Given a pre-fence successful or condition-failed mutation response is lost, - when its request ID is retried against the committed destination, assert the - original result is returned and no condition is re-evaluated. Invariant: - merge preserves the deduplication and CAS-result retention boundary. E2E test. -- Given a parent scan token before merge commit, when it is resumed after the - destination becomes current, assert it returns refresh-required rather than - scanning the widened range. Invariant: merge cannot silently change a scan's - range or pagination history. Integration test. -- Given merge preparation, fencing, replay, publication, abort, ambiguity, and - GC, when metrics are collected, assert each outcome is attributed to the - transition and both parents. Invariant: all material merge states are - observable. Unit test. - -Required gates: - -- `pixi run tree-fmt` -- `pixi run tree-lint` -- `pixi run test-tree-ct` -- `pixi run test-tree-ffi` -- `pixi run -- cargo fmt --all -- --check` -- `pixi run rs-lint` -- `pixi run -- cargo test -p crowdb-chunk-stream --all-targets` -- `pixi run -- cargo test -p crowdb-chunk-kv --all-targets` -- `pixi run -- cargo test -p crowdb-chunk-kv-client --all-targets` -- `pixi run -- cargo test -p crowdb-chunk-kv-server --all-targets` -- `pixi run clean --runtime-only && pixi run test-server` - -## Open Questions - -- Should the destination normally stay on the left owner, the less-loaded - parent owner, or a third server selected by the balance score? Reusing one - parent owner reduces warm-up; choosing by load can avoid an immediate follow- - up transfer. -- Should merge initially be operator-triggered only, or may the balance monitor - propose and automatically execute merges after a long low-load window? - Automatic merge controls partition overhead but adds a second autonomous - topology-changing policy beside split. -- Is native two-manifest composition required for the first merge release, or - may it fall back to a bounded background rebuild when page topology prevents - safe reuse? Composition minimizes IO; a rebuild fallback is simpler but may - make large merges impractically slow. -- What maximum dual-fence duration is acceptable before merge must abort? A - short bound protects foreground availability; a longer bound tolerates slow - final replay and checkpoint publication. diff --git a/doc/backlog/R147-tree-chunk-gc.md b/doc/backlog/R147-tree-chunk-gc.md deleted file mode 100644 index c45c91fd8..000000000 --- a/doc/backlog/R147-tree-chunk-gc.md +++ /dev/null @@ -1,139 +0,0 @@ - - - -### R147: crowdb-tree / chunkdb — Reclaim B+tree chunk strips - -**Status: Deferred.** Implement after R140 establishes immutable page packs, -manifest retention, and durable reclaim candidates. R146 must make abandoned -active chunks sealable before they can enter physical reclamation. - -## Problem - -The existing crowdb-tree GC removes unreachable page IDs from the mapping table -and eventually releases local page-store extents. For the R140 backend, that is -only logical reclamation: page packs reside in chunk strips whose DiskDB blocks -remain allocated until chunkdb releases them. Deleting a mapping entry without -this second step leaks chunk capacity; releasing a strip too early can destroy -pages still referenced by another manifest, split child, or pinned snapshot. - -A strip can also contain both dead and live page packs. Chunkdb's proposed -in-chunk GC in `doc/backlog/R92-chunkdb-in-chunk-gc.md` is the correct ownership -boundary for releasing strip storage, but crowdb-tree must first remove or -relocate every live reference and publish that fact durably. The release flow -must survive tree, KV-server, and chunkdb restart and must tolerate an ambiguous -release response. - -The root storage contracts are -`doc/design/tree/design-crowdb-tree-storage.md` and -`doc/design/chunkdb/design-crowdb-chunkdb.md`. - -## Solution - -Use a two-phase, manifest-fenced flow: crowdb-tree proves a strip unreachable, -then chunkdb idempotently reclaims it through a durable cleanup intent. - -1. Extend R140 logical GC to emit a durable reclaim candidate containing tree - identity, source manifest generation, chunk ID and revision, strip sequence - and byte range, and the page-pack ordinals formerly stored there. Emit it - only after a newer manifest that removes those references is durable. -2. Advance a candidate to releasable only after every retained manifest, - split-child manifest, and in-memory snapshot pin that can name any listed - page pack is past the reclamation watermark. Recompute reachability from the - current manifest before dispatch; a reachable ordinal cancels the candidate - as stale. -3. Add or complete the generic R92 `ReclaimStrip` chunkdb operation for both - `BtreePage` and `PageIndex` chunks. Its request carries a deterministic - reclaim ID, chunk ID, expected chunk revision, and exact - strip identities; under the existing lifecycle guard it verifies that the - chunk is Sealed and the identities still match, removes the strips from the - published layout, and persists the reclaim ID plus qualified DiskDB cleanup - intents. Retired segments are freed only after the layout-validity window. - A retry with the same reclaim ID returns the persisted result. -4. Delete an entire sealed chunk when all of its strips are releasable. For a - partially dead strip, do not punch arbitrary byte holes in the first - release. Copy its live page packs to fresh R140 B+tree chunks, verify the - copies, atomically publish a new manifest with replacement references, wait - for the old generation and reader-layout window to expire, and then reclaim - the whole old strip. This gives a workable compaction flow without changing - stable chunk offsets or requiring R92 `CollapseStrip` and `MergeStrips`. -5. Persist GC state as `Prepared`, `Releasing`, or `Complete` in the injected - root catalog before issuing chunkdb mutations. On startup, the current tree - owner resumes non-complete records; chunkdb independently resumes its - segment cleanup intents. An ambiguous RPC result is resolved by querying - the chunk revision and strip identities before retrying. -6. Bound candidates, bytes copied, concurrent chunk mutations, and work per - maintenance pass. GC and repacking run outside page lookup, apply, and - mapping-resolution hot paths and add no lock to them. Foreground checkpoint - publication wins the existing manifest gate; stale GC output becomes - orphan work and is retried from the newer generation. -7. Expose logical bytes retired, physical bytes released, pending and oldest - candidate age, live bytes copied, reclaim conflicts, ambiguous outcomes, - cleanup retries, and bytes retained by manifest or snapshot pins. - -## Dependencies - -- Depends on R140 for generation-addressed page packs, immutable manifests, - reference retention, logical GC, and the injected root catalog. -- Depends on R146 so crashed writers eventually produce Sealed chunks. GC skips - Active chunks and remains safe while sealing is unavailable. -- Uses R92's generic in-chunk `ReclaimStrip` boundary. If the broader collapse - and merge work is still unlanded, R147 implements only the idempotent - whole-strip operation needed here and leaves those compaction operations - deferred. -- R142 supplies the group-backed production root catalog and schedules tree - maintenance. Tests may use R140's in-memory epoch-fenced catalog. -- DiskDB qualified frees and chunkdb cleanup-intent reconciliation remain the - physical block-release authority. - -## Acceptance - -- Given tree GC removes a page from the current mapping but an older manifest - or pinned snapshot still references its pack, when GC evaluates the strip, - assert no chunkdb reclaim is issued and retained-byte metrics identify the - pin. Invariant: logical death does not imply physical unreachability. - Integration test. -- Given all packs in one strip are unreachable past the watermark, when the - persisted candidate is dispatched, assert chunkdb removes that exact strip, - waits out layout validity, frees its qualified segments, and marks the record - Complete. Invariant: physical release follows durable reference removal. - Integration test. -- Given every strip of a sealed chunk is releasable, when GC runs, assert it - uses whole-chunk deletion and all cleanup intents complete idempotently. - Invariant: an empty sealed chunk retains no allocated strips. Integration - test. -- Given a strip mixes live and dead packs, when GC compacts it, assert live - packs are copied and verified, the replacement manifest is published before - the old strip is retired, and reads remain correct across the cutover. - Invariant: partial-strip reclamation never creates a page-reference hole. - E2E test. -- Given a checkpoint publishes generation `g + 1` while GC works from `g`, when - GC reaches publication, assert it cannot replace `g + 1`, its unreferenced - copied packs become orphan candidates, and work restarts from the current - manifest. Invariant: GC never rolls back acknowledged tree state. - Integration test. -- Given the KV-server restarts in each of `Prepared` and `Releasing`, when the - partition reopens, assert it resumes the same candidate without an early or - duplicate free. Invariant: tree-side GC progress is restart-safe. E2E test. -- Given chunkdb restarts after layout retirement but before DiskDB cleanup, - when reconciliation resumes, assert the retired segments are freed after the - validity window and the cleanup intent clears. Invariant: physical release - survives chunkdb restart. E2E test. -- Given a reclaim RPC response is lost, when the tree owner queries and retries - the same reclaim ID, assert the persisted result identifies the exact retired - strips; a different ID or replacement identity conflicts. Invariant: - ambiguous completion cannot reclaim a replacement strip. Integration test. -- Given bounded GC budgets and sustained foreground traffic, when many strips - become reclaimable, assert each pass stays within its byte, candidate, and - concurrency limits while candidates eventually complete. Invariant: physical - GC is bounded and makes progress without entering tree hot paths. Integration - test. - -Required gates: - -- `pixi run tree-fmt` -- `pixi run tree-lint` -- `pixi run test-tree-ct` -- `pixi run test-tree-ffi` -- `pixi run -- cargo test -p crowdb-chunkdb --all-targets` -- `pixi run -- cargo fmt --all -- --check` -- `pixi run rs-lint` diff --git a/doc/backlog/R148-chunk-stream-scale-out.md b/doc/backlog/R148-chunk-stream-scale-out.md deleted file mode 100644 index 56a5b1a94..000000000 --- a/doc/backlog/R148-chunk-stream-scale-out.md +++ /dev/null @@ -1,154 +0,0 @@ - - - -### R148: chunk-stream/chunk-kv — Partition metadata scale-out and sealed-chunk EC - -**Status: Deferred.** Implement after the R141/R142/R143 single-group, -three-way-mirror production path is complete and measured. All features in -this requirement are disabled by default. - -## Problem - -R141 deliberately binds one stream's metadata to one nonzero KV group and -writes each active stream chunk as three mirrors. R142 stores the same -partition's R140 tree root catalog in that metadata group: tree page packs and -mapping data remain in ChunkDB/DiskIO chunks, while the Paxos KV records hold -the owner authority, current root manifest, immutable manifest generations, -and reference-segment indexes that locate those chunks. Group 0 contains only -the authoritative partition/range ownership and metadata-group binding. - -That baseline keeps one partition's checkpoint metadata together and keeps -high-frequency root publication out of group 0, but it does not move a hot or -oversized partition metadata namespace between groups, shard its stream extent -index and tree reference indexes across groups, or reduce the long-term -storage cost of sealed chunks. - -These changes require explicit cutover protocols. Moving keys without fencing -the authoritative binding can expose mixed metadata generations, while -converting an active chunk to erasure coding can race acknowledged-cursor -advancement and make the writer's durability contract ambiguous. - -## Solution - -1. Treat each partition's metadata placement as one binding-generation unit. - The selected nonzero KV group owns both the R141 stream namespace and the - R142 tree root namespace. The tree namespace contains - `authority(tree_id) = (owner_epoch, current_generation)`, the current - manifest, immutable generation manifests, and reference segments. A tree - root publish CASes the authority record and atomically writes the new - authority value, current manifest, and immutable generation in one Paxos KV - batch. Old epochs and same-generation concurrent publishers fail closed. -2. Add a binding-generation migration state machine for moving the complete - partition metadata unit from one KV group to another. Bulk-copy immutable - stream pages, tree manifest generations, and reference segments while the - source serves. Then fence and drain partition mutation admission, publish a - final checkpoint, copy the remaining current stream/root records, and - verify the destination contains the exact tree generation, applied - mutation sequence, acknowledged stream cursor, and replay-retention floor. - Atomically publish the destination group, higher binding generation, and - higher ownership epoch in group 0 before the destination can serve. Retain - the source read-only for a grace interval and delete it through bounded, - idempotent cleanup. Append and root publication never dual-write groups. -3. Make reopen resolve group 0 first and accept stream and tree metadata only - from the group named by the same binding generation. A prepared destination - may verify copied metadata and prewarm chunk routes but cannot append, - checkpoint, reclaim, or serve until the exact group-0 proof arrives. A - timeout around group-0 publication keeps the source fenced and both sides - non-writable until an authoritative read proves commit or abort. -4. Add optional per-partition metadata sharding only after whole-unit migration - works. Keep one small generation-fenced directory in the partition's - primary metadata group. Route stream extent pages by logical offset and - tree reference segments by stable object ID; keep the tree authority and - current root manifest together in the primary group so one CAS remains the - publication point. Random stream seek and tree-root open remain logarithmic, - and trim/reclaim can advance nonzero shard floors without scanning or - rewriting earlier shards. -5. Keep new and active chunks three-way mirrored. After a chunk is sealed and - no writer can advance its cursor, optionally submit it to chunkdb's existing - mirror-to-EC conversion workflow. Verify the EC layout before atomically - switching the chunk layout; retry or crash leaves the mirror layout - readable. Chunk-stream metadata continues to name the chunk, not its strips - or current layout. -6. Gate migration, sharding, and EC conversion behind separate configuration - switches that default to disabled. Old bindings and mirror-only chunks keep - their current meaning, and disabling a switch stops new background work - without invalidating already published layouts. -7. Bound copy, verification, conversion, and cleanup work across both metadata - namespaces. Expose binding age, primary and shard group IDs, stream/tree - metadata bytes, migration phase/lag, copied and verified bytes, source - retention, sealed mirror backlog, EC conversion latency/failure, retained - mirror bytes, and cleanup retries. - -## Dependencies - -- Depends on R141 for the production stream metadata and mirrored chunk - baseline, including stable logical offsets and sealed-chunk lifecycle. -- Depends on R142 for the chunk-backed tree, generation-CAS root catalog, - checkpoint tuple, and epoch-fenced partition lifecycle. -- Depends on R143 for authoritative group-0 binding publication and operator - configuration. -- Reuses chunkdb's durable layout-conversion state machine and R103's - copy/verify/cutover principles, but metadata-group migration is a distinct - namespace operation. -- The metadata publication fence selected in R141 must apply to binding and - shard-directory generations before this requirement begins. - -## Acceptance - -- Given a populated stream, when its metadata binding migrates to another KV - group with a checkpointed chunk-backed tree, and failures are injected before - bulk copy, final fence, verification, group-0 publication, and source cleanup, - assert every reopen selects one metadata group and recovers the exact same - tree generation, applied sequence, stream cursor, retained retry results, and - key/value contents. Invariant: migration never combines stream metadata and - a tree root from different binding generations. E2E test. -- Given an old source owner and a destination owner, when group 0 publishes the - higher epoch and destination group, assert the source cannot append, publish - a tree root, or reclaim metadata, while the destination cannot do any of - those operations before exact publication proof. Invariant: metadata - placement migration never creates two writable authorities. Integration test. -- Given two same-epoch checkpoint publishers with the same expected tree - generation, when both publish concurrently, assert exactly one authority CAS - and its matching current and immutable manifests commit. Repeat after moving - ownership to a higher epoch and assert the old epoch always fails. Invariant: - a root generation and its authority fence are one atomic Paxos decision. - Integration test. -- Given a sharded partition metadata namespace spanning several KV groups, when - readers seek near the stream head, middle, and tail, open the current tree - root, resolve reference segments, trim complete leading extent shards, and - reclaim old tree generations, assert lookup remains logarithmic and no live - metadata page is rewritten or lost. Invariant: sharding preserves logical - addressing and one primary root publication point. Integration test. -- Given a sealed mirrored chunk, when EC conversion succeeds, fails, or crashes - around layout publication, assert readers see either the verified mirror - layout or the verified EC layout and the stream's logical extent is - unchanged. Invariant: conversion cannot expose a partial layout. E2E test. -- Given default configuration, when streams append, roll, and reopen, assert no - metadata migration, sharding, or EC background task starts. Invariant: R141's - measured mirror-only behavior remains the default. Integration test. -- Given configured work limits and a large backlog, when maintenance runs, - assert foreground append/tree-read latency remains within the selected - regression threshold and stream copy, tree metadata copy, verification, EC, - and cleanup queues each report bounded progress. Invariant: coordinated - scale-out work cannot create unbounded foreground interference. Benchmark. - -Required gates: - -- `pixi run -- cargo fmt --all -- --check` -- `pixi run rs-lint` -- `pixi run -- cargo test -p crowdb-chunk-stream --all-targets` -- `pixi run -- cargo test -p crowdb-chunk-kv --all-targets` -- `pixi run -- cargo test -p crowdb-chunkdb --all-targets` -- `pixi run clean --runtime-only && pixi run test-server` - -## Open Issues - -- The CI log for - `small_object_writer_e2e::eight_closed_mirror_strips_become_one_durable_ec_strip_without_reread` - failed at `location strip`, while the exact test, its 16-test suite and the - full `test-server` task pass locally at default test concurrency. CI run - `36449749925` completed its "Upload test logs on failure" step, but the run - artifact list contains only `docker-preview-1`; the `runtime-server` artifact - is absent. The test now prints the queried chunk and location on this failure. - Wait for a recurrence and use those diagnostics to identify the first - divergent state before changing the assertion or retry policy. diff --git a/doc/backlog/R168-s3-shared-object-reclamation.md b/doc/backlog/R168-s3-shared-object-reclamation.md deleted file mode 100644 index e44291a0f..000000000 --- a/doc/backlog/R168-s3-shared-object-reclamation.md +++ /dev/null @@ -1,73 +0,0 @@ - - - -### R168: access server / S3 — Shared small-object reclamation - -## Status - -**Deferred until R95 qualified chunk-range deletion is implemented and basic -S3 deletion is stable.** Before then, the basic delete path removes metadata -and records exact -pending ranges as safe logical garbage. - -## Problem - -Small objects use the shared chunk writer to avoid private-chunk waste. Their -bytes cannot be reclaimed by deleting the whole chunk, and an unqualified byte -range could erase neighboring live objects or a reused generation. The basic -delete path needs -a restart-safe way to turn pending logical garbage into safe range deletion. - -The lifecycle boundary is -`doc/design/access-server/s3/design-crowdb-access-s3.md` §5. - -## Solution - -1. Persist a range-cleanup record containing object generation, chunk identity, - writer/layout generation, exact offset/length, reader-validity deadline, and - stable cleanup identity before invoking physical deletion. -2. After metadata invisibility and the grace period, call R95's idempotent - `DeleteChunkRange(chunk_id, offset, size)` interface. The RPC exists before - R95 but returns not-implemented and performs no mutation until chunkdb owns - the required range validation and used-bitmap lifecycle. -3. Reconcile unknown results from chunk state, retry bounded transient failures, - and quarantine generation/ownership mismatches for inspection. Never widen, - merge, or align a range across live neighbors merely to reclaim space. -4. Track logical garbage, eligible bytes, completed bytes, retry age, and - quarantine without high-cardinality labels. Apply backpressure to cleanup - independently of foreground DELETE. -5. Mark the S3 cleanup complete only after the qualified range is known deleted; - absence of cleanup never changes the object's already-absent visibility. - -## Dependencies - -- Depends on R95 and the basic S3 deletion contract. -- Uses immutable generation/data references and bounded cleanup admission. -- R169 may compact fragmented shared chunks after individual dead ranges are - marked; it is not required for correctness here. - -## Acceptance - -- Given adjacent live and deleted small objects in one shared chunk, when the - qualified range deletion runs after grace, assert only the deleted object's - exact bytes become reclaimable. Invariant: shared reclamation cannot damage - a neighbor. E2E test. -- Given stale chunk/writer generation or a range that has been reused, when - cleanup runs, assert R95 rejects/quarantines it without mutation. Invariant: - offset and length alone are never deletion authority. Integration test. -- Given lost replies, restart, and repeated cleanup delivery, when the - reconciler settles, assert one terminal result is recorded and visibility - stays absent throughout. Invariant: range cleanup is idempotent and separate - from logical deletion. E2E test. -- Given a sustained delete backlog, when configured cleanup limits are reached, - assert foreground object operations retain their reserved resources and - garbage metrics remain bounded-cardinality. Invariant: reclamation cannot - starve the data path. Integration test. - -Required gates: - -- `pixi run -- cargo test -p crowdb-access-s3 --all-targets` -- `pixi run -- cargo test -p crowdb-chunk-client --all-targets` -- `pixi run -- cargo test -p crowdb-chunkdb --all-targets` -- `pixi run -- cargo fmt --all -- --check` -- `pixi run rs-lint` diff --git a/doc/backlog/R169-s3-shared-chunk-tree-gc.md b/doc/backlog/R169-s3-shared-chunk-tree-gc.md deleted file mode 100644 index 00b747303..000000000 --- a/doc/backlog/R169-s3-shared-chunk-tree-gc.md +++ /dev/null @@ -1,77 +0,0 @@ - - - -### R169: access server / S3 — B+tree and shared-chunk garbage collection - -## Status - -**Deferred until R147, R168, and the basic S3 workload provide measured garbage -and fragmentation data.** It is unblocked when qualified dead ranges and -metadata tombstones can be enumerated safely and compaction benefit is known. - -## Problem - -Qualified range deletion can mark shared object bytes dead, but mixed-live -chunks may remain fragmented and S3 metadata trees accumulate obsolete object -generations, tombstones, completed upload intents, and cleanup records. Simple -age-based deletion can race snapshots/readers or discard the only recovery -evidence for an ambiguous operation. - -The lifecycle scope is -`doc/design/access-server/s3/design-crowdb-access-s3.md` §5; generic tree -strip reclamation is R147. - -## Solution - -1. Measure dead/live ratios and plan bounded compaction of shared object chunks. - Copy only still-reachable ranges into new shared chunks, verify contents, - atomically replace generation references, then retire old layouts after the - reader-validity fence. -2. Feed unreachable storage strips and chunks into the qualified generic - in-chunk/whole-chunk lifecycle rather than freeing DiskDB ranges directly - from S3. -3. Add S3 metadata retention rules for obsolete generations, tombstones, - completed/aborted upload intents, and completed cleanup records. Prove they - are no longer visibility, retry, reader, or recovery authority before - deleting their Chunk-KV keys. -4. Integrate tree logical-GC output with R147's manifest-fenced strip - reclamation. Compaction and metadata GC are restart-safe, idempotent, - throttled, and lower priority than foreground traffic. -5. Expose reclaimable, copied, freed, pinned, quarantined, and amplification - metrics. Disable compaction when its projected benefit does not exceed the - configured threshold. - -## Dependencies - -- Depends on R147 and R168; R92/R95 provide qualified in-chunk/range lifecycle - mechanisms where applicable. -- Uses the basic milestone's generation metadata, retry identities, tombstones, - and background admission. -- It is not a prerequisite for basic S3 visibility or large-object deletion. - -## Acceptance - -- Given a shared chunk with interleaved live/dead ranges and pinned old readers, - when compaction runs, assert live bytes remain exact, metadata switches - atomically, and the old layout is reclaimed only after pins expire. - Invariant: compaction never changes visible object bytes. E2E test. -- Given obsolete and still-authoritative generations, intents, tombstones, and - cleanup records, when metadata GC evaluates them, assert only records proven - irrelevant to visibility/retry/recovery are removed. Invariant: GC cannot - erase recovery authority. Integration test. -- Given a crash after copy, metadata switch, or old-layout retirement, when GC - restarts, assert it resumes to one valid layout without leaks outside the - recorded candidate set. Invariant: every compaction phase is idempotent. - E2E test. -- Given low dead ratio or foreground pressure, when the planner evaluates work, - assert it skips/throttles compaction and respects reserved resources. - Invariant: GC benefit and load gates protect foreground service. Integration - test. - -Required gates: - -- `pixi run -- cargo test -p crowdb-access-s3 --all-targets` -- `pixi run test-tree-ct` -- `pixi run -- cargo test -p crowdb-chunkdb --all-targets` -- `pixi run -- cargo fmt --all -- --check` -- `pixi run rs-lint` diff --git a/doc/backlog/R185-access-iceberg-cache-invalidation.md b/doc/backlog/R185-access-iceberg-cache-invalidation.md deleted file mode 100644 index cf2f3d28d..000000000 --- a/doc/backlog/R185-access-iceberg-cache-invalidation.md +++ /dev/null @@ -1,129 +0,0 @@ - - - -### R185: access server / Iceberg — Bounded cache and invalidation - -## Status - -Deferred until R178 through R184 stabilize identities, generations, digests, -lifecycles, clear fences, and the measured uncached correctness baseline. Cache is -not a dependency of the first correct Iceberg milestone. - -## Problem - -Authoritative reads on every request create an active-catalog hot key and repeat -name, head, metadata, manifest, and format parsing work. Independent caches would -multiply memory budgets and stale-data rules. Cross-server notification can reduce -staleness but cannot be authority because instances disconnect, register late, and -receive duplicated or reordered messages. - -The native Iceberg design resolves lease-plus-grace clear semantics and requires per-class limits chosen -by focused benchmarks. This requirement adds one Iceberg-owned cache manager while -preserving correct behavior when notifications or the complete cache are disabled. - -## Solution - -- **CACHE-I1 — Authority isolation:** a hit can avoid I/O but cannot create, - publish, rename, drop, clear, recover, authorize, or reclaim a resource. -- **CACHE-I2 — Bounded ownership:** total and class bytes, entries, one-entry size, - fills, queues, batches, fanout, retries, and retained owners have separate caps. -- **CACHE-I3 — Qualified identity:** mutable entries carry CatalogId plus epoch or - generation; immutable entries carry stable identity plus generation/digest. -- **CACHE-I4 — Optional notification:** loss, duplication, reorder, late join, and - unreachable receivers preserve behavior through expiry and authoritative refresh. -- **CACHE-I5 — Lock-free hit:** a hot hit takes no global or cross-class lock. -- **CACHE-I6 — Rename is not aliasing:** an old table name can retain a bounded - tombstone hint but never resolves as a normal alias. - -1. Add `cache/manager.rs`, `budget.rs`, `policy.rs`, `entry.rs`, `fill.rs`, and - `metrics.rs`. Reserve a validated global byte budget and class budgets for active - catalog, names, authorities, heads, negative/tombstone entries, parsed metadata, - serialized responses, manifests, format structures, and file records. Oversized - entries bypass cache. -2. Configure maximum entries, bytes, one-entry bytes, idle TTL, absolute lease, - negative TTL, fill concurrency, and retained-owner accounting per class. Derive - checked-in initial defaults from focused cold/hot and adversarial benchmarks; a - missing, zero-invalid, or unbounded setting fails startup. -3. Treat active catalog, names, namespace authorities, and TableHeads as mutable. - An expired entry must refresh authoritatively or fail closed. A mutation always - validates current durable revision and cannot succeed from cache alone. -4. Cache metadata projections under CatalogId, TableId, generation, JSON digest, - and projection version. Cache manifests and parsed Parquet/ORC/Avro/Puffin - structures under FileId, digest, and parser version. Whole large files never - enter this cache. -5. Add one bounded typed invalidation coalescer. After a durable mutation completes, - coalesce transitions by resource to the newest generation or strongest state. - Immutable files emit no event. Queue saturation drops optimization events rather - than delaying the mutation. -6. Start one internal `crowdb-rpc` listener per Access Server and publish its - endpoint through the Group-0 service registry. Use service-specific FlatBuffer - messages, bounded control size, deadlines, connection/fanout concurrency, and - retries. External Iceberg HTTP never accepts invalidation RPC. -7. Receivers validate the complete batch before applying any entry, then apply it - idempotently. Older generations are ignored. Table rename converts an existing - old-name entry to an authorization-neutral tombstone; only request-time current - authorization may disclose the destination. -8. Integrate clear with the native Iceberg contract: notification prompts eviction, but completion waits - for R178's persisted maintenance deadline and admitted/delegated grace. Start - lease age before the authoritative root read, never when a delayed reply arrives; - maintenance prevents fresh leases, while an existing lease may admit old-context - requests only until its original expiry. New-domain admission opens after the - grace proof. Retired-catalog GC uses - durable fences and never waits for physical cache eviction acknowledgements. -9. Expose per-class hit, miss, stale, fill, bypass, bytes, entries, eviction, expiry, - rebuild, notification, fanout, drop, and refresh-failure metrics. Benchmark hit - latency and contention for unrelated cache classes before enabling by default. - -## Dependencies - -- Depends on R177 through R184 and their final identities and measured baseline. -- Reuses Group-0 service discovery and `crowdb-rpc`; if registration is unavailable, - notification is disabled and TTL-only correctness remains. Static instance lists - are forbidden. -- Does not depend on R82 watch coalescing. This coalescer batches typed post-commit - cache transitions, not KV watch keys. -- Canonical JSON and file bytes remain the fallback for every missing, corrupt, - unknown-version, or evicted projection/cache entry. - -## Acceptance - -- Given mixed entries and fill pressure beyond every total and class limit, when - admission, eviction, and bypass run concurrently, assert retained owners, bytes, - entries, queues, and work stay within each configured cap. Invariant: CACHE-I2. - Integration test. -- Given an expired active-catalog or TableHead entry while Chunk-KV is unreachable, - when a request arrives, assert it fails closed and never reuses stale authority. - Invariants: CACHE-I1 and CACHE-I4. Integration test. -- Given duplicate, reordered, dropped, oversized, and malformed heterogeneous - invalidation batches, when receivers apply them, assert validation is atomic, - current entries only advance, and TTL refresh converges missed entries. Invariants: - CACHE-I3 and CACHE-I4. Integration test. -- Given a successful table rename and later authorized and unauthorized old-name - requests, when a cached tombstone exists and then expires, assert it never loads - the table, disclosure is authorization-filtered, and an uncached old name is - ordinary not-found. Invariant: CACHE-I6. E2E test. -- Given valid and corrupt metadata/format projections, when cached loads execute, - assert qualifiers prevent cross-generation reuse and every invalid entry falls - back to canonical bytes without authority mutation. Invariants: CACHE-I1 and - CACHE-I3. Integration test. -- Given a hot commit stream, unrelated mutations, queue saturation, disconnected - instances, and a late server, when fanout runs, assert work remains bounded, - newest generations converge, committed mutation latency is unaffected, and the - late server loads authority before readiness. Invariant: CACHE-I4. E2E test. -- Given a disconnected old-root lease holder and delayed cache fills, when clear - enters maintenance and publishes a new pointer, assert no lease extension and - no old-context response after the persisted completion boundary, even across a - recovering clear coordinator. Invariants: CACHE-I3 and CACHE-I4. E2E test. -- Given concurrent hits, fills, invalidations, expiry, and eviction across classes, - when contention benchmarks run, assert hot lookup takes no global lock and its - latency is independent of unrelated class activity. Invariant: CACHE-I5. - Integration test. - -Required gates: - -- `pixi run -- cargo test -p crowdb-access-iceberg --all-targets` -- `pixi run -- cargo test -p crowdb-access-server --all-targets` -- `pixi run -- cargo test -p crowdb-kv-client --all-targets` -- `pixi run -- cargo test -p crowdb-rpc-ffi --all-targets` -- `pixi run -- cargo fmt --all -- --check` -- `pixi run rs-lint` diff --git a/doc/backlog/R186-access-iceberg-orc-validation.md b/doc/backlog/R186-access-iceberg-orc-validation.md deleted file mode 100644 index 9e850c092..000000000 --- a/doc/backlog/R186-access-iceberg-orc-validation.md +++ /dev/null @@ -1,68 +0,0 @@ - - - -### R186: access server / Iceberg — Selected ORC validation - -## Status - -Deferred by user decision as an independent, unimplemented ORC follow-up. -Resume when ORC support is selected for implementation; it is not a completion -gate for the completed Parquet container/client workflows. Those workflows do -not absorb selected ORC validation. Selected-file validation continues to reject ORC explicitly -rather than representing an unchecked file as validated. - -## Problem - -Native FileIO can retain immutable ORC bytes, but container recognition does not -prove selected schema, row counts or delete semantics. Accepting a container hint -as proof would violate the canonical-file contract in -[the root design](../design/access-server/iceberge/design-crowdb-iceberg.md). -An official client configured to write ORC needs a distinct, tested validation -capability rather than an implicit fallback to Parquet checks. - -## Solution - -1. Extend `crowdb-access-iceberg::file` with bounded canonical ORC decoding. Use - the official ORC protobuf and pinned Iceberg mappings and SDK as authorities; - decode compression framing correctly and reject unsupported codecs or - encryption explicitly. Independently limit encoded/decoded bytes, protobuf - work, type depth/count and stripe count. -2. Extend `crowdb-access-iceberg::manifest` selected-file validation with field-ID, - historical schema, row-count and supported delete checks. Bind the manifest's - semantic kind without changing immutable file authority. Never trust hints - instead of canonical bytes. -3. Integrate ORC into complete selected snapshot validation only after its format - gates pass. Failure or cancellation publishes no validation result or head. - Keep unsupported selected ORC explicit before this capability lands; ordinary - immutable upload is not a table-selection or commit proof. - -## Dependencies - -- R180 supplies immutable files and canonical range reads. -- R181/R182 supply trusted table metadata and selected snapshot validation. -- Extend the completed REST/SDK conformance profile with ORC after this - requirement passes; initial Parquet-only acceptance is already complete. -- R185 caches are optional; uncached canonical reads remain correct. - -## Acceptance - -- Given official SDK ORC fixtures and supported compression variants, when - canonical metadata is decoded, assert schemas and row counts match the writer. - Invariant: canonical authority. Integration test. -- Given malformed framing, excessive expansion, deep types, invalid stripes, - encryption or unsupported codecs, when decoded, assert bounded failure with no - successful proof. Invariant: independent resource limits. Unit test. -- Given data and delete manifests with historical schemas and false counts, when - selected, assert compatible files pass and mismatched kind, identity, fields or - counts fail. Invariant: manifest-to-file binding. Integration test. -- Given a complete snapshot containing ORC and a failure or cancellation during - validation, when publication is attempted, assert no head advances; before ORC - support, assert selection fails explicitly. Invariant: fail-closed publication. - Integration test. - -Required gates: - -- `pixi run -- cargo test -p crowdb-access-iceberg --all-targets` -- `pixi run -- cargo test -p crowdb-access-server --all-targets` -- `pixi run rs-fmt-check` -- `pixi run rs-lint` diff --git a/doc/backlog/R193-chunkdb-node-failure-budget.md b/doc/backlog/R193-chunkdb-node-failure-budget.md deleted file mode 100644 index e2c5159f5..000000000 --- a/doc/backlog/R193-chunkdb-node-failure-budget.md +++ /dev/null @@ -1,47 +0,0 @@ - - - -### R193: chunkdb — Configurable node failure budget and EC placement - -#### Status - -Planned. The explicit single-node and three-node profiles, persisted strip layouts, and degraded-placement repair baseline are complete. - -#### Problem - -The current system defines two deployment contracts: one test node with no node-failure tolerance and three production nodes tolerating one failed node. The current EC selector checks the maximum fragments on one node against the parity count. That check cannot express a larger failure budget: with six nodes and two allowed failures, `4+2` and `8+4` can survive any two nodes, while `2+1` cannot. Mirror copy counts and journal/tree write policies are also configured independently rather than derived from one system protection contract. See [chunkdb placement](../design/chunkdb/design-crowdb-chunkdb.md) and [chunk IO](../design/chunkio/design-crowdb-chunkio.md). - -Operators need to choose a node failure budget for a deployment without accidentally admitting an EC shape or mirror layout that loses committed data within that budget. When nodes fail, new placement must use the remaining budget and record any loss of the full-cluster protection target for repair after recovery. - -#### Solution - -The system property `max_node_failures` is the number of unavailable nodes the configured deployment promises to tolerate from its complete topology. It is a protection target, not an automatic instruction to reduce copies each time a node fails. The remaining budget is the target minus the nodes already unavailable. A production profile must have enough voting KV replicas to retain quorum at that target and enough distinct storage nodes to place its selected layouts. The explicit test-single-node profile has a zero budget. Startup and allocation reject mismatched service policy, impossible budgets, and unsafe layouts; failure never silently switches deployment mode. - -For a mirror strip, the full protection target requires at least `max_node_failures + 1` copies on distinct nodes. Continue to use the full copy count after a failure when placement permits it. For an EC strip with `k` data and `m` parity fragments, sort per-node fragment counts descending; the sum of the largest `max_node_failures` counts must be at most `m` in a healthy topology. Apply the corresponding remaining-budget check to new allocations after failures. Keep fragments spread across available nodes even when no further node-failure budget remains. Persist actual strip geometry and its full protection target separately so readers use the real layout and repair can restore the target. - -1. Validate the deployment property and KV/storage topology consistently in deployment configuration, ChunkDB startup, and access/chunk writer policy. Preserve the existing one-node/zero-failure and three-node/one-failure profiles. -2. Replace the one-node EC bound in ChunkDB placement and physical validation with the worst-case sum across the configured number of failed nodes. Select mirror copy counts from the deployment contract, while retaining explicit per-strip policy only when it meets or exceeds the target. -3. During an outage within the configured budget, try full protection first. If it cannot fit, permit a layout that meets the remaining budget, persist a degraded-placement marker, and create a durable placement task. Reject allocation if even the remaining-budget layout or KV quorum is unavailable. Never claim full protection for a degraded strip. -4. After capacity returns, use fenced placement tasks to move or rebuild fragments until the original target holds. Reads and writes follow each strip's persisted geometry throughout migration; a restart resumes unfinished tasks without accepting stale placement. - -#### Dependencies - -- The existing two deployment profiles, strip dispatch, and placement-repair baseline are the starting point for generalized validation. -- R103 is responsible for ChunkDB range-owner migration after a ChunkDB instance failure. This requirement's storage placement budget does not replace metadata service failover; end-to-end availability depends on both. -- R139 may later distribute the system property through Group 0. Until then, startup must reject inconsistent local configuration rather than assume a remote configuration service exists. - -#### Acceptance - -- Given six voting KV/storage nodes and `max_node_failures = 2`, start services; startup accepts the budget and reports the configured target. Given an impossible budget or too few voters, startup rejects it before writes. Integration test. -- Given six healthy nodes with budget two, allocate `4+2` and `8+4` EC strips; every pair of nodes owns at most two and four fragments respectively. Request `2+1`; allocation rejects it because a pair can own more than one fragment. Integration test. -- Given six healthy nodes with budget two, allocate mirror strips used by object data, journal, and tree pages; each new strip has at least three copies on distinct nodes and reads from its persisted layout. Integration test. -- Given a six-node budget-two cluster, stop one node and allocate new mirror and EC strips; placement retains the full target where possible, otherwise uses the remaining one-failure budget and persists a repair task without changing deployment mode. E2E test. -- Given the same cluster with two nodes unavailable, allocate while KV quorum and a valid remaining-budget placement exist; operations succeed without claiming that a third failure is tolerated. Remove enough further capacity or quorum; allocation returns an error. E2E test. -- Given a degraded EC strip and recovered nodes, restart the repair worker and complete its task; data remains readable during movement, the task survives restart, and the final placement again passes the full two-node-failure check. E2E test. -- Given the existing single-node and three-node configurations, run their write/read and one-node-out suites after introducing the generalized policy; their configured budgets and persisted layouts remain valid. E2E test. - -Run `pixi run rs-fmt-check`, `pixi run rs-lint`, `pixi run cargo test -p crowdb-kv`, `pixi run cargo test -p crowdb-chunkdb`, `pixi run cargo test -p crowdb-chunk-client`, and `pixi run cargo test -p crowdb-chunk-stream` for the implemented scope. - -#### Open Questions - -None. diff --git a/doc/backlog/R196-refine-benchmark-tests.md b/doc/backlog/R196-refine-benchmark-tests.md deleted file mode 100644 index a5de1d89d..000000000 --- a/doc/backlog/R196-refine-benchmark-tests.md +++ /dev/null @@ -1,282 +0,0 @@ - - - -### R196: benchmark — Refine benchmark tests - -#### Problem - -Service benchmarks currently mix native regression fixtures, standalone Rust -bench binaries, and `crowdb-cli bench` commands. Result parsing, phase checks, -CPU accounting, and reports are repeated in shell/Python drivers. The existing -S3 CLI already sends authenticated HTTP requests to a selected cluster, but -does not observe server resource usage. Iceberg lacks an equivalent reusable -upload regression. These differences make storage and access measurements hard -to reproduce or relate to one another. - -The current Stream measurements demonstrate why deployment and CPU conditions -belong with every result: a client and services sharing CPUs 0–3 behave very -differently from the same binaries allowed on CPUs 0–31. Chunk-KV uses Stream -for its WAL, so Stream waits must also be observable in the KV workload. Object -performance needs its own measurements above ChunkIO. Neither an isolated -Stream microbenchmark nor an HTTP upload result describes all these layers. - -This requirement unifies the service benchmark regressions under -`tools/benchmark/`, including RPC, KV, DiskDB, ChunkDB, ChunkIO/object, -Chunk-Stream, Chunk-KV, S3, and Iceberg. Relevant contracts are the -[OCI cluster lifecycle](../../container/oci-image/README.md), -[Stream design](../design/chunkds/design-crowdb-chunk-stream.md), -[Chunk-KV design](../design/chunkds/design-crowdb-chunk-kv.md), -[object writer design](../design/chunkio/design-crowdb-chunkio-small-object-writer.md), -and [upload flow](../design/access-server/iceberge/design-crowdb-iceberg-upload-flow.md). - -#### Solution - -The default regression deployment is a **three-node cluster with one node per -container**. A host-side `crowdb-cli bench` process is the external benchmark -tool. Publish the test servers' required ports and advertise endpoints reachable -by that client. All service regression drivers adopt this deployment and CLI -path; the current native Stream fixture remains only until this work lands. - -Responsibilities are explicit: - -- Existing OCI/cluster lifecycle code builds and starts the three containers, - prepares storage, checks full application readiness, exposes endpoints, and - stops only the invocation's owned resources. -- `crowdb-cli bench` owns workload execution, sampling, resource/metrics - collection, accounting, validation, and the common structured report. Reuse - common-library system counters and existing management/status APIs. -- Each driving script selects cases and deployment/CPU profiles, invokes the - external CLI, retains its exit status and artifacts, and records measured - results in that same script's comments. Do not maintain separate tracked - result documents or baseline JSON files, or duplicate collectors in scripts. - -The following invariants define the benchmark: - -- **I1 — Actual service path.** The external CLI reaches the selected cluster - over published ports. Storage workloads use the deployed services; S3 and - Iceberg use authenticated HTTP. The CLI does not create an in-process cluster - as a substitute. A successful write means the layer's configured durability - and publication contract is satisfied. -- **I2 — Controlled input and timing.** Record payload seed, size, concurrency, - count/duration, warmup, read-cache policy, and media/durability profile. Bound - client payload memory independently of large object size. Separate cluster - setup, dataset preparation, warmup, measurement, drain, verification, and - cleanup. A duration-limited case cannot silently finish early due to a small - operation limit. Timed 4-KiB single-/32-producer write cases run at least - 60 seconds by default; short pipeline smoke tests are labelled separately. -- **I3 — Complete accounting.** Track admitted, completed, failed, and incomplete - operations, drain admitted work, and verify deterministic bytes outside the - timed interval. A timeout, missing result/required metric, failed cleanup, - or unverified write fails the case. Missing cases cannot disappear from a - passing summary. Bounded memory and queue limits remain regression gates. -- **I4 — Attributable measurements.** Identify client, all three nodes, service - instances, and measurement windows. Counts and sums are interval deltas; - interval percentiles come from matching samples/histogram buckets, never - subtraction of cumulative percentiles. Do not average node percentiles into - a cluster percentile. Concurrent phases overlap and cannot be summed as if - they partitioned request latency. Counter resets or replaced processes are - explicit. Optional unavailable counters are `unsupported`, never zero. -- **I5 — Comparable conditions.** A result includes exact commands and software, - image, topology, CPU, runtime, storage, and durability identities. Compare - only matching profiles with documented tolerances. Default placement and a - constrained cpuset are separate profiles; native MemDisk and container disk - results are also distinct. The accepted implementation is the baseline; - the runner does not build or run an older binary to produce a comparison. -- **I6 — Owned three-node deployment.** Readiness requires all three members - and usable storage. Port conflicts, missing peers, or unreachable advertised - endpoints fail before measurement, without singleton/native fallback. - Failure/interruption retains diagnostics and cleans only resources owned by - this invocation. Image builds, compilation, tests, and other workloads run - outside performance measurement windows. -- **I7 — Results live with the driver.** Record the dated measured summary and - its conditions directly in each driving script. The CLI can emit JSON/TSV - and raw artifacts into ignored runtime directories and generate a comment - block for the driver. Retain useful previous measurements there with their - profile/revision so comparisons have context. Do not create an additional - tracked result file; optional raw traces do not replace the script summary. - -Work items: - -1. Reuse `container/oci-image/` and the existing CLI cluster lifecycle to run - the owned three-container cluster. Publish HTTP and the management/RPC/data - endpoints needed by each selected workload; verify client reachability of - every address returned by discovery, routing, and redirects. Supply the - accepted public origins when creating the deployment. The CLI runs outside - all node containers, using explicit target bindings and private credentials. - Record exact container/image identities and port bindings. Do not implement - another bootstrap or silently attach to an unrelated running cluster. -2. Refine the commands under `app/crowdb-cli/src/commands/bench/`, extending its - existing `metrics` and `result` responsibilities and reusing workload code - in `lib/crowdb-console-shared/src/ops/` and owning storage libraries. Add - Stream and Chunk-KV entry points where absent; move the reusable Stream - workload behind the CLI. Preserve S3's existing real HTTP path and extend - it for bounded large uploads and multipart. Add the Iceberg FileIO adapter. - All workload families share accounting, collector lifetimes, output schema, - and failure handling instead of parsing human-readable metrics in scripts. -3. Implement one CLI collection lifecycle around preparation, measurement, - drain, and teardown. Take service snapshots before/after the measured window - and flush final client metrics. Use bounded histograms and existing periodic - system metrics; make per-operation traces optional. Report collection cost - and window boundaries, with no request-path file/process polling. Retrieve - server process/cgroup evidence through the owned deployment or management - APIs, mapping container identities explicitly; never interpret a container - PID as a host PID or report CLI-only CPU as cluster CPU. -4. Capture the report contents and workload matrix below. Reuse shared system - counters for CPU and C6; do not add another hardware counter implementation - to the scripts or CLI. Permission failures leave optional fields explicitly - `unsupported`; default benchmarks need no extra perf privileges or power - policy changes. A missing required workload/phase field still fails. -5. Convert every service regression driver under `tools/benchmark/` to the - three-node container plus external CLI path, retaining case filters, bounded - timeouts, and complete failure output. The scripts carry named case/gate - configuration and measured result comments; collection, latency math, and - report validation belong in the CLI. Preserve correctness gates during the - conversion. Recapture results for the new container profile rather than - treating native measurements as directly comparable. Support an explicitly - selected remote cluster through the same CLI, with unavailable local-host - measurements marked `unsupported` and no remote deployment cleanup. - -Required report contents: - -- **Conditions:** UTC date; CLI command; source revision and dirty status; - client/service binary hashes and image digest; host CPU model, kernel, - physical/logical cores, SMT/NUMA topology; three node identities, placement, - ports, mirror/EC policy, media type/device, KV/WAL backend, fsync/durability, - dataset/cache policy, and runtime/RPC worker counts. Retained MemDisk, null - data sources, sparse files, and real disks must be named accurately. -- **CPU placement and usage:** requested and effective cpusets for client, - containers, and each service's worker threads; cgroup CPU quota/period and - throttling deltas; client and per-node/service user/system CPU seconds, - average CPU %, RSS/peak/growth, context switches and migrations where - available. State that 100% means one logical CPU. Separate timed-window - usage from whole-command usage that includes dataset preparation. -- **System evidence:** disk and network bytes/operations and resource pressure - when available; per-core C6 and per-package C6 residency with units, elapsed - interval and availability. Counter support/permission/reset states stay - distinct. Optional detailed perf profiles have explicit collection settings - and their own comparable profile; no per-operation privileged probes. -- **Workload outcome:** logical/physical bytes, operations/s and MiB/s, actual - duration, admission/drain counts, errors/retries/timeouts, error classes, - count/mean/p50/p95/p99/max latency and a mergeable histogram. State sample - counts and resolution; omit unsupported percentiles with a reason. Separate - client payload generation/signing/hash work from transfer/server stages. -- **Latency attribution:** queue depth/bytes and wait; RPC round-trip and - response scheduling; append batch/mirror waits; asynchronous capacity and - candidate preparation versus foreground wait; seal/rollover, StreamChunkRecord - publication/build/get/scan/CAS, checkpoint and replay; cache/physical reads - and reclaimed bytes. Chunk-KV adds WAL append/commit, tree read/write, - flush/snapshot/compaction, and read-cache evidence. Object adds admission, - batching, reservation/preparation, payload write, parity, and metadata - publication. Use the actual available stage counters and identify required - fields by workload; capture count, sum, distribution, and max separately. -- **Protocol stages:** S3/Iceberg direct PUT, UploadPart, multipart completion, - and optional table commit are separate operations. A successful part does - not count as a published object. Upload completion includes protocol metadata - publication; FileIO latency does not include an unrelated snapshot commit. - -Minimum workload matrix: - -- **Existing RPC/KV/DiskDB/ChunkDB cases:** retain their meaningful workload and - correctness coverage while moving startup and measurement to the common path. -- **Stream:** preserve the current seven cases: 4-KiB append at 1/32 producers, - 1-MiB append at 8, 4-MiB rollover, random 4-KiB read at 32, 320-MiB replay, - and 576-MiB GC. Ordinary append must not checkpoint the cursor; metadata - publication scales with chunk transitions, not historical record count. -- **Chunk-KV:** 4-KiB values at 1/32 clients for put/get, plus batch and ordered - scan; record key/value sizes, batch size, read/write mix and cache state. - Include a run crossing WAL chunk rollover and a bounded restart/replay case. - A sustained case must expose background flush/snapshot/compaction work and - cannot end before queued durable work drains. Correlate end-to-end KV latency - with its Stream WAL metrics in the same report. -- **Object/ChunkIO:** small and large write/read plus mixed reads through the - existing object writers/readers; include 4-KiB at 1/32 clients, 1-MiB at 8, - and the configured small/large boundary. Record physical write amplification, - batching, placement and verification rather than inferring performance from - Stream or KV alone. -- **S3/Iceberg:** small-path and threshold-boundary uploads, direct 100-MiB PUT, - and multipart 100-MiB upload, at single and concurrent clients. Reuse bounded - deterministic payload blocks and precompute expected integrity data outside - timing; record the integrity mode and part size. Preserve S3 read/range/list - and mix coverage through the same collector. Final bounded byte/range checks - occur outside upload timing. - -#### Dependencies - -- Existing three-node OCI cluster lifecycle, readiness, target bindings, and - port/public-origin configuration. Add missing external RPC/data exposure in - that lifecycle; an unreachable route blocks the affected benchmark rather - than falling back to native services. -- Existing storage benchmark workloads, Stream/Chunk-KV APIs, object writers - and readers, and S3/Iceberg authentication/publication paths. Reuse these - contracts and existing native results as recorded evidence; this requirement - owns their CLI/container regression conversion and new profile measurements. -- Shared metrics/system-status counters and management snapshots provide - collection. Hardware-dependent counters remain optional if unavailable; - required stage coverage is implemented or its case fails explicitly. -- Official-client compatibility remains in the existing E2E suites. A - client-inclusive profile can use the common result schema without replacing - the controlled CLI workload or mixing their results. - -#### Acceptance - -- Given an owned three-container cluster, run every converted driver with the - host CLI; assert three ready members, published reachable endpoints and real - service requests, with no native/in-process replacement (I1, I6). E2E test. -- Given a port collision, absent third member, or an unreachable advertised - RPC/HTTP address, start a case; assert failure before timing, useful - diagnostics, and no singleton fallback (I1, I6). E2E test. -- Given an existing unrelated cluster and an interrupted/failed benchmark, - stop the run; assert only owned containers/data are cleaned under the declared - retention policy and all failure logs remain available (I3, I6). E2E test. -- Given each minimum workload and fixed deterministic inputs, execute through - the CLI; assert complete operation/byte accounting, bounded memory/queues, - real stored-byte verification, and a common report (I1–I4). E2E test. -- Given S3 and Iceberg multipart uploads, complete both; assert distinct part, - completion and optional table-commit measurements, correct integrity checks, - and no part counted as a final object (I1, I3, I4). E2E test. -- Given a 60-second 4-KiB case and a small operation cap, execute the duration - profile; assert it cannot receive a passing full-duration verdict early and - that setup/warmup/drain have distinct windows (I2, I3). Integration test. -- Given permission-denied hardware counters and remote endpoints, collect - metrics; assert per-core and package C6/local-resource availability is - `unsupported` where appropriate, with no fabricated zeros or default sudo - requirement (I4). Integration test. -- Given independent client/container cpusets and a throttled node, run a case; - assert effective per-service/thread placement, quota, throttling, CPU time, - RSS and timing windows are attributed correctly (I4, I5). E2E test. -- Given overlapping stage histograms, low sample counts, process replacement - and counter reset, build the report; assert interval aggregation is valid, - unsupported percentiles are identified, and neither percentile subtraction - nor exclusive summation of overlapping phases occurs (I4). Unit test. -- Given Chunk-KV WAL rollover/replay and object small/large-boundary workloads, - run the CLI cases; assert their own verified outcomes, latency distributions, - Stream WAL/object stages, and drained durable work (I1–I4). E2E test. -- Given a failed request, missing phase/result/case, malformed report, timeout - or cleanup failure, summarize the run; assert nonzero status and retained - diagnostic evidence instead of a partial passing result (I3, I4). Integration test. -- Given matching and mismatched measurement profiles, assess recorded results; - assert only matching profiles receive a tolerance verdict and that the current - baseline does not require running an older binary (I5). Unit test. -- Given a completed run, generate its driver comment block; assert it includes - results, date, exact command, image/revision, cpusets, CPU usage and artifact - references, and can update the driving script without another tracked result - file. Assert scripts delegate collection/validation to the CLI (I7). Integration test. - -Run `pixi run rs-fmt-check`, `pixi run rs-lint`, -`pixi run cargo test -p crowdb-console-shared`, -`pixi run cargo test -p crowdb-cli`, and -`pixi run test-container-e2e-fixture`. -Run each converted service regression serially through -`pixi run bash tools/benchmark/bench-chunk-stream-regression.sh`, -`pixi run bash tools/benchmark/bench-chunk-kv-regression.sh`, -`pixi run bash tools/benchmark/bench-chunkio-write-regression.sh`, -`pixi run bash tools/benchmark/bench-chunkio-small-write-regression.sh`, -`pixi run bash tools/benchmark/bench-chunkio-read-regression.sh`, -`pixi run bash tools/benchmark/bench-rpc-regression.sh`, -`pixi run bash tools/benchmark/bench-kv-write-regression.sh`, -`pixi run bash tools/benchmark/bench-kv-read-regression.sh`, -`pixi run bash tools/benchmark/bench-kv-scan-regression.sh`, -`pixi run bash tools/benchmark/bench-diskdb-regression.sh`, and -`pixi run bash tools/benchmark/bench-chunkdb-regression.sh`. -Verify the S3/Iceberg adapters with `pixi run test-e2e-s3` and -`pixi run test-e2e-iceberg`, including their new CLI regression cases. diff --git a/doc/backlog/R207-chunkdb-repo-metadata-chunk-kv.md b/doc/backlog/R207-chunkdb-repo-metadata-chunk-kv.md deleted file mode 100644 index 0703d6da3..000000000 --- a/doc/backlog/R207-chunkdb-repo-metadata-chunk-kv.md +++ /dev/null @@ -1,318 +0,0 @@ - - - -### R207: chunkdb — Repo chunk metadata and tasks on chunk-kv - -Status: Deferred by user beyond the current direct-KV scope. This document -records the selected future architecture and its unresolved contracts. Start -implementation only when this follow-up is selected and its publication, -range-layout and migration policies have been settled. R202 does not depend -on completing this requirement. - -#### Problem - -All production ChunkDB Chunk records, maintenance tasks, task indexes and -reservations currently use `CrowdbKvClient` and direct Paxos KV groups. -Their routing cache loads an independently published 1024-slot storage map -whose eligible destinations are selected nonzero groups; group 0 holds only -control-plane bindings. System and user-data lifecycle/task runtimes have -separate authority and execution capacity. S3 and IcebergTable have distinct -chunk types, -but neither has a chunk-kv metadata backend. Existing chunk-kv functionality -must not be mistaken for an already connected ChunkDB persistence path. -See [ChunkStore](../../app/crowdb-chunkdb/src/storage.rs), -[TaskStore](../../app/crowdb-chunkdb/src/task/store.rs), -[routing](../../app/crowdb-chunkdb/src/routing.rs) and -[startup](../../app/crowdb-chunkdb/src/main.rs). - -Growing user-data metadata and maintenance traffic therefore grow Paxos -storage and write load. The goal is to move that user-data state onto chunk-kv -while retaining the system state needed to operate and recover chunk-kv on -direct KV. Changing the client alone cannot preserve the current publication, -task and recovery contracts. - -The [ChunkDB root design](../design/chunkdb/design-crowdb-chunkdb.md) describes -the existing lifecycle; [fixed slot routing](../design/chunkdb/design-crowdb-chunkdb-range-binding.md) owns the -current type/layer definitions and direct-KV partition contract. The -[chunk-kv design](../design/chunkds/design-crowdb-chunk-kv.md) supplies ordered -storage ranges. This follow-up owns their integration, not a replacement -partition engine or a global transaction manager. - -##### Audited gaps and potential failures - -Source inspection on 2026-10-03 found the following; no future-path performance -or runtime correctness is claimed by this audit. - -- **Backend and lifecycle dispatch:** ChunkStore and TaskStore use concrete - direct-KV clients. Allocation, updates, listing, reservations, task scanning - and owner queries have no complete repo backend path. A shared backend switch - would not establish the required separate system/repo operation logic. -- **Recursive dependency:** chunk-kv's journal uses chunk-stream. Allocation, - owner validation and reopen now preserve an explicit system `Wal` or business - `Stream` purpose. Backend dispatch must preserve that distinction: routing - system WAL metadata to chunk-kv would make journal metadata depend on the - storage it must recover. The binding's `owner_kind="chunk-kv-partition"` - does not itself select the allocated type. See - [Stream allocation](../../lib/crowdb-chunk-stream/src/production_chunk/allocation.rs), - [owner validation](../../lib/crowdb-protocol/src/chunk_stream.rs) and - [storage assembly](../../app/crowdb-chunk-kv-server/src/storage.rs). -- **Atomic publication:** chunk creation writes a Chunk, finalize task and - index using direct-KV `batch_write_cas`. Task transitions update canonical - state and secondary indexes together; reservation operations conditionally - publish chunk and reservation records together. Chunk-kv's current - [batch_mutate](../../lib/crowdb-chunk-kv-client/src/compose/batch.rs) is - explicitly non-transactional; its point mutations do not expose an equivalent - conditional multi-key commit. Same-range placement alone does not add it. -- **Ordered key placement:** `/chunk/`, `/reservation/` and binary task tags - have different key prefixes; task indexes order priority/time before chunk - identity. Direct KV explicitly selects their destination group. Routing the - same bytes independently by ordered range may separate related records. - Even a shared prefix does not prevent a later split inside that prefix. - This is a future chunk-kv integration concern, not a cross-group transaction - gap in R202: its hash of the owning chunk ID routes all associated records - to the same direct-KV group regardless of their encoded key order. - `ChunkTaskKey.partition_id` currently carries a chunk identity, not a - chunk-kv catalog partition ID. See - [task keys](../../lib/crowdb-protocol/src/key/chunk_task.rs) and - [reservation storage](../../app/crowdb-chunkdb/src/storage/reservation.rs). -- **Task isolation:** current startup wires a shared task store/manager/scanner/ - executor pipeline. Common ready/finalize/expired-lease scans do not define - separate persistent system/repo task domains. Moving only Chunk records - would leave tasks behind or make them undiscoverable. -- **Unsafe reclamation:** `SegmentOwnerResolver` reads both chunk references - and active task targets. An unavailable repo backend must remain an error; - interpreting it as absence could free a repair target or live payload. - See [owner resolver](../../app/crowdb-chunkdb/src/task/owner.rs). -- **Residual Paxos traffic:** stream cursor advancement still calls - `advance_chunk_write`, persisting lower-layer Chunk metadata through direct - KV. Stream manifests/extents and tree root catalogs also use direct KV. - Offloading upper-layer records does not prove that total Paxos operations - per user mutation fall. See - [cursor advancement](../../lib/crowdb-chunk-stream/src/production_chunk.rs), - [stream metadata](../../lib/crowdb-chunk-stream/src/kv.rs) and storage assembly. -- **Ambiguous migration:** S3/Iceberg records and tasks already in direct KV - cannot be switched merely by changing a type-routing constant. Old and new - clients, expired task claims, incomplete copies and lost cutover replies - need one durable storage authority. Source and destination revisions are - backend-local and must not be treated as interchangeable CAS tokens. - -#### Solution - -##### Selected future distribution - -Use [AGENTS.md terminology](../../AGENTS.md#terminology): repo chunk is a -collective discussion term for user-data chunks, never a generic implementation -type. Inherit the type and separate operation contracts from -[ChunkDB type and operation domains](../design/chunkdb/design-crowdb-chunkdb.md#39-chunk-types-for-different-use-cases). - -- **System:** Wal journal chunks, BtreePage data chunks and PageIndex chunks - holding persistent tree page mapping tables keep their Chunk metadata, - associated maintenance tasks, indexes and reservations in hash-selected - nonzero direct Paxos KV groups under the R202 routing contract. -- **Repo:** S3, IcebergTable and business Stream chunk metadata, together with - their tasks, indexes and reservations, move to chunk-kv. Future Dataset chunks - follow this layer with their own type; this requirement does not create a - Dataset API or claim that its type is already implemented. -- **Payload:** all chunk bytes remain on DiskIO-managed storage allocated by - DiskDB. This requirement changes the location of metadata and task state, - not the physical payload placement. -- **Bootstrap/control state:** group-0 bindings/catalogs, stream manifests and - extent pages, and tree root catalogs keep their explicit lower-layer - authorities. Page mapping payload in PageIndex chunks is distinct from - those chunks' metadata and from the root references used to locate them. - Relocating these control collections is outside this requirement. - -The dependency direction is: - -```text -repo operation domain -> chunk-kv metadata range -> system WAL/tree/PageIndex chunks - -> direct KV metadata + DiskIO payload -``` - -System recovery must not traverse the repo operation domain. A system task -may repair a physical tree chunk containing repo records; it remains a system -chunk task. It must not claim or execute the repo tasks stored inside that tree. - -##### Operation and range contracts - -- Dispatch by explicit type/purpose into separate system and repo lifecycle - orchestration. Reads, mutations, listing, task RPCs, reservations and owner - reconciliation all retain the layer. Codecs, wire envelopes, payload IO and - task algorithms may be reused; persistence authority and admission remain - independent. A repo outage never triggers direct-KV fallback after cutover. -- Chunk-kv already partitions by ordered range. Map the repo domain to those - ranges while preserving ChunkDB service routing as a separate authority. - R202 owns service partition topology; this requirement consumes it rather - than redefining a storage range whenever a service owner changes. -- Define the unit of conditional publication within one storage range. Either - add explicit range-local multi-record atomic commit with split-safe placement, - or use canonical single-key publication plus a reviewed recoverable index/ - task-discovery design. The latter must retain required lifecycle outcomes. - Do not wrap non-transactional batch_mutate and label it atomic. -- Define range-local task namespaces, due/lease scan scopes, claim ownership - and dispatch admission. Ordinary repo lifecycle does not require a global - due-time queue or cross-range transactions. Split/transfer must preserve the - chosen publication unit and task discoverability; a key prefix alone is not - a split fence. Namespace routing must not accidentally claim existing S3/ - Iceberg object or catalog keys hosted by chunk-kv. -- Separate system/repo task runtimes so upper-layer backlog or unavailable - storage cannot exhaust system task admission. Same-process versus separate - processes and exact capacity budgets remain open implementation choices. -- Coordinate a tree snapshot's page and mapping generations using the R202 - PageIndex contract. Do not publish a root that requires its own unavailable - mapping table to find the mapping chunks during cold recovery. - -##### Upgrade and failure outcomes - -- Bootstrap the lower system path before enabling repo operations. Verify the - target catalog, publication contract and task namespace before activation. - Until explicit enablement, the current direct-KV behavior remains unchanged. -- Backend migration covers the canonical chunk, tasks, indexes and reservations - as one recoverable logical unit. Persist migration identity and generation, - define source/destination authority at each stage, fence old writes/claims, - reconcile lost replies, and retire source records only after recovery proves - the destination authoritative. Exact online/offline policy is still open. -- Repo storage unavailable after cutover: return explicit retryable failures; - do not accept writes in direct KV or report data absent for reclamation. - Independently provisioned system operations and task admission remain usable. -- Task execution or client response is uncertain: preserve operation identity - and reconcile durable state before retrying; duplicate work must not publish - two state transitions or release a resource twice. -- Old service/storage generations after split or handoff: reject stale mutation - authority and refresh routing. Read fallback is allowed only by the reviewed - migration/read-consistency contract, not by endpoint reachability alone. - -##### Invariants - -- **I1 — Explicit distribution.** System metadata/task state stays in direct KV; - enabled repo metadata/task state uses chunk-kv, with unchanged payload placement. -- **I2 — Acyclic recovery.** System lookup, journal progress, mapping recovery - and maintenance never require repo metadata service availability. -- **I3 — Separate operation domains.** System and repo lifecycles, task scans, - claims and admission retain independent authority despite shared code. -- **I4 — Range-local consistency.** Publication and legal split boundaries - preserve required chunk/task/reservation outcomes and complete task discovery. -- **I5 — Durable authority.** Cutover, restart, stale caches and lost replies - retain acknowledged state and admit only the current writer/task authority. -- **I6 — Safe reclamation.** Unavailable metadata is not absence; current chunk - references and active task targets protect their exact allocation incarnation. -- **I7 — Measured relief.** Report total Paxos bytes, records and operations per - user mutation, including lower-layer cursor/checkpoint/task traffic, at equal - durability and workload. Do not claim savings from record relocation alone. - -##### Work items - -1. In ChunkDB lifecycle/storage/routing and startup, add the repo chunk-kv - operation path and explicit enablement while retaining the R202 system path. - Cover point operations, listing, reservations and administrative/task RPCs. -2. In the protocol key model and chunk-kv client/server, select and implement - range-local publication, namespace placement, legal split boundaries and - recovery. Define backend-specific revision and retry identity semantics. -3. In TaskStore/manager/scanner/executor and maintenance coordinators, connect - the isolated repo task domain to chunk-kv. Preserve claim fencing, finalize - discovery, conversion, repair, relocation and reservation cleanup. -4. In SegmentOwnerResolver and DiskDB ownership queries, retain the complete - chunk/task proof across both domains, failures, storage transfers and retry. -5. In routing catalogs and migration control, implement the selected backend - conversion and restart protocol. Reuse R103's service handoff and direct-KV - partition migration authority/recovery principles; never - use a service binding flip as proof that records were copied or fenced. -6. In chunk-kv startup/storage assembly, verify typed system WAL and PageIndex - prerequisites, bootstrap order and cold recovery before serving repo data. - Existing legacy identities follow the reviewed R202 compatibility policy. -7. Measure the current direct-KV baseline and complete integrated path under - equivalent S3/Iceberg and business-stream workloads. Include task load, - outage/recovery behavior, memory, latency and all residual Paxos traffic. - Update permanent designs only with the implemented contract. - -#### Dependencies - -- [fixed slot routing](../design/chunkdb/design-crowdb-chunkdb-range-binding.md) owns current direct-KV routing, - type cleanup, typed Wal/PageIndex and separate operation/task domains. This - requirement consumes those boundaries. It must not block R202 completion or - force a premature change to its all-KV-group scope. -- [R103](R103-chunkdb-range-migration.md) owns service handoff and direct-KV - partition data migration for group expansion/shrink. This requirement extends - its authority/recovery principles to a different backend; direct-KV group - migration and direct-KV-to-chunk-kv conversion remain separate operations. -- Existing chunk-kv client/server ordered-range split/transfer are primitives. - [R144](R144-chunk-kv-partition-merge.md) merge remains deferred; do not promise - merge for repo metadata before a reviewed implementation exists. -- Publication semantics and key/split rules are prerequisites to enabling the - new path. Their absence keeps this follow-up disabled; it does not justify - weakened atomicity or an implicit runtime fallback. -- Preserve the implemented [tree engine](../design/tree/design-crowdb-tree-engine.md) - concurrent admission and prefix-safe flush handoff. Backend migration must - retain its acknowledged-write and recovery guarantees. - -#### Acceptance - -- Given current deployment configuration without repo backend enablement, - allocate/update chunks and tasks; assert all remain in direct KV and require - no new repo backend connection (I1). Integration test. -- Given enabled system/repo domains, exercise allocation, query, listing, - mutations and reservations for supported types; assert selected metadata/ - task destinations, independent operation paths and unchanged payload route - (I1, I3). Integration test. -- Given chunk-kv cold startup and system Wal/BtreePage/PageIndex chunks, recover - tree and journal before repo activation; assert no recursion into repo - metadata and consistent page/mapping recovery (I2). Integration test. -- Given repo storage outage and task backlog, run system lookup/maintenance; - assert system admission remains usable and repo operations fail explicitly - without direct-KV fallback or cross-domain claims (I1–I3). Integration test. -- Given chunk/task/reservation publication, interrupt each selected commit or - recovery boundary and lose replies; restart/retry and assert preserved - lifecycle outcomes and discoverable required tasks (I4, I5). Integration test. -- Given related canonical/index keys and existing non-ChunkDB application keys, - route them and attempt legal/illegal splits; assert namespace isolation and - preservation of the selected publication unit (I3, I4). Integration test. -- Given ready and leased tasks in a transferred or split range, resume scanning - and stale workers; assert complete discovery, current claims only and no - duplicate resource release (I3–I5). Integration test. -- Given a DiskDB segment named by a repo repair task, query ownership while - chunk-kv is unavailable and after recovery; assert uncertainty retains the - segment and the exact task target remains protected (I6). Integration test. -- Given existing direct-KV repo chunks and associated state, run the selected - migration with crashes before/after cutover and lost replies; assert one - authoritative backend, preserved acknowledgements and safe cleanup (I1, I5, - I6). Integration test. -- Given concurrent service handoff and a supported storage transfer, issue - stale client/task operations; assert generations are reconciled independently - and service ownership changes do not invent storage authority (I5). - Integration test. -- Given invalid target configuration or an unsupported legacy migration, enable - the backend; assert rejection before authority changes and preservation of - source records (I1, I5). Integration test. -- Given equal-durability representative workloads, compare baseline and - integrated paths; assert the user-approved performance budget and report all - residual Paxos traffic and task/recovery overhead (I7). Integration test. - -#### Open Questions - -- Range-local conditional multi-key commit, or canonical single-key state with - recoverable derived indexes/tasks? Choose based on publication/recovery - complexity and storage-engine support; same-range routing alone is insufficient. -- What exact key namespace and indivisible split unit preserve publication and - due/lease discovery? Prefer a range-local design; compare the cost of aligned - index placement with explicit recoverable index maintenance. -- Should existing repo state move through online fenced migration, a maintenance - window, or an explicitly approved reset for development deployments? Define - rollback only where authority has not been irrevocably transferred. -- Separate processes or separately admitted runtimes in one process, and what - capacity budgets keep system repair available under repo backlog? -- What measured Paxos byte/operation and latency budget constitutes success? - Backend separation and lower metadata cardinality alone do not quantify it. - -The following are implementation verification commands, not tests run by this -documentation-only change. Add reviewed migration/outage cases to the affected -suites when implementing this requirement. - -```sh -pixi run cargo test -p crowdb-chunk-kv --test partition_test --test journal_failure_test -pixi run cargo test -p crowdb-chunk-kv-client --test routing_test --test compose_test -pixi run cargo test -p crowdb-chunk-stream --test production_chunk_test --test stream_test -pixi run cargo test -p crowdb-chunkdb --test routing_test --test lifecycle_test --test owner_metadata_test --test conversion_test -pixi run cargo test -p crowdb-protocol --test chunk_task_key_test --test chunk_task_value_test -pixi run rs-fmt-check -pixi run cargo clippy -p crowdb-protocol -p crowdb-chunk-client -p crowdb-chunk-stream -p crowdb-chunk-kv -p crowdb-chunk-kv-client -p crowdb-chunk-kv-server -p crowdb-chunkdb -p crowdb-chunkdb-client --all-targets -- -D warnings -``` diff --git a/doc/backlog/R233-sdk-local-cache-checkpoint.md b/doc/backlog/R233-sdk-local-cache-checkpoint.md deleted file mode 100644 index 27dd8e85d..000000000 --- a/doc/backlog/R233-sdk-local-cache-checkpoint.md +++ /dev/null @@ -1,91 +0,0 @@ - - - -### R233: SDK — Local disk cache and checkpoint staging - -Status: Deferred until dedicated HTTP upload/read workload measurements. -The public Rust/Python HTTP SDK and real-service functional acceptance are -available; their delivery did not include performance baselines. - -Problem: - -- Training reads may reuse immutable payloads; checkpoint writers may need a - short local-write pause before background upload. The - [Dataset contract](../design/access-server/dataset/spec-crowdb-dataset.md) - defines remote Snapshot publication, not local checkpoint completion. -- Treating a local write as a remotely durable checkpoint would report success - before another machine can recover it. Local staging can absorb bursts but - cannot compensate indefinitely for remote bandwidth below sustained writes. - -Solution: - -1. Add optional Rust cache/staging components under `sdk/`, exposed through the - same Python wrapper as the HTTP client. Rust owns journal recovery, disk - quotas, eviction and background upload; Python exposes configuration and - completion state without duplicating these mechanisms. Keep default HTTP - clients diskless. Key read cache entries by endpoint/security scope, Dataset, - immutable Snapshot, field/range and digest. Validate bytes before admission; - never cache mutable latest as immutable identity. A cache hit does not extend - remote retention or bypass server authorization/lease policy. -2. Separate evictable clean cache from uncommitted staging. Bound disk usage, - memory and upload concurrency. Evict clean entries first; apply explicit - backpressure or fail admission when staging reaches quota. Never silently - evict the only pending copy. Keep hot reads lock-free; any proposed lock - requires the repository's explicit contention/trade-off review. -3. Use a recoverable local journal and durable file/directory synchronization. - Expose distinct Staging, LocalDurable, Uploading, RemoteCommitted and Failed - states. LocalDurable means local bytes and replay information are durable; - it does not survive loss of that disk. RemoteCommitted requires payload - durability and successful Snapshot publication. `flush`/remote wait returns - that Snapshot or an error/ambiguous operation identity, never local success. -4. Resume after process restart using the same upload and publication tokens. - Local corrupt/incomplete files fail validation; partial remote uploads remain - invisible. Retain pending local data after transient failure and report quota, - disk-full and shutdown outcomes. Cleanup follows remote commitment or explicit - discard; remote ownership rules still govern uploaded bytes. -5. Keep model serialization and coordinated multi-rank checkpoint assembly in - adapters. Publish one complete Snapshot only after all declared checkpoint - components exist. Measure pause time, remote completion latency, sustained - throughput, disk/memory bounds and restart recovery against direct streaming. - Do not claim a performance improvement without these workload measurements. - -Dependencies: - -- Existing upload sessions, publication recovery and the public HTTP SDK; - dedicated workload metrics remain a prerequisite. - HTTP streaming remains the fallback without local storage or when caching is - disabled. No RDMA or fat client is required for this feature. -- Native Dataset integration may reuse these components after proving the - same visibility contract. - -Acceptance: - -1. Given identical fields in different snapshots/security scopes -> populate - and corrupt cache entries and advance latest -> assert no cross-scope reuse, - checksum rejection and fixed Snapshot identity. Cache invariant. Integration test. -2. Given a slow server and small disk quota -> stage repeatedly and exhaust the - disk -> assert bounded usage, clean eviction, explicit backpressure and - preservation of pending bytes. Capacity invariant. Integration test. -3. Given a local checkpoint -> stop after each durable state and restart -> - assert LocalDurable never implies RemoteCommitted, journal replay preserves - identity, and remote wait returns only a published Snapshot. Durability invariant. Integration test. -4. Given response loss, partial upload and corrupt local data -> recover/retry - or shut down -> assert no partial publication, duplicate Snapshot or silent - discard and explicit failure outcomes. Recovery invariant. Integration test. -5. Given a declared multi-rank checkpoint and measured baseline -> delay one - component and compare staged/direct runs -> assert complete-only publication - and record pause/completion latency, throughput and resource bounds. - Checkpoint completeness invariant. E2E test. -6. Given the same staged operation -> inspect and resume it through Rust and - the Python wrapper -> assert identical journal identity, durability states, - quota errors and remote completion. Shared implementation invariant. Integration test. - -Verification commands: - -- `pixi run -- cargo test --manifest-path sdk/Cargo.toml --workspace` -- `pixi run -- python -m unittest discover -s sdk -p 'test_*.py'` -- `pixi run clean --runtime-only && pixi run test-e2e-dataset` -- `pixi run -- cargo fmt --manifest-path sdk/Cargo.toml --all -- --check` -- `pixi run -- cargo clippy --manifest-path sdk/Cargo.toml --workspace --all-targets -- -D warnings` -- `pixi run rs-fmt-check` -- `pixi run rs-lint` diff --git a/doc/backlog/R4-bounded-mempool.md b/doc/backlog/R4-bounded-mempool.md deleted file mode 100644 index baca1936b..000000000 --- a/doc/backlog/R4-bounded-mempool.md +++ /dev/null @@ -1,27 +0,0 @@ - - - -### R4: Bounded memory pool for `buffer` allocations - -**Problem**: `buffer::allocate` uses unbounded `std::malloc`. When crowdbtree is -embedded in crowdb-kv-server, a burst of large writes can spike RSS without -backpressure. - -**Approach**: Admission control at `Crowdbtree::apply()`/`apply_batch()` entry -via `Config.mem_budget_bytes` (0 = unlimited). Track outstanding buffer -bytes atomically; reject with `Status::resource_exhausted()` when over -budget. Flush/snapshot path is exempt (must always succeed). - -**Priority**: Low — needed only when memory-bound deployment is a -requirement. - -**Complexity**: Medium — atomic counter, budget accounting in apply path, -test for rejection + recovery. - -**Files**: `lib/crowdb-tree/include/crowdb-tree/config.h`, -`lib/crowdb-tree/include/crowdb-tree/buffer.h`, -`lib/crowdb-tree/src/btree/crowdb-tree.cpp` - -**Acceptance**: Unit test: set `mem_budget_bytes=1MiB`, apply 2 MiB of -values, verify `resource_exhausted` after budget exceeded, verify success -after budget freed (entries applied + flushed). diff --git a/doc/backlog/R60-tree-scan-sibling-leaf-readahead.md b/doc/backlog/R60-tree-scan-sibling-leaf-readahead.md deleted file mode 100644 index 061f5fe7a..000000000 --- a/doc/backlog/R60-tree-scan-sibling-leaf-readahead.md +++ /dev/null @@ -1,137 +0,0 @@ - - - -### R60: Scan — Sibling-Leaf Readahead on Cold Scans - -**Status**: Deferred pending a cold file/block-backed scan benchmark that -forces leaf eviction and records page reads, scan retries, latency, and -throughput. Implement readahead only if that baseline shows serialized leaf -loads are a material part of scan latency or limit NVMe throughput. - -**Problem**: the scan path demand-loads each L1 leaf inline. The sync -path (`Crowdbtree::scan`) resolves a leaf via the page cache when the -merge loop reaches it — the read stalls the loop until the leaf is -resident. The async path (`scan_async_attempt`) resolves one pending -page per reactor round trip and retries the whole scan, so a multi-leaf -cold range pays one reactor round trip per cold leaf, serialized with -the merge work on prior leaves. - -A scan knows its next leaf before finishing the current one: the merge -loop reads `base->right_sibling()` in `crowdb-tree.cpp` right after -descending into a leaf, before iterating that leaf's entries. So -the page id of the next leaf is available while the current leaf is -still being merged. Today nothing is done with it until the current -leaf exhausts and `refill_l1` walks to the next — at which point the -read stalls. - -**Solution**: issue a readahead (prefetch) for the right-sibling leaf -as soon as its page id is known, overlapping the next leaf's I/O with -the current leaf's merge work. - -- **Sync path** (`Crowdbtree::scan` / `try_scan_no_load`): after reading - `right_sibling`, call the page cache's prefetch/async-resolve seam - for that page id (non-blocking — enqueues the read, does not wait). - When `refill_l1` later walks to that leaf, the read is already in - flight or complete, so the stall is hidden. If the leaf is already - resident the prefetch is a no-op. -- **Async path** (`scan_async_attempt`): instead of resolving one pending page - per reactor round trip and waiting until the current leaf is exhausted to - discover the next miss, submit the right-sibling read as soon as the current - resident leaf exposes its page id. This fixed one-leaf lookahead overlaps - the next NVMe read with current-leaf merge and packing work while bounding - memory and unused reads. It does not create multiple in-flight reads within - one scan because the following sibling id is not known yet. -- **Prefetch depth**: start with a fixed depth of one. Tunable 2–4-leaf windows - are out of scope until the cold benchmark shows one-leaf lookahead is useful - but insufficient. -- **Eligibility invariant**: only a range scan may request sibling readahead. - Issue it only when the scan can continue beyond the current leaf, has not - exhausted its item limit or byte budget, has time remaining before its - deadline, and uses a file/block-backed async page store. A resident sibling - needs only the existing CPU prefetch; an unloaded or already-loading sibling - must not receive a duplicate read. Point `get` operations and memory-mode - scans never request sibling I/O. -- **I/O mechanism**: use a request-scoped engine-managed asynchronous read, - not a process-wide operating-system sequential-access hint. The block path - uses `O_DIRECT`, and a global hint could be ineffective or pollute unrelated - buffered-file workloads. At most one speculative sibling page may be loaded - and then left unused when the current leaf satisfies the scan. - -**Scope**: -- `lib/crowdb-tree/src/btree/crowdb-tree.cpp` — `scan` / `try_scan_no_load`: - after `page_id = base->right_sibling()`, issue a prefetch. The - prefetch seam must already exist or be added to the page cache - using the configured `AsyncPageStore` and reactor. If no request-scoped - async-resolve seam exists, this is blocked on adding one. -- `lib/crowdb-tree/src/btree/crowdb-tree.cpp` — `scan_async_attempt`: submit - the right-sibling read while merging the current resident leaf and reuse or - await that read if the scan reaches the sibling before completion. -- Benchmark prerequisite: add a file/block-backed multi-leaf scan case that - forces eviction and records page reads, scan retries, latency, and - throughput before changing the read path. -- Tests: `test-tree-ct` scan tests must pass unchanged (readahead is - observational — output is identical). Extend or mirror - `AsyncScan.MissAfterEvictionCompletesViaReactor` for fixed one-leaf - readahead only after the benchmark meets the implementation threshold. - -**Complexity**: Medium. The prerequisite benchmark is required because the -maintained scan regression runs in memory mode and cannot establish a cold -NVMe benefit. The sync-path prefetch is small IF a -prefetch/async-resolve seam exists in the page cache (likely needs a -small addition — the cache today resolves on demand). The async-path -coordination is the bulk: the retry loop must distinguish a sibling read that -is already in flight from an unloaded page, avoid duplicate submissions, and -resume from the correct key when the read completes. -Measurement is required to confirm the win (cold ranges only — the -bench today runs mem-mode where leaves are resident, so a cold/disk -bench config is a prerequisite for validation). - -**Dependencies**: -- Blocked on the cold file/block-backed benchmark and baseline measurements. -- May need a page-cache prefetch seam (check first). If the cache has - no non-blocking resolve, add one as a sub-task. -- Independent of R57 (staging copies) and R58 (merge loop compares) — - they touch different parts of the scan path. R60 overlaps I/O with - merge work; R57/R58 speed up the merge work itself. Complementary. -- If the existing cold-scan regression measurements expose a regression - without identifying its source, profile the affected case on demand before - investing in readahead. - -**Acceptance**: -- Scan output is byte-identical to today across all `test-tree-ct` scan - tests — Integration test. -- With a cold file/block-backed tree, perform a point `get`, a memory-mode - scan, and a scan whose range ends in the current leaf; assert that none - submits a sibling read, preserving the scan-only eligibility invariant — - Integration test. -- With an unloaded sibling already being loaded, scan through the current - leaf; assert that the scan reuses or awaits the existing operation and does - not submit a duplicate read, preserving the one-read-per-page invariant — - Integration test. -- With an expired scan deadline or an exhausted item/byte budget, finish the - current leaf; assert that no sibling read is submitted, preserving the - request-bound work invariant — Integration test. -- A cold file/block-backed benchmark forces eviction and reports page reads, - scan retries, latency, and throughput for the no-readahead baseline — - Integration test. -- Before the production read path changes, a fixed one-leaf experiment shows - a material reduction in latency or increase in throughput against that - baseline; otherwise R60 is closed without implementation — Integration - test. -- Readahead memory is bounded (per-scan in-flight cap, default window - = 1); a full-keyspace cold scan does not grow unbounded RSS — Integration - test. -- No regression on `tools/benchmark/bench-kv-scan-regression.sh` (mem-mode configs - unchanged — readahead is a no-op when leaves are resident) — Integration - test. - -**Note**: the gap lives in -`doc/design/kv/kv-scan-flow-analysis.md` Gap Analysis → Performance → -"No sibling-leaf readahead on cold scans". - -Verification commands: -- `pixi run test-tree-ct` -- `pixi run tree-fmt` -- `pixi run tree-lint` -- `pixi run rs-fmt -- --check` -- `pixi run rs-lint` diff --git a/doc/backlog/R80-diskdb-rebalance.md b/doc/backlog/R80-diskdb-rebalance.md deleted file mode 100644 index 75dd7f5d4..000000000 --- a/doc/backlog/R80-diskdb-rebalance.md +++ /dev/null @@ -1,127 +0,0 @@ - - - -### R80: diskdb — Disk Space Rebalance Convergence - -**Problem**: DiskDB can now place new allocations on the least-used eligible -disk and can move one committed block through its durable relocation journal. -The current intra-disk-group planner, however, reacts to one utilization -sample, chooses the hottest and coldest disks without projecting the selected -block's effect, and has no end-to-end proof that repeated cycles converge and -then stop. A large block can overshoot the balance point, transient skew can -start unnecessary movement, and passing handoff or restart tests does not show -that a newly added or recovered empty disk actually absorbs old allocations. - -The implemented allocation, relocation, and owner-fencing baseline is defined -by the [DiskDB design](../design/diskdb/design-crowdb-diskdb.md), especially -sections 3.2, 9, and 10. ChunkDB's protection-preserving cross-disk-group -planner is defined by the -[ChunkDB design](../design/chunkdb/design-crowdb-chunkdb.md#74-active-cross-domain-rebalance). -This requirement is limited to completing and proving DiskDB's space-balance -behavior within one disk-group. - -**Solution**: - -1. **Define an improving move**. `RebalancePlannerTask` computes normalized - utilization as `busy_bytes / capacity_bytes` for every allocatable disk. It - selects the hottest source and coldest eligible target deterministically, - but admits a source block only when projecting that exact block onto the - target strictly reduces the disk-group's utilization spread. Require the - target to retain `rebalance.min_target_free_bytes` after reservation. If no - committed block improves the spread, emit no relocation and report the - group as stalled; never oscillate a block across the balance point merely - because the current spread exceeds the threshold. - -2. **Require sustained skew**. `RebalancePlannerTask` tracks when each owned - disk-group first exceeds `rebalance.imbalance_threshold_pct` and starts new - work only after it remains above the threshold for - `rebalance.hysteresis_secs`. A balanced observation clears the timer. This - observation state is an optimization and may reset on restart; - `RelocationJournalValue` remains the durable recovery authority and resumes - before any new decision. - -3. **Converge at a bounded rate**. `RebalancePlannerTask` re-evaluates live - usage after each completed relocation, starts no more than - `rebalance.max_jobs_per_cycle`, and keeps at most one active relocation per - disk-group. `RebalanceZonePacer` keeps disk-groups and zones serially paced - by `rebalance.zone_delay_secs`. The planner stops creating journals once the - spread is below the configured threshold. A source or target that is no - longer allocatable invalidates the candidate without weakening - `RelocationWorker` safety rules. - -4. **Make convergence observable**. `DiskdbMetrics` keeps the existing - imbalance and relocation metrics and distinguishes groups that are observing - hysteresis, actively moving, balanced, or stalled because no safe improving - block fits. Status is derived from `DdbDiskGroup::aggregate_usage`, the - hysteresis observation, and durable journals, not from a process-local job - set. Per-disk-group keepalive summaries remain the cluster-wide source for - placement and operator inspection. - -5. **Prove passive and active balance together**. Preserve the lock-free - `DdbDiskGroup::allocate_block` load-aware policy and verify its behavior over - sequences rather than a single allocation. Add a full-stack fixture with - real DiskIO and the ChunkDB owner path that repeatedly runs the production - planner until a hot disk-group converges or truthfully reports that block - granularity prevents further improvement. - -The relocation invariants remain unchanged: ChunkDB alone publishes layout -changes; the exact target is copied, fsynced, published, and confirmed before -the source is freed; and timeout, age, or utilization never authorizes a free. - -**Dependencies**: - -- Uses DiskDB's existing load-aware allocator, per-disk usage snapshots, - relocation journal, DiskIO copy/fsync path, and tentative-owner scanner. -- Uses ChunkDB's existing exact-source revision fence and durable relocation - task. No new metadata publication path is introduced. -- Cross-disk-group selection is already owned by ChunkDB. This requirement - does not change rack, node, or physical-disk placement policy. -- Dynamic disk membership and status come from the existing group-0 sync. A - disk leaving `Up` is excluded on the next planning decision. - -**Acceptance**: - -- Given equal-capacity disks at 90% and 0%, when 100 single-block allocations - run with load-aware allocation enabled, assert the colder disk receives the - allocations until its utilization catches up; repeat with the policy - disabled and assert cursor-order round-robin, and with equal utilization and - exclusions to assert deterministic ties and strict anti-affinity. Invariant: - passive balancing changes selection only and preserves allocation safety. - Integration test. -- Given unequal-capacity disks, when allocation uses `free_bytes` and then - `inverse_used_pct`, assert each policy follows its documented absolute or - normalized ordering without division-by-zero or overflow. Invariant: mixed - capacity produces deterministic, configured behavior. Unit test. -- Given a disk-group whose spread crosses the threshold for less than the - hysteresis interval, when planner cycles run, assert no relocation journal is - created; after sustained skew, assert exactly one improving journal is - admitted. Invariant: transient skew cannot trigger data movement. - Integration test. -- Given a candidate block whose projected move would preserve or increase the - spread, or leave less than the configured target headroom, when the planner - evaluates it, assert no target is reserved and the group reports stalled. - Invariant: every admitted move strictly improves usable balance. Unit test. -- Given a hot disk and a newly added or recovered empty peer, when production - planner cycles execute through real DiskIO and ChunkDB, assert each - `SourceFreed` move lowers projected spread, restart once with a non-terminal - journal, and eventually observe spread below threshold with no subsequent - journal. Invariant: active rebalance converges, survives restart, and stops. - E2E test. -- Given balanced disks, a single-disk group, all disks non-allocatable, or a - source/target status change between cycles, when the planner runs, assert it - creates no unsafe new journal and retains or safely resumes any existing - journal. Invariant: topology edge cases are no-op or retry, never unsafe - relocation. Integration test. -- Given several owned disk-groups, when some are observing, moving, balanced, - and stalled, assert metrics/status report each state consistently with - current usage and non-terminal journals. Invariant: operators can distinguish - progress from inability to improve. Integration test. - -Verification commands: - -- `pixi run test-diskdb` -- `pixi run test-chunkdb` -- `pixi run test-diskdb-client` -- `pixi run test-chunkdb-client` -- `pixi run rs-fmt -- --check` -- `pixi run rs-lint` diff --git a/doc/backlog/R82-kv-watch-notify-coalescing.md b/doc/backlog/R82-kv-watch-notify-coalescing.md deleted file mode 100644 index 37714c440..000000000 --- a/doc/backlog/R82-kv-watch-notify-coalescing.md +++ /dev/null @@ -1,191 +0,0 @@ - - - -### R82: kv — Watch/Notify Coalescing (Debounce) - -**Problem**: - -- **Current behavior + impact** — the watch/notify extension ships - without coalescing: `PxLearner::apply_entry` calls `WatchRegistry::emit` - directly, producing one notify frame per changed key per matching - prefix. A burst write to a watched prefix (e.g. a diskdb - `batch_write` touching 10 disks under `/hw/disk/`) generates 10 - separate `WatchNotify` frames, each waking the subscriber's - keepalive and triggering a re-read via the normal `Get` path. Under - sustained burst load (periodic discovery scans, bulk status - updates), this amplifies both notify traffic and downstream read - load — the notify mechanism can be strictly worse than the - fixed-interval polling it replaces. Not a correctness issue: the - safety-net poller covers any missed notifies. The impact is purely - load/efficiency under bursty workloads. -- **Design pointers** — - [`doc/design/kv/design-crowdb-kv-watch-notify.md`](../design/kv/design-crowdb-kv-watch-notify.md) - §3 (Coalescing — deferred) states the rationale and points - back to this backlog item. The original coalescer design (struct - shape, timer-task lifecycle, per-prefix debounce, step-down flush) - was specified in the same design draft but removed from the - watch/notify extension's scope after the timer-task wiring proved - incomplete (the spawned timer captured no registry/coalescer refs, so - buffered keys were silently dropped). New work; the reference model - is etcd watch (which coalesces at the mvcc backend, not the apply - path). -- **Use scenarios** — - - **Disk discovery burst** — diskdb runs a periodic discovery - scan; one `batch_write` updates 10+ disk records under - `/hw/disk/`; the owning diskdb's keepalive is woken once (not 10 - times) and performs one re-read sweep. - - **Bulk ownership transfer** — an operator reassigns 5 - disk-groups in one batch; the `/hw/dg_owner/` watcher receives - one coalesced notify with all 5 changed keys, not 5 separate - frames. - - **Continuous churn** — a prefix under continuous write pressure - (writes arriving faster than the debounce window) is - rate-limited to one notify per `debounce_ms` interval, bounding - subscriber wakeups. - -**Solution**: - -- **No clear solution yet — deferred to design.** The core design - questions (timer-task ownership model, Arc/Weak capture, per-prefix - vs per-watcher debounce, step-down flush semantics) need a design - draft to resolve. The high-level shape is known from the original - watch/notify design draft, but the timer-task wiring gap that caused - the extension to drop the feature must be addressed properly. - -- **One-line summary**: per-prefix debounce coalescer with - timer-task flush, wired into the apply path between - `apply_entry` and `WatchRegistry::emit`. - -- **Numbered work items**: - 1. **`WatchCoalescer` struct** - (`lib/crowdb-kv/src/cluster/watch_registry.rs`) — new struct - holding `debounce_ms: AtomicU64` + `pending: - Mutex, (HashSet>, - Option>)>>`. The apply-path hook calls - `record_chosen` instead of `emit` directly; `record_chosen` - either emits immediately (`debounce_ms == 0`) or buffers keys - per prefix and spawns a timer task. - 2. **Timer task wiring** — the spawned `tokio::time::sleep` task - must capture `Weak` + `Weak` so - it survives group drop mid-debounce (weak upgrade fails → - no-op). On wake: lock `pending`, drain the prefix's key set, - call `registry.emit`. This is the gap that caused the - coalescer's removal from the initial watch/notify extension — - the original code captured no refs and silently dropped buffered - keys. - 3. **`PxLearner` wiring** (`lib/crowdb-kv/src/paxos/learner.rs`) — - the `watch_registry` `OnceLock` gains a third element - (`Arc`); `set_watch_registry` takes the - coalescer; the apply-path hook calls `coalescer.record_chosen` - instead of `registry.emit`. - 4. **`PxGroup` wiring** (`lib/crowdb-kv/src/cluster/group.rs`) — - `watch_coalescer: Arc` field; constructed in - `PxGroup::new` with the configured `debounce_ms`; wired into - the learner via `set_watch_registry`. - 5. **Step-down flush** - (`lib/crowdb-kv/src/cluster/group_election.rs`, - `lib/crowdb-kv/src/cluster/group_propose.rs`) — `step_down` and - the two propose-path direct `become_follower` calls call - `watch_coalescer.flush_and_clear()` before - `watch_registry.clear()`, emitting any pending notifies so no - notify is lost on leader change. - 6. **Configuration** (`lib/crowdb-kv/src/common/config.rs`, - `app/crowdb-diskdb/src/ddb_config.rs`) — add - `watch_notify_debounce_ms` to `CrowDBConfig` (default 100, - dynamic) and `notify_debounce_ms` to `NotifyConfig` (default - 100, documentation/operator intent — the crowdb-kv config is - authoritative). Live reload via `set_from_config` propagates - to the coalescer's `AtomicU64`. - -- **Flow diagram**: - -``` - apply_entry (PxLearner) - │ - ▼ - ┌─────────────────────┐ - │ WatchCoalescer │ - │ debounce_ms == 0? │ - │ ├─ yes → emit now │ - │ └─ no → buffer │ - │ + spawn │ - │ timer │ - └────────┬────────────┘ - │ timer fires (debounce_ms) - ▼ - ┌─────────────────────┐ - │ drain pending[pfx] │ - │ registry.emit(...) │ - └─────────────────────┘ - │ - ▼ - ┌─────────────────────┐ - │ WatchRegistry.emit │ - │ → watcher channels │ - └─────────────────────┘ -``` - -- **Edge cases at a glance**: - - Group dropped mid-debounce → timer task's `Weak` upgrade fails → - no-op (no dangling emit). - - Leader step-down with pending keys → `flush_and_clear` emits - final notifies before registry clears. - - Second write to same prefix within window → adds to existing - `HashSet`, does not reset timer (fixed trailing-edge). - - Write to unwatched key → no pending entry created, no timer - spawned. - - `debounce_ms` changed live → new calls read the new value; - existing timers fire at their original schedule. - -**Dependencies**: - -- Depends on the watch/notify extension (`WatchRegistry`, - `WatchCoalescer` apply-path hook, `WatchNotifyClient` — see - [`doc/design/kv/design-crowdb-kv-watch-notify.md`](../design/kv/design-crowdb-kv-watch-notify.md)). - That extension must be landed first; this item adds the coalescer - layer on top. -- No items depend on R82 — it is a pure load optimization. - -**Acceptance**: - -**WatchCoalescer** (`lib/crowdb-kv/src/cluster/watch_registry.rs`): -- `debounce_ms=0`, `record_chosen` a payload with 2 keys → - immediate emit (no timer). Unit test. -- `debounce_ms=50`, `record_chosen` 3 payloads to the same prefix - within 10 ms → one `WatchNotify` with all keys after ~50 ms (use - `tokio::time::pause`). Unit test. -- `record_chosen` writes to two prefixes → two separate notifies - (one per prefix timer). Unit test. -- `flush_and_clear` with 2 buffered keys → one final notify with - the buffered keys + no timers remain. Unit test. -- `record_chosen` a payload whose key matches no registered prefix - → no pending entry created, no notify. Unit test. -- Group dropped mid-debounce (drop all `Arc`s, advance timer) → no - panic, no emit (weak upgrade fails). Unit test. - -**Step-down flush**: -- Buffer 2 keys, call `flush_and_clear` → assert one notify with - both keys, then `registry.clear()` → channels closed. Unit test. - -**E2E (diskdb)**: -- `notify_debounce_ms=100`, burst-write 10 disks in one - `batch_write` → one coalesced notify wakes keepalive once, not 10 - times. E2E test. -- `notify_debounce_ms=0`, burst-write 10 disks → 10 separate - notifies (proves coalescing is the debounce path, not emit). - E2E test. - -- `pixi run cargo fmt --all -- --check` -- `pixi run cargo clippy --all-targets -- -D warnings` - -**Open Questions**: - -- **Q1 — `debounce_ms` ownership: crowdb-kv config vs per-subscription.** - The coalescer runs on the crowdb-kv leader (group 0). Options: (a) - one global `debounce_ms` on `CrowDBConfig` — simple, but all - watchers share the same window; (b) per-subscription debounce sent - as a `WatchSubscribe` parameter — more ergonomic, adds per-watcher - debounce timer state. Recommendation: (a) for v1 — one global - debounce on the leader; revisit if per-subscription debounce is - needed. Cannot be resolved autonomously — it is an API-ergonomics - trade-off that needs a human decision. diff --git a/doc/backlog/R83-chunkdb-complete-recovery-flow.md b/doc/backlog/R83-chunkdb-complete-recovery-flow.md deleted file mode 100644 index cfff04f5e..000000000 --- a/doc/backlog/R83-chunkdb-complete-recovery-flow.md +++ /dev/null @@ -1,287 +0,0 @@ - - - -### R83: chunkdb — Complete Recovery Flow (Real Data Recovery + Speed Control) - -**Problem**: - -- **Current behavior + impact** — diskdb's recovery story is - disk-layer only and stops at the hand-off boundary. diskdb has zone - bitmap reconstruction (zone-management §6, strategies 1/2/3) and the - per-disk failure recovery scan (R76, `RecoveryScanTask` in - `app/crowdb-diskdb/src/recovery/disk_recovery.rs`). The R76 scan does - the right *discovery* work — it iterates a `Bad` disk's zones, lists - live `BusyBlockValue`s, and collects each impacted block's - `owner_chunk` (`ChunkId`, 192-bit, §3.4) — but the *repair* step is a - placeholder: `RecoveryAction` has exactly one variant `LogOnly`, and - `recover_zone_blocks` logs the impacted blocks + owner chunks and - does no data repair or relocation (disk_recovery.rs:378). The - doc-comment states the intent: "Future versions add - `Relocate`/`RebuildFromEc` when the `diskio` service exists" - (disk_recovery.rs:37). There is **no chunkdb yet** — chunks (logical - blocks composed of mirror or EC strips, each strip a set of diskdb - `Segment`s) are only a reserved fbs schema surface - (`chunkdb.fbs`). So when a disk goes - `Bad`, its impacted blocks are identified and handed to a - "future recovery/relocation path" (§8) that does not exist: the owner - is notified nowhere, no surviving replica/parity is read, no rebuilt - data is written to new segments, and the chunk's strips are never - updated. Full data recovery is missing. Recovery speed is also - uncontrolled — once real rebuild I/O ships, an unthrottled rebuild - would starve foreground allocate/free/read traffic; the throttle - belongs at the chunkdb layer (the component that owns the - chunk→strip→segment mapping and drives rebuild I/O), not at diskdb - (which has no data I/O envelope, §2). -- **Design pointers** — - [`doc/design/diskdb/design-crowdb-diskdb.md`](../design/diskdb/design-crowdb-diskdb.md) - §2 (Non-Goals — no data I/O; "a future diskio-like component does - data I/O"), §3.4 (records are the source of truth; - `BusyBlockValue` carries `owner_chunk` — the recovery hand-off - handle), §8 (disk failure detection + bad-disk handling — - "relocation/rebuild is a follow-up requirement"; the impacted list - is "handed to a future recovery/relocation path: the data-IO layer - rebuilds from EC/mirror, or the owner is notified to re-allocate - elsewhere"), - [`design-crowdb-diskdb-zone-management.md`](../design/diskdb/design-crowdb-diskdb-zone-management.md) - §6 (crash recovery strategies 1/2/3 — diskdb-layer bitmap - reconstruction only). Fbs schema surfaces: - `chunkdb.fbs` (`Chunk`, `ChunkStrip`, `MirrorStrip`, - `EcStrip`, `StripType`, `ECState`, `ChunkState`, - `ChunkdbService` — `AllocateChunk` / - `AppendChunk` / `SealChunk` / `DeleteChunk` / `ReplaceChunkStripRange` / - `ListChunks`), `diskio.fbs` (`DiskWrite` / `DiskRead`). - CROWDB's chunkdb owns disk-failure recovery: it rebuilds lost blocks - from mirror/EC at the chunk layer, driving rebuild I/O and pacing it. -- **Use scenarios** — - - **Mirror replica rebuild after disk failure** — a disk goes `Bad`; - diskdb's R76 scan lists the impacted busy blocks + their - `owner_chunk`. chunkdb (the owner) reads each impacted chunk's - strip layout, finds the surviving mirror replica(s) on healthy - disks, reads the surviving data via `diskio` `DiskRead`, allocates - a new segment on a healthy disk via diskdb `AllocateBlocks`, writes - the rebuilt replica via `diskio` `DiskWrite`, and updates the - chunk's strip (`ReplaceChunkStripRange`) to reference the new `Segment`. - The old segment on the `Bad` disk is freed (`FreeBlocks`) once the - disk is replaced/removed. The chunk is whole again. - - **EC data-block reconstruction** — an EC strip (`data_num` + `code_num`) - loses one data block when its disk goes `Bad`. chunkdb reads the - surviving data + parity blocks via `diskio`, reconstructs the lost - data block, writes it to a newly-allocated segment, and updates - the strip. Parity is recomputed if a parity block was lost. - - **Recovery speed control during foreground load** — a cluster is - serving foreground writes while a disk is `Bad`. The operator sets - a recovery throttle (bandwidth / IOps / concurrent rebuilds). - chunkdb paces the rebuild reads + writes so foreground - allocate/free/read latency is unaffected; recovery progress + - throttle state are visible to the operator. - - **Restart mid-recovery** — chunkdb crashes or restarts while a - rebuild is in progress. On restart it reads persisted per-chunk / - per-strip recovery progress from KV and resumes the rebuild from - the last completed strip, without re-rebuilding completed strips - or losing the in-flight one. - - **Unrecoverable data loss** — a mirror strip loses *all* replicas, - or an EC strip loses more than `code_num` blocks. chunkdb detects - the loss is unrecoverable, reports it (metric + log), and does not - attempt a rebuild that would silently corrupt data. - -**Solution**: - -- **No clear solution yet — deferred to design.** chunkdb does not - exist yet — the chunkdb server component (`ChunkdbService` - implementation) is itself unbuilt, and the recovery flow's shape - depends on chunkdb's architecture (chunk→strip→segment storage, - ownership model, KV schema) plus the `diskio` service (also unbuilt). - The high-level shape is known from R76's placeholder + the reserved - fbs schema surfaces, but the detailed design (rebuild orchestration, - hand-off mechanism, throttle mechanism, progress schema) needs a - design draft once chunkdb's core is defined. - -- **One-line summary**: chunkdb owns real data recovery — rebuild lost - mirror replicas / EC data+parity from surviving strips via `diskio`, - pace rebuild I/O at the chunkdb layer so foreground traffic is not - starved, and persist per-chunk/per-strip progress for restart-resume. - -- **Numbered work items**: - 1. **chunkdb server component (prerequisite)** — - `app/crowdb-chunkdb` (new crate, not yet existing) implementing - `ChunkdbService` (`AllocateChunk` / `AppendChunk` / `SealChunk` / - `DeleteChunk` / `ReplaceChunkStripRange` / `ListChunks`). This is a - prerequisite for R83, not part of it — it should be filed as its - own backlog item. R83 assumes chunkdb exists and owns the - chunk→strip→segment mapping persisted to CROWDB KV. - 2. **Recovery orchestration** (`app/crowdb-chunkdb/src/recovery/`) — - consume diskdb's impacted-blocks + `owner_chunk` hand-off (R76 - `RecoveryScanTask` output); for each impacted chunk, read its - strip layout, classify each impacted strip (`MirrorStrip` vs - `EcStrip`), and build a rebuild plan (which surviving - replicas/blocks to read, how to reconstruct, where to write the - new segment). Replaces R76's `RecoveryAction::LogOnly` with - `Relocate` / `RebuildFromEc` variants. - 3. **`diskio` integration** — read surviving blocks via - `DiskioService::DiskRead` and write rebuilt blocks via - `DiskioService::DiskWrite` to newly-allocated segments (diskdb - `AllocateBlocks` on healthy disks, with `exclude_disks` anti- - affinity against the `Bad` disk). Prerequisite: the `diskio` - server component (also unbuilt; should be its own backlog item). - 4. **Strip update + old-segment free** — after a rebuilt block is - durably written to its new segment, `ReplaceChunkStripRange` the chunk's - strip to reference the new `Segment`, then `FreeBlocks` the old - segment on the `Bad` disk (after the disk is replaced/removed, or - once the strip no longer references it). - 5. **Recovery speed control** (`app/crowdb-chunkdb/src/recovery/throttle.rs`) - — pace rebuild I/O at the chunkdb layer: configurable - `recovery.max_bandwidth_bytes`, `recovery.max_iops`, - `recovery.max_concurrent_rebuilds` (live-reloadable). The - orchestrator gates each rebuild read/write through the throttle so - foreground allocate/free/read traffic is not starved. Emit - recovery-progress + throttle metrics (`recovery.rebuilt_blocks`, - `recovery.pending_chunks`, `recovery.throttle_utilization`). - 6. **Recovery progress persistence** — per-chunk / per-strip - recovery progress written to CROWDB KV (schema follows R76's - `RecoveryScanProgressValue` precedent, keyed by `ChunkId` / - strip index). On chunkdb restart, resume from the last completed - strip; no double-rebuild of completed strips. - -- **Flow diagram**: - -``` - disk goes Bad - │ - ▼ - diskdb RecoveryScanTask (R76) - │ lists impacted BusyBlockValue + owner_chunk per zone - ▼ - chunkdb recovery orchestrator (item 2) - │ for each impacted chunk: - │ read strip layout → classify Mirror / EC - │ build rebuild plan (surviving blocks + reconstruct fn) - ▼ - ┌──────────────────────────────────────────┐ - │ throttle gate (item 5) │ - │ max_bandwidth / max_iops / max_concurrent │ - └──────────────┬───────────────────────────┘ - ▼ - diskio DiskRead (surviving blocks) ──► reconstruct (mirror copy / EC) - │ - ▼ - diskdb AllocateBlocks (healthy disk, exclude Bad) ──► diskio DiskWrite (new segment) - │ - ▼ - ReplaceChunkStripRange (new Segment) ──► FreeBlocks (old segment on Bad disk) - │ - ▼ - persist per-chunk/strip progress (item 6) ──► next strip / next chunk -``` - -- **Edge cases at a glance**: - - Mirror strip loses all replicas → unrecoverable; report data loss, - do not rebuild. - - EC strip loses more than `code_num` blocks → unrecoverable; report - data loss, do not rebuild. - - Rebuild target disk goes `Bad` mid-rebuild → re-pick a healthy - target on the next cycle; the in-flight write to the now-`Bad` - target is abandoned. - - chunkdb restart mid-rebuild → resume from persisted per-strip - progress; no double-rebuild, no lost in-flight strip (the strip is - re-attempted). - - Recovery throttle set very low → recovery takes long; operator - sees progress via metrics; no foreground impact. - - `owner_chunk` missing on a busy block (legacy / corrupt record) → - skip the block, report it (cannot rebuild without the owner). - - EC parity not yet computed (`EC_STATE_NO_PARITY`) on a strip that - loses a data block → cannot reconstruct from parity; rebuild from - mirror fallback or report unrecoverable (design decision). - - Disk moved (R81 Part 2) with segments still on the original paxos - group → recovery must read the moved disk's records via the - per-disk bind, not the disk-group's default bind. - -**Dependencies**: - -- **chunkdb server component (prerequisite, unlanded)** — R83 assumes - the chunkdb server exists (`ChunkdbService` impl, chunk→strip→segment - KV schema). This must be filed as its own backlog item and landed - first; without it there is no chunk owner to drive recovery. -- **`diskio` service (prerequisite, unlanded)** — real data recovery - needs `DiskRead` (surviving blocks) + `DiskWrite` (rebuilt blocks). - The `diskio` server is a future component (§2, `diskio.fbs` - reserved). Must be filed as its own backlog item. Fallback without - diskio: R76's `LogOnly` placeholder stays (current state). -- **R76** — `RecoveryScanTask`, impacted-blocks + `owner_chunk` - collection, `RecoveryAction` placeholder (`LogOnly` → `Relocate` / - `RebuildFromEc`), `RecoveryScanProgressValue` schema as the progress - precedent. -- **R72** — diskdb `AllocateBlocks` / `FreeBlocks`, `Segment` record - model, `BusyBlockValue` / `owner_chunk`. -- **R81 Part 2** — disk move with per-disk bind; recovery must account - for moved disks whose records live on the original paxos group - (per-disk bind routing, not the disk-group's default bind). -- Nothing depends on R83 yet. - -**Acceptance**: - -- **Mirror replica rebuild**: a disk goes `Bad`; a mirror strip with 2 - replicas (one on the `Bad` disk, one on a healthy disk) → chunkdb - reads the surviving replica via `diskio`, allocates a new segment on - a healthy disk (excluding the `Bad` disk), writes the rebuilt - replica, `ReplaceChunkStripRange` updates to the new `Segment`, and frees the old - segment → the chunk reads back correctly from the new replica. E2E - test (pending chunkdb + diskio). -- **EC data-block reconstruction**: an EC strip (`data_num=4`, - `code_num=2`) loses 1 data block → chunkdb reconstructs the lost - block from surviving data + parity, writes it to a new segment, and - updates the strip → the chunk reads back correctly. E2E test. -- **EC parity-block reconstruction**: an EC strip loses 1 parity block - → chunkdb recomputes parity from the surviving data blocks, writes - it to a new segment, updates the strip → `EC_STATE_PARITY` restored. - E2E test. -- **Unrecoverable — mirror all replicas lost**: a mirror strip with 2 - replicas both on `Bad` disks → chunkdb reports data loss (metric + - log), does not attempt a rebuild, no silent corruption. E2E test. -- **Unrecoverable — EC loses more than `code_num`**: an EC strip - (`code_num=2`) loses 3 blocks → chunkdb reports data loss, does not - rebuild. E2E test. -- **Recovery speed control**: `recovery.max_iops` set to N → rebuild - read/write rate stays ≤ N; foreground allocate/free/read latency - stays within the pre-recovery baseline (no starvation). E2E test. -- **Restart resume**: chunkdb restarts mid-rebuild → on restart it - resumes from the persisted per-strip progress; completed strips are - not re-rebuilt, the in-flight strip is re-attempted and completes. - E2E test. -- **`owner_chunk` missing**: a busy block with no `owner_chunk` → - chunkdb skips it and reports it (metric); no crash, no rebuild - attempt. E2E test. -- **Moved-disk recovery**: a disk moved via R81 Part 2 (records on the - original paxos group via per-disk bind) goes `Bad` → chunkdb reads - the impacted blocks via the per-disk bind and rebuilds correctly. - E2E test. -- `pixi run cargo fmt --all -- --check` and - `pixi run cargo clippy --all-targets -- -D warnings` clean. -- `pixi run test-chunkdb` (relevant integration tests pass, pending - chunkdb crate) + `pixi run test-diskdb` (R76 hand-off integration). - -**Open Questions**: - -- **chunkdb + diskio as separate backlog items?** R83 depends on both - the chunkdb server and the diskio server existing. These are - prerequisites that should be filed as their own backlog items - (chunkdb core, diskio core) before R83 can be designed in detail. - Cannot be resolved autonomously — it is a scope/sequencing decision - that needs a human to confirm the prerequisite items should be - filed and in what order. -- **Recovery hand-off mechanism** — how does diskdb's - `RecoveryScanTask` hand the impacted-blocks + `owner_chunk` list to - chunkdb? Options: (a) push — diskdb notifies chunkdb (the owner) via - an owner-notification mechanism (does not exist yet); (b) pull — - chunkdb subscribes to disk `Bad` transitions (watch/notify, R78) and - pulls the impacted list from diskdb. Trade-off: (a) couples diskdb - to chunkdb; (b) keeps diskdb unaware of chunkdb but adds latency. - Design decision. -- **Recovery speed control granularity** — per-chunk, per-disk, or - per-instance throttle? Per-instance is simplest (one global budget); - per-disk allows prioritizing hot disks; per-chunk is the finest. - Recommendation: per-instance for v1, revisit if per-disk priority is - needed. Design decision. -- **EC strip with no parity yet (`EC_STATE_NO_PARITY`) loses a data - block** — cannot reconstruct from parity. Rebuild from a mirror - fallback (if the chunk has one), or report unrecoverable? Design - decision. diff --git a/doc/backlog/R84-chunkdb-post-disk-move-placement-scanner.md b/doc/backlog/R84-chunkdb-post-disk-move-placement-scanner.md deleted file mode 100644 index cc559582d..000000000 --- a/doc/backlog/R84-chunkdb-post-disk-move-placement-scanner.md +++ /dev/null @@ -1,246 +0,0 @@ - - - -### R84: chunkdb — Post-Disk-Move Chunk Placement Scanner - -**Problem**: - -- **Current behavior + impact** — R81 Part 2 adds disk move with a - stable `DiskId` (UUID): a physical disk is relocated from - `(old_rack, old_node, old_dg)` to `(new_rack, new_node, new_dg)`, - its zone/busy/free records are copied from the old disk-group's bind - to the new disk-group's bind during Maintenance (no concurrent - writes), and the disk becomes available at the new placement without - a full recovery scan. The move is placement-only — the disk's block - data is intact on the physical medium, and the records are keyed - only by `DiskId` (globally unique, §3.9), so copying them between - paxos groups is a literal key-value copy. What is missing is any - verification that chunk placement is still consistent after a move. - Chunks reference the blocks they own via `Segment { disk_id, - zone_index, unit_offset, unit_count }` (one `Segment` per allocated - range, embedded in `MirrorStrip` / `EcStrip` inside each - `ChunkStrip`, `chunkdb.fbs`). After a disk move, every chunk - that has a `Segment` on the moved disk must still be able to reach - that segment: the `disk_id` must resolve to a disk that is `Up` at - its new placement. There is no scanner that walks - chunk→strip→segment placement after a move and verifies - reachability + health. A move that leaves a chunk's segments on a - disk that is now unreachable (e.g. the move partially failed, the - disk went `Missing` at the new placement) would silently break chunk - reads with no detection until a client hits the broken segment. This - is the chunk-placement analog of diskdb's background scanner (§10) — - but diskdb's scanner checks disk-layer record/bitmap drift, not - chunk→segment placement integrity, which is chunkdb's domain. -- **Design pointers** — - [`doc/backlog/R81-sysdata-id-reuse-safety-and-disk-move.md`](R81-sysdata-id-reuse-safety-and-disk-move.md) - Part 2 (disk move, record copy during Maintenance, records keyed - by `DiskId`, no full scan), - [`doc/design/diskdb/design-crowdb-diskdb.md`](../design/diskdb/design-crowdb-diskdb.md) - §3.9 (unit-based sizes; disk-id key routing — record keys carry - `DiskId`, no `node_id`/`disk_group_id`), §8 (disk status management - — effective status = `max(node, group, disk)`; `Missing` detection - by absence from a group-0 sync response), §10 (Background Scanner — - `ScannerTask` / `BgRunner` / `BackgroundTask` pattern, KV-persisted - progress, resume after restart, admin RPCs `TriggerScan` / - `GetScanStatus`). Fbs schema surfaces: `chunkdb.fbs` (`Chunk`, - `ChunkStrip`, `MirrorStrip`, `EcStrip`, `Segment` reference), - `chunkdb.fbs` (`ListChunks` — paginated chunk scan). - CROWDB's disk-move model (R81) is new, so the post-move chunk - placement scanner is new work shaped on diskdb's `ScannerTask` - precedent. -- **Use scenarios** — - - **Post-move placement verification** — an operator moves a disk - from disk-group A to disk-group B (R81 Part 2). The move completes - and the disk is `Up` at the new placement. A scanner walks every - chunk that has a `Segment` on the moved `DiskId`, resolves each - segment's disk to its current placement (group-0 `DiskValue`), - and verifies the disk is `Up`. All segments reachable → the move - is placement-clean; - the scanner reports 0 inconsistencies. - - **Detect partially-failed move** — a disk move left the disk - `Missing` at the new placement (the new disk-group's instance has - not picked it up via keepalive reconcile). The scanner finds - chunks whose segments reference the moved `DiskId`, sees the disk - is not `Up` at the new placement, and reports the unreachable - segments + their owning chunks so the operator can fix the move - or trigger recovery (R83). - - **Detect orphaned segment after disk removal (not move)** — a - segment references a `DiskId` that no longer exists in group 0 - (the disk was removed, not moved). The scanner reports the - orphaned segment + owning chunk as a data-loss risk; the chunk - needs recovery (R83) if it has surviving replicas/parity. - - **Periodic placement integrity sweep** — independent of any - specific move, a periodic sweep verifies that every chunk's - segments resolve to a live, `Up` disk, catching placement drift - from any cause (move, removal, status change). - -**Solution**: - -- **No clear solution yet — deferred to design.** chunkdb does not - exist yet (see R83 prerequisites), so the scanner's host (chunkdb-side - vs diskdb-side), the chunk iteration mechanism, and the trigger - model need a design draft once chunkdb's core is defined. The - high-level shape is known from diskdb's `ScannerTask` precedent (§10) - + the disk-move model (R81). - -- **One-line summary**: a placement-integrity scanner that, after a - disk move (and periodically), walks chunk→strip→segment placement, - resolves each segment's `DiskId` to its current group-0 placement, - and reports any segment that is unreachable or orphaned. - -- **Numbered work items**: - 1. **Placement scanner task** (`app/crowdb-chunkdb/src/scanner/placement.rs`, - new — or `app/crowdb-diskdb/src/scanner/` if the design picks - diskdb as the host) — a background task following the - `BgRunner` + `BackgroundTask` pattern (R75 / §10 `ScannerTask`). - Runs on a configurable interval (`scanner.placement_interval_secs`, - default 600) and is triggerable on-demand after a disk move - (admin RPC `TriggerPlacementScan`). - 2. **Chunk→strip→segment walk** — iterate chunks via - `ChunkdbService::ListChunks` (paginated, `ChunkId` order); for - each chunk, iterate its `ChunkStrip`s and the `Segment`s inside - each `MirrorStrip` / `EcStrip`. Collect the set of `DiskId`s - referenced by live (non-deleted) chunks. - 3. **Segment reachability check** — for each referenced `DiskId`, - resolve its current placement from group 0 (`HardwareClient` - `DiskValue` lookup). Verify: the disk exists in group 0 (not - orphaned); the disk's effective status is `Up` (not - `Missing`/`Bad`/`Offline`). A segment whose disk is - `Bad`/`Missing` is reported + handed to recovery (R83). - 4. **Placement integrity report + metrics** — emit a - `PlacementScanSummary` (counts: `reachable_segments`, - `unreachable_segments`, `orphaned_segments`) + gauges - (`scanner.placement.unreachable`, - `scanner.placement.orphaned`). Admin RPCs `TriggerPlacementScan` - / `GetPlacementScanStatus` mirror diskdb's scanner admin surface. - 5. **Move trigger** — the scanner is triggered on disk-move - completion via watch/notify (R78, `/hw/disk/` prefix) when a - `DiskValue` moves path, with periodic sweep as a safety net - (same pattern as diskdb's watch/notify + polling safety net, §8). - 6. **Progress persistence + resume** — KV-persisted scan progress - (last scanned `ChunkId`, following the existing - `RecoveryScanProgressValue` checkpoint pattern); on restart the scanner - resumes from the last checkpoint, no full re-walk. - -- **Flow diagram**: - -``` - disk move completes (R81 Part 2) ──or── periodic interval - │ - ▼ - PlacementScannerTask (item 1) - │ - ▼ - ListChunks (paginated by ChunkId) ──► for each chunk: - │ iterate ChunkStrips → MirrorStrip / EcStrip → Segments - │ collect referenced DiskId set - ▼ - for each referenced DiskId (item 3): - ├─ group-0 DiskValue lookup (HardwareClient) - │ ├─ disk missing in group 0 → orphaned segment (report) - │ ├─ effective status != Up → unreachable (report + hand to R83) - │ └─ status Up → segment OK - ▼ - PlacementScanSummary + gauges (item 4) - │ - ▼ - persist progress (item 6) ──► next ChunkId page / done -``` - -- **Edge cases at a glance**: - - Segment on a moved disk, disk `Up` at new placement → reachable; - scanner reports 0 issues for that segment. - - Segment on a moved disk, disk `Missing` at new placement (move - partially failed) → unreachable; report + hand to recovery (R83). - - Segment on a disk removed from group 0 (not moved) → orphaned; - report data-loss risk + hand to recovery (R83). - - Segment on a `Bad` disk → unreachable; hand to R76/R83 recovery - (the placement scanner does not rebuild, it detects + routes). - - chunkdb restart mid-scan → resume from persisted - `last_scanned_chunk_id`; no full re-walk. - - Chunk deleted mid-scan (`ChunkState::Deleted`) → skip its segments - (freed); no false orphan. - - No chunks reference the moved disk → scanner reports 0 segments - scanned for that disk; no overhead beyond the chunk walk. - - Disk moved multiple times → only the final placement matters; the - scanner resolves the current `DiskValue` each run. - -**Dependencies**: - -- **R81 Part 2** — disk move with record copy during Maintenance; - the scanner verifies placement integrity after a move. Without R81 - there is no disk move to scan. -- **chunkdb server component (prerequisite, unlanded)** — the scanner - walks chunks via `ListChunks`; chunkdb must exist (same prerequisite - as R83). Must be filed as its own backlog item. -- **R75** — `BgRunner` + `BackgroundTask` pattern; the placement - scanner follows `ScannerTask`'s structure (interval trigger, - on-demand `TriggerScan`, `ScanState`, in-progress flag). -- **R78** — watch/notify on `/hw/disk/` for the move trigger (optional; - polling fallback if watch/notify is not landed). -- **R83** — recovery flow; the placement scanner detects unreachable - / orphaned segments and hands them to R83 for rebuild. R83 must - exist for the hand-off to do real work; without R83 the scanner - reports only (same placeholder shape as R76's `LogOnly`). -- **R72** — `Segment` record model; the scanner reads `Segment`s from - chunk strips. -- Nothing depends on R84 yet. - -**Acceptance**: - -- **Post-move reachability**: move a disk via R81 Part 2 to a new - disk-group, disk `Up` at the new placement → the scanner walks all - chunks with segments on the moved `DiskId`, verifies each segment - is reachable, reports 0 unreachable / 0 orphaned. E2E test (pending - chunkdb). -- **Partially-failed move detection**: move a disk but leave it - `Missing` at the new placement → the scanner reports the chunks + - segments referencing the moved `DiskId` as unreachable. E2E test. -- **Orphaned segment after removal**: a chunk's segment references a - `DiskId` removed from group 0 (not moved) → the scanner reports the - segment as orphaned with its owning chunk. E2E test. -- **`Bad`-disk segment routing**: a chunk's segment references a disk - that is `Bad` → the scanner reports the segment as unreachable and - hands it to recovery (R83); the scanner does not rebuild. E2E test - (pending R83). -- **Deleted chunk skipped**: a chunk transitions to - `ChunkState::Deleted` mid-scan → its segments are not reported as - orphaned (they are freed). E2E test. -- **Restart resume**: chunkdb restarts mid-scan → on restart the - scanner resumes from the persisted `last_scanned_chunk_id`; no full - re-walk, no missed chunks. E2E test. -- **On-demand trigger**: `TriggerPlacementScan` after a move → the - scan runs immediately (not waiting for the interval) and returns the - current summary + `scan_in_progress` flag. E2E test. -- **No chunks reference the moved disk**: a disk is moved but no chunk - has a segment on it → the scanner completes with 0 segments scanned - for that disk; no false positives. E2E test. -- `pixi run cargo fmt --all -- --check` and - `pixi run cargo clippy --all-targets -- -D warnings` clean. -- `pixi run test-chunkdb` (relevant integration tests pass, pending - chunkdb crate). - -**Open Questions**: - -- **Scanner host: chunkdb-side or diskdb-side?** The scanner walks - chunk→strip→segment placement (chunkdb's data model) but checks disk - reachability (diskdb's / group-0's domain). Options: - (a) chunkdb-side — chunkdb owns the chunk→segment mapping, so it is - the natural host; it calls `HardwareClient` to resolve disk - placement. (b) diskdb-side — diskdb has the scanner infrastructure - (`ScannerTask`) and the disk records, but it does not know which - chunks reference which disks (that is chunkdb's domain), so it would - need a reverse lookup chunkdb does not expose. Recommendation: - chunkdb-side (a) — the chunk→segment walk is the core of the scan - and only chunkdb has it. Design decision. -- **Trigger model: watch/notify vs periodic-only?** Watch/notify - (R78) on `/hw/disk/` gives immediate post-move triggering; periodic- - only is simpler but adds detection latency. Recommendation: - watch/notify trigger + periodic safety net (same as diskdb's model, - §8). Depends on R78 being landed. Design decision. -- **Should the placement scanner be bundled with diskdb's existing - background scanner (§10) or kept separate?** Bundling reuses one - `ScannerTask` loop + admin surface; keeping it separate respects - the chunkdb/diskdb component boundary. Recommendation: separate - (chunkdb-owned), mirroring diskdb's scanner structure. Design - decision. diff --git a/doc/backlog/R92-chunkdb-in-chunk-gc.md b/doc/backlog/R92-chunkdb-in-chunk-gc.md deleted file mode 100644 index 08b377e4b..000000000 --- a/doc/backlog/R92-chunkdb-in-chunk-gc.md +++ /dev/null @@ -1,17 +0,0 @@ - - - -### R92: chunkdb — In-Chunk GC Operations - -**Problem**: Shared chunks accumulate unused space as objects are deleted. -Without in-chunk GC, this space is never reclaimed, leading to storage -waste. CROWDB needs localized GC operations confined to individual chunks -to avoid global merge overhead. - -**Solution**: Implement in-chunk GC operations (ReclaimStrip, CollapseStrip, -MergeStrips) for shared chunks. Add logical-to-physical offset mapping -to support GC while keeping chunk IDs stable. R95 owns the chunk-centered -orphan scan and qualified range deletion; this requirement provides the -in-chunk reclamation operations after R95 proves a range unreachable. - -**Scope**: Placeholder - detailed design to be refined before implementation. diff --git a/doc/backlog/R95-chunkdb-chunk-range-delete.md b/doc/backlog/R95-chunkdb-chunk-range-delete.md deleted file mode 100644 index d3994c0f5..000000000 --- a/doc/backlog/R95-chunkdb-chunk-range-delete.md +++ /dev/null @@ -1,201 +0,0 @@ - - - -### R95: chunkdb — Qualified chunk range deletion and orphan scan - -## Problem - -Shared chunks contain ranges owned by different objects or multipart parts. -Deleting a whole chunk for one unreachable range can erase live neighbors. -Uploads may also write a chunk and fail before their part or final object -reference is recorded. A cleanup queue populated by each MPU mutation would -add metadata writes to the upload path and still miss those pre-record crashes. -The [ChunkDB design](../design/chunkdb/design-crowdb-chunkdb.md) separates -logical range retirement from physical strip release. A successful notification -does not supply independent orphan discovery or publication fencing. - -Current user-data frames contain length and write time but no final resource -identity. Dataset upload expiry discovers ranges through upload/part records; -Iceberg discovers pre-publication ranges through block intents. Removing these -records before an independent discovery path exists would leak writes that -crash before publishing their result. A second failure window is publication -admission before expiry followed by an indefinitely delayed final write after -GC has reclaimed its physical range. A local clock check and a long grace -period do not fence that delayed write. - -## Solution - -1. Complete `DeleteChunkRange(chunk_id, offset, size)` in the chunkdb protocol, - client and server. Access submits one notification without a preliminary - QueryChunk or whole-chunk deletion. An optional observed chunk revision - adds a strict layout precondition; otherwise ChunkDB obtains and atomically - conditions on its current layout. Only sealed user-data chunks accept exact - byte ranges; a dedicated chunk additionally requires its complete sealed - object extent. Stale explicit layouts and active writers are rejected or - deferred. Persist merged dead intervals and the sealed physical - identity as resource metadata, atomically fenced by the chunk revision. - This background resource state is approved; per-upload process records are - not. Successful retirement completes the caller's notification responsibility; - Access must not wait for physical reclamation or poll the returned physical - status. ChunkDB owns subsequent release and restart recovery independently. - Never round a dead frame outward to the usage bitmap's coarser granularity. -2. Use a bounded, chunk-centered scanner to find old chunks and ranges whose - bytes have no live owner. Do not create per-MPU cleanup intents, candidate - records or a durable cleanup queue. Start with a configurable age threshold - of one day; a candidate must be older than the threshold after its last - write. Age alone never authorizes deletion. Enumerate complete, CRC-validated - frames only within a sealed chunk's acknowledged extent. A malformed frame, - incomplete boundary or uncertain authority defers its range; never scan - past corruption by guessing the next object boundary. - New frame formats must carry enough resource identity to resolve the exact - final publication authority and its fixed eligibility deadline. Encode this - in the existing data write, without a separate foreground intent/index RPC. - A digest of a variable key is insufficient unless it directly addresses the - final authority. Header extensions change per-frame payload capacity: the - location format and native reader/writer arithmetic must select the layout - unambiguously. Continue reading old frames and retain their existing minimum - discovery facts until their ranges can be independently qualified. -3. Compare each candidate against published S3 objects, Iceberg file - descriptors, committed Dataset/Split manifests and still-eligible completed - parts. Unpublished parts have an explicit commit deadline. Final publication - must be fenced against expiry reclamation, including a request admitted - before expiry whose publication arrives later. A lost part-publication - response or missing intermediate MPU record is not proof that a chunk is - unused. Refuse deletion when references or the publication fence cannot be confirmed. - Dataset's immutable part authority permits a direct boundary: an expired - missing part and its delayed final creation compete on the same part key. - GC installs a small permanent retirement value before retiring the range; - the delayed create-if-absent must then fail. Do not delete this value after - the range notification, which would reopen the key to an arbitrarily late - creation. If a final part exists with different exact locations, a losing - physical write cannot replace that immutable result. Existing owners and - complete final references protect the matching locations beyond the temporary - deadline. This eliminates process progress, not final resource facts. -4. Keep S3 MPU session, current-part and immutable part-generation keys under - a common upload prefix in Chunk-KV so the scanner can enumerate one upload's - references with a bounded prefix scan. The S3 multipart authority owns that - key layout and Abort/expiry semantics. Only completed data resources and - published references are required by this contract; it does not authorize - durable upload workflows. Old part generations remain available until the - scanner proves their bytes are unreachable. An aborted or expired upload becomes a candidate only after - the age gate and reference checks. -5. Recheck chunk identity, layout generation and exact range against current - authority immediately before reclaim. Treat lost delete replies - idempotently and keep an in-memory scan cursor and bounded work budget. - Restart may rescan old chunks. Report examined, - eligible, retired, deferred and physically reclaimed bytes without per-upload - metric labels. Snapshot deletion denies new requests and waits its persisted - grace period; readers are not registered or renewed. Layout retirement uses - its bounded validity interval; overdue readers may fail explicitly. -6. Establish overwrite publication authority before removing S3/Iceberg - discovery records. S3 ordinary objects are directly overwritten at their - tenant/bucket/object key and have no globally addressed immutable file - authority. Current Iceberg files publish at a canonical table path and file - name; the catalog/file-ID lookup is a legacy path, not the authority for - every new file. Neither can safely use a permanent tombstone at a mutable - name alone: a legitimate replacement must remain possible, while an old - generation must never return. A separate generation index alone also leaves - a cross-key race between checking eligibility and publishing the final name. - Select an authoritative conditional mutation or an immutable payload - generation plus an atomic reference boundary before implementing this part. - Specify the additional resource fields and foreground RPC cost for review; - do not add a request ledger, journal or response receipt by default. -7. Keep complete-result expiry discoverable. Dataset Snapshot candidate and - Split revision roots need fixed creation/eligibility facts in their existing - resource writes; retries must not extend their original deadline. Complete - unreferenced results remain readable by ID until retirement. Discover and - fence both partial and complete expired resources, preserving latest/stable, - retention, live children and inherited/shared data. An owner acquired before - the final root completes must not pin an abandoned payload forever. These - authority facts are a dependency of orphan qualification, not a reason to - restore editing/evaluation workflows. - -## Dependencies - -- R92 supplies in-chunk strip reclamation after R95 qualifies dead ranges. -- The S3 multipart authority supplies grouped MPU keys and completed part - and final object references. The scanner also recognizes Iceberg's frozen - MPU selection. -- Published references, part deadlines and deletion/layout grace must be - verifiable before physical deletion is enabled; R168 may use the qualified range-delete - interface for ordinary S3 object deletion. -- Access retains the existing minimum physical-range discovery facts until - the frame scanner and final publication fences are verified. Access cleanup - does not implement background physical scanning or wait for physical free. -- The overwrite authority choice below and complete-resource expiry must land - before removing their only orphan-discovery path. Without them, defer unsafe - ranges and retain the existing facts; do not report the scanner as complete. - -## Acceptance - -- Given a shared chunk with live and unreachable ranges, when the scanner - evaluates a candidate older than one day, assert only the exact unreachable - range is passed to `DeleteChunkRange`. Invariant: a live neighbor is never - reclaimed. E2E test. -- Given an MPU part that was replaced, aborted or expired, when its grouped KV - prefix is scanned, assert old locations remain protected by any active or - frozen selection and become eligible only after the age and reference - checks. Invariant: no extra MPU cleanup record is required. Integration test. -- Given a write that crashed before part metadata publication, when the chunk - scan runs, assert it defers the chunk until the age gate and in-flight writer - proof settle, then discovers the orphan without a part record. Invariant: - missing upload metadata alone cannot reclaim data. Integration test. -- Given an unexpired deletion/layout grace, changed layout generation or uncertain authority, - when range deletion is attempted, assert no bytes are reused. After the - grace expires and identity is confirmed, repeat the same delete through a - lost reply and assert one idempotent result. Invariant: reclamation is fenced - by current chunk and publication authority, without reader sessions. Integration test. -- Given a large backlog and foreground load, when scanning runs, assert it - respects item, byte and time budgets and reports deferred/reclaimed progress. - Invariant: cleanup cannot monopolize the data path. Integration test. - -- Given adjacent, overlapping and unaligned byte ranges, when the server records - their retirement and restarts, assert exact union, unchanged live gaps and - idempotent replay. A new stale-layout range fails without mutation; an already - retired range can be queried again. Invariant: monotonic exact retirement. - Integration test. -- Given retirement success with blocks still allocated, when the client receives - that response, assert Accepted without further polling. Invariant: notification - completion is independent of physical capacity recovery. Unit test. -- Given an expired Dataset part whose final creation is paused in transit, when - GC seals the same resource key and notifies range retirement, then resume the - write and retry it after worker restart, assert both creations fail and the - permanent resource fence remains. Invariant: retirement cannot resurrect. - Integration test. -- Given a complete part shared by live Snapshot owners and an abandoned root - that acquired an owner before completing, when expiry scans run after restart, - assert only the abandoned owner's reference is retired, matching live bytes - survive, and a complete resource's original deadline is unchanged by retry. - Invariant: final reference facts determine liveness. Integration test. -- Given an old S3 PUT or Iceberg file publication paused after its eligibility - check, when its generation expires and a new generation is published, resume - the old mutation and assert it cannot replace the new value or reference the - retired bytes. Repeat with a lost mutation reply. Invariant: overwrite - publication and generation retirement share an authoritative boundary. - Integration test. -- Given old frames and new frames with extended resource headers, when native - and HTTP readers request first, interior and final frame ranges before and - after restart, assert exact logical bytes without collecting a copy or using - the old fixed-header offset arithmetic for the new layout. Corrupt or - incomplete resource headers defer reclamation. Invariant: format-compatible - physical discovery preserves readable data. Integration test. - -## Open Questions - -- Which authoritative overwrite boundary should S3 and current Iceberg file - publication use? An in-authority conditional mutation must fence the exact - retired generation without an extra eligibility-read race. An immutable - payload-generation resource adds at least one resource write per generation - and still requires an atomic final reference boundary. Describe the concrete - failure flow and RPC cost before selecting either; an independent index or a - local expiry check is insufficient. - -Required gates: - -- `pixi run clean-env && pixi run cargo test -p crowdb-chunkdb --all-targets` -- `pixi run cargo test -p crowdb-chunk-client --all-targets` -- `pixi run cargo test -p crowdb-access-s3 --all-targets` -- `pixi run cargo test -p crowdb-access-dataset --all-targets` -- `pixi run cargo test -p crowdb-access-iceberg --all-targets` -- `pixi run cargo fmt --all -- --check` -- `pixi run rs-lint` diff --git a/doc/backlog/R96-chunkdb-console-cli-integration.md b/doc/backlog/R96-chunkdb-console-cli-integration.md deleted file mode 100644 index 0c6d2a3a9..000000000 --- a/doc/backlog/R96-chunkdb-console-cli-integration.md +++ /dev/null @@ -1,12 +0,0 @@ - - - -### R96: chunkdb — Console/CLI Integration - -**Problem**: chunkdb has no management interface for operators. There is no way to view chunk status, query chunk distribution, or manage chunk lifecycle via web UI or CLI. - -**Solution**: Add chunkdb panel to crowdb-web console with chunk overview, distribution visualization, and query capabilities. Implement crowdb-cli chunkdb subcommands for chunk management operations. Integrate with group-0 for topology display. - -**Scope**: The complete Web UI, including Chunk listing and Strip placement, -is specified by [Console UI specification](../design/console/design-crowdb-console-ui.md). This item retains CLI -integration and shared operation reuse; do not implement a second Web panel. diff --git a/doc/backlog/backlog.md b/doc/backlog/backlog.md deleted file mode 100644 index 26c83c222..000000000 --- a/doc/backlog/backlog.md +++ /dev/null @@ -1,95 +0,0 @@ - - - -# CROWDB Backlog Index - -This index contains only requirements that are not complete. Completed -requirements and their temporary execution plans are removed during the final -`implement-requirement` cleanup. Requirement detail and acceptance criteria -live in the linked document; implementation follows -[`implement-requirement`](../../.agents/skills/implement-requirement/SKILL.md). - -**Next R number: R245** - -## Dataset and access - -- **[R233](R233-sdk-local-cache-checkpoint.md)** — optional local disk read cache - and recoverable checkpoint staging with explicit remote durability. - **Area:** SDK. **Complexity:** High. **Status:** Deferred until dedicated HTTP workload measurements. - -## Access server and external data - -- **[R168](R168-s3-shared-object-reclamation.md)** — shared small-object - reclamation. **Area:** access server / S3 / chunkdb. **Complexity:** High. - **Status:** Deferred until R95 and the basic S3 delete path are stable. -- **[R169](R169-s3-shared-chunk-tree-gc.md)** — shared-chunk and B+tree garbage - collection. **Area:** access server / S3 / crowdb-tree / chunkdb. - **Complexity:** High. **Status:** Deferred until R147, R168, and workload - measurements are complete. -- **[R185](R185-access-iceberg-cache-invalidation.md)** — bounded Iceberg cache - and invalidation. **Area:** access server / Iceberg / Group 0 / Chunk-KV. - **Complexity:** Medium. **Status:** Deferred pending focused cache measurements. -- **[R186](R186-access-iceberg-orc-validation.md)** — selected ORC validation. - **Area:** access server / Iceberg. **Complexity:** Medium. **Status:** - Deferred as an independent follow-up by user decision. -- **[R196](R196-refine-benchmark-tests.md)** — refine benchmark tests: external - CLI workloads and shared collection against a three-node container cluster; - record results in each driver. **Area:** CLI / storage / access / benchmarks. - **Complexity:** Medium. **Dependencies:** existing OCI and service contracts. - -## Chunk, Chunk-KV, and recovery - -- **[R83](R83-chunkdb-complete-recovery-flow.md)** — complete data recovery and - recovery-speed control. **Area:** chunkdb / diskdb / DiskIO. - **Complexity:** High. **Dependencies:** chunkdb server and DiskIO recovery path. -- **[R84](R84-chunkdb-post-disk-move-placement-scanner.md)** — post-move chunk - placement scanner. **Area:** chunkdb / diskdb. **Complexity:** Medium. - **Dependencies:** R81 Part 2 and the chunkdb server. -- **[R92](R92-chunkdb-in-chunk-gc.md)** — in-chunk garbage-collection operations. - **Area:** chunkdb / chunk IO. **Complexity:** High. **Dependencies:** chunk - lifecycle and qualified range ownership. -- **[R95](R95-chunkdb-chunk-range-delete.md)** — qualified chunk-range deletion - and orphan scanning. **Area:** chunkdb / chunk IO / S3. - **Complexity:** High. **Dependencies:** chunk-range ownership records. -- **[R96](R96-chunkdb-console-cli-integration.md)** — chunkdb CLI management - integration. **Area:** chunkdb / console / CLI. **Complexity:** Medium. - **Dependencies:** stable chunkdb management surface. -- **[R147](R147-tree-chunk-gc.md)** — reclaim B+tree chunk strips. **Area:** - crowdb-tree / chunkdb / diskdb. **Complexity:** High. **Status:** Deferred - until immutable page packs and durable reclaim candidates are available. -- **[R148](R148-chunk-stream-scale-out.md)** — stream metadata scale-out and - sealed-chunk EC. **Area:** chunk-stream / chunk-kv / KV / chunkdb. - **Complexity:** High. **Status:** Deferred until the measured single-group - mirror path is complete. -- **[R193](R193-chunkdb-node-failure-budget.md)** — configurable node-failure - budget and EC placement. **Area:** KV / chunkdb / chunk IO / deployment. - **Complexity:** High. **Status:** Planned. -- **[R207](R207-chunkdb-repo-metadata-chunk-kv.md)** — repo metadata and tasks on - chunk-kv. **Area:** chunkdb / chunk-kv. **Complexity:** High. - **Status:** Deferred beyond the direct-KV stage. - -## KV, DiskDB, and topology - -- **[R80](R80-diskdb-rebalance.md)** — disk space rebalance convergence. - **Area:** diskdb / DiskIO. **Complexity:** Medium. -- **[R82](R82-kv-watch-notify-coalescing.md)** — watch/notify coalescing. - **Area:** KV / diskdb. **Complexity:** Medium. -- **[R102](R102-diskdb-dynamic-binding-migration.md)** — dynamic disk-group - binding migration. **Area:** diskdb / KV. **Complexity:** High. -- **[R103](R103-chunkdb-range-migration.md)** — dynamic slot ownership and - KV-group expansion/shrink. **Area:** chunkdb / KV. **Complexity:** High. - **Status:** Deferred by user decision until dynamic ownership is requested. -- **[R139](R139-group0-service-config.md)** — single-container and cluster - performance profiles, test tuning, and Group-0 distributed service configuration. **Area:** configuration / control plane. **Complexity:** Medium. - **Status:** Deferred until the file-backed service configuration contract. -- **[R144](R144-chunk-kv-partition-merge.md)** — adjacent partition merge. - **Area:** chunk-kv / KV / group 0. **Complexity:** High. **Status:** Deferred - until manifest reuse, transfer fencing, and transition recovery are stable. - -## Tree and memory - -- **[R4](R4-bounded-mempool.md)** — bounded memory pool for tree allocations. - **Area:** crowdb-tree engine. **Complexity:** Medium. -- **[R60](R60-tree-scan-sibling-leaf-readahead.md)** — sibling-leaf readahead on - cold scans. **Area:** crowdb-tree / scan / DiskIO. **Complexity:** Medium. - **Status:** Deferred pending cold file/block-backed measurements. diff --git a/doc/design/README.md b/doc/design/README.md new file mode 100644 index 000000000..0612747a9 --- /dev/null +++ b/doc/design/README.md @@ -0,0 +1,64 @@ + + + +# CROWDB Design + +These overviews explain CROWDB's architecture, capabilities, and design trade-offs. Start with the [product design](design-crowdb.md) for workloads and advantages, then read the [architecture](design-crowdb-architecture.md) to see how the components fit together. Component READMEs provide API usage, configuration, and operational instructions. + +For a shorter path through the design: + +- **Application access:** choose [Iceberg](design-crowdb-access-iceberg.md), [Dataset](design-crowdb-access-dataset.md), or [S3](design-crowdb-access-server.md). Trusted Dataset and S3 applications can also use the [native SDK](design-crowdb-sdk-native.md). +- **Metadata and recovery:** compare [Paxos-KV](design-crowdb-paxos-kv.md) and [Chunk-KV](design-crowdb-chunk-kv.md), then follow their links to Tree and ChunkStream. +- **Payload I/O:** start with [Chunk storage](design-crowdb-chunkdb.md), then follow allocation, device I/O, and transport into DiskDB, DiskIO, and RPC. +- **Operations:** read [cluster management](design-crowdb-monitor.md) and [container deployment](design-crowdb-deployment.md). + +## Product and architecture + +| Document | Scope | +| --------------------------------------------- | ---------------------------------------------------------------------------------------------- | +| [Product design](design-crowdb.md) | Workloads, shared storage, independent scaling, coordinated recovery, and design advantages. | +| [Architecture](design-crowdb-architecture.md) | Component roles, logical/physical placement, serving capacity, recovery, and resource control. | + +## Components + +| Document | Scope | +| ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------- | +| [Iceberg storage](design-crowdb-access-iceberg.md) | Native writes, direct-URI reads, table commits, and integrated file lifecycle. | +| [Dataset storage](design-crowdb-access-dataset.md) | Sample-oriented storage, small payloads, Snapshot/Split selection, and direct training I/O. | +| [Access Server and S3](design-crowdb-access-server.md) | Host, S3 buckets and objects, streaming, multipart, and authentication. | +| [Native SDK](design-crowdb-sdk-native.md) | Direct Dataset/S3 access, RDMA/TCP, retained host buffers, capacity budgets, and future GPU work. | +| [Paxos-KV](design-crowdb-paxos-kv.md) | Concurrent slot writes, per-key apply ordering, stores/groups/replicas, and durable metadata. | +| [Chunk-KV](design-crowdb-chunk-kv.md) | Ordered metadata, independent payload placement, smooth split/transfer, and serving capacity. | +| [Tree storage engine](design-crowdb-tree.md) | Concurrent skiplists, page mapping, fork/join flush, checkpoints, and external-WAL recovery. | +| [ChunkStream](design-crowdb-chunk-stream.md) | Reusable durable byte streams, journal composition, rollover/recovery/trim, and future interfaces. | +| [Chunk storage](design-crowdb-chunkdb.md) | Chunk placement, large/small write pipelines, EC, read flow, and repair. | +| [DiskDB allocation](design-crowdb-diskdb.md) | Disk/zone allocation, durable records, capacity accounting, and recovery. | +| [DiskIO](design-crowdb-diskio.md) | Addressed byte I/O, asynchronous backends, completion, and buffer lifetime. | +| [Storage RPC](design-crowdb-rpc.md) | RDMA data paths, buffer ownership, resource backpressure, asynchronous completion, and TCP fallback. | +| [Cluster management](design-crowdb-monitor.md) | System metadata in Group 0, node discovery, service supervision, and console operations. | +| [Container deployment](design-crowdb-deployment.md) | OCI packaging, runtime independence, host networking/devices, and CPU/memory limits. | + +## Usage and validation + +| Document | Scope | +| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | +| [Dataset HTTP SDK](../../sdk/README.md) | Portable Dataset HTTP SDK for Rust/Python, examples, packaging, and limits. | +| [Native Rust SDK](../../lib/crowdb-sdk-native/README.md) | Native Rust API usage, memory budgets, transport options, and shutdown. | +| [Native Python SDK](../../lib/crowdb-sdk-native-python/README.md) | Trusted native Python packaging, buffers, and application lifetime. | +| [Local node guide](../../container/single-node-container/README.md) | Local node storage, listener setup, and client examples. | +| [OCI image guide](../../container/oci-image/README.md) | OCI build, launch, host deployment, persistent data, and lifecycle. | +| [Access acceptance guide](../../app/crowdb-access-server/tests/README.md) | Access acceptance evidence and client/engine validation boundaries. | +| [E2E guide](../../e2e-test/README.md) | Container suites, deployment/restart checks, service ownership, and artifacts. | +| [Ecosystem matrix](../../e2e-test/clients/ecosystem/README.md) | Pinned external client/engine matrix and manual acceptance. | +| [Tree benchmark guide](../../lib/crowdb-tree/bench/README.md) | Targeted Tree benchmark conditions, reproduction, and interpretation. | +| [Tools guide](../../tools/README.md) | Tool directories, Pixi task entry points, and validation commands. | + +## Project files + +| Document | Scope | +| ---------------------------------------------- | ---------------------------------------------------------- | +| [README.md](../../README.md) | Product introduction and deployment options. | +| [CONTRIBUTING.md](../../CONTRIBUTING.md) | Environment, development workflow, and contribution rules. | +| [VERSION](../../VERSION) | Canonical development version. | +| [SECURITY.md](../../SECURITY.md) | Vulnerability reporting. | +| [CODE_OF_CONDUCT.md](../../CODE_OF_CONDUCT.md) | Community conduct. | diff --git a/doc/design/access-server/dataset/design-crowdb-access-dataset.md b/doc/design/access-server/dataset/design-crowdb-access-dataset.md deleted file mode 100644 index 0a7da892d..000000000 --- a/doc/design/access-server/dataset/design-crowdb-access-dataset.md +++ /dev/null @@ -1,589 +0,0 @@ - - - -# CROWDB - Design: Dataset - -Dataset is CROWDB's sample-oriented data domain for AI, analytics, and data -preparation. It owns dataset metadata, immutable snapshots, sample selection, -field projection, bounded reads, and payload lifetime while using CROWDB KV and -chunks for durable state and bytes. - -The normative object and behavior contract is [CROWDB Dataset Specification](spec-crowdb-dataset.md). This design document describes the implementation architecture that realizes that contract; it does not redefine Dataset semantics. - -This design is independent of S3 and Iceberg. A Dataset may record an external -source reference, but that reference does not replace Dataset publication, -selection, or reclamation authority. - -## Table of contents - -1. [Scope and boundary](#1-scope-and-boundary) -2. [Authority and naming](#2-authority-and-naming) -3. [Dataset and snapshot model](#3-dataset-and-snapshot-model) -4. [Manifest, schema, samples, and fields](#4-manifest-schema-samples-and-fields) -5. [Selection and read plans](#5-selection-and-read-plans) -6. [Access surfaces and bounded delivery](#6-access-surfaces-and-bounded-delivery) -7. [Progress and lifecycle](#7-progress-and-lifecycle) -8. [Interactions](#8-interactions) -9. [Correctness invariants](#9-correctness-invariants) -10. [Implementation and I/O flows](#10-implementation-and-io-flows) -11. [Risks and open questions](#11-risks-and-open-questions) -12. [Empty Dataset retirement](#12-empty-dataset-retirement) - -## 1. Scope and boundary - -Dataset provides a stable logical model for samples and heterogeneous fields. It -supports HTTP access through an independently packaged Rust SDK and Python -wrapper, plus trusted direct storage access through the shared native runtime -and its optional Python extension. Both surfaces preserve the same selection -and integrity semantics. It does not own disk placement, replication, -erasure coding, repair, or physical chunk layout. It does not become a query -engine or a training-framework-specific storage protocol. - -The core objects are Dataset, Snapshot, Manifest, Schema, Sample, Field, -Split revision, Selection, Read Plan, and Cursor. Worker scheduling, tensor decoding, collation, -and model checkpoints remain in adapters or execution systems above Dataset. - -## 2. Authority and naming - -Dataset authority always supplies the fixed `dataset-ns` default/root namespace. Every Dataset -identity carries that namespace, even though the first implementation exposes no -namespace create, delete, or mutation operation. Dataset names are unique within -the supplied namespace. The representation keeps namespace in the key so a -future multi-namespace authority can be added without changing Dataset identity, -snapshot inheritance, chunk placement, or read ordering. - -## 3. Dataset and snapshot model - -A Dataset owns metadata, snapshots, manifests, samples, fields, and the payload -references needed to read them. A Snapshot is an immutable published view -identified by an opaque `snapshot_id`. `latest` and `stable` are mutable names -that resolve to real snapshot IDs; cursors and read plans store the real ID. - -A Snapshot has at most one `parent_snapshot_id`. A child applies additions, -deletions, and membership rules to its parent; the numeric order of IDs never -implies ancestry. Any committed same-Dataset Snapshot can be the parent; multiple -parentless roots form a forest. Multi-parent merging is not supported. Old -Snapshots remain readable until reclamation has passed their protection checks. - -Publication builds and validates the manifest, schema binding, indexes, payload -references, checksums, and durability state before atomically advancing a -mutable name. Readers observe either the old complete snapshot or the new -complete snapshot. - -## 4. Manifest, schema, samples, and fields - -The Manifest is the authority for a Snapshot's schema binding, sample membership, -field locators, checksums and parent delta. Split sets are separate immutable -revisions owned by that Snapshot. The Manifest -may be partitioned and indexed; a single hot key cannot be required for all -reads. The current Schema records a version and field names/requiredness; it -does not declare types, MIME, nested structures or semantic media metadata. - -`sample_id` is a stable opaque byte sequence supplied by the importer, represented -as canonical padded base64 in JSON. Within one -snapshot, `(sample_id, field)` has one logical value. A field is either an inline -KV value for bounded metadata or an opaque chunk location string for larger -payloads. Dataset passes the location to the chunk system and never parses or -constructs its physical meaning. Copy-on-write shares unchanged values between -snapshots; field and sample deletion uses tombstones so historical snapshots -remain valid. - -A sample may contain image, video, audio, text, label, embedding, or metadata -fields. Fields are aligned by snapshot and sample identity, while decoding and -tensor composition belong to the consumer adapter. - -## 5. Selection and read plans - -A Split is a named logical subset of a Snapshot. One immutable Split revision -contains the complete set of names, definitions, membership and overlap/coverage -policy. A Snapshot's currentSplit points to one revision or is absent. Publishing -a revision changes only this pointer, never Dataset latest or the data Snapshot. -A Selection in a Read Plan chooses -samples using prefix, range, explicit IDs, keyword, field match, and boolean -composition. Prefix, range, and explicit IDs can be checked for overlap directly; -metadata predicates require bounded work against one fixed Snapshot. Defaults -are disjoint membership and allowed unassigned Samples. A committed empty set -differs from no Split revision. All committed revisions live until owner reclaim. - -The [control lifecycle](design-dataset-control.md) specifies durable composition, -frozen Split inheritance, publication attempts, bounded inspection and cleanup. - -A Read Plan binds one snapshot ID to selection, ordering, optional shuffle -parameters, and batch policy. The implementation supports bounded sample and -group shuffle. -Sample shuffle loads only sample IDs and lightweight metadata into bounded groups; -a selection that fits one group completes in one pass, while a larger selection -uses multiple groups with the same deterministic contract. Group shuffle remains -an explicit plan type for prefix, metadata, session, device, time-window, or -association-key grouping. Every shuffle has sample-count, retained-byte, batch, -prefetch, and in-flight limits and is rejected before payload reads when a limit -would be exceeded. Shuffle never moves or rewrites published payload. - -## 6. Access surfaces and bounded delivery - -HTTP is the portable Dataset surface used by the public Rust SDK and its -Python wrapper. The trusted native facade reuses in-process Dataset authority -and reaches Chunk-KV and ChunkIO/DiskIO through the shared runtime. Its bounded -versioned routing observations do not replace storage authority. See the -[native runtime contract](../design-crowdb-native-sdk.md) and -[native SDK guide](../../../../lib/crowdb-sdk-native/README.md). - -`GetSample`, `GetBatch`, and bounded Scan resolve metadata first, apply field -projection, and then read inline values or chunk ranges/streams. Batch size, -read window, prefetch, cache, and in-flight requests are bounded and propagate -backpressure. Direct GPU delivery may use short-lived capabilities and a CPU -fallback, but it must preserve the same selection and integrity result. - -## 7. Progress and lifecycle - -A cursor is valid only for its bound Read Plan and snapshot. Ordered reads store a -sample position; grouped shuffle stores group identity or index and an offset -within the group. The client advances a cursor after batch confirmation. A crash -may replay the last unconfirmed batch, but may not skip or change the snapshot. -Epoch and sampler state remain client state. - -Read admission checks the selected snapshot's authoritative deletion state without -registering a reader. Deletion durably fences new reads and records a configured grace-period -`reclaim_after` deadline (twenty minutes by default); retries and restarts preserve that deadline. Existing -streams can finish using resolved metadata and payloads during the grace period. -Unusually long reads may fail after reclamation; partial delivery is never complete -success. Access Servers and native clients do not acknowledge deletion. Client -checkpoints and local request/buffer admission have independent lifetimes. EOF, -cancel and drop release local request admission without remote cleanup writes; -caller-held buffers remain valid and charged until their final release. - -Dataset owns the lifetime of its metadata and payload references. Reclamation -preserves explicit retention, live snapshot ancestry and shared payload owners. -External S3 or Iceberg references are provenance unless an explicit cross-model -retention contract says otherwise. Physical I/O identity checks and buffer ownership -remain necessary when storage is reclaimed or reused. - -## 8. Interactions - -Dataset metadata uses KV authority and can use chunk-backed range state for -large manifests or indexes. Chunk I/O owns location validation, range/stream -reads, checksums, and physical recovery. Access Server exposes HTTP and health -boundaries; a direct client observes topology but never publishes placement. -Framework adapters map Read Plans to PyTorch `IterableDataset`, TensorFlow/JAX -iterators, Hugging Face streaming examples, or Ray blocks without adding storage -objects. - -S3 and Iceberg remain peer metadata domains. Dataset can consume immutable data -from either through an explicit source reference, but cannot commit their tables, -overwrite their objects, or reclaim their owned bytes. - -The Dataset library owns these semantics and storage clients. The existing Access -Server may expose Dataset HTTP on its own configured listener, alongside S3 and -Iceberg. The Dataset SDK has an independent Rust workspace and release version -in the CrowDB repository. Its HTTP client uses reqwest, Tokio and rustls; the -Python package calls that implementation through PyO3. Clients share pooled -connections, atomic request/byte admission and process-owned runtime resources. -Blocking native waits release the GIL; live handles inherited across fork are -rejected. File transfers and batched iteration bound memory independently of -file size. The optional native Python extension calls the shared Rust native -facade, retaining the same bounded ownership and runtime shutdown contracts. -Direct metadata and chunk access use the runtime's configured TCP/RDMA policy -without implicit HTTP fallback. Tensor decoding and collation stay above both -SDK surfaces; optional local staging remains a separate extension. - -## 9. Correctness invariants - -- **DATA-I1 — Namespace identity:** every Dataset carries the configured - default/root namespace, and a Dataset name is unique within it. -- **DATA-I2 — Snapshot consistency:** one operation resolves exactly one immutable - snapshot and never mixes fields from another snapshot. -- **DATA-I3 — Atomic publication:** readers observe no manifest or one complete - published snapshot. -- **DATA-I4 — Explicit ancestry:** only the selected snapshot's parent chain - explains inherited state; ID ordering is irrelevant. -- **DATA-I5 — Opaque locations:** Dataset does not interpret physical chunk - locations. -- **DATA-I6 — Logical shuffle:** ordering and shuffle do not move published - payload. -- **DATA-I7 — Bounded delivery:** batch, window, prefetch, cache, and in-flight - limits bound memory and registration. -- **DATA-I8 — Cursor binding:** a cursor and Read Plan cannot resume against a - different snapshot or selection identity. -- **DATA-I9 — Safe reclamation:** retention and live ownership protect reachable - metadata and payload. Deletion rejects new reads and delays reclamation for the - grace period; overdue reads may fail without returning truncated success. -- **DATA-I10 — Surface equivalence:** HTTP and native access return the same - logical selection and integrity result. - -## 10. Implementation and I/O flows - -The implementation is split into a `crowdb-access-dataset` library and a thin -`crowdb-access-server` Dataset listener, following the S3 and Iceberg boundary. -`DatasetAuthority` owns keys, records, publication state, ancestry, retention, -through the `DatasetStore` interface; read checkpoints stay with the caller. `DatasetReadService` -owns admission, bounded delivery, plan/cursor execution, and request lifetime. -`DatasetHttpService` owns HTTP framing, authentication, route/status mapping, -shutdown, and health wiring. `ChunkKvDatasetStore` adapts Dataset metadata and -payload access to the chunk-backed KV client without exposing physical layout -to the authority. - -### 10.1 Upload and publication flow - -`DatasetUploads` persists bounded upload resources, immutable numbered part -receipts and Dataset-scoped ownership in `DatasetStore`. -`ChunkPayloadStorage` writes parts through the shared small-object pipeline with -Dataset chunk type 7. Each part is at most 4 MiB; 10000 parts bound an upload to -39.0625 GiB. A write intent records the exact physical location before payload I/O. -Successful durable completion accepts the receipt. Recovery accepts an intent -only after the chunk's acknowledged cursor covers the location and the bytes -pass integrity checks. Readable but unconfirmed bytes remain pending. A fenced -abandoned allocation is reclaimed before replacing it under the same part ID. -Upload resources hold at most 256 active and 4096 retired owner identities. A full -retirement list closes admission to new owners while existing owner retries remain -valid. Retiring an owner not recorded in a full list upgrades the same resource to -version 5 before success, even if that owner has not yet claimed it; an active owner -is removed in that CAS. This prevents an older writer paused before its claim from -reintroducing the owner after cleanup. Readers supporting only versions 2 through 4 -reject this final fence. Older Access servers and native SDKs must upgrade before -reading or writing that saturated resource. New mutators retain version 5 and -require its retirement list to remain full. -Ordinary resources use version 4. The first upgrade costs one GET and one CAS; -repeating retirement of an absent owner in version 5 needs one GET and no write. -Upload storage uses an independent binary envelope: a 52-byte fixed header -followed by sixteen bytes per active or retired owner. Counts and exact encoded -length are checked before allocating identity vectors; the maximum authority -contains 4352 identities in 69,684 bytes. Public `UploadRecord` JSON and its -logical versions remain unchanged. Part receipts retain their JSON encoding. -Readers accept legacy JSON versions 2 through 5 with the same validation and -identity representation; writers emit the binary envelope. Conditions compare -the actual stored bytes, so a paused JSON mutation cannot overwrite a converted -resource. JSON-only Access servers and direct native clients reject this -envelope; their storage readers and writers require a coordinated upgrade. -Completion verifies every part and the whole declared MD5 with bounded buffers. -For one part whose declared length and MD5 match the whole upload, verification -hashes the stored bytes once and reuses that exact-range result. Multiple part -digests cannot be concatenated into the whole-upload MD5. - -Expiry scans bounded pages of upload authorities directly. Each caller retains -its scan cursor locally; restart repeats the scan and loses no cleanup obligation. -Discovery reads at most twenty keys / 1 MiB and seeks past each upload's child -subtree. A bounded local queue schedules uploads in turn; each cleanup window -scans at most 32 actual part records / 1 MiB and attempts at most 64 range -notifications. Discovery and cleanup each have a two-second deadline. Sparse -uploads do not perform a point GET for every declared but absent part. -Each cleanup authority point read has a fixed 352 KiB budget. Canonical JSON needs -at most 66 bytes per owner including a separator, plus 1 KiB for other fields; -4352 owner identities and a metadata key shorter than 64 KiB fit this bound. -Noncanonical or otherwise oversized records still return a visible limit error. -Accepted range retirement permits deletion of that aborted upload's part record; -the upload cannot accept a new part or owner. Deferred or uncertain notification -retains the exact range. A full queue can discard a hint only after serving it, -leaving a discovery slot; removed part records ensure useful progress survives -hint eviction and restart. There is no persistent scan or queue record. -Independent upload creation writes only its own authority. No shared registry tail, -registration generation, registry fence or durable scan cursor participates in -creation, part upload or owner release. Corrupt/failed entries are reported and -retried next cycle without starving subsequent records. Legacy registry pages are -ignored. Claimed payloads and terminal owner revocation remain authoritative; -removing an owner cannot release another owner's bytes. Immutable Accepted part -records are the sole part-listing facts. Listing and completion scan at most -128 records / 1 MiB per page, with sixteen concurrent disjoint prefixes of 256 -part IDs. A part write updates only its own record; it does not rewrite a shared -index page. Legacy index records are ignored. - -Prepare validates the complete current sample membership, schema, parent and -operation token, then persists an invisible candidate and claims completed -uploads. New caller tokens directly select the immutable Snapshot ID; they do -not create a separate operation-to-Snapshot mapping. Existing legacy mappings -remain readable and select their original Snapshot. Retirement retains a small -negative fence for legacy delayed writers and the selected Snapshot resource. -Omitted samples disappear; surviving samples inherit omitted optional -fields through explicit ancestry. Tombstones remove inherited values. Required -field names cannot be removed from a child schema. - -Publish reuses the prepared Snapshot ID, marks its complete record Published, -and CAS-advances the head. A Published Snapshot record can precede head -advancement; a successful publish response establishes the completed operation. -Lost responses are resolved by inspecting status and retrying identical content -with the same operation token. Historical retries cannot rewind a newer head. -Divergent retries conflict. The first Snapshot resource write includes a fixed -internal `gc_after` deadline twenty-four hours after creation; content bindings -inherit it. Retries and publication-state changes preserve that deadline. The -background worker scans Snapshot resources directly, including writes interrupted -before the content binding exists. Each page has at most twenty rows, nine MiB -and a two-second deadline; its position is discardable and restarts from the -beginning. Head, retention and ancestry checks protect referenced resources -without writes. Eligible resources use the ordinary authoritative reclamation -fence and configured deletion grace. Eligibility alone does not revoke reads. -An old resource inherits its existing content-binding deadline. If neither -record has one, the worker assigns a fixed deadline twenty-four hours after its -first observation. Legacy opaque resources without a supported Manifest remain -excluded from automatic retirement because their payload references are unknown. -No per-resource discovery task is created. -Upload expiry does not reclaim claimed candidate payloads. - -### 10.2 Read flow - -`DatasetReads` resolves latest/stable and optional currentSplit into immutable -versions and returns a self-contained continuation. The server stores no read -record, selected-ID pages, progress, TTL or cancellation state. Every next request -supplies its continuation and passes ordinary authentication and deletion admission. -The token binds Dataset, Snapshot, Split, selection, projection, worker partition, -shuffle seed/group size and position. It is a cursor, not an authorization grant. - -Selection uses bounded inspection windows without fetching a full Manifest. -Within one next request, selection windows share admission, immutable summaries, -schema and compiled predicates. Explicit IDs are validated once. This prepared -query is local to the request and binds its authority, Dataset and Snapshot; -independent page requests still perform their own admission. Per-window storage -budgets and the overall group deadline remain enforced. -Split filtering precedes deterministic shuffle and worker partitioning. The cursor -carries a group start and offset, not the Dataset's selected IDs. A partly consumed -shuffle group is reconstructed from immutable metadata; this bounds memory but -adds metadata reads when groups are much larger than delivered batches. Groups -are capped at 100000 selected IDs and 8 MiB of IDs; selection work has a two-second -deadline and at most 5000 scan windows of up to 20 candidates. Serialized cursor -content is capped at 256 KiB before base64 encoding. Invalid or excessive work -returns an explicit error. Stream metadata and headers remain capped at 8 MiB. - -Repeating a continuation replays its batch, including from another Access Server -or native client. After successful application processing, the caller saves the -returned next continuation in its own checkpoint. EOF and buffer release never -save progress. There is no remote ack, resume or close; cancelling a request stops -its local producer. The SDK iterable's acknowledgement is a local operation. - -The public binary response streams a bounded JSON header, integrity-checked -payload frames, field terminators and a batch terminator. Success also requires -HTTP EOF with no trailing bytes. Each active transfer owns local admission -credits through delivery/drop. Snapshot reads perform no persistent activity -updates or heartbeat writes. Cancellation stops the producer and returns credits; -retained SDK-owned buffers keep their byte credits until released. -For an uploaded field, HTTP and native readers validate one immutable upload -description per request, then fetch each part record once. Part indices, lengths -and payload digests remain checked; EOF does not reload metadata. The description -is request-owned and registers no remote reader or retention. New requests still -perform admission, and reclaimed parts fail instead of producing successful EOF. -Native delivery reuses a single part's successful-EOF length and MD5 proof only -when the part is the complete field range and matches the field's expected digest. -Partial data and failed streams provide no proof. Multiple parts still require -the independent whole-field digest, and inline fields remain checked directly. -The internal legacy scan adapter also returns a caller-owned cursor without -loading or persisting progress. Requests have no shared inactivity TTL. -Legacy scan planning retains one flat ordered ID vector, extracts only the -current window by moving its IDs, and releases the remaining vector before -payload delivery. Batch adapters move IDs into windows. Selection and budget -accounting share one pass; sample shuffle computes its plan identity once and -orders by the existing MD5 seed/epoch/plan/ID key with binary-ID tie breaking. -The legacy path still scans its full selection; flat planning does not provide -the incremental metadata behavior of inspection-backed reads. - -Binding reuses its committed-Snapshot admission. Independent selection, preview -and batch calls retain their own admission. Combined continuation selection and -delivery share one request-local Snapshot/schema/membership proof under separate -work budgets; wrappers do not repeat the same deletion lookup. Empty and legacy -batch entry points retain independent admission. No proof is cached across requests. - -Legacy Snapshots without a stored visibility proof are checked against head and -anchor ancestry using bounded, cancellable reads. The traversal validates the -Snapshot identity and publication state, rejects cycles and never fills a -persistent proof cache. Existing proofs remain readable; modern unpublished -candidates cannot use legacy ancestry as evidence of publication. - -### 10.3 Retention and reclaim flow - -Reclaim atomically fences new reads, invalidates stale head publishers with a -head nonce, and rechecks latest, stable, explicit retention and all live committed -child branches. Temporary child admission fences do not prove retirement. A -reclaimer owns a CAS token; only that owner can reopen admission or enter the -irreversible retiring phase. Concurrent stale reclaimers cannot undo retirement. -After the terminal deletion mark, cleanup waits until the persisted grace -deadline. The Access Server scans bounded pages of deletion records and resumes -cleanup independently of reader processes. An Aborted Snapshot tombstone preserves -the original Manifest while payload cleanup is pending and prevents publication retry resurrection. -REST and bincode deletion requests only establish this mark and observe durable -completion; repeated requests do not drive metadata deletion or accept a client -cleanup cursor. Background workers release upload ownership for that operation -only; other Snapshot owners preserve shared bytes. Owner, legacy Split-draft, -index-fence and metadata scan positions are local, bound to one Dataset/Snapshot. -Lost positions repeat idempotent retirement without a stored progress record. - -Background discovery reads at most twenty deletion marks / 512 KiB and keeps -at most 32 local cleanup hints. Each window has a two-second deadline. Pending -work runs at a 50 ms interval; failures back off to 500 ms and rotate the queue, -so one slow Snapshot cannot starve another. An empty backlog waits fifteen -seconds. Restart rescans durable marks. Metadata deletion starts only after the -owner cursor completes; clients cannot skip that boundary. Accepted range -notifications end Access responsibility without waiting for physical capacity -recovery. Shared range retirement does not count as a reclaimed whole chunk. - -Local queues reserve a full discovery page against failed work. Each hint tracks -whether its previous window failed. Successful incomplete windows keep their -positions. Failed hints may occupy at most `32 - discovery_page_items` slots; -overflow discards the oldest failed hint, leaving every durable resource fact -untouched. This opens discovery capacity even when 32 operations never respond. -An outer deadline marks a hint failed only while its window is active, not while -discovery or legacy-checkpoint sweeping runs. Dropped positions repeat idempotent -retirement. After discarding a hint, a completed discovery pass restarts when its -remaining hints have all failed, so no discarded resource is permanently skipped. Empty -passes retain the normal fifteen-second idle interval. No scheduler position or -failure bit is persisted. - -Metadata cleanup deletes at most twenty keys per local window. It removes -obsolete Manifest storage and inspection projections while retaining small -permanent CAS fences for all possible immutable index and Split revision/member -writes. A failed fence write does not advance the local row position. Commit -receipts, head fences and terminal retirement evidence remain. A retry helps the -final status marker if cleanup completed before the response or status write -failed. Old upload, metadata, Split-discovery and index checkpoint keys are swept -in pages of twenty only after authoritative Snapshot completion. The sweep reads -no checkpoint payload and never deletes the completion or immutable-write fences; -its scan position is also local. Empty legacy scans wait fifteen seconds. - -Legacy Composition cleanup scans existing draft facts into a separate local queue -of at most 32 operations, discovering eight drafts / 9 MiB at a time. Each -two-second window advances candidate fences, upload owner retirement, scratch -fences or registered edit fences. The queue rotates failures and slow operations; -restart rediscovers every obligation from the retained draft, work-page and edit -registration facts. No candidate, scratch, edit or upload-retirement checkpoint is -written. Complete result owners remain independent of draft abandonment. Old -checkpoint keys are deleted after the local pass; immutable discovery facts and -negative fences remain. The retired HTTP Composition cleanup action returns 501 -before storage, and no client supplies or confirms background cleanup progress. - -Composition evaluation work pages record their possible row writes before applying -those writes. Obsolete-window cleanup seals rows before sealing their work page, -then seals the first absent work page to exclude delayed next-window creation. -Split evaluation similarly registers immutable page keys/digests before writing -members. Split cleanup processes at most twenty page references per local window. A -separate background queue discovers eight historical drafts / 9 MiB at a time and -keeps at most 32 hints. Snapshot and Composition cleanup embed the same local -Split cursor; no Split checkpoint or work-format migration is written. Restart -repeats idempotent fences. Complete final roots preserve their member pages; an -absent root competes with cleanup on that same key before member retirement. -Readable Snapshot owners wait for the persisted deletion grace. A permanently -sealed incomplete root can retire without creating or waiting for a reader -deadline. Grace waiting and initial work still protected by a live owner leave -the current queue pass and are rescanned after fifteen seconds. The retired -HTTP Split cleanup action returns 501 before storage. Work-intent facts and -negative fences remain; old drafts without that format are retired with their -Snapshot metadata instead of receiving a process-format upgrade. Terminal Snapshot cleanup walks Dataset draft -roots in resumable single-root windows, including drafts without a published -revision summary. Under that terminal fence it can upgrade an older root's work -marker before retiring its stored pages. Split revision summaries remain valid -small records and are filtered by owner/commit visibility. - -The composed reclaim operation advances upload owners, Split drafts, index fences -and ordinary metadata cleanup with a ten-second deadline. Each component retains -its own restart cursor; completion does not claim physical upload deletion. - -Obsolete composition candidates are discovered by deterministic operation IDs, -even when preparation failed before storing a pointer in the draft. Missing -operation/Snapshot reservations become permanent fences. Existing uncommitted -candidates enter terminal retirement before cleanup enumerates every possible -Manifest/index key from the retained input. This seals writes not yet sent by a -failed preparer. Small valid retired inventory/child summaries and write fences -survive general cleanup. Candidate recovery caps its original record at 9 MiB -and metadata window at 9 MiB with a two-second operation deadline. - -Managed composition uploads register staging and candidate owners durably before -claiming payloads. Publication requires complete candidate ownership. Terminal -cleanup retires the staging owner and obsolete candidate owners, preserving -published candidates and independent owners of shared uploads. Snapshot upload -reclaim walks at most twenty retained Manifest field positions per step after -terminal admission fencing. It revokes only that Snapshot publication's owner; -physical cleanup stays discoverable from durable upload authorities. The retained Manifest -allows owner cleanup even after inspection indexes have been retired. - -Composition version 3 registers each edit ID in the root before writing its -payload page. Up to 4096 registered edits, including failed attempts, are allowed. -Terminal cleanup seals these pages in bounded windows and keeps immutable edit -receipts. Older composition writers reject version 3 rather than ignoring the -registration protocol. Existing records remain readable; editing upgrades the -root while preserving its journal and payload ownership. - -`retire_owner` permanently revokes one publication's upload claim with the same -session CAS that removes its ownership. A delayed claim for that operation then -fails; other owners remain valid. Resources retain at most 4096 retired owner IDs. -Below that limit, retirement records a negative fence even for an absent owner. -At the limit, new claims report capacity; retirement upgrades the resource to -version 5 before removing an unrecorded owner or acknowledging its absence. -The full list remains unchanged, so retired owners cannot rejoin and the final -active owner can still be removed before abort or expiry. Ordinary `release` -remains reversible below saturation; a released owner cannot rejoin a full resource. - -Upload expiry uses local continuation/queue hints and existing upload/part resource -facts. Shared registry registration and removal are absent. Snapshot deletion -sweeps also yield after a two-second page deadline, keeping durable deletion marks -and advancing past a stalled item. The listener schedules snapshot pages every -fifteen seconds and upload catch-up windows fifty milliseconds apart; failed -upload windows back off five hundred milliseconds while other work remains. -Idle discovery runs every fifteen seconds. Limits bound work per window rather -than promise a maximum reclamation latency or measured foreground p99. - -### 10.4 Transport and compatibility - -The public versioned HTTP surface uses JSON control records, binary payload -streams and bearer authentication. Legacy bincode routes remain an internal -compatibility interface. HTTP/1.1 keep-alive has separate bounded idle connection -and executing request admission, header deadlines and graceful shutdown. -Default executing admission is eight requests. Early authentication/capacity -rejection drains a bounded body without materializing it so valid part writers -receive the structured error. Oversized or stalled rejected bodies close their -connection after a bounded drain; they cannot consume unbounded resources. -Uploads and downloads stream with bounded concurrency and backpressure. The SDK -retries only documented idempotent requests, preserving replayable bytes and -operation IDs under one overall deadline. Authentication, integrity and semantic -conflicts are not blanket retried. File downloads use a private temporary file -and rename only after successful integrity verification. - -Head envelopes decode legacy versions and use version 3 for commit-attempt -markers. Access envelopes decode versions 1 through 4 and write version 4 with -reclaim ownership, terminal deletion, grace deadline, cleanup target and completion. -Legacy reader activity is ignored; all authority writers must use this contract. -Upload sessions decode versions 2 through 4 and write version 4 without registry -fields. Permanent retired-owner identities still reject delayed claims. This -metadata upgrade is one-way: do not downgrade an upgraded authority or mix old -and new authority writers against the same Dataset store. Immutable Dataset/Snapshot/Manifest encodings -remain compatible. Persisted read-session records and legacy cursor keys are -unused. Session read routes are absent; clients use `POST reads/bind` and -`POST reads/next`. Upgraded clients and servers must deploy together. Control v2 -advertises `stateless-reads` so an SDK cannot treat the retired session capability -as support for this contract. Physical purging of unused keys is an explicit -maintenance operation, not a read-path side effect. - -Hot-path synchronization uses atomics, immutable owned views, CAS and bounded -notification channels. No application-wide request lock serializes independent -operations. Producer admission includes queued native buffers and metadata -parsing, while caller-retained completed results belong to the application. -Functional resource checks verify these ownership rules. Benchmark drivers, -measurements and performance thresholds are outside this implementation. - -## 11. Risks and open questions - -Future namespaces add metadata fan-out and authorization scope. The first -implementation intentionally has no namespace operations; it only carries the -configured default/root namespace in every identity. - -Manifest partitioning, predicate indexes, cache invalidation, and direct GPU -transport require workload measurements. They are independent extensions and do -not alter the Dataset identity or snapshot contract. - -## 12. Empty Dataset retirement - -- Dataset root records carry a deletion receipt and bounded creation reservations. - Snapshot/upload creation reserves its child key through the root CAS before the - first child CAS. Deletion settles materialized reservations, checks all live - content, then compares the same root version. Concurrent new creation therefore - invalidates the emptiness decision. No per-Dataset lock is required. -- At most 32 outstanding creation reservations are admitted. An abandoned reservation - expires after twenty minutes, independently of deletion grace. Before removing an - expired reservation, the worker CAS-installs a permanent child tombstone if the - child is still absent. A late creator cannot revive an operation after name reuse. -- Marked roots are hidden from discovery and normal opens. The cleanup worker uses - bounded pages and an expendable cursor; roots own durable deadlines and progress. - Because acceptance requires no live content, due cleanup only retires the namespace - reservation. It preserves the minimal root receipt and existing child fences. -- `dataset.reclaim_grace_seconds` is shared by Dataset, Snapshot and Split revision - deletion, defaults to 1200, and is positive in Access Server configuration. The - worker polls Dataset/Snapshot marks at min(grace, 15) seconds. Configuration changes - affect future marks; persisted deadlines never move. Restarted workers can resume - from the start, and duplicate workers use CAS rather than unconditional deletion. -- HTTP v2 advertises `dataset-delete`. DELETE of the Dataset requires `{id}`; - `GET .../deletion/{id}` reconciles its receipt. Nonempty returns 409 - `dataset_not_empty`; an identity mismatch returns 409. Native `DatasetClient::delete` - uses the same authority and expected-ID contract. diff --git a/doc/design/access-server/dataset/design-dataset-control.md b/doc/design/access-server/dataset/design-dataset-control.md deleted file mode 100644 index 7bd7bd809..000000000 --- a/doc/design/access-server/dataset/design-dataset-control.md +++ /dev/null @@ -1,438 +0,0 @@ - - - -# CROWDB - Design: Dataset Control Lifecycle - -Depends on: [Dataset architecture](design-crowdb-access-dataset.md). -Satisfies: [Dataset specification](spec-crowdb-dataset.md). -Wire contract: [control API v2](openapi-crowdb-dataset-v2.yaml). - -This document describes Snapshot composition, whole-set Split revisions and their -bounded control operations. Payload delivery retains the existing binary framing. - -## Table of contents - -1. [Identity and authority](#1-identity-and-authority) -2. [Composition and inheritance](#2-composition-and-inheritance) -3. [Split evaluation and publication](#3-split-evaluation-and-publication) -4. [Bounded observation and reads](#4-bounded-observation-and-reads) -5. [Ownership and cleanup](#5-ownership-and-cleanup) -6. [Compatibility and invariants](#6-compatibility-and-invariants) - -## 1. Identity and authority - -Control v2 advertises `dataset-identity-fence`. A caller may bind a named route to -one active Dataset ID with `X-Crowdb-Dataset-Id`, a single canonical nonzero -32-character lowercase hex value. The request-owned authority carries that -expected ID through Dataset reads and root resource reservation CAS; checking -only the initial HTTP observation would leave a deletion/name-reuse race. A -replacement returns 409 `dataset_replaced`, a marked/missing Dataset returns -404, and malformed/duplicate headers return 400. Omission preserves existing -clients. This active-instance guard is omitted when retrying deletion receipts -after a successful mark; those operations already bind their immutable ID. - -A Dataset has opaque latest/stable Snapshot references. A Snapshot may declare any -complete available same-Dataset parent or no parent. Its parent is independent of the head -generation used for publication CAS. Multiple roots and off-latest branches remain -discoverable. A preparation binding or a Published flag alone is not a complete result. -The final Snapshot resource root is installed after Manifest/index rows, the initial -Split revision and necessary payload ownership are ready. The current root uses -version 2 of the existing candidate binding; new staging uses version 3 and is not -readable. Old version 1 bindings use existing publication evidence, or read-only -validation of their Published content identity/digest and complete initial Split. -They never acquire new publication proofs during reads or head replacement. Complete results are -readable, enumerable, retainable and usable as parents/stable targets without winning -latest. ID reads consult this root rather than publication receipts or mutable heads. - -Publication selects a complete candidate with an exact expected head generation. -It writes only the fenced head, without attempt/response records, success proofs -or Composition committing/published progress. Compatibility attempt IDs are echoed -in the immediate response and do not create request history. Retrying the same -current latest confirms its current effect; after another Snapshot replaces it, -the stale condition returns the observed latest/stable/generation without rewinding -the head or querying again. Unknown storage outcomes propagate unchanged. The -caller decides how to use an observed base; the server never rebases content. -Existing receipts and attempt-bearing heads are decoded read-only. Before replacing -an old head without complete resource roots, the existing bounded legacy ancestry -anchor preserves those resources; new version 4 heads need no receipt helping or -anchor lookup. Stable-only legacy roots are included in that compatibility index. - -The v1 JSON adapter prepares content once before acquiring payload owners. It then -completes that resource and selects it using the existing latest-parent condition; -it does not rerun Manifest/index preparation. Authority validation replaces the -old operation/head preflight. Responses use the known immutable metadata plus one -latest observation, without rereading operation/Snapshot records or copying the -Manifest for a second preparation. - -`PUT .../datasets/{dataset}/head` selects an existing complete Snapshot from -`snapshot` and explicit nullable `expected_head`, under `snapshot-head-update`. -It reads no draft or attempt and returns the same latest/stable/generation shape -as GET head. Stable is preserved; failed conditions include `observed_head` in that -shape. GET head reads Dataset identity and head once each. Rust exposes -`ControlClient::update_snapshot_head` and `Head::from_conflict`; Python exposes -`update_snapshot_head` and `ConflictError.observed_head` for synchronous and async -clients. Missing capability never invokes a composition or v1 publication fallback. - -Each Snapshot owns immutable Split-set revisions and an independently fenced -currentSplit pointer. Names are scoped to Dataset/Snapshot/revision. Publishing a -new revision replaces the complete set, including rename/removal; old revisions -remain readable until explicit deletion or owner retirement closes admission. -An empty set is a revision, distinct from an absent pointer. - -## 2. Composition and inheritance - -The native `prepare_composition_resource` accepts a caller-owned Snapshot ID and -complete parent/source/schema/changes/Split inputs. It computes a bounded final -Manifest directly, then writes its immutable partitions and inspection index, -initial Split membership and final payload owners. It completes the existing -resource root last; latest/stable are updated separately. There is no operation -mapping, draft, edit receipt, working row, window intent, staging owner or durable -computation cursor. The native input and final Manifest are bounded to 8 MiB and -the request to 30 seconds, with twenty-Sample source windows. Partial field patches -keep other effective parent fields; source filtering does not prevent an explicit -upsert from reintroducing a parent Sample. Each patch ID appears once. Frozen -Split sources and inherited definitions name exact revisions; retries never resolve -currentSplit again. Patch application moves existing rows and tracks encoded size -incrementally rather than serializing the growing Manifest after every change. - -A retry recomputes incomplete work and reuses matching complete content. The -existing content binding follows all final Manifest/index rows, so retrying a -failed owner claim does not rewrite those rows; it still checks unfinished initial -membership and owners before completion. The Snapshot ID fixes its Manifest and initial Split binding; different final content -conflicts, including removing an existing initial Split. Identical complete retries -write no state. Unknown completion replies return directly without a confirmation -read. Resource preparation checks Dataset identity without loading the head. -Managed upload metadata is validated once per unique upload before output writes, -and the final Snapshot owner is acquired before completion. The final Manifest -finds every owner for cleanup; there is no parallel ownership-progress list. Opaque -legacy locators inherited from a parent retain their existing storage contract; -new chunk patches must be valid managed upload locators. - -Resource prefixes and the immutable Snapshot record expose interrupted output to -owner retirement. Existing member pages are sealed after the grace period, and -recomputation cannot complete a retired Snapshot. The first Snapshot write stores -its fixed internal `gc_after`; a bounded full scan discovers abandoned resources -even before their content binding exists. Terminal deletion seals that binding, -including a missing key, so delayed creation cannot complete the resource. -`PUT snapshots/{snapshot}` exposes this operation with `composition-preparation`, -when an upload service is configured. Rust `PrepareSnapshot` and sync/async Python -`prepare_snapshot` preserve the same ID/input and never query workflow status or -downgrade. Explicit nullable parent/schema and strict DTOs reject process IDs. -Upload values carry id/length/md5, converted to locators without a metadata query; -the authority checks each unique upload before claiming it. Descriptor mismatches -return conflict. HTTP body/input/Manifest are bounded to 8 MiB. Declared oversized -bodies use the existing bounded discard before response, avoiding an unread-body -reset that can truncate the 413 JSON. Composition and standalone Split workflow -writes return 501 before storage access, and the workflow SDK methods are removed. -Only legacy HTTP observation, abandonment and cleanup remain for existing records. - -Legacy `CompositionRecord` stores old input revisions, source IDs, edits, work -progress, candidate and ownership lists. These formats remain cleanup inputs; -current routes neither append edits nor resume them. Status only reads the stored -draft and never helps receipts, queries commit ledgers or advances publication. -Legacy Committing is presented as Prepared without changing stored bytes. A caller -may select an already complete resource with a direct head update; the status does -not prove that a historical request selected the head. - -The default source is all parent Samples. Selection filters and a frozen named -Split can restrict the source, with their intersection when both are present. -A source S1/r2/train produces a child with parent S1. It copies exact frozen -membership/provenance, not the future currentSplit value. Inherited Split members -intersect surviving child IDs; newly added IDs remain unassigned unless explicit -assignments or reevaluated definitions include them. - -Sample upsert replaces the supplied field bindings while preserving other effective -fields; remove excludes its ID. Effective inherited values are resolved server-side -and materialized as bindings in the complete child Manifest. Inspection reports the -child as the binding source after this copy; chunk payloads remain shared. Sparse -Manifests created through direct publication can still resolve omitted optional -fields from an ancestor and report that ancestor as the source. An unchanged -required field must remain represented in the final complete membership. Tombstone, missing and empty inline -value remain distinct. The current Schema contains only version/name/requiredness. - -Computation reads windows of at most twenty Samples into a bounded request-local -Manifest builder. It writes final content directly; no redo window or working row -precedes it. Final resource keys reject conflicting immutable content. The complete -root makes data and initial Splits available together, and the independent data -head transition selects that resource as latest. - -## 3. Split evaluation and publication - -Standalone Split preparation accepts an owner Snapshot, caller-owned revision ID, -policy and complete definitions in one resource PUT. Preparation checks source -availability; deletion may interrupt unfinished work. It requires valid overlap/ -coverage policy. Defaults are disjoint and allow-unassigned. Frozen inherited -membership is explicit; reevaluation is a deliberate definition change. The caller -captures the intended currentSplit revision/generation before computation and keeps -that base separate from the immutable result. - -Predicates support all, explicit IDs, prefix, range, keyword, inline field match and -boolean composition over the selected universe. Effective optional inheritance and -tombstones are resolved through metadata. Payload contents are not searched. - -Weighted partition uses evaluator version 1: SHA-256 of `crowdb-split-v1\0`, seed -as u64 big endian, Sample-ID length as u32 big endian, then exact ID bytes. Interpret -the first eight digest bytes as u64 big endian. Its bucket is -`(u128(hash) * sum(weights)) >> 64`; ordered cumulative weights select the name. -Weight sums are checked u32 values. This is deterministic approximate partitioning, -not a promise of exact cardinality. Overlap/coverage validation remains independent. - -Membership pages are immutable and checksum-bound. Preparation persists the whole -revision before currentSplit CAS. A conflict preserves the candidate and caller -inputs; retry never silently rebases definitions or overwrites another revision. -The native `prepare_split_revision` operation accepts caller-owned revision ID, -policy and complete definitions. It recomputes incomplete work from those inputs, -writing only final member pages, the resource enumeration index and complete -revision root. The index precedes pages; the root is installed last. No draft, -edit receipt, window intent or phase cursor is written. Input and final bindings -are capped at 1 MiB; computation uses twenty-Sample windows within a 30-second -request deadline. It retains one window and the bounded bindings, encodes each -member page once, and tracks binding size incrementally. An identical complete -resource is reusable without recomputation; different input at that ID conflicts. -Unknown writes propagate without confirmation reads. Frozen inherited sources use -the supplied revision, regardless of later head changes. Snapshot retirement -finds both complete and partial output through resource prefixes without drafts; -complete revisions have no independent automatic expiry. Explicit revision deletion -and its bounded resweeps are described under ownership and cleanup. -The complete `SplitRevisionRecord` is the final result: an explicit revision ID -can be read, listed or used as an inherited source without winning currentSplit. -Read admission still checks the owning Snapshot. Reads load that immutable record -directly without Split publication proofs, currentSplit lookups or helper writes. -Standalone Split publication writes only currentSplit. It creates no attempt, -response receipt, per-revision success proof or draft committing/published state. -A retry returns the current generation when it still names the same revision; -after a later revision replaces it, the stale request returns a precondition -conflict without replaying historical success or rewinding the pointer. A definite -CAS conflict uses its returned head; a transport error remains unknown. Legacy -receipts and attempt-bearing heads remain readable without backfilling receipts. -`PUT snapshots/{snapshot}/split-revisions/{revision}` exposes preparation with the -`split-preparation` capability. Rust `PrepareSplitRevision` and synchronous/async -Python `prepare_split_revision` retain the exact resource ID and complete input on -retry; an error carries the revision identity. Missing capability never invokes a -draft or v1 mutation. Standalone begin/edit/evaluate/prepare/publish workflow routes -return 501 before storage access; SDK workflow methods are removed. Only legacy -draft observation, abandonment and cleanup remain for existing records. Initial Split computation in `prepare_snapshot` shares the same pure result -preparation against its fixed incomplete owner; it writes no draft or progress. -Legacy internal draft drivers remain for migration/cleanup and are not called by -the resource preparation routes. -`PUT snapshots/{snapshot}/split-head` accepts the target revision and both explicit -nullable expected revision/generation. It requires the `split-head-update` -capability and has no draft or attempt identity. The failed head condition is -returned as `observed_head`, including an observed absence. It is an observation, -not a promise that another writer has not changed the head since. The caller may -review the new base and use it in a new conditional request; no rebase is implicit. -Rust exposes the observation through `SplitHead::from_conflict`; Python exposes -`ConflictError.observed_head`. Missing legacy observations do not trigger an -automatic head query or a mutation downgrade. - -## 4. Bounded observation and reads - -`inspection_split_page` and `POST snapshots/{snapshot}/split-preview` evaluate one -Split definition over a bounded committed membership window. They reuse the same -rule compiler and evaluator as Split preparation, including deterministic weighted -partitions and frozen inheritance. The cursor binds the Snapshot and full rule; -zero matching rows may still carry a scan continuation. Preview writes no resource, -progress or read session and reads no uploaded payload bytes. The `split-preview` -capability advertises this operation; the console proxy permits it in readonly mode. - -Inspection projections contain complete summaries, Schema rows, Sample IDs, field -metadata and separate value bindings. Ordinary inspection and indexed read planning -do not load a complete Manifest or chunk payload. Inventory and parent/child indexes -filter visibility through the complete resource root, with read-only compatibility -for old publication evidence, not merely candidate record state. - -Preparing an index overlaps at most sixteen immutable row writes, with at most -2 MiB of encoded keys and values queued. Encoding holds at most one additional -row under the existing per-row bound. No row is copied into another batch buffer. -All row writes must succeed before inventory, parent/child and ready summaries -are installed. Failure leaves the index unavailable; repeating the request checks -the same immutable rows. This bounds parallel work while preserving the existing -number of conditional mutations and each row's identity. - -List pages default to twenty items and cap at one hundred with a 1 MiB metadata -window. Exact matrices admit twenty IDs by twelve fields. IDs preserve up to 16 KiB -of binary data; Schema admits up to 256 fields with names up to 1024 bytes. Projection -and selection use aggregate byte/deadline/cancellation budgets. Native bounded reads -reject oversized external values before materializing their contents. - -Continuation delivery resolves selection and field locators within one request. -It reuses that request's Snapshot admission, schema and already scanned membership; -Split visibility checks reuse admission for the same Snapshot. No admission handle -is returned to callers. A separate request checks deletion again, including empty -results. Explicit batch reads remain independent admitted requests. HTTP delivery -moves inline payload owners from resolved locators into output buffers without -cloning the payload. - -Delivery resolves membership and field bindings in windows of at most sixteen -cells. Independent point reads overlap within one ancestry level; unresolved -fields proceed to the parent only after its read-only admission. Ancestor summaries -remain local to the request. Each read reserves its response cap before dispatch, -with at most sixteen requests and 2 MiB of response capacity in flight. Completed -rows consume the aggregate metadata budget, including responses waiting behind -an earlier read. Unused reservations are released before admitting a smaller final -read. Request order, missing fields, tombstones and locator integrity checks remain -independent of completion order. All windows share the request deadline and -cancellation; no reader state is persisted. - -The bounded compatibility part reader checks the declared logical length before -allocation. A single buffer covering that complete object is transferred with its -allocation owner after successful EOF. Fragmented objects allocate their final -buffer once; each source buffer is copied and released before the next storage -pull, so a one-slot reader can make progress. Short, oversized and failed terminal -reads remain errors. Whole-part digest verification still runs on the returned -bytes before acceptance. - -Old snapshots without projections require an explicit complete inspection rebuild. -One request bounds the source record at 9 MiB, its embedded Manifest at 8 MiB and -the entire rebuild at thirty seconds. Final rows use at most sixteen pending -writes and 2 MiB of encoded keys and values in flight. The builder writes complete -rows, inventory and parent-child discovery before installing the immutable summary -root. Partial rows are invisible without that root. A failed request returns its -error and a retry recomputes from the source; it saves no preparation cursor or -request receipt. An existing summary returns without writes. Cleanup and the -finalizer conditionally replace the same root, so a retired root rejects a delayed -finalizer. - -Status can decode historical preparation progress when no summary exists, but a -complete rebuild never resumes or rewrites that progress. The summary takes -precedence over historical progress, including corrupt progress. New successful -responses retain zero revision and row-count fields for wire compatibility. -Unsupported or oversized sources return errors instead of empty Snapshots. Field -preview is explicit and capped at 64 KiB after read-only admission; it does not -create a read plan or acknowledge application consumption. - -V2 read planning pins Snapshot and optional Split revision once, intersects Selection -with Split membership, then orders/shuffles and partitions workers before fetching -projected payloads. Planning advances twenty-Sample windows with fixed total ID/byte -limits. Fetch/preview/status never save consumption progress. The client saves the next -continuation after processing; acknowledgement is local. Cancel stops the current -request without a reader registry or remote close. - -## 5. Ownership and cleanup - -Incomplete-candidate cleanup and finalization conditionally replace the same -resource root. A cleanup seal prevents a delayed finalizer from exposing retired -rows. If the complete root wins, abandoning or superseding the draft preserves its -Manifest, initial Split members and payload owners; ordinary Snapshot reclamation -later retires them under deletion admission and grace rules. Completion transport -errors propagate without confirmation reads. Definite cleanup conflicts use the -returned value without reloading the root. - -The parent/child index is installed before the child's complete root. Finalization -rechecks parent availability after that index write. Parent deletion conservatively -protects any indexed child without terminal retirement, including an incomplete -child awaiting cleanup. Thus deletion either observes the child or the child's -last parent check rejects completion. Readers never register in this protocol. -Reclaim admission may remove an exact child-index entry after validating its -binding and the child's permanent retirement authority. Completed removals shrink -later bounded scans; the retirement authority remains even if a delayed writer -recreates the derived index. A failed or timed-out index deletion returns its -unknown outcome without compensation writes. A fresh request rechecks authority -and can resume this resource cleanup; ordinary inspection remains read-only. -Discovery and per-resource positions stay in memory. Restart may repeat already -sealed prefixes, so progress requires enough consecutive bounded windows to pass -those prefixes. Grace deadlines specify earliest reclamation, not a completion -time or a guarantee under arbitrary restart frequency. - -Uploads are Dataset-scoped and content-verified. Resource preparation claims only -the final Snapshot owner for each unique managed upload in its final Manifest, -including unchanged parent uploads. Completion requires all those owners. The -Manifest supplies cleanup's owner references without a second process list. -Retiring one owner preserves other owners; permanent revocation rejects delayed -claims. Legacy staging/candidate lists remain cleanup inputs for old records. - -Legacy draft abandonment and cleanup use their existing bounded cursors and seals -for old work, including keys that could still receive delayed writes. Stored edit -receipts and progress can be decoded and reclaimed, but resource preparation never -creates or resumes them. Their independent discovery and replacement remain part -of migration work; removing the last discovery path would lose owned garbage. - -Legacy draft cleanup competes with preparation on the final revision key. If the -complete root already exists, including a root returned by a failed cleanup CAS, cleanup -preserves its member pages and summary while retiring evaluation intents. If -cleanup seals an absent root first, a delayed preparation cannot publish it. -Abandoning the draft does not delete an already complete revision. Retiring the -owning Snapshot still permits revision cleanup and closes new read admission. -Sources use read-only availability checks. Legacy `source_lease_deadline` is only -a compatibility field; resource preparation has no reader lease or draft. - -Complete Split revisions normally remain owned by their Snapshot and are reclaimed -with it; there is no independent revision TTL. Explicit -`DELETE snapshots/{snapshot}/split-revisions/{revision}` takes only those two IDs -and an empty body. It rejects the current head and the Snapshot's immutable -initial revision with 409, including an initial revision that is no longer current. -A missing revision returns 404. A successful mark immediately closes new revision -reads and frozen-source admission; ordinary requests return 404. Existing bounded -read limits still apply and may return a limit error before the closed record is -decoded. Previously admitted -work may continue until cleanup, and may fail if it outlives the grace period. - -Deletion state and the fixed configured (default twenty-minute) deadline replace the complete root on -the same revision key, retaining its page bindings for cleanup. The head generation -is fenced against a publisher that observed the revision before deletion. There -is no separate deletion task, reader session or request ledger. Repeating DELETE -preserves the stored deadline; an unknown mutation returns without a confirmation -read, and the client retries the same two IDs. The bounded background worker scans -revision resources using discardable local cursors, seals membership metadata and -rescans terminal resources for delayed member writes. It does not delete Samples, -upload owners or chunk payloads. If Snapshot deletion has already closed every -owner read and its own configured (default twenty-minute) grace has elapsed, revision cleanup may use -that earlier owner fence without waiting for the revision's later deadline. - -DELETE returns 200 with `snapshot`, `revision`, `complete` and `reclaim_after`. -Pending deletion reports `complete=false` and the fixed absolute Unix-seconds -deadline as a canonical decimal string. A terminal resource reports -`complete=true` and an explicit null deadline; it retains no historical response -from which to reconstruct the old deadline. The `split-revision-delete` capability -is independent of revision reads and preparation. Rust and synchronous/async Python -expose `delete_split_revision(snapshot, revision)`; missing capability never invokes -legacy draft abandonment. The native adapter calls the same authority under local -admission with no additional metadata GET and no automatic unknown-outcome retry. - -Snapshot reclaim checks latest/stable, retained Snapshots, live children, candidate -ownership under a terminal owner fence. New deletion marks have a durable configured (default twenty-minute) grace deadline; -existing marks keep their persisted absolute deadline. The grace period -separates deletion admission from cleanup. After that deadline it retires upload owners, -Snapshot-scoped Split drafts, possible index writes and ordinary metadata in phases. -Completion reports logical retirement; physical upload deletion remains deferred -to the bounded upload-authority scan. Complete Split metadata also supports the -explicit revision deletion described above. - -`GET snapshots/{snapshot}/reclaim` is bounded, read-only observation. It returns -publication operation identity where available, latest/stable/retained flags, -`reclaim_after` and the compatibility `active_read_lease=false` field, and not_started/pending/completed retirement state. Parent reachability is -explicitly not evaluated in this small read; observed flags are never permission -to reclaim. Rust/Python `control().lifecycle(snapshot)` share this contract. -Metadata HTTP reads remain capped at five seconds, while bounded mutation steps -use their configured mutation deadline (normally 60 seconds). - -## 6. Compatibility and invariants - -Control v2 exposes strict DTOs and capability discovery. Required null preconditions -cannot be omitted; u64s use decimal strings; binary IDs use canonical base64. Clients -check capabilities before mutation and never downgrade a Split-aware operation to v1. -Existing v1 routes, persisted read interpretation and binary framing remain available. -Rust and Python control clients share transport/admission/retry/cancellation semantics. -SDK clones share a fixed atomic bitset of capabilities previously advertised by -the versioned endpoint. Positive hints remove discovery GETs from warm control -calls; absent capabilities are rediscovered, never downgraded. Explicit -`capabilities()` queries replace hints after a valid response and clear them for -an unsupported API version. Unknown transport outcomes do not imply capability -revocation. Concurrent cold calls may duplicate discovery under the existing -HTTP budget; there is no discovery lock or server session. Every actual operation -still passes server authentication, capacity, deletion admission and mutation -preconditions. Server redeployment can invalidate a hint and make the operation -fail; explicit discovery refreshes it. A preflight GET could not eliminate that -deployment window either. - -- **I1 — Immutable data:** committed Snapshot content never changes. -- **I2 — Independent revision:** Split publication cannot move Dataset latest. -- **I3 — Frozen source:** source revision identity survives later pointer changes. -- **I4 — Complete publication:** data, initial membership and upload ownership are - ready before the visible head transition. -- **I5 — Exact retry:** a successful attempt cannot create another Snapshot or - rewind a later head. -- **I6 — Bounded work:** page, byte and time budgets constrain underlying storage - work as well as response length. -- **I7 — Terminal retirement:** delayed writers cannot restore retired work or - acquire a revoked owner after cleanup. diff --git a/doc/design/access-server/dataset/openapi-crowdb-dataset-v1.yaml b/doc/design/access-server/dataset/openapi-crowdb-dataset-v1.yaml deleted file mode 100644 index 31359f8b6..000000000 --- a/doc/design/access-server/dataset/openapi-crowdb-dataset-v1.yaml +++ /dev/null @@ -1,2535 +0,0 @@ -# Copyright 2026-present Gian -# Licensed under the Apache License, Version 2.0. -{ - "openapi": "3.1.0", - "info": { - "title": "CrowDB Dataset HTTP API", - "version": "1.0.0", - "description": "Public Dataset REST v1 wire contract. Unknown optional JSON properties are ignored. Missing required properties and unknown variants are rejected. This document does not expose persisted bincode records." - }, - "security": [ - { - "Bearer": [] - } - ], - "paths": { - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}": { - "put": { - "operationId": "CreateDataset", - "description": "Duplicate identity returns conflict. A lost create reply is recovered by OpenDataset; never retry creation with a new name.", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Dataset" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - } - }, - "get": { - "operationId": "OpenDataset", - "description": "", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Dataset" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/snapshots/prepare": { - "post": { - "operationId": "PrepareSnapshot", - "description": "Same canonical content/token recovers the same candidate. Complete preparation persists the manifest but never advances latest. Payload uploads are fenced against cleanup before binding.", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Snapshot" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - }, - "requestBody": { - "required": true, - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Publication" - } - } - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/snapshots/publish": { - "post": { - "operationId": "PublishSnapshot", - "description": "Returns only after complete publication and head advancement. Lost replies are recovered with the same token and content. A published operation with is_latest=false remains addressable; status does not claim head success.", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Snapshot" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - }, - "requestBody": { - "required": true, - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Publication" - } - } - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/operations/{operation}": { - "get": { - "operationId": "GetOperation", - "description": "", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "operation", - "required": true, - "schema": { - "$ref": "#/components/schemas/Id" - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Snapshot" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/snapshots": { - "get": { - "operationId": "ListSnapshots", - "description": "Latest ancestry, newest first. next is the next immutable ancestor. An omitted starting cursor resolves latest once.", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "query", - "name": "after", - "schema": { - "$ref": "#/components/schemas/Id" - } - }, - { - "in": "query", - "name": "limit", - "schema": { - "type": "integer", - "minimum": 1, - "maximum": 1000, - "default": 100 - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/SnapshotPage" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/snapshots/{snapshot}": { - "get": { - "operationId": "GetSnapshot", - "description": "", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "snapshot", - "required": true, - "schema": { - "$ref": "#/components/schemas/Id" - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Snapshot" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/snapshots/{snapshot}/manifest": { - "get": { - "operationId": "GetManifest", - "description": "Bounded JSON manifest; response over metadata limit is rejected, never truncated.", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "snapshot", - "required": true, - "schema": { - "$ref": "#/components/schemas/Id" - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Manifest" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/names/stable": { - "put": { - "operationId": "SetStable", - "description": "", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Dataset" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - }, - "requestBody": { - "required": true, - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/SnapshotRequest" - } - } - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/snapshots/{snapshot}/retain": { - "post": { - "operationId": "RetainSnapshot", - "description": "", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "snapshot", - "required": true, - "schema": { - "$ref": "#/components/schemas/Id" - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Snapshot" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/snapshots/{snapshot}/release": { - "post": { - "operationId": "ReleaseSnapshot", - "description": "", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "snapshot", - "required": true, - "schema": { - "$ref": "#/components/schemas/Id" - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Snapshot" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/snapshots/{snapshot}/reclaim": { - "post": { - "operationId": "ReclaimSnapshot", - "description": "Deletion marks the snapshot unavailable to new requests and persists a twenty-minute grace deadline before background cleanup. This empty-body action only marks deletion and observes status; repeated requests do not drive metadata cleanup or accept a client progress cursor. Latest/stable/retained/parent snapshots conflict. Active readers do not block deletion; overdue reads may fail. Physical capacity recovery remains a separate ChunkDB responsibility.", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "snapshot", - "required": true, - "schema": { - "$ref": "#/components/schemas/Id" - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Reclaim" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - } - }, - "get": { - "operationId": "GetReclaimStatus", - "description": "", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "snapshot", - "required": true, - "schema": { - "$ref": "#/components/schemas/Id" - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Reclaim" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/uploads": { - "post": { - "operationId": "CreateUpload", - "description": "Dataset-scoped token. At most 10000 parts, each at most 4 MiB. Retrying identical declarations returns status without extending expiry.", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Upload" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - }, - "requestBody": { - "required": true, - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/UploadRequest" - } - } - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/uploads/{upload}": { - "get": { - "operationId": "GetUpload", - "description": "", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "upload", - "required": true, - "schema": { - "$ref": "#/components/schemas/Id" - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Upload" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - } - }, - "delete": { - "operationId": "AbortUpload", - "description": "Idempotent abort for unclaimed data. Claimed data conflicts. Cleanup may remain pending; metadata may not claim physical completion before storage reclamation acknowledges it.", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "upload", - "required": true, - "schema": { - "$ref": "#/components/schemas/Id" - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Upload" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/uploads/{upload}/parts/{index}": { - "put": { - "operationId": "PutUploadPart", - "description": "Exactly the declared part length. Immutable index/digest: identical retry succeeds, divergent content conflicts. Allocation intents and storage acknowledgements survive crash. Returns one bounded part receipt, never a growing upload list.", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "upload", - "required": true, - "schema": { - "$ref": "#/components/schemas/Id" - } - }, - { - "in": "path", - "name": "index", - "required": true, - "schema": { - "type": "integer", - "minimum": 0, - "maximum": 9999 - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Part" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - }, - "requestBody": { - "required": true, - "content": { - "application/octet-stream": { - "schema": { - "type": "string", - "format": "binary", - "maxLength": 4194304 - } - } - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/uploads/{upload}/complete": { - "post": { - "operationId": "CompleteUpload", - "description": "Checks contiguous parts, full length and whole MD5 before publication eligibility. Completion is not snapshot publication; open/complete uploads expire after 24 hours of server time.", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "upload", - "required": true, - "schema": { - "$ref": "#/components/schemas/Id" - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Upload" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/reads/next": { - "post": { - "operationId": "NextBatch", - "description": "Replay the caller continuation. Save next only after successful processing; no remote acknowledgement or close.", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/vnd.crowdb.dataset-batch.v1": { - "schema": { - "type": "string", - "format": "binary" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - }, - "requestBody": { - "required": true, - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/ReadContinuation" - } - } - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/reads/bind": { - "post": { - "operationId": "BindRead", - "description": "Resolve immutable versions into a client-owned continuation. No read record or progress is stored.", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/ReadBinding" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - }, - "requestBody": { - "required": true, - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/ReadPlan" - } - } - } - } - } - }, - "/api/dataset/v1/namespaces/{namespace}/datasets/{dataset}/reads": { - "post": { - "operationId": "GetBatch", - "description": "GetSample uses one sample ID. One streaming response for all requested fields in sample/projection order, with field and frame integrity. Unknown samples/projections reject before payload reads. Missing and tombstone fields are explicit.", - "parameters": [ - { - "in": "path", - "name": "namespace", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - }, - { - "in": "path", - "name": "dataset", - "required": true, - "schema": { - "type": "string", - "minLength": 1, - "maxLength": 255 - } - } - ], - "responses": { - "200": { - "description": "Successful operation", - "content": { - "application/vnd.crowdb.dataset-batch.v1": { - "schema": { - "type": "string", - "format": "binary" - } - } - } - }, - "400": { - "$ref": "#/components/responses/Error" - }, - "401": { - "$ref": "#/components/responses/Error" - }, - "404": { - "$ref": "#/components/responses/Error" - }, - "409": { - "$ref": "#/components/responses/Error" - }, - "413": { - "$ref": "#/components/responses/Error" - }, - "429": { - "$ref": "#/components/responses/Error" - }, - "500": { - "$ref": "#/components/responses/Error" - }, - "501": { - "$ref": "#/components/responses/Error" - }, - "503": { - "$ref": "#/components/responses/Error" - }, - "504": { - "$ref": "#/components/responses/Error" - }, - "499": { - "$ref": "#/components/responses/Error" - } - }, - "requestBody": { - "required": true, - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/BatchRequest" - } - } - } - } - } - } - }, - "components": { - "securitySchemes": { - "Bearer": { - "type": "http", - "scheme": "bearer" - } - }, - "schemas": { - "Id": { - "type": "string", - "pattern": "^(?!0{32}$)[0-9a-f]{32}$" - }, - "Md5": { - "type": "string", - "pattern": "^[0-9a-f]{32}$" - }, - "SampleId": { - "type": "string", - "format": "byte", - "minLength": 1, - "maxLength": 1368, - "description": "Canonical padded RFC 4648 base64 of 1..1024 opaque sample-ID bytes." - }, - "Error": { - "type": "object", - "properties": { - "code": { - "type": "string", - "maxLength": 255 - }, - "message": { - "type": "string", - "maxLength": 1024 - }, - "request_id": { - "$ref": "#/components/schemas/Id" - }, - "retryable": { - "type": "boolean" - } - }, - "required": [ - "code", - "message", - "request_id", - "retryable" - ] - }, - "Dataset": { - "type": "object", - "properties": { - "id": { - "$ref": "#/components/schemas/Id" - }, - "namespace": { - "const": "dataset-ns" - }, - "name": { - "type": "string", - "maxLength": 255 - }, - "latest": { - "anyOf": [ - { - "$ref": "#/components/schemas/Id" - }, - { - "type": "null" - } - ] - }, - "stable": { - "anyOf": [ - { - "$ref": "#/components/schemas/Id" - }, - { - "type": "null" - } - ] - } - }, - "required": [ - "id", - "namespace", - "name", - "latest", - "stable" - ] - }, - "FieldDefinition": { - "type": "object", - "properties": { - "name": { - "type": "string", - "maxLength": 255 - }, - "required": { - "type": "boolean" - } - }, - "required": [ - "name", - "required" - ] - }, - "FieldValue": { - "oneOf": [ - { - "type": "object", - "properties": { - "kind": { - "const": "inline" - }, - "data": { - "type": "string", - "format": "byte", - "maxLength": 87384 - } - }, - "required": [ - "kind", - "data" - ] - }, - { - "type": "object", - "properties": { - "kind": { - "const": "payload" - }, - "upload": { - "$ref": "#/components/schemas/Id" - } - }, - "required": [ - "kind", - "upload" - ] - }, - { - "type": "object", - "properties": { - "kind": { - "const": "tombstone" - } - }, - "required": [ - "kind" - ] - } - ] - }, - "Sample": { - "type": "object", - "properties": { - "id": { - "$ref": "#/components/schemas/SampleId" - }, - "fields": { - "type": "object", - "maxProperties": 256, - "additionalProperties": { - "$ref": "#/components/schemas/FieldValue" - } - } - }, - "required": [ - "id", - "fields" - ] - }, - "Manifest": { - "type": "object", - "properties": { - "schema": { - "type": "array", - "items": { - "$ref": "#/components/schemas/FieldDefinition" - }, - "maxItems": 256 - }, - "samples": { - "type": "array", - "items": { - "$ref": "#/components/schemas/Sample" - }, - "maxItems": 1000000 - } - }, - "required": [ - "schema", - "samples" - ] - }, - "Publication": { - "type": "object", - "properties": { - "operation": { - "$ref": "#/components/schemas/Id" - }, - "parent": { - "anyOf": [ - { - "$ref": "#/components/schemas/Id" - }, - { - "type": "null" - } - ] - }, - "manifest": { - "$ref": "#/components/schemas/Manifest" - } - }, - "required": [ - "operation", - "parent", - "manifest" - ] - }, - "Snapshot": { - "type": "object", - "properties": { - "id": { - "$ref": "#/components/schemas/Id" - }, - "parent": { - "anyOf": [ - { - "$ref": "#/components/schemas/Id" - }, - { - "type": "null" - } - ] - }, - "operation": { - "$ref": "#/components/schemas/Id" - }, - "state": { - "enum": [ - "prepared", - "published", - "aborted" - ] - }, - "is_latest": { - "type": "boolean" - } - }, - "required": [ - "id", - "parent", - "operation", - "state", - "is_latest" - ] - }, - "SnapshotPage": { - "type": "object", - "properties": { - "snapshots": { - "type": "array", - "items": { - "$ref": "#/components/schemas/Snapshot" - }, - "maxItems": 1000 - }, - "next": { - "anyOf": [ - { - "$ref": "#/components/schemas/Id" - }, - { - "type": "null" - } - ] - } - }, - "required": [ - "snapshots", - "next" - ] - }, - "UploadRequest": { - "type": "object", - "properties": { - "id": { - "$ref": "#/components/schemas/Id" - }, - "length": { - "type": "integer", - "minimum": 1, - "maximum": 41943040000 - }, - "part_size": { - "type": "integer", - "minimum": 1, - "maximum": 4194304 - }, - "md5": { - "$ref": "#/components/schemas/Md5" - } - }, - "required": [ - "id", - "length", - "part_size", - "md5" - ] - }, - "Part": { - "type": "object", - "properties": { - "index": { - "type": "integer", - "minimum": 0, - "maximum": 9999 - }, - "length": { - "type": "integer", - "minimum": 1, - "maximum": 4194304 - }, - "md5": { - "$ref": "#/components/schemas/Md5" - } - }, - "required": [ - "index", - "length", - "md5" - ] - }, - "Upload": { - "type": "object", - "properties": { - "id": { - "$ref": "#/components/schemas/Id" - }, - "length": { - "type": "integer" - }, - "part_size": { - "type": "integer" - }, - "md5": { - "$ref": "#/components/schemas/Md5" - }, - "state": { - "enum": [ - "open", - "complete", - "claimed", - "aborted" - ] - }, - "expires_at": { - "type": "integer" - }, - "parts": { - "type": "array", - "items": { - "$ref": "#/components/schemas/Part" - }, - "maxItems": 10000 - }, - "cleanup_complete": { - "type": "boolean", - "description": "True only after all unowned physical ranges have been reclaimed. Deferred cleanup remains false." - } - }, - "required": [ - "id", - "length", - "part_size", - "md5", - "state", - "expires_at", - "parts" - ] - }, - "BatchRequest": { - "type": "object", - "properties": { - "snapshot": { - "$ref": "#/components/schemas/Id" - }, - "sample_ids": { - "type": "array", - "items": { - "$ref": "#/components/schemas/SampleId" - }, - "maxItems": 10000 - }, - "fields": { - "type": "array", - "items": { - "type": "string", - "maxLength": 255 - }, - "maxItems": 256 - } - }, - "required": [ - "snapshot", - "sample_ids", - "fields" - ] - }, - "BatchField": { - "type": "object", - "properties": { - "sample": { - "$ref": "#/components/schemas/SampleId" - }, - "field": { - "type": "string", - "maxLength": 255 - }, - "state": { - "enum": [ - "value", - "missing", - "tombstone" - ] - }, - "length": { - "type": "integer", - "minimum": 0 - }, - "md5": { - "anyOf": [ - { - "$ref": "#/components/schemas/Md5" - }, - { - "type": "null" - } - ] - } - }, - "required": [ - "sample", - "field", - "state", - "length", - "md5" - ] - }, - "BatchHeader": { - "type": "object", - "properties": { - "snapshot": { - "$ref": "#/components/schemas/Id" - }, - "sample_ids": { - "type": "array", - "items": { - "$ref": "#/components/schemas/SampleId" - }, - "maxItems": 10000 - }, - "fields": { - "type": "array", - "items": { - "$ref": "#/components/schemas/BatchField" - }, - "maxItems": 10000 - }, - "next": { - "anyOf": [ - { - "type": "string", - "maxLength": 4096 - }, - { - "type": "null" - } - ] - }, - "end": { - "type": "boolean" - } - }, - "required": [ - "snapshot", - "sample_ids", - "fields", - "next", - "end" - ] - }, - "WorkerPartition": { - "type": "object", - "properties": { - "worker": { - "type": "integer", - "minimum": 0, - "maximum": 65535 - }, - "workers": { - "type": "integer", - "minimum": 1, - "maximum": 65535 - }, - "rank": { - "type": "integer", - "minimum": 0, - "maximum": 65535 - }, - "world_size": { - "type": "integer", - "minimum": 1, - "maximum": 65535 - } - }, - "required": [ - "worker", - "workers", - "rank", - "world_size" - ] - }, - "Selection": { - "oneOf": [ - { - "type": "object", - "properties": { - "kind": { - "const": "explicit" - }, - "ids": { - "type": "array", - "items": { - "$ref": "#/components/schemas/SampleId" - }, - "maxItems": 10000 - } - }, - "required": [ - "kind", - "ids" - ] - }, - { - "type": "object", - "properties": { - "kind": { - "enum": [ - "prefix", - "keyword" - ] - }, - "value": { - "$ref": "#/components/schemas/SampleId" - } - }, - "required": [ - "kind", - "value" - ] - }, - { - "type": "object", - "properties": { - "kind": { - "const": "range" - }, - "start": { - "$ref": "#/components/schemas/SampleId" - }, - "end": { - "$ref": "#/components/schemas/SampleId" - } - }, - "required": [ - "kind", - "start", - "end" - ] - }, - { - "type": "object", - "properties": { - "kind": { - "const": "field_match" - }, - "field": { - "type": "string", - "maxLength": 255 - }, - "value": { - "$ref": "#/components/schemas/SampleId" - } - }, - "required": [ - "kind", - "field", - "value" - ] - }, - { - "type": "object", - "properties": { - "kind": { - "enum": [ - "and", - "or" - ] - }, - "items": { - "type": "array", - "items": { - "$ref": "#/components/schemas/Selection" - }, - "maxItems": 64 - } - }, - "required": [ - "kind", - "items" - ] - }, - { - "type": "object", - "properties": { - "kind": { - "const": "not" - }, - "item": { - "$ref": "#/components/schemas/Selection" - } - }, - "required": [ - "kind", - "item" - ] - } - ] - }, - "Shuffle": { - "type": "object", - "properties": { - "seed": { - "type": "integer", - "minimum": 0, - "maximum": 18446744073709551615 - }, - "group_size": { - "type": "integer", - "minimum": 1, - "maximum": 1000000 - } - }, - "required": [ - "seed", - "group_size" - ] - }, - "ReadPlan": { - "type": "object", - "properties": { - "snapshot": { - "type": "string", - "maxLength": 32 - }, - "selection": { - "$ref": "#/components/schemas/Selection" - }, - "fields": { - "type": "array", - "items": { - "type": "string", - "maxLength": 255 - }, - "maxItems": 256 - }, - "batch_size": { - "type": "integer", - "minimum": 1, - "maximum": 10000 - }, - "partition": { - "$ref": "#/components/schemas/WorkerPartition" - }, - "shuffle": { - "anyOf": [ - { - "$ref": "#/components/schemas/Shuffle" - }, - { - "type": "null" - } - ] - } - }, - "required": [ - "snapshot", - "selection", - "fields", - "batch_size", - "partition", - "shuffle" - ] - }, - "SnapshotRequest": { - "type": "object", - "properties": { - "snapshot": { - "$ref": "#/components/schemas/Id" - } - }, - "required": [ - "snapshot" - ] - }, - "Reclaim": { - "type": "object", - "properties": { - "snapshot": { - "$ref": "#/components/schemas/Id" - }, - "state": { - "enum": [ - "pending", - "completed" - ] - }, - "chunks_reclaimed": { - "type": "integer", - "minimum": 0 - } - }, - "required": [ - "snapshot", - "state", - "chunks_reclaimed" - ] - }, - "ReadBinding": { - "type": "object", - "required": [ - "snapshot", - "cursor" - ], - "properties": { - "snapshot": { - "type": "string" - }, - "cursor": { - "type": "string", - "maxLength": 349530 - } - } - }, - "ReadContinuation": { - "type": "object", - "additionalProperties": false, - "required": [ - "cursor" - ], - "properties": { - "cursor": { - "type": "string", - "maxLength": 349530 - } - } - } - }, - "responses": { - "Error": { - "description": "Stable structured failure; credentials and payload contents are never included. Codes: invalid_request, unauthorized, not_found, conflict, request_too_large, capacity, integrity, cancelled, unavailable, timeout, unsupported. Only capacity/unavailable are retryable when the operation permits replay.", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/Error" - } - } - } - } - } - }, - "x-crowdb-wire": { - "identity": "IDs are lowercase 32-hex nonzero 16-byte identities. MD5 is lowercase 32 hex; MD5 is an integrity check, not authentication. Dataset names are UTF-8, percent-encoded as one path segment, compared without Unicode normalization. Only dataset-ns exists in v1. Sample IDs are canonical padded base64, never interpreted as paths.", - "bounds": "JSON request/response at most 8 MiB before allocation; inline decoded values at most 64 KiB, schema/projection 256 fields, sample IDs at most 1024 bytes. Resource credits cover active request bodies and actual response delivery. Request/stream deadlines and deployment admission budgets are explicit configuration. Stateless read groups admit 100000 selected IDs and 8 MiB of ID bytes; continuation JSON is capped at 256 KiB before encoding. There are no persisted read pages or progress records. Manifest membership is the complete current sample set; omitted samples are removed, while omitted optional fields on surviving samples inherit by ancestry. Required fields must be present in each sample; explicit tombstones hide parent fields.", - "batch": "Content-Type application/vnd.crowdb.dataset-batch.v1. Magic is 8 ASCII bytes CDBDS001. Each frame: type u8, body length u32 big endian, then body. Types: 1 JSON BatchHeader (at most 8 MiB); 2 data (16-byte MD5 followed by 1..4 MiB bytes); 3 field end (zero length); 0 batch end (zero length); 4 terminal JSON Error (at most 64 KiB, only after the header). Header declares every sample/field in requested order. Value fields are delivered sequentially with one or more type-2 frames and one type-3; empty values have just type-3. Missing/tombstone have no data frames. Validate each frame MD5, exact whole-field length/MD5, field ends, batch end and HTTP EOF with no trailing bytes. A batch is successful only after completion. A streaming destination may contain verified partial data on error and must be discarded; download-to-path publishes only a verified temporary file. A type-4 frame reports a failed operation, may follow verified partial data, and never permits successful completion. It carries the same structured error model as control responses. The SDK must report the error and discard partial output.", - "retry": "SDK uses one overall deadline and stable logical tokens; bounded attempts/backoff only on documented idempotent operations with replayable bodies. Do not retry authorization, integrity or semantic conflicts. Ambiguous mutation errors retain operation/upload identity for status recovery. Never invent new tokens on retry.", - "integrity": "Seekable file convenience upload makes a bounded digest pass then a bounded transfer pass. Upload complete independently verifies durable parts and whole MD5. These are deliberate integrity passes, not whole-file buffering. Nonseekable uploads require known length/digest or later explicit streaming-session design.", - "lifecycle": "Remote durability is guaranteed after successful snapshot publication; local staging/checkpoint flush semantics are future SDK work. Reads perform no persistent activity registration or renewal. Snapshot deletion rejects new requests and defers cleanup for twenty minutes; admitted reads may complete or fail if overdue. Local byte credits survive until streaming completes/drops. Keep-alive uses bounded idle/header timeouts and shutdown drain. Upload ownership and reclaim are durable authority state, never client authority. Whole-file integrity completion has a 30-minute server deadline; clients use their configurable transfer deadline. Candidate reclamation retains an aborted operation tombstone; publication retry cannot resurrect it. chunks_reclaimed counts whole chunks, not retired shared ranges." - } -} diff --git a/doc/design/access-server/dataset/openapi-crowdb-dataset-v2.yaml b/doc/design/access-server/dataset/openapi-crowdb-dataset-v2.yaml deleted file mode 100644 index 43e050fa1..000000000 --- a/doc/design/access-server/dataset/openapi-crowdb-dataset-v2.yaml +++ /dev/null @@ -1,2002 +0,0 @@ -# Copyright 2026-present Gian -# Licensed under the Apache License, Version 2.0. -openapi: 3.1.0 -info: - title: CrowDB Dataset control API v2 - version: 2.0.0 - description: >- - Capability-gated Dataset control operations. Clients must discover required - capabilities before mutation. An absent capability is unavailable; clients - must not downgrade a v2 mutation to v1. The capability list advertises only - implemented operation families. Existing v1 control and binary framing remain - available under their original routes. Metadata GET routes require - bounded-inspection; explicit POST inspection preparation requires - inspection-preparation. Dataset creation, stable/retention changes and receipts - require dataset-management; direct Snapshot head updates require snapshot-head-update; - Split reads require split-revisions; read-only rule previews require split-preview; complete Split preparation requires split-preparation; - direct Split head updates require split-head-update; explicit complete-revision - deletion requires split-revision-delete and never falls back to draft abandonment; versioned - stateless reads require stateless-reads. Upload, Snapshot composition preparation and retirement - require uploads, composition-preparation and snapshot-reclaim respectively, advertised only - when the upload service is configured. Query parameters are strict and duplicate - keys fail. Snapshot preparation JSON/input/Manifest are limited to 8 MiB; - standalone Split preparation JSON and final bindings are limited to 1 MiB. - Composition and standalone Split begin/edit/evaluate/prepare/publish workflow routes return 501; - legacy draft reads, abandonment and cleanup remain available for old records. - The dataset-identity-fence capability supports X-Crowdb-Dataset-Id on named - Dataset routes. The active-instance guard is checked during request admission - and root creation reservation CAS; a replacement returns 409 dataset_replaced, - missing/marked identities return 404, malformed/duplicate headers return 400. - Existing clients may omit it. Deletion receipt/status retries after marking - omit this active guard and retain their own expected immutable ID. -servers: - - url: /api/dataset/v2 -security: - - bearerAuth: [] -paths: - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - put: - operationId: prepareSnapshot - summary: Prepare complete Snapshot content from retained caller inputs - description: >- - Requires composition-preparation and a configured upload service. Caller - chooses the Snapshot ID and retains the complete parent/source/schema/ - Sample patches/Split definitions. Parent and schema are required nullable - fields; null schema inherits the fixed parent's schema. Frozen sources and - inherited definitions name exact revisions; currentSplit is never resolved - during retries. Body, native input and final Manifest are bounded to 8 MiB, - with 30 seconds for computation. Each patch ID occurs once; upsert patches - specified fields and preserves other effective parent fields. Uploaded - fields carry id/length/md5 from a completed result; conversion performs no - metadata query and the authority validates the descriptor before claiming - its final Snapshot owner. The complete root is installed last, after final - index, initial Split membership and payload ownership. Complete results - are readable by ID before head publication. Retry the same ID and full - input after unknown outcomes; different content or initial Split binding - conflicts. No editing/evaluation progress, operation mapping or staging - owner is stored. Capture the intended head base before computation, then - update the Dataset head separately. There is no workflow or v1 fallback. - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/snapshot'} - requestBody: - required: true - content: - application/json: - schema: {$ref: '#/components/schemas/PrepareSnapshot'} - responses: - '200': - description: Complete immutable result; latest/stable are unchanged - content: - application/json: - schema: {$ref: '#/components/schemas/PreparedSnapshot'} - '409': - description: Resource content, descriptor or Split policy conflicts - $ref: '#/components/responses/InspectionError' - '413': - description: Body, native input, final Manifest or computation exceeds bounds - $ref: '#/components/responses/InspectionError' - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/head: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: datasetHead - summary: Read mutable names and their current opaque head generation - description: >- - Requires head-inspection capability. Generation covers the exact head - including its internal fence. An absent head has null generation. Stable - changes and reclaim fences may change generation without changing latest. - Clients must treat the token as opaque and preserve it without decoding. - parameters: - - in: path - name: dataset - required: true - schema: {type: string} - responses: - '200': - description: One consistent observation of the current head - content: - application/json: - schema: {$ref: '#/components/schemas/Head'} - '404': - description: Dataset or namespace is unavailable - put: - operationId: updateSnapshotHead - summary: Select a complete Snapshot using an explicit head condition - description: >- - Requires snapshot-head-update. The input contains only the complete resource ID - and explicit nullable expected_head; no draft or attempt identity is read. - Stable is preserved. Same-current-target retries confirm the current effect; - stale conditions after another publication return observed_head without - a confirmation query or implicit content rebase. Unknown outcomes do not - create response history. The JSON body is limited to 4096 bytes. - parameters: - - {$ref: '#/components/parameters/dataset'} - requestBody: - required: true - content: - application/json: - schema: {$ref: '#/components/schemas/UpdateSnapshotHead'} - responses: - '200': - description: Current effect confirmed without a historical response receipt - content: - application/json: - schema: {$ref: '#/components/schemas/Head'} - default: - $ref: '#/components/responses/InspectionError' - /capabilities: - get: - operationId: datasetCapabilities - summary: Discover supported Dataset control operation families - responses: - '200': - description: Supported API version and capability names - headers: - x-request-id: - schema: {type: string} - content: - application/json: - schema: {$ref: '#/components/schemas/Capabilities'} - '401': - description: Missing or invalid configured bearer token - '429': - description: API admission capacity exhausted - '504': - description: Request deadline expired - /namespaces/dataset-ns/datasets: - get: - operationId: listDatasets - summary: Discover existing Dataset identities - parameters: - - {$ref: '#/components/parameters/after'} - - {$ref: '#/components/parameters/limit'} - responses: - '200': - description: Bounded metadata result; counts and byte lengths are decimal strings. - content: - application/json: - schema: {$ref: '#/components/schemas/DatasetPage'} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/snapshot-inventory: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: snapshotInventory - summary: List complete Snapshot inventory - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/after'} - - {$ref: '#/components/parameters/limit'} - responses: - '200': - description: Bounded metadata result; counts and byte lengths are decimal strings. - content: - application/json: - schema: {$ref: '#/components/schemas/SnapshotPage'} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/roots: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: snapshotRoots - summary: List complete parentless Snapshot roots - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/after'} - - {$ref: '#/components/parameters/limit'} - responses: - '200': - description: Bounded metadata result; counts and byte lengths are decimal strings. - content: - application/json: - schema: {$ref: '#/components/schemas/SnapshotPage'} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}/children: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: snapshotChildren - summary: List complete children of a concrete Snapshot - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/snapshot'} - - {$ref: '#/components/parameters/after'} - - {$ref: '#/components/parameters/limit'} - responses: - '200': - description: Bounded metadata result; counts and byte lengths are decimal strings. - content: - application/json: - schema: {$ref: '#/components/schemas/SnapshotPage'} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}/summary: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: snapshotSummary - summary: Read bounded complete Snapshot metadata - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/snapshot'} - responses: - '200': - description: Bounded metadata result; counts and byte lengths are decimal strings. - content: - application/json: - schema: {$ref: '#/components/schemas/SnapshotSummary'} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}/schema: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: snapshotSchema - summary: Read an ordered Schema window - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/snapshot'} - - {$ref: '#/components/parameters/after'} - - {$ref: '#/components/parameters/limit'} - responses: - '200': - description: Bounded metadata result; counts and byte lengths are decimal strings. - content: - application/json: - schema: {$ref: '#/components/schemas/SchemaPage'} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}/samples: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: snapshotSamples - summary: Read exact binary Sample IDs in byte order - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/snapshot'} - - {$ref: '#/components/parameters/after'} - - {$ref: '#/components/parameters/limit'} - responses: - '200': - description: Bounded metadata result; counts and byte lengths are decimal strings. - content: - application/json: - schema: {$ref: '#/components/schemas/SamplePage'} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}/split-preview: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - post: - operationId: previewSnapshotSplit - summary: Preview one Split definition without creating a revision - description: >- - Requires split-preview. Read-only bounded evaluation on a complete Snapshot, - including frozen inherited membership. Limit bounds scanned Samples, not - returned matches; an empty page can have a continuation. The cursor binds - the immutable scope and complete definition. No draft, evaluation progress, - resource or read session is written. Uploaded payload bytes are not read. - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/snapshot'} - - {$ref: '#/components/parameters/after'} - - {$ref: '#/components/parameters/limit'} - requestBody: - required: true - content: - application/json: - schema: {$ref: '#/components/schemas/DraftSplit'} - responses: - '200': - description: Matching base64 Sample IDs and scan continuation. - content: - application/json: - schema: {$ref: '#/components/schemas/SamplePage'} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}/matrix: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: snapshotMatrix - summary: Read effective field metadata without payload access - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/snapshot'} - - {$ref: '#/components/parameters/after'} - - {$ref: '#/components/parameters/limit'} - - in: query - name: fields - required: true - description: JSON-encoded array of 1 to 12 exact field names; continuation binds this projection. - schema: {type: string} - responses: - '200': - description: Bounded metadata result; counts and byte lengths are decimal strings. - content: - application/json: - schema: {$ref: '#/components/schemas/MatrixPage'} - default: - $ref: '#/components/responses/InspectionError' - post: - operationId: exactSnapshotMatrix - summary: Project a bounded explicit ID window without creating a read session - description: Read-only query, including Sample IDs obtained from a Split member page. Retains request order. - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/snapshot'} - requestBody: - required: true - content: - application/json: - schema: - type: object - additionalProperties: false - required: [sample_ids, fields] - properties: - sample_ids: {type: array, minItems: 1, maxItems: 20, uniqueItems: true, items: {type: string}} - fields: {type: array, minItems: 1, maxItems: 12, uniqueItems: true, items: {type: string}} - responses: - '200': - description: Effective field metadata only; no payload or consumption progress mutation. - content: - application/json: - schema: - type: object - required: [entries] - properties: - entries: {type: array, maxItems: 20, items: {$ref: '#/components/schemas/MatrixRow'}} - default: - $ref: '#/components/responses/InspectionError' - - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}/contains: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: snapshotContains - summary: Look up exact binary Sample membership - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/snapshot'} - - in: query - name: sample - required: true - description: Canonical padded base64 of a nonempty Sample ID, at most 16384 decoded bytes. - schema: {type: string} - responses: - '200': - description: Bounded metadata result; counts and byte lengths are decimal strings. - content: - application/json: - schema: {$ref: '#/components/schemas/Contains'} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}/inspection: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: snapshotInspection - summary: Read explicit metadata preparation status - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/snapshot'} - responses: - '200': - description: Bounded metadata result; counts and byte lengths are decimal strings. - content: - application/json: - schema: {$ref: '#/components/schemas/InspectionPreparation'} - default: - $ref: '#/components/responses/InspectionError' - post: - operationId: prepareSnapshotInspection - summary: Rebuild complete legacy inspection metadata - description: Explicit mutation with a thirty-second deadline, a 9 MiB source-record limit and an 8 MiB embedded-Manifest limit. Writes complete index rows and installs the summary last without durable preparation progress. Retry rebuilds from the source or reuses an existing summary without writes. Errors propagate without confirmation reads. Does not change publication heads or acknowledge reads. - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/snapshot'} - responses: - '200': - description: Complete metadata readiness; revision and rows_written are zero compatibility fields. - content: - application/json: - schema: {$ref: '#/components/schemas/InspectionPreparation'} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}/split-head: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - put: - operationId: updateSplitHead - summary: Conditionally update currentSplit from a complete resource - description: Requires split-head-update. Caller supplies the target and both explicit nullable preconditions, without a draft or attempt ID. A conflict carries observed_head for caller review; the server does not rebase or query again to confirm a CAS outcome. - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/snapshot'} - requestBody: - required: true - content: - application/json: - schema: {$ref: '#/components/schemas/UpdateSplitHead'} - responses: - '200': - description: Current effect confirmed, without a historical response receipt. - content: - application/json: - schema: {$ref: '#/components/schemas/SplitHead'} - default: - $ref: '#/components/responses/InspectionError' - get: - operationId: splitHead - parameters: - - in: path - name: dataset - required: true - schema: {type: string} - - in: path - name: snapshot - required: true - schema: {type: string} - responses: - '200': - description: Revision-bound operation result - content: - application/json: - schema: {$ref: '#/components/schemas/SplitHead'} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}/split-revisions: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: splitHistory - description: Lists complete revisions, including results that have not updated currentSplit. Snapshot read admission applies. - parameters: - - in: path - name: dataset - required: true - schema: {type: string} - - in: path - name: snapshot - required: true - schema: {type: string} - - {$ref: '#/components/parameters/after'} - - {$ref: '#/components/parameters/limit'} - responses: - '200': - description: Revision-bound operation result - content: - application/json: - schema: {$ref: '#/components/schemas/SplitRevisionPage'} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}/split-revisions/{revision}: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - delete: - operationId: deleteSplitRevision - summary: Mark one historical complete Split revision for delayed metadata reclamation - description: >- - Requires split-revision-delete. Accepts only the two resource IDs, with no - query parameters or request body. The current Split and the Snapshot's - initial Split are protected. A successful mark rejects new reads and - persists one fixed reclaim_after deadline using dataset.reclaim_grace_seconds (default 1200). Repeating the - same DELETE preserves that deadline. Background cleanup seals membership - metadata; it does not delete sample payloads. The owner Snapshot may - reclaim this metadata after its own read admission closes and its own - configured (default twenty-minute) grace expires. Complete revisions otherwise follow their - owner Snapshot's lifetime, without an independent automatic TTL. No - request, response or progress session is persisted. Unknown outcomes - return directly; an explicit client retry repeats the same two-ID DELETE. - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/snapshot'} - - in: path - name: revision - required: true - schema: {$ref: '#/components/schemas/OpaqueId'} - responses: - '200': - description: Fixed deletion mark or terminal metadata reclamation status - content: - application/json: - schema: {$ref: '#/components/schemas/SplitRevisionDeletion'} - '404': - description: The scoped revision was not found - $ref: '#/components/responses/InspectionError' - '409': - description: Current or initial revision is protected, or the resource changed concurrently - $ref: '#/components/responses/InspectionError' - default: - $ref: '#/components/responses/InspectionError' - put: - operationId: prepareSplitRevision - summary: Prepare complete immutable membership from caller-owned inputs - description: >- - Requires split-preparation. The caller retains revision ID, policy and - complete definitions. No operation, edit or evaluation cursor is accepted. - Computation is bounded to 30 seconds and 1 MiB input/final bindings. - Retry the same ID and input after an unknown outcome; incomplete work is - recomputed, while an identical complete result is reused. Different input - at a complete ID conflicts. Frozen sources retain their supplied revision. - This operation does not update currentSplit; capture its intended base - before computation and update split-head separately. No draft fallback. - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/snapshot'} - - in: path - name: revision - required: true - schema: {$ref: '#/components/schemas/OpaqueId'} - requestBody: - required: true - content: - application/json: - schema: {$ref: '#/components/schemas/PrepareSplitRevision'} - responses: - '200': - description: Complete immutable resource, readable before head publication - content: - application/json: - schema: {$ref: '#/components/schemas/SplitRevision'} - '409': - description: Resource ID already has different input, or validation conflicts - $ref: '#/components/responses/InspectionError' - '413': - description: Request body exceeds 1 MiB - $ref: '#/components/responses/InspectionError' - default: - $ref: '#/components/responses/InspectionError' - get: - operationId: splitRevision - description: Reads a complete immutable revision by ID without requiring currentSplit publication. Snapshot read admission applies. Deletion-marked revisions are unavailable, normally with 404; existing bounded-work errors may take precedence. - parameters: - - in: path - name: dataset - required: true - schema: {type: string} - - in: path - name: snapshot - required: true - schema: {type: string} - - in: path - name: revision - required: true - schema: {type: string} - responses: - '200': - description: Revision-bound operation result - content: - application/json: - schema: {$ref: '#/components/schemas/SplitRevision'} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}/split-revisions/{revision}/splits/{name}/members: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: splitMembers - parameters: - - in: path - name: dataset - required: true - schema: {type: string} - - in: path - name: snapshot - required: true - schema: {type: string} - - in: path - name: revision - required: true - schema: {type: string} - - in: path - name: name - required: true - schema: {type: string} - - {$ref: '#/components/parameters/after'} - - {$ref: '#/components/parameters/limit'} - responses: - '200': - description: Revision-bound operation result - content: - application/json: - schema: {$ref: '#/components/schemas/SamplePage'} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/split-drafts/{operation}: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - deprecated: true - description: Read-only observation of an existing legacy draft; never resumes work. - operationId: splitDraft - parameters: - - in: path - name: dataset - required: true - schema: {type: string} - - in: path - name: operation - required: true - schema: {type: string} - responses: - '200': - description: Revision-bound operation result - content: - application/json: - schema: {$ref: '#/components/schemas/SplitDraft'} - default: - $ref: '#/components/responses/InspectionError' - delete: - deprecated: true - description: Abandon an existing legacy draft for cleanup. - operationId: abandonSplitDraft - parameters: - - in: path - name: dataset - required: true - schema: {type: string} - - in: path - name: operation - required: true - schema: {type: string} - requestBody: - required: true - content: - application/json: - schema: {$ref: '#/components/schemas/DraftRevision'} - responses: - '200': - description: Revision-bound operation result - content: - application/json: - schema: {$ref: '#/components/schemas/SplitDraft'} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/reads/bind: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - post: - operationId: bindRead - description: Resolve immutable Snapshot and Split versions without creating read state. - parameters: - - {$ref: '#/components/parameters/dataset'} - requestBody: - required: true - content: - application/json: - schema: {$ref: '#/components/schemas/ReadPlanV2'} - responses: - '200': - description: Client-owned continuation; no TTL, remote acknowledgement or close. - content: - application/json: - schema: {$ref: '#/components/schemas/ReadBinding'} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/reads/next: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - post: - operationId: nextReadBatch - description: Replay this continuation; save the returned next cursor after processing. - parameters: - - {$ref: '#/components/parameters/dataset'} - requestBody: - required: true - content: - application/json: - schema: {$ref: '#/components/schemas/ReadContinuation'} - responses: - '200': - description: Bounded batch; null next means end of selection. - content: - application/vnd.crowdb.dataset-batch.v1: - schema: {type: string, format: binary} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}/preview: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: snapshotFieldPreview - summary: Read a bounded binary field prefix for text or hex presentation - description: >- - Never acknowledges an application read. Response bytes are capped at 64 KiB; - uploaded parts are independently integrity-checked with a 4 MiB per-part - allocation cap and five-second payload deadline. Full-field MD5 identifies - the entire value, not the truncated prefix. External legacy locators without - supported payload storage return an explicit error, never an arbitrary URL. - parameters: - - {$ref: '#/components/parameters/dataset'} - - {$ref: '#/components/parameters/snapshot'} - - in: query - name: sample - required: true - schema: {type: string, description: Canonical padded base64 Sample ID} - - in: query - name: field - required: true - schema: {type: string} - - in: query - name: max_bytes - schema: {type: integer, minimum: 1, maximum: 65536, default: 65536} - responses: - '200': - description: Binary prefix or explicit missing/tombstone state - content: - application/json: - schema: - type: object - required: [state, length, md5, data, truncated] - properties: - state: {enum: [value, missing, tombstone]} - length: {type: [string, 'null']} - md5: {type: [string, 'null']} - data: {type: [string, 'null'], description: Canonical padded base64; empty string is an empty value} - truncated: {type: boolean} - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/deletion/{id}: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: datasetDeletionStatus - summary: Reconcile a deletion for the exact Dataset identity - parameters: - - $ref: '#/components/parameters/dataset' - - name: id - in: path - required: true - schema: {type: string, pattern: '^[0-9a-f]{32}$'} - responses: - '200': - description: Null before deletion, otherwise the durable receipt; identity mismatch is 409. - content: - application/json: - schema: - type: object - required: [deletion] - properties: - deletion: - anyOf: - - $ref: '#/components/schemas/DatasetDeletion' - - type: 'null' - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - delete: - operationId: deleteDatasetV2 - summary: Mark an empty Dataset deleted - description: >- - Requires dataset-delete capability and exact Dataset ID. Nonempty (including - pending creation/cleanup) returns 409 dataset_not_empty. Successful marking - immediately hides the Dataset. Cleanup follows dataset.reclaim_grace_seconds - (default 1200). The name remains reserved until cleanup completes. Retries - preserve the deadline; stale IDs cannot delete a replacement Dataset. - parameters: - - $ref: '#/components/parameters/dataset' - requestBody: - required: true - content: - application/json: - schema: - type: object - additionalProperties: false - required: [id] - properties: - id: {type: string, pattern: '^[0-9a-f]{32}$'} - responses: - '200': - description: Durable deletion receipt, including idempotent retries. - content: - application/json: - schema: - $ref: '#/components/schemas/DatasetDeletion' - default: - $ref: '#/components/responses/InspectionError' - put: - operationId: createDatasetV2 - summary: Create a Dataset - parameters: - - $ref: '#/components/parameters/dataset' - responses: - '200': - description: Successful bounded operation. Integer counts are exact decimal strings. - content: - application/json: - schema: - $ref: '#/components/schemas/DatasetInfo' - default: - $ref: '#/components/responses/InspectionError' - get: - operationId: getDatasetV2 - summary: Read Dataset metadata - parameters: - - $ref: '#/components/parameters/dataset' - responses: - '200': - description: Successful bounded operation. Integer counts are exact decimal strings. - content: - application/json: - schema: - $ref: '#/components/schemas/DatasetInfo' - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/names/stable: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - put: - operationId: setStableSnapshot - summary: Set stable without changing latest - parameters: - - $ref: '#/components/parameters/dataset' - responses: - '200': - description: Successful bounded operation. Integer counts are exact decimal strings. - content: - application/json: - schema: - $ref: '#/components/schemas/DatasetInfo' - default: - $ref: '#/components/responses/InspectionError' - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/StableSnapshot' - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}/retain: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - post: - operationId: retainSnapshotV2 - summary: Retain Snapshot retention - parameters: - - $ref: '#/components/parameters/dataset' - - in: path - name: snapshot - required: true - schema: &id001 - $ref: '#/components/schemas/OpaqueId' - responses: - '200': - description: Successful bounded operation. Integer counts are exact decimal strings. - content: - application/json: - schema: - $ref: '#/components/schemas/RetainedSnapshot' - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}/release: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - post: - operationId: releaseSnapshotV2 - summary: Release Snapshot retention - parameters: - - $ref: '#/components/parameters/dataset' - - in: path - name: snapshot - required: true - schema: *id001 - responses: - '200': - description: Successful bounded operation. Integer counts are exact decimal strings. - content: - application/json: - schema: - $ref: '#/components/schemas/RetainedSnapshot' - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/snapshots/{snapshot}/reclaim: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: observeSnapshotLifecycleV2 - summary: Observe bounded protection and retirement facts without mutation - parameters: - - $ref: '#/components/parameters/dataset' - - in: path - name: snapshot - required: true - schema: *id001 - responses: - '200': - description: Observed facts, not permission to reclaim; parent reachability is not evaluated. - content: - application/json: - schema: - $ref: '#/components/schemas/SnapshotLifecycle' - default: - $ref: '#/components/responses/InspectionError' - post: - operationId: reclaimSnapshotV2 - summary: Mark Snapshot deletion and observe background retirement - parameters: - - $ref: '#/components/parameters/dataset' - - in: path - name: snapshot - required: true - schema: *id001 - responses: - '200': - description: Durable deletion status. Background workers advance cleanup after the grace deadline. - content: - application/json: - schema: - $ref: '#/components/schemas/SnapshotReclaim' - default: - $ref: '#/components/responses/InspectionError' - description: Latest, stable, retained or live-parent Snapshots are protected. - This empty-body action only marks deletion and observes durable status; repeated calls - do not drive cleanup. Nonempty bodies, including cleanup cursor or completion inputs, - are rejected. Background workers retire owners before metadata. Physical capacity - recovery remains a separate ChunkDB responsibility. - /namespaces/dataset-ns/datasets/{dataset}/commits/{attempt}: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: commitStatusV2 - summary: Read a legacy publication receipt - description: Returns existing legacy receipts without backfill. New Snapshot and Split publication creates no historical receipts; retry the same conditional request to confirm its current effect. - parameters: - - $ref: '#/components/parameters/dataset' - - in: path - name: attempt - required: true - schema: *id001 - responses: - '200': - description: Successful bounded operation. Integer counts are exact decimal strings. - content: - application/json: - schema: - $ref: '#/components/schemas/CommitStatus' - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/split-drafts/{operation}/cleanup: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - post: - deprecated: true - operationId: cleanupSplitDraft - summary: Retired Split cleanup action - parameters: - - $ref: '#/components/parameters/dataset' - - in: path - name: operation - required: true - schema: *id001 - description: Returns unsupported after authentication and route validation, before storage. Background workers discover historical Split draft and work facts using local cursors. Readable Snapshot owners retain their deletion grace; permanently sealed incomplete roots do not require a reader deadline. - responses: - '501': - $ref: '#/components/responses/InspectionError' - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/uploads: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - post: - operationId: createUploadV2 - summary: Create a Dataset-scoped multipart upload - parameters: - - $ref: '#/components/parameters/dataset' - responses: - '200': - description: Successful bounded operation. Integer counts are exact decimal strings. - content: - application/json: - schema: - $ref: '#/components/schemas/UploadStatus' - default: - $ref: '#/components/responses/InspectionError' - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/CreateUpload' - /namespaces/dataset-ns/datasets/{dataset}/uploads/{upload}: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - operationId: getUploadV2 - summary: Inspect upload progress - parameters: - - $ref: '#/components/parameters/dataset' - - in: path - name: upload - required: true - schema: *id001 - responses: - '200': - description: Successful bounded operation. Integer counts are exact decimal strings. - content: - application/json: - schema: - $ref: '#/components/schemas/UploadStatus' - default: - $ref: '#/components/responses/InspectionError' - delete: - operationId: abortUploadV2 - summary: Abort an unowned upload - parameters: - - $ref: '#/components/parameters/dataset' - - in: path - name: upload - required: true - schema: *id001 - responses: - '200': - description: Successful bounded operation. Integer counts are exact decimal strings. - content: - application/json: - schema: - $ref: '#/components/schemas/UploadStatus' - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/uploads/{upload}/complete: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - post: - operationId: completeUploadV2 - summary: Verify and complete upload parts - parameters: - - $ref: '#/components/parameters/dataset' - - in: path - name: upload - required: true - schema: *id001 - responses: - '200': - description: Successful bounded operation. Integer counts are exact decimal strings. - content: - application/json: - schema: - $ref: '#/components/schemas/UploadStatus' - default: - $ref: '#/components/responses/InspectionError' - /namespaces/dataset-ns/datasets/{dataset}/uploads/{upload}/parts/{index}: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - put: - operationId: uploadPartV2 - summary: Write one bounded upload part - parameters: - - $ref: '#/components/parameters/dataset' - - in: path - name: upload - required: true - schema: *id001 - - in: path - name: index - required: true - schema: - $ref: '#/components/schemas/Decimal' - responses: - '200': - description: Successful bounded operation. Integer counts are exact decimal strings. - content: - application/json: - schema: - $ref: '#/components/schemas/UploadPart' - default: - $ref: '#/components/responses/InspectionError' - requestBody: - required: true - content: - application/octet-stream: - schema: - type: string - format: binary - maxLength: 4194304 - /namespaces/dataset-ns/datasets/{dataset}/compositions/{operation}: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - get: - deprecated: true - operationId: getComposition - summary: Observe an existing legacy composition without resuming it - parameters: - - $ref: '#/components/parameters/dataset' - - in: path - name: operation - required: true - schema: *id001 - responses: - '200': - description: Successful bounded operation. Integer counts are exact decimal strings. - content: - application/json: - schema: - $ref: '#/components/schemas/CompositionDraft' - default: - $ref: '#/components/responses/InspectionError' - delete: - deprecated: true - operationId: abandonComposition - summary: Abandon an unpublished composition - parameters: - - $ref: '#/components/parameters/dataset' - - in: path - name: operation - required: true - schema: *id001 - responses: - '200': - description: Successful bounded operation. Integer counts are exact decimal strings. - content: - application/json: - schema: - $ref: '#/components/schemas/CompositionDraft' - default: - $ref: '#/components/responses/InspectionError' - description: Background workers clean existing draft resources after abandonment; no client cleanup action is required. Complete resource owners remain independent of the draft. - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/DraftRevision' - /namespaces/dataset-ns/datasets/{dataset}/compositions/{operation}/cleanup: - parameters: - - {$ref: '#/components/parameters/datasetInstance'} - post: - deprecated: true - operationId: cleanupComposition - summary: Retired Composition cleanup action - parameters: - - $ref: '#/components/parameters/dataset' - - in: path - name: operation - required: true - schema: *id001 - responses: - '501': - $ref: '#/components/responses/InspectionError' - default: - $ref: '#/components/responses/InspectionError' - description: Returns unsupported after authentication and route validation, before storage. Background workers discover historical draft facts and retire them using local cursors; no client cleanup state or completion confirmation is accepted. -components: - parameters: - datasetInstance: - in: header - name: X-Crowdb-Dataset-Id - description: Optional active Dataset instance; omit after marking when retrying deletion receipts/status. - schema: {type: string, pattern: '^(?!0{32}$)[0-9a-f]{32}$'} - dataset: - in: path - name: dataset - required: true - schema: {type: string} - snapshot: - in: path - name: snapshot - required: true - schema: {type: string, pattern: '^[0-9a-f]{32}$'} - after: - in: query - name: after - description: Opaque canonical base64 continuation; preserve query and immutable scope. Empty pages may continue. - schema: {type: string, maxLength: 43692} - limit: - in: query - name: limit - schema: {type: integer, minimum: 1, maximum: 100, default: 20} - responses: - InspectionError: - description: >- - 400 invalid or duplicate query; 401 unauthorized; 404 unavailable; - 409 inspection_not_prepared; 413 read_limit; 422 inspection_unsupported; - 429 capacity; 499 cancelled; 500 integrity; 503 unavailable; 504 timeout. - Unknown query keys are rejected. No error triggers unbounded fallback. - content: - application/json: - schema: - type: object - required: [code, message, request_id, retryable] - properties: - code: {type: string} - message: {type: string} - request_id: {type: string} - retryable: {type: boolean} - observed_head: - oneOf: - - {$ref: '#/components/schemas/SplitHead'} - - {$ref: '#/components/schemas/Head'} - securitySchemes: - bearerAuth: - type: http - scheme: bearer - schemas: - Head: - type: object - required: [latest, stable, generation] - properties: - latest: - type: [string, 'null'] - pattern: '^[0-9a-f]{32}$' - stable: - type: [string, 'null'] - pattern: '^[0-9a-f]{32}$' - generation: - type: [string, 'null'] - pattern: '^[0-9a-f]{64}$' - UpdateSnapshotHead: - type: object - additionalProperties: false - required: [snapshot, expected_head] - properties: - snapshot: {$ref: '#/components/schemas/OpaqueId'} - expected_head: - type: [string, 'null'] - pattern: '^[0-9a-f]{64}$' - Capabilities: - type: object - required: [api_version, capabilities] - properties: - api_version: - type: integer - const: 2 - capabilities: - type: array - uniqueItems: true - items: {type: string} - description: Unknown names may be ignored; required names must be present. - DatasetDeletion: - type: object - required: [id, reclaim_after, complete] - properties: - id: {type: string, pattern: '^[0-9a-f]{32}$'} - reclaim_after: {type: string, pattern: '^[0-9]+$'} - complete: {type: boolean} - DatasetInfo: - type: object - required: [id, namespace, name, latest, stable] - properties: - id: {type: string} - namespace: {type: string} - name: {type: string} - latest: {type: [string, 'null']} - stable: {type: [string, 'null']} - SnapshotSummary: - type: object - required: [id, parent, schema_version, field_count, sample_count, manifest_sha256, inspection, reason] - properties: - id: {type: string} - parent: {type: [string, 'null']} - schema_version: {type: integer} - field_count: {type: integer} - sample_count: {type: string, pattern: '^[0-9]+$'} - manifest_sha256: {type: string, pattern: '^[0-9a-f]{64}$'} - inspection: {enum: [ready, unsupported]} - reason: {type: [string, 'null']} - FieldDefinition: - type: object - required: [name, required] - properties: - name: {type: string} - required: {type: boolean} - FieldMetadata: - type: object - required: [name, storage, length, md5, source_snapshot] - properties: - name: {type: string} - storage: {enum: [inline, payload, tombstone]} - length: {type: string, pattern: '^[0-9]+$'} - md5: {type: [string, 'null']} - source_snapshot: {type: string} - MatrixRow: - type: object - required: [id, fields] - properties: - id: {type: string, description: Canonical padded base64 Sample ID} - fields: - type: array - description: Requested field order; null means missing, distinct from tombstone. - items: - anyOf: - - {$ref: '#/components/schemas/FieldMetadata'} - - {type: 'null'} - Contains: - type: object - required: [contains] - properties: - contains: {type: boolean} - InspectionPreparation: - type: object - required: [snapshot, revision, rows_written, state, reason] - properties: - snapshot: {type: string} - revision: {type: string, pattern: '^[0-9]+$', description: Zero for complete rebuilds; historical GET progress can retain a nonzero value.} - rows_written: {type: string, pattern: '^[0-9]+$', description: Zero for complete rebuilds; historical GET progress can retain a nonzero value.} - state: {enum: [preparing, ready, unsupported], description: Preparing is observable only for historical progress through GET. New POST results describe a complete index.} - reason: {type: [string, 'null']} - DatasetPage: - type: object - required: [entries, after] - properties: - after: {type: [string, 'null']} - entries: - type: array - maxItems: 100 - items: {$ref: '#/components/schemas/DatasetInfo'} - SnapshotPage: - type: object - required: [entries, after] - properties: - after: {type: [string, 'null']} - entries: - type: array - maxItems: 100 - items: {$ref: '#/components/schemas/SnapshotSummary'} - SchemaPage: - type: object - required: [entries, after] - properties: - after: {type: [string, 'null']} - entries: - type: array - maxItems: 100 - items: {$ref: '#/components/schemas/FieldDefinition'} - SamplePage: - type: object - required: [entries, after] - properties: - after: {type: [string, 'null']} - entries: - type: array - maxItems: 100 - items: {type: string, description: Canonical padded base64 Sample ID} - MatrixPage: - type: object - required: [entries, after] - properties: - after: {type: [string, 'null']} - entries: - type: array - maxItems: 100 - items: {$ref: '#/components/schemas/MatrixRow'} - Decimal: - type: string - pattern: '^(0|[1-9][0-9]*)$' - description: Exact unsigned 64-bit integer encoded as a canonical decimal string. - OpaqueId: - type: string - pattern: '^[0-9a-f]{32}$' - description: Nonzero opaque identity. - NullableRevision: - type: [string, 'null'] - pattern: '^[0-9a-f]{32}$' - NullableGeneration: - type: [string, 'null'] - pattern: '^[0-9a-f]{64}$' - Predicate: - oneOf: - - type: object - additionalProperties: false - required: [kind] - properties: - kind: {const: all} - - type: object - additionalProperties: false - required: [kind, ids] - properties: - kind: {const: explicit} - ids: {type: array, maxItems: 10000, items: {type: string}} - - type: object - additionalProperties: false - required: [kind, value] - properties: - kind: {enum: [prefix, keyword]} - value: {type: string, description: Nonempty canonical padded base64} - - type: object - additionalProperties: false - required: [kind, start, end] - properties: - kind: {const: range} - start: {type: string} - end: {type: string} - - type: object - additionalProperties: false - required: [kind, field, value] - properties: - kind: {const: field_match} - field: {type: string} - value: {type: string, description: Canonical padded base64; empty is an exact empty inline value} - - type: object - additionalProperties: false - required: [kind, items] - properties: - kind: {enum: [and, or]} - items: {type: array, minItems: 1, items: {$ref: '#/components/schemas/Predicate'}} - - type: object - additionalProperties: false - required: [kind, item] - properties: - kind: {const: not} - item: {$ref: '#/components/schemas/Predicate'} - description: Maximum depth 32 and 1024 nodes. IDs are binary base64, never text normalization. - SplitPolicy: - type: object - additionalProperties: false - required: [overlap, coverage] - properties: - overlap: {enum: [disjoint, allow_overlap]} - coverage: {enum: [require_complete, allow_unassigned]} - SplitSource: - type: object - additionalProperties: false - required: [snapshot, revision, name] - properties: - snapshot: {$ref: '#/components/schemas/OpaqueId'} - revision: {$ref: '#/components/schemas/OpaqueId'} - name: {type: string, minLength: 1, description: Exact UTF-8 name, at most 255 bytes} - SplitOrigin: - oneOf: - - type: object - additionalProperties: false - required: [kind] - properties: - kind: {const: evaluated} - - type: object - additionalProperties: false - required: [kind, source] - properties: - kind: {const: inherited} - source: {$ref: '#/components/schemas/SplitSource'} - SplitDefinition: - oneOf: - - type: object - additionalProperties: false - required: [kind, predicate] - properties: - kind: {const: selection} - predicate: {$ref: '#/components/schemas/Predicate'} - - type: object - additionalProperties: false - required: [kind, seed, weights, index] - properties: - kind: {const: weighted_partition} - seed: {$ref: '#/components/schemas/Decimal'} - index: {type: integer, minimum: 0} - weights: - type: array - minItems: 1 - items: - type: object - additionalProperties: false - required: [name, weight] - properties: - name: {type: string} - weight: {type: integer, minimum: 1, maximum: 4294967295} - description: Uses sample-id-sha256-v1 with ordered weights and checked u32 sum. - DraftSplit: - type: object - additionalProperties: false - required: [name, definition, origin] - properties: - name: {type: string} - definition: {$ref: '#/components/schemas/SplitDefinition'} - origin: {$ref: '#/components/schemas/SplitOrigin'} - PrepareSplitRevision: - type: object - additionalProperties: false - required: [policy, splits] - properties: - policy: {$ref: '#/components/schemas/SplitPolicy'} - splits: {type: array, items: {$ref: '#/components/schemas/DraftSplit'}} - DraftRevision: - type: object - additionalProperties: false - required: [expected_revision] - properties: - expected_revision: {$ref: '#/components/schemas/Decimal'} - UpdateSplitHead: - type: object - additionalProperties: false - required: [revision, expected_revision, expected_head] - properties: - revision: {$ref: '#/components/schemas/OpaqueId'} - expected_revision: {$ref: '#/components/schemas/NullableRevision'} - expected_head: {$ref: '#/components/schemas/NullableGeneration'} - CommitTarget: - oneOf: - - type: object - required: [kind, snapshot] - properties: - kind: {const: snapshot} - snapshot: {$ref: '#/components/schemas/OpaqueId'} - - type: object - required: [kind, snapshot, revision] - properties: - kind: {const: split_revision} - snapshot: {$ref: '#/components/schemas/OpaqueId'} - revision: {$ref: '#/components/schemas/OpaqueId'} - DraftProgress: - type: object - required: [state] - properties: - state: {enum: [editing, evaluating, evaluated, prepared, committing, published, abandoned]} - input_revision: {$ref: '#/components/schemas/Decimal'} - next_page: {$ref: '#/components/schemas/Decimal'} - binding: {type: string} - target: {$ref: '#/components/schemas/CommitTarget'} - attempt: {$ref: '#/components/schemas/OpaqueId'} - SplitDraft: - type: object - required: [operation, snapshot, expected_revision, revision, policy, splits, progress] - properties: - operation: {$ref: '#/components/schemas/OpaqueId'} - snapshot: {$ref: '#/components/schemas/OpaqueId'} - expected_revision: {$ref: '#/components/schemas/NullableRevision'} - revision: {$ref: '#/components/schemas/Decimal'} - policy: {$ref: '#/components/schemas/SplitPolicy'} - splits: {type: array, items: {$ref: '#/components/schemas/DraftSplit'}} - progress: {$ref: '#/components/schemas/DraftProgress'} - SplitRevision: - type: object - required: [snapshot, revision, operation, evaluator_version, policy, splits] - properties: - snapshot: {$ref: '#/components/schemas/OpaqueId'} - revision: {$ref: '#/components/schemas/OpaqueId'} - operation: {$ref: '#/components/schemas/OpaqueId'} - evaluator_version: {const: 1} - policy: {$ref: '#/components/schemas/SplitPolicy'} - splits: - type: array - items: - type: object - required: [name, definition, origin, count] - properties: - name: {type: string} - definition: {$ref: '#/components/schemas/SplitDefinition'} - origin: {$ref: '#/components/schemas/SplitOrigin'} - count: {$ref: '#/components/schemas/Decimal'} - SplitHead: - type: object - required: [snapshot, revision, generation] - properties: - snapshot: {$ref: '#/components/schemas/OpaqueId'} - revision: {$ref: '#/components/schemas/NullableRevision'} - generation: {$ref: '#/components/schemas/NullableGeneration'} - SplitRevisionDeletion: - type: object - additionalProperties: false - required: [snapshot, revision, reclaim_after, complete] - properties: - snapshot: {$ref: '#/components/schemas/OpaqueId'} - revision: {$ref: '#/components/schemas/OpaqueId'} - reclaim_after: - description: Fixed Unix-seconds cleanup deadline while pending; null after terminal reclamation. - anyOf: - - {$ref: '#/components/schemas/Decimal'} - - {type: 'null'} - complete: {type: boolean} - SplitRevisionPage: - type: object - required: [entries, after] - properties: - after: {type: [string, 'null']} - entries: - type: array - maxItems: 100 - items: - type: object - required: [snapshot, revision, operation, split_count] - properties: - snapshot: {$ref: '#/components/schemas/OpaqueId'} - revision: {$ref: '#/components/schemas/OpaqueId'} - operation: {$ref: '#/components/schemas/OpaqueId'} - split_count: {type: integer} - SplitPreview: - type: object - required: [input_revision, next_page, complete, member_counts, validation] - properties: - input_revision: {$ref: '#/components/schemas/Decimal'} - next_page: {$ref: '#/components/schemas/Decimal'} - complete: {type: boolean} - member_counts: {type: array, items: {$ref: '#/components/schemas/Decimal'}} - validation: - type: object - required: [scanned, assigned, unassigned, overlapping, unassigned_examples, overlap_examples] - properties: - scanned: {$ref: '#/components/schemas/Decimal'} - assigned: {$ref: '#/components/schemas/Decimal'} - unassigned: {$ref: '#/components/schemas/Decimal'} - overlapping: {$ref: '#/components/schemas/Decimal'} - unassigned_examples: {type: array, items: {type: string}} - overlap_examples: {type: array, items: {type: string}} - CommitReceipt: - type: object - required: [attempt, target, expected_head, committed_head] - properties: - attempt: {$ref: '#/components/schemas/OpaqueId'} - target: {$ref: '#/components/schemas/CommitTarget'} - expected_head: {$ref: '#/components/schemas/NullableGeneration'} - committed_head: {type: string, pattern: '^[0-9a-f]{64}$'} - ReadPlanV2: - type: object - additionalProperties: false - required: [snapshot, selection, fields, batch_size, partition, shuffle, split] - description: >- - Snapshot latest/stable and null Split revision resolve into the returned - continuation. Selection and Split filtering precede shuffle and worker - partitioning. There is no persisted read plan, preparation or progress. - properties: - snapshot: {type: string, description: Concrete ID or latest/stable} - selection: {$ref: '#/components/schemas/Predicate'} - fields: {type: array, maxItems: 256, uniqueItems: true, items: {type: string}} - batch_size: {type: integer, minimum: 1, maximum: 10000} - partition: - type: object - additionalProperties: false - required: [worker, workers, rank, world_size] - properties: - worker: {type: integer, minimum: 0, maximum: 65535} - workers: {type: integer, minimum: 1, maximum: 65535} - rank: {type: integer, minimum: 0, maximum: 65535} - world_size: {type: integer, minimum: 1, maximum: 65535} - shuffle: - oneOf: - - {type: 'null'} - - type: object - additionalProperties: false - required: [seed, group_size] - properties: - seed: {$ref: '#/components/schemas/Decimal'} - group_size: {type: integer, minimum: 1, maximum: 100000} - split: - oneOf: - - {type: 'null'} - - type: object - additionalProperties: false - required: [revision, name] - properties: - revision: {$ref: '#/components/schemas/NullableRevision'} - name: {type: string} - ReadBinding: - type: object - required: [snapshot, split, cursor] - properties: - snapshot: {$ref: '#/components/schemas/OpaqueId'} - split: - oneOf: - - {type: 'null'} - - {$ref: '#/components/schemas/SplitSource'} - cursor: {type: string, maxLength: 349530} - ReadContinuation: - type: object - additionalProperties: false - required: [cursor] - properties: - cursor: {type: string, maxLength: 349530} - PrepareSnapshot: - type: object - additionalProperties: false - required: [parent, source, schema, samples, splits, policy] - properties: - parent: {$ref: '#/components/schemas/NullableRevision'} - source: {$ref: '#/components/schemas/CompositionSource'} - schema: - anyOf: - - {$ref: '#/components/schemas/CompositionSchema'} - - {type: 'null'} - samples: {type: array, items: {$ref: '#/components/schemas/SnapshotChange'}} - splits: {type: array, maxItems: 64, items: {$ref: '#/components/schemas/DraftSplit'}} - policy: {$ref: '#/components/schemas/SplitPolicy'} - SnapshotValue: - oneOf: - - type: object - additionalProperties: false - required: [kind, data] - properties: - kind: {const: inline} - data: {type: string, description: Canonical padded base64 of at most 65536 decoded bytes} - - type: object - additionalProperties: false - required: [kind, id, length, md5] - properties: - kind: {const: upload} - id: {$ref: '#/components/schemas/OpaqueId'} - length: {$ref: '#/components/schemas/Decimal'} - md5: {type: string, pattern: '^[0-9a-f]{32}$'} - - type: object - additionalProperties: false - required: [kind] - properties: - kind: {const: tombstone} - SnapshotField: - type: object - additionalProperties: false - required: [name, value] - properties: - name: {type: string, minLength: 1, description: At most 1024 UTF-8 bytes} - value: {$ref: '#/components/schemas/SnapshotValue'} - SnapshotChange: - oneOf: - - type: object - additionalProperties: false - required: [kind, id] - properties: - kind: {const: remove} - id: {type: string} - - type: object - additionalProperties: false - required: [kind, id, fields] - properties: - kind: {const: upsert} - id: {type: string} - fields: {type: array, maxItems: 256, items: {$ref: '#/components/schemas/SnapshotField'}} - description: Canonical base64 Sample IDs decode to 1 through 16384 bytes. Each ID occurs once; an upsert preserves unspecified effective parent fields. - PreparedSnapshot: - type: object - required: [snapshot, initial_split, sample_count] - properties: - snapshot: {$ref: '#/components/schemas/OpaqueId'} - initial_split: {$ref: '#/components/schemas/NullableRevision'} - sample_count: {$ref: '#/components/schemas/Decimal'} - CompositionSchema: - type: object - additionalProperties: false - required: - - version - - fields - properties: - version: - type: integer - minimum: 0 - maximum: 4294967295 - fields: - type: array - items: - $ref: '#/components/schemas/FieldDefinition' - maxItems: 256 - CompositionSource: - oneOf: - - type: object - additionalProperties: false - required: - - kind - properties: - kind: - const: all - - type: object - additionalProperties: false - required: - - kind - - predicate - properties: - kind: - const: selection - predicate: - $ref: '#/components/schemas/Predicate' - - type: object - additionalProperties: false - required: - - kind - - source - properties: - kind: - const: split - source: - $ref: '#/components/schemas/SplitSource' - selection: - anyOf: - - $ref: '#/components/schemas/Predicate' - - type: 'null' - CompositionUploadStatus: - type: object - additionalProperties: false - required: - - publication_ready - - staged_uploads - - candidate_uploads - properties: - publication_ready: {type: boolean} - staged_uploads: {$ref: '#/components/schemas/Decimal'} - candidate_uploads: {$ref: '#/components/schemas/Decimal'} - CompositionDraft: - type: object - additionalProperties: false - required: - - operation - - revision - - parent - - source - - parent_split_revision - - schema - - splits - - policy - - progress - - source_lease_deadline - - preparation - - uploads - properties: - operation: {$ref: '#/components/schemas/OpaqueId'} - revision: {$ref: '#/components/schemas/Decimal'} - parent: - $ref: '#/components/schemas/NullableRevision' - source: - $ref: '#/components/schemas/CompositionSource' - parent_split_revision: - $ref: '#/components/schemas/NullableRevision' - schema: - $ref: '#/components/schemas/CompositionSchema' - splits: {type: array, maxItems: 64, items: {$ref: '#/components/schemas/DraftSplit'}} - policy: - $ref: '#/components/schemas/SplitPolicy' - progress: - $ref: '#/components/schemas/DraftProgress' - source_lease_deadline: - <<: {$ref: '#/components/schemas/Decimal'} - description: Compatibility field; new drafts and edits use zero and hold no source lease. - preparation: - anyOf: - - type: object - additionalProperties: false - required: - - snapshot - - split_preview - properties: - snapshot: {$ref: '#/components/schemas/OpaqueId'} - split_preview: - anyOf: - - $ref: '#/components/schemas/SplitPreview' - - type: 'null' - - type: 'null' - uploads: - $ref: '#/components/schemas/CompositionUploadStatus' - CreateUpload: - type: object - additionalProperties: false - required: - - id - - length - - part_size - - md5 - properties: - id: {$ref: '#/components/schemas/OpaqueId'} - length: {$ref: '#/components/schemas/Decimal'} - part_size: - type: integer - minimum: 1 - maximum: 4194304 - md5: &composition006 - type: string - pattern: ^[0-9a-f]{32}$ - UploadPart: - type: object - additionalProperties: false - required: - - index - - length - - md5 - properties: - index: - type: integer - minimum: 0 - length: {$ref: '#/components/schemas/Decimal'} - md5: *composition006 - UploadStatus: - type: object - additionalProperties: false - required: - - id - - length - - part_size - - md5 - - state - - expires_at - - cleanup_complete - - parts - properties: - id: {$ref: '#/components/schemas/OpaqueId'} - length: {$ref: '#/components/schemas/Decimal'} - part_size: - type: integer - md5: *composition006 - state: - enum: - - open - - complete - - claimed - - aborted - expires_at: {$ref: '#/components/schemas/Decimal'} - cleanup_complete: {type: boolean} - parts: - type: array - items: - $ref: '#/components/schemas/UploadPart' - maxItems: 10000 - StableSnapshot: - type: object - additionalProperties: false - required: - - snapshot - properties: - snapshot: {$ref: '#/components/schemas/OpaqueId'} - RetainedSnapshot: - type: object - additionalProperties: false - required: - - snapshot - - retained - properties: - snapshot: {$ref: '#/components/schemas/OpaqueId'} - retained: {type: boolean} - SnapshotLifecycle: - type: object - required: [snapshot, operation, latest, stable, retained, active_read_lease, state, physical_cleanup_deferred, parent_protection] - properties: - snapshot: *id001 - operation: - type: [string, 'null'] - latest: - type: boolean - stable: - type: boolean - retained: - type: boolean - active_read_lease: - type: boolean - const: false - description: Compatibility field; readers do not register persistent activity. - reclaim_after: - type: [integer, 'null'] - minimum: 0 - description: Durable Unix-seconds deadline for destructive cleanup. - state: - type: string - enum: [not_started, pending, completed] - physical_cleanup_deferred: - type: boolean - parent_protection: - type: string - const: not_evaluated - SnapshotReclaim: - type: object - additionalProperties: false - required: - - snapshot - - phase - - complete - - physical_cleanup_deferred - properties: - snapshot: {$ref: '#/components/schemas/OpaqueId'} - phase: - enum: - - grace_period - - upload_owners - - split_drafts - - index_fences - - metadata - - complete - complete: {type: boolean} - reclaim_after: - type: integer - minimum: 0 - physical_cleanup_deferred: - const: true - CommitStatus: - type: object - additionalProperties: false - required: - - receipt - properties: - receipt: - anyOf: - - $ref: '#/components/schemas/CommitReceipt' - - type: 'null' diff --git a/doc/design/access-server/dataset/spec-crowdb-dataset.md b/doc/design/access-server/dataset/spec-crowdb-dataset.md deleted file mode 100644 index e0a5a7fcf..000000000 --- a/doc/design/access-server/dataset/spec-crowdb-dataset.md +++ /dev/null @@ -1,537 +0,0 @@ - - - -# CROWDB - Design: Dataset Specification - -This document is the normative Dataset contract. It defines the objects, -identities, state transitions, observable behavior, and compatibility rules -shared by Dataset authorities, HTTP clients, native clients, and SDKs. -The implementation design is described in -[Design: Dataset](design-crowdb-access-dataset.md); when the two documents -appear to disagree, this specification is authoritative. - -## Table of contents - -1. [Scope and terms](#1-scope-and-terms) -2. [Identity and names](#2-identity-and-names) -3. [Dataset records](#3-dataset-records) -4. [Operations](#4-operations) -5. [Manifest and field values](#5-manifest-and-field-values) -6. [Snapshot publication](#6-snapshot-publication) -7. [Selection, plans, and cursors](#7-selection-plans-and-cursors) -8. [Read behavior](#8-read-behavior) -9. [Retention and reclamation](#9-retention-and-reclamation) -10. [HTTP and native surfaces](#10-http-and-native-surfaces) -11. [Errors and compatibility](#11-errors-and-compatibility) -12. [Normative invariants](#12-normative-invariants) - -## 1. Scope and terms - -Dataset is a logical, sample-oriented collection. It stores metadata and -references to payload bytes; it does not define physical chunk layout, -replication, erasure coding, or framework tensor types. - -The following terms have one meaning throughout this specification: - -- **Dataset** is the named authority for immutable snapshots and their - metadata. -- **Snapshot** is an immutable view of a Dataset. It is identified by an - opaque `SnapshotId` and may name one parent Snapshot. -- **Manifest** is the complete, validated description of one Snapshot's - schema, sample membership, field values, partitions, and integrity metadata. -- **Schema** defines the fields, requiredness, and compatibility rules for a - Manifest. -- **Sample** is an opaque `sample_id` and its field map in one Snapshot. -- **Field** is a named value of one Sample. A missing field and a tombstone are - distinct: missing means no value is supplied by the visible ancestry; - tombstone explicitly hides an inherited value. -- **Locator** identifies payload bytes. Inline values are stored in metadata; - chunk locators are opaque strings interpreted only by the chunk subsystem. -- **Split revision** is an immutable complete set of named subsets, definitions, - membership and policies owned by one Snapshot. currentSplit is its mutable pointer. -- **Selection** identifies samples from one Snapshot. -- **Read Plan** binds a Selection, ordering, projection, batching, and optional - shuffle to one immutable Snapshot. -- **Cursor** records progress for one Read Plan and is acknowledged only after - the corresponding batch is accepted by the caller. -- **Deletion grace period** separates a durable Dataset, Snapshot or Split revision deletion - mark from destructive cleanup; it does not guarantee completion of every admitted read. - -Dataset does not infer semantics from a locator, a Snapshot ID's numeric -ordering, a sample ID's encoding, or an external source URI. - -## 2. Identity and names - -A Dataset identity is `(namespace, dataset_name)`. The initial authority uses -the configured `dataset-ns` root namespace for every Dataset. Names are unique -within that namespace and are compared as opaque UTF-8 names after validation. - -`SnapshotId` is immutable and opaque. `latest` and `stable` are mutable names -whose values are Snapshot IDs. A Read Plan, Cursor, or Manifest stores -the resolved Snapshot ID and never stores a mutable name in its place. - -A Snapshot has zero or one declared parent. Parentage is explicit; ID ordering -never implies ancestry. A child may add, replace, or tombstone samples and -fields while inheriting all values not overridden by its visible parent chain. -Any complete available same-Dataset Snapshot may be the parent, independently of latest. -Multiple parentless roots are permitted. Sample IDs are opaque bytes, represented -as canonical padded base64 in JSON; they MUST NOT be normalized as text or paths. - -## 3. Dataset records - -A Dataset record establishes that the identity exists and owns its metadata. -Creating an existing identity is a conflict. Opening a missing identity is a -not-found error. Whole-Dataset deletion requires the expected immutable Dataset ID. -A nonempty Dataset MUST be rejected: latest/stable, unreclaimed Snapshots (including -unpublished candidates), uploads, legacy drafts, or pending creation all count as -content. Empty deletion atomically marks the Dataset unavailable and persists a -`reclaim_after` deadline. Read/list hide it immediately; creation at that name is -rejected until the background worker completes retirement. Repetition preserves -the receipt and deadline. Name reuse creates a different Dataset ID; a stale -DELETE MUST conflict. Minimal identity receipts and retired child fences remain -for retry safety, rather than resurrecting old operations. - -Control v2's optional `X-Crowdb-Dataset-Id` active-instance guard MUST reject a -same-name replacement with 409 `dataset_replaced`. Malformed, zero, noncanonical -or duplicated values MUST fail with 400. Missing/marked identities return 404. -The advertised `dataset-identity-fence` capability covers both request admission -and creation reservation CAS, so deletion/name reuse cannot redirect a delayed -upload or Snapshot creator after its initial identity observation. Requests that -omit the header retain their existing contracts; deletion receipt/status retries -after marking use their own immutable ID without this active-instance guard. - -The Access Server setting `dataset.reclaim_grace_seconds` defaults to 1200 and -must be positive. Dataset, Snapshot and Split revision deletion use this value -when first marked. Existing deadlines are immutable across configuration changes -and restarts. Tests may set it to 1. Direct authorities expose -`with_reclaim_grace_seconds`; native clients default to 1200. - -A Snapshot record contains the Snapshot ID, optional parent, publication -operation identity, Manifest binding, and publication state. Publication states -are: - -- `Prepared`: the operation and candidate Snapshot are durable but the - Snapshot is not visible through `latest`. -- `Published`: the complete Manifest and binding are durable and the Snapshot - may become visible through `latest`. -- `Aborted`: the candidate is permanently unavailable for publication. - -A Manifest binding records the Snapshot it belongs to, its partition count, and -its integrity metadata. A binding for another Snapshot is invalid. Partitions -are an internal representation; readers observe one logical Manifest. - -## 4. Operations - -The Dataset authority provides these operations. Each operation is scoped by a -Dataset identity; operations that take a Snapshot ID MUST use the immutable ID, -not `latest` or `stable` as an implicit mutable reference. -The table summarizes the legacy publication/read surface. Control v2 adds bounded -catalog/tree inspection, immutable composition resources and independent Split revisions as -specified below and in its versioned OpenAPI contract. - -| Operation | Input | Successful result | -| --- | --- | --- | -| `CreateDataset` | Dataset identity | Creates the identity; duplicate creation conflicts. | -| `OpenDataset` | Dataset identity | Returns the Dataset record. | -| `PrepareSnapshot` | Parent, Manifest, operation token | Returns one durable candidate Snapshot ID that is not visible through `latest`. | -| `PublishSnapshot` | Parent, complete Manifest, operation token | Returns the published Snapshot ID and advances `latest` atomically. | -| `GetLatest` | Dataset identity | Returns the current Snapshot ID or no value for an empty Dataset. | -| `ListSnapshots` | Dataset identity | Returns published Snapshot records in declared ancestry order. | -| `GetSnapshot` | Snapshot ID | Returns the immutable Snapshot record. | -| `GetManifest` | Snapshot ID | Returns the complete validated logical Manifest. | -| `SetStable` | Published Snapshot ID | Sets the mutable `stable` name. | -| `RetainSnapshot` / `ReleaseSnapshot` | Snapshot ID | Adds or removes an explicit retention reference idempotently. | -| `ReclaimSnapshot` / `GetReclaimStatus` | Snapshot ID | Reclaims eligible state or returns durable reclaim progress. | -| `GetSample` / `GetBatch` | Snapshot ID, sample IDs, projection | Returns ordered samples and verified field values. | -| `Scan` | Read Plan, optional Cursor | Returns one bounded batch and the next Cursor. | -| `BindRead` / `NextBatch` | Query or continuation | Pins immutable versions or replays a caller-owned position without read-side writes. | -| `Cancel` | Bound read or service | Stops local admission and production without a remote reader release. | - -`PrepareSnapshot` and `PublishSnapshot` share the same operation token and -validation rules. A complete publication may be implemented as one call that -performs preparation and completion internally; it MUST have the same durable -state transitions. Data reads MUST NOT persist activity, selected-ID copies, progress, TTL or -reader registration. Checkpoints are owned by the client. - -## 5. Manifest and field values - -A Manifest MUST validate before publication. Validation rejects an empty -sample set, duplicate sample IDs, duplicate field names, missing required -fields, invalid checksums, invalid parent references, and incomplete or -inconsistent partition metadata. - -Within one Snapshot, `(sample_id, field_name)` has at most one value. A Schema -field can be required or optional. Required field names MUST remain in a child schema. Removing an inherited -value requires an explicit tombstone; readers never silently combine -incompatible schemas. A Manifest declares the complete current sample -membership: omitted samples disappear, while surviving samples may inherit -omitted optional fields through their declared parent. - -A Field value is exactly one of: - -- **Inline**: bounded bytes plus the declared digest. -- **Chunk**: an opaque location, range or stream metadata as defined by the - chunk subsystem, and the declared digest. -- **Tombstone**: an explicit deletion marker with no payload. - -Materialized reads verify the declared field digest before success. Streaming -reads return verified frames; whole-field integrity is established at field end -and partial output must be discarded if completion fails. External S3 or -Iceberg references are provenance and input metadata; they do not transfer -ownership of those systems' objects to Dataset. - -## 6. Snapshot publication - -Publication is an idempotent operation identified by an operation token. A -retry with the same token and equivalent parent and Manifest returns the same -Snapshot ID. A retry with divergent content is a conflict. - -The authority MUST perform publication in this order: - -1. Validate the Dataset, declared parent, operation token, and decoded Manifest. -2. Allocate or recover one candidate Snapshot ID for the operation. -3. Persist the prepared Snapshot record. -4. Persist every Manifest partition and the Snapshot binding. -5. Finish inspection metadata, initial Split membership and required payload owners. -6. Persist the published Snapshot record and install the final complete resource root. -7. Advance `latest` atomically to the complete Snapshot. - -Prepare completes the durable candidate without advancing latest. A failure -before step 6 leaves the candidate invisible and recoverable by the same -operation token. The Published flag alone does not authorize reads. A complete root -allows ID reads and references before step 7 succeeds; an -operation status alone is not proof that the head advanced. A successful Publish -completes the head transition. Retry identical content to finish an ambiguous -outcome; retries of historical publication MUST NOT rewind a newer head. A reader of `latest` MUST see either the previous -complete Snapshot or the new complete Snapshot, never a prepared or partial -Manifest. - -`stable` may be set only to an existing complete available Snapshot. Publishing a new -Snapshot never changes `stable` implicitly. A published Snapshot remains -addressable by ID until reclamation removes its metadata under Section 9. -Complete results that never won latest, including head-CAS losers, MUST also be -readable by ID, enumerable and usable as parents or stable targets. Preparation -records alone MUST NOT authorize these operations. Draft abandonment or supersession -MUST preserve the complete result and its payload owners. Incomplete cleanup MUST -seal the same final resource key that completion conditionally installs, preventing -late finalization from exposing reclaimed data. - -Control v2 separates candidate creation from publication attempts. An attempt MUST -bind the exact candidate and expected opaque head generation. Initial Split -membership and candidate upload ownership MUST be ready before the head transition. -New Snapshot publication MUST NOT persist attempt/response ledgers, success proofs -or draft publication progress. A response may echo the compatibility attempt ID; -it does not establish historical response replay. Retrying the same current target -can confirm its effect; a stale request after a later publication MUST NOT rewind -latest. A head conflict MUST return the observed latest/stable/generation without -a confirmation query. A reviewed retry uses that observation with caller-chosen -content; the server MUST NOT rebase. Unknown storage outcomes MUST propagate. -Existing legacy receipts remain readable without backfill. -Native Composition resource preparation MUST accept complete caller-owned inputs -and a fixed Snapshot ID without persisting editing or evaluation progress. Its -fixed source/Split revisions MUST survive retries unchanged. It MUST install the -complete resource root only after final Manifest/index, initial Split membership -and managed payload owners are ready; it MUST NOT move latest/stable. Input and -final Manifest are bounded to 8 MiB and computation to 30 seconds. Incomplete -computation is redone; identical complete content is reusable with zero writes. -Different resource content, including its initial Split binding, MUST conflict. -Unknown completion outcomes MUST NOT cause a confirmation query. REST exposes -this operation through `PUT snapshots/{snapshot}` with the -`composition-preparation` capability when uploads are configured. The JSON body -is also bounded to 8 MiB. Parent/schema are explicit nullable fields. Upload input -MUST include completed id/length/md5; DTO conversion MUST NOT add metadata queries. -The authority validates unique upload descriptors and returns conflict on mismatch. -Rust/Python SDKs MUST retain the same ID/full input without status queries or a -workflow fallback. Legacy Composition begin/edit/evaluate/prepare/publish and -cleanup routes return 501 before storage; only observation and abandonment remain -for old records. Background cleanup MUST rediscover old draft, work-page and edit -registration facts without storing candidate, scratch, edit or upload-retirement -progress. A lost local cursor repeats idempotent retirement and preserves complete -resource ownership and permanent fences against delayed writes. - -Composition and Split status reads MUST NOT reconstruct a publication request, -help edit receipts or persist recovery state. Legacy Committing drafts are -presented as Prepared without rewriting the stored draft or querying attempt, -receipt or head records. A caller may select a complete resource with the direct head API; this does not -prove that the historical request changed a head. -Direct Snapshot head updates MUST accept the complete Snapshot ID and explicit -nullable expected generation without an operation or draft identity. They MUST -preserve stable and require the `snapshot-head-update` capability in portable SDKs. -Their response and conflict observation use the latest/stable/generation head -shape. Unsupported direct updates MUST NOT fall back to composition or v1 writes. - -Publishing an independent Split revision MUST update only its Snapshot's currentSplit -pointer with an expected revision/head CAS. It MUST NOT create a data Snapshot or -change latest/stable. Old revisions remain addressable until explicit deletion -or owner reclamation. -Every complete immutable Split revision MUST also be readable by ID, enumerable -and usable as a frozen source before publication or after a publication conflict. -Abandoning a draft MUST NOT remove its complete result. A missing or cleanup-sealed -final revision record is unavailable; incomplete membership pages alone do not -constitute a result. Snapshot deletion admission still applies to each new request. -Independent Split publication MUST NOT persist an attempt, response receipt or -draft publication progress. A retry that observes the same current revision can -confirm the current effect; if a later revision has replaced it, the stale expected -base MUST fail without rewinding currentSplit. It does not promise historical -response replay. Unknown storage outcomes MUST propagate without a confirmation -query. Existing legacy receipts remain readable; new Split attempts are absent -from commit-status history. -Independent Split preparation MUST accept caller-owned revision ID, policy and -complete definitions through `PUT snapshots/{snapshot}/split-revisions/{revision}` -with the `split-preparation` capability. It MUST NOT persist editing/evaluation -progress or invoke draft APIs. Input and final bindings are bounded to 1 MiB and -computation to 30 seconds. An identical complete result MUST be reusable without -recomputation; different input at that ID MUST conflict. Incomplete work is retried -from client inputs. SDKs MUST retain the same ID/input and never downgrade to a -workflow mutation. Standalone begin/edit/evaluate/prepare/publish and cleanup -routes return 501; legacy draft observation and abandonment remain available. -Background cleanup MUST use local revision/page/item cursors and retain permanent -final-root and member fences. Reader-visible Snapshot owners wait for their -persisted deletion grace; incomplete content sealed at its final root has no -admitted readers and MUST NOT wait for a missing deadline. -Direct Split head updates MUST accept the complete resource ID and explicit -nullable expected revision and head generation without an operation/attempt ID or -draft lookup. A failed head condition MUST return the observed revision/generation, -including absence, without a confirmation query. The caller decides whether the -observed base is suitable for another update; the SDK MUST NOT rebase automatically -or fall back to a draft write when the direct-update capability is unavailable. - -## 7. Selection, plans, and cursors - -A Selection is evaluated against exactly one Snapshot. Supported selection -forms are explicit sample IDs, prefix or range, and bounded metadata predicates; -compound forms preserve their declared order and duplicate policy. Invalid or -ambiguous selections are rejected before payload reads. - -A Read Plan contains the resolved Snapshot ID, Selection, ordering, projection, -batch size, and optional deterministic shuffle parameters. The plan identity -includes all of those values. A plan cannot be resumed with a different -Snapshot, Selection, projection, ordering, or shuffle configuration. -Control v2 additionally pins a selected Split revision/name. Membership filtering -precedes shuffle and worker partitioning, which precede payload reads. A currentSplit -change MUST NOT alter an admitted plan or composition source. - -A Cursor is either at the start or contains a plan-bound group and offset. The -caller acknowledges a batch by saving the next Cursor in its own checkpoint. A failed or -unacknowledged batch may be replayed; a successful acknowledgement may not be -skipped or applied to another plan. - -Batch size, metadata bytes, prefetch, in-flight requests, and shuffle memory -are bounded. The authority rejects a plan that exceeds configured limits before -reading payload bytes. - -## 8. Read behavior - -`GetSample`, `GetBatch`, and bounded Scan first resolve Manifest metadata, -selection, projection, and ancestry, then fetch only the requested values. -Requested sample and projection order is preserved unless the plan explicitly -requests another ordering. Unknown samples, invalid projections, checksum -mismatches, and malformed cursors are errors. - -Each request checks authoritative snapshot availability at admission and MUST NOT -register, renew or release persistent reader activity. A durable deletion mark -rejects new requests. Previously admitted transfers may continue; neither EOF nor -cancellation waits for a remote cleanup operation. Local byte/request credits and -caller-owned buffers retain their existing lifetime guarantees. - -HTTP and native reads use the same authority semantics. Transport framing, -serialization, and topology discovery may differ, but the resolved Snapshot, -logical samples, field values, checksums, cursor result, and error class are -equivalent. - -## 9. Retention and reclamation - -`retain` and `release` apply to an immutable Snapshot ID. The latest Snapshot, -stable Snapshot, retained Snapshots, parents of live complete Snapshots on any -branch are protected from reclamation. Active reads do not prevent deletion. -The parent/child index MUST precede final child completion, which MUST recheck parent -availability. An indexed child protects its parent until terminal retirement even -while its own completion is pending; cleanup removes this conservative protection. - -A new Snapshot resource MUST persist its fixed twenty-four-hour `gc_after` -retirement eligibility deadline in its first write; content bindings inherit it. -Identical retries and publication-state changes MUST NOT renew it. Bounded -background scanning includes resources without a content binding and MAY mark -an expired complete or interrupted result only through the same authoritative -reference checks and publication fencing as explicit reclamation. Protected -results remain available; expiry eligibility alone is not a read-admission TTL. -Legacy resources inherit an existing binding deadline; otherwise their first -background observation assigns a fixed twenty-four-hour deadline. Unsupported -legacy opaque Manifest formats remain excluded from automatic retirement. - -Snapshot deletion first persists a terminal mark and a configured (default twenty-minute) -`reclaim_after` deadline. -Cleanup MUST NOT remove snapshot-owned metadata or payload references before -that deadline. The default applies to new marks; existing persisted absolute -deadlines remain unchanged. Restart and retry MUST preserve it. No Access Server or client -acknowledgements are required. Reads that exceed the grace period may return -an error after partial delivery; they MUST NOT report successful completion or -switch snapshots. Deletion completion does not revoke bytes already delivered. - -Reclamation is idempotent. It removes only unreachable Dataset-owned metadata -and payload references. A payload shared by two retained or readable -Snapshots remains available until its final Dataset owner is unreachable. -External S3, Iceberg, or other source references are never deleted by Dataset -reclamation. - -A reclaim status record reports the Snapshot, pending or completed state, and -reclaimed-count progress. Repeated requests and authority restarts preserve -that status and do not double-delete a payload. - -Complete Split revisions MUST default to reclamation with their owning Snapshot; -they MUST NOT acquire an independent automatic TTL. Explicit revision deletion -MUST name both Snapshot and revision IDs. The current Split head and the Snapshot's -initial Split revision MUST be protected, even when the initial revision is no -longer current. Deletion MUST close new revision reads and source admission, -preserve its fixed configured (default twenty-minute) grace across retries and restart, and fence -publishers admitted against an older head generation. Deletion state and exact -page bindings MUST reside on the existing revision resource. Deletion MUST NOT -create a reader session, request ledger or separate deletion task. - -Background cleanup MUST discover marked and terminal revision resources in bounded -pages with local cursors and preserve permanent fences against delayed member -writes. If the owning Snapshot has closed all read admission and completed its -own configured (default twenty-minute) grace, cleanup MAY use that owner fence before the revision's -later deadline. Revision deletion removes only Split membership and revision -metadata; it MUST NOT delete Samples, payload ownership or chunk data. - -## 10. HTTP and native surfaces - -The versioned public wire schemas, resource paths, binary framing and retry -contract are defined by [Dataset HTTP API v1](openapi-crowdb-dataset-v1.yaml) and -[Dataset control API v2](openapi-crowdb-dataset-v2.yaml). -The public surface uses JSON control records and binary payload streams; -persisted authority bincode records are not public DTOs. - -Control v2 MUST negotiate capabilities before mutation and MUST NOT downgrade -Split-aware inputs to v1. Required nullable preconditions MUST be supplied explicitly. -Unsigned 64-bit control values use canonical decimal strings. Legacy routes and -read-plan interpretations remain compatible; binary framing is unchanged. - -`DELETE snapshots/{snapshot}/split-revisions/{revision}` MUST accept an empty body -and require `split-revision-delete` in portable SDKs. A protected current or initial -revision returns 409; an absent resource or a new read of a closed revision returns -404. Existing bounded-work errors may take precedence over the closed-revision -404 response. Successful deletion and repeated deletion of a terminal resource return 200 -with `snapshot`, `revision`, `complete` and required nullable `reclaim_after`. -Pending status MUST carry the original absolute Unix-seconds deadline as a -canonical decimal string and `complete=false`; terminal status MUST carry null -and `complete=true`. Clients MUST NOT substitute legacy draft abandonment when -the capability or route is unavailable. Retrying after an unknown outcome uses -the same two IDs without a status or head query. The native adapter MUST add no -metadata lookup or implicit mutation retry to the shared authority operation. - -Composition takes retained complete caller inputs and creates an immutable -resource without edit/evaluation progress. Parent data defaults to all Samples; -Selection, frozen named Split and their intersection can restrict it. Source -S1/r2/train creates a child with parent S1 and frozen source membership. Inherited -Splits intersect surviving IDs; new IDs are unassigned until explicit assignment or -reevaluation. Defaults are disjoint/allow-unassigned; inheritance is explicit and -frozen to the supplied revision. Weighted -partition follows the versioned SHA-256 algorithm in the -[control lifecycle](design-dataset-control.md#3-split-evaluation-and-publication). - -Inspection MUST bound underlying metadata work, not merely paginate a fully loaded -Manifest. Committed inventory/children, Schema, Samples and projected field metadata -are separate bounded reads. Legacy inspection preparation MUST rebuild complete -metadata in one request without durable preparation progress. The source record -limit is 9 MiB, the embedded Manifest limit is 8 MiB and the request deadline is -thirty seconds. Final index writes MUST remain bounded at sixteen pending rows and -2 MiB of encoded keys and values. The immutable summary root MUST become visible -only after all rows and discovery entries are complete; cleanup MUST prevent a -delayed finalizer from replacing a retired root. Failures propagate without -confirmation reads, and retries rebuild from the source or reuse an existing root -without writes. Historical progress MAY be observed when the root is absent, but -MUST NOT drive or override a complete rebuild. Successful rebuilds return zero -revision and row-count compatibility fields. Unsupported or oversized sources MUST -return errors rather than empty results. Preview is opt-in, capped at 64 KiB and -MUST NOT acknowledge read consumption. - -The HTTP surface is the portable wire contract. It exposes Dataset creation and -opening, latest and Snapshot inspection, publication and preparation, Manifest -retrieval, retention, stable selection, reclamation status, bounded reads, read -queries and client-owned continuations. Cancellation is request-local. - -The native surface exposes the same operations through typed calls. Every -operation identifies the Dataset explicitly or through a bound service and -carries a Snapshot ID where the operation is Snapshot-scoped. - -HTTP uploads MUST use a Dataset-scoped opaque session ID, declared length, -part size and MD5. Numbered parts are immutable: identical retries reuse the -receipt; divergent bytes conflict. Declared part sizes are 1 byte through 4 MiB; -the final part may be shorter. At most 10000 parts are accepted. Completion -independently verifies part lengths/digests and the whole digest before producing a completed reference. -Preparation rejects incomplete, aborted or foreign references. Claimed payloads -remain protected until every candidate/Snapshot owner is released. Unowned -sessions expire; physical cleanup may remain pending and MUST report that state. - -Public reads bind the resolved immutable Snapshot, optional Split revision, -selection, projection, shuffle and worker partition in a caller-owned continuation. -Bind and next requests MUST NOT create persistent or required in-memory sessions. -The same continuation MUST replay its batch across independent server instances. -A successful consumer saves the next continuation; fetching, EOF and releasing -buffers MUST NOT advance shared progress. There is no remote ack, resume or close. -Missing next continuation means end of selection. Cancellation affects only the -current request. Each request authenticates and checks deletion admission. - -Selection windows, shuffle memory and request deadlines MUST remain bounded. -Groups admit at most 100000 selected IDs and 8 MiB of ID bytes; token JSON is capped -at 256 KiB before base64 encoding. Selected IDs are not copied into persistent -read records or the token. JSON metadata and binary headers are capped at 8 MiB. -Large fields stream rather than being materialized as a whole batch. - -Binary frame integrity is established at each frame, field integrity at field -end, and successful batch completion requires the batch terminator followed by -HTTP EOF. A caller-owned writer may contain a verified prefix on failure and -MUST discard partial output. File convenience downloads expose their destination -only after successful verification. SDK cancellation propagates to the same -Rust operation and returns producer resources; caller-retained completed data -has its own application lifetime. - -A configured HTTP bearer credential protects all Dataset routes. Deployments -that need separate management and data policies may enforce that distinction at -the service boundary without changing Dataset semantics. - -## 11. Errors and compatibility - -Errors preserve their logical class across surfaces: - -- invalid input, Manifest, selection, or cursor → client-invalid; -- missing Dataset, Snapshot, or Manifest → not-found; -- duplicate operation, parent race, protected Snapshot, or stale cursor → - conflict; -- checksum or corrupt durable state → integrity/unavailable; -- cancelled read or expired admission → cancelled; -- delivery-window or configured bound exceeded → backpressure/resource - exhausted; -- transient storage or service failure → unavailable and retryable only when - the operation is documented idempotent. - -Wire records are versioned. Unknown optional fields may be ignored, but a -reader MUST reject an unsupported version, missing required field, invalid -checksum, or incompatible state transition. New surfaces must preserve the -logical behavior in this specification before adding implementation-specific -extensions. - -## 12. Normative invariants - -- **DATA-S1 — Snapshot identity:** every operation resolves one immutable - Snapshot ID. -- **DATA-S2 — Complete visibility:** `latest` exposes only a published, - complete Manifest. -- **DATA-S3 — Explicit ancestry:** only declared parent links provide - inheritance. -- **DATA-S4 — Manifest uniqueness:** one Snapshot has one valid value per - `(sample_id, field_name)`. -- **DATA-S5 — Opaque location:** Dataset never parses or manufactures chunk - locations. -- **DATA-S6 — Plan binding:** a Cursor is valid only for its exact Read Plan. -- **DATA-S7 — Bounded delivery:** configured memory, batch, and in-flight - limits are enforced before payload admission. -- **DATA-S8 — Delayed reclamation:** deletion rejects new reads and defers - cleanup to its durable grace deadline; overdue reads may fail without - successful truncation or snapshot substitution. -- **DATA-S9 — Ownership-safe GC:** reclamation deletes only unowned Dataset - data and never external source objects. -- **DATA-S10 — Surface parity:** HTTP and native surfaces preserve the same - logical results and error classes. diff --git a/doc/design/access-server/design-crowdb-access-server.md b/doc/design/access-server/design-crowdb-access-server.md deleted file mode 100644 index bfe282e31..000000000 --- a/doc/design/access-server/design-crowdb-access-server.md +++ /dev/null @@ -1,320 +0,0 @@ - - - -# CROWDB - Design: Access Server - -CROWDB presents three first-class upper access models: S3, Iceberg, and -Dataset. Each model owns its semantics while sharing the same durable CROWDB -storage substrate. HTTP is the portable server interface. A trusted internal -native runtime exposes topology-aware storage and host-buffer delivery, with -S3 as its first typed frontend. Dataset retains its own semantic authority -when attaching to this shared runtime. - -## Table of contents - -1. [Intent](#1-intent) -2. [Access portfolio](#2-access-portfolio) -3. [Architecture](#3-architecture) -4. [Semantic ownership](#4-semantic-ownership) -5. [Data paths](#5-data-paths) -6. [Shared storage boundary](#6-shared-storage-boundary) -7. [Scale-out and security](#7-scale-out-and-security) -8. [Correctness invariants](#8-correctness-invariants) -9. [Direction and risks](#9-direction-and-risks) - -## 1. Intent - -The access architecture gives different users the model that fits their work -without forcing all data through an object-store abstraction: - -- existing applications use an S3-compatible object interface; -- table engines use an Iceberg catalog and table interface; and -- AI and data-intensive applications use Dataset concepts and can choose a - portable HTTP path or a direct native path. - -The Access Server terminates the HTTP surfaces. It is a stateless translator, -not a storage authority or a required relay for every client. The HTTP engine -is an implementation choice and is not part of the architecture contract. - -## 2. Access portfolio - -| Model | Semantic authority | Portable surface | Performance surface | -| ------- | ------------------------------------------ | ---------------- | ---------------------------- | -| S3 | Buckets, objects, multipart, S3 lifecycle | HTTP | Trusted native storage SDK | -| Iceberg | Catalogs, namespaces, tables, snapshots | HTTP | Delegated immutable FileIO | -| Dataset | Namespaces, snapshots, samples, fields | HTTP | Topology-aware native client | - -Related designs: - -- [S3](s3/design-crowdb-access-s3.md) -- [Trusted native SDK](design-crowdb-native-sdk.md) -- [Iceberg](iceberge/design-crowdb-iceberg.md) -- [Dataset](dataset/design-crowdb-access-dataset.md) -- [Accelerated transfer](s3/design-crowdb-access-s3-rdma.md) -- [Chunk I/O](../chunkio/design-crowdb-chunkio.md) -- [Chunk-KV](../chunkds/design-crowdb-chunk-kv.md) -- [RPC](../rpc/design-crowdb-rpc.md) - -## 3. Architecture - -```text - portable, stateless server access - - S3 client Iceberg client Dataset HTTP client - | HTTP | HTTP | HTTP - v v v -+------------+ +---------------+ +---------------+ -| S3 module | | Iceberg module| | Dataset module| -+------+-----+ +-------+-------+ +-------+-------+ - | | | - +---------------------+------------------------+ - | - CROWDB storage clients - | - v - Chunk-KV + chunk services - ^ - | - routed RPC / optional RDMA data plane - | - Shared native runtime - ^ - | - AI runtime / CPU or GPU memory - - topology-aware direct access -``` - -S3 and Iceberg expose HTTP protocols. The trusted native S3 facade reuses S3 -execution over the shared runtime and contacts metadata/data services directly. -Dataset's native adapter uses this same runtime while retaining Dataset -selection and acknowledgement semantics. The native runtime removes the -Access Server routing and payload hop. - -## 4. Semantic ownership - -S3, Iceberg, and Dataset are peers. None is implemented as a metadata convention -on another model: - -- S3 owns object names, overwrite behavior, listing, and S3 compatibility. -- Iceberg owns catalog, table, snapshot, commit, immutable file, and - reachability semantics. Its S3-shaped file locations do not make S3 the - authority. -- Dataset owns namespace-scoped identity, snapshot publication, sample and field - selection, batching, streaming, prefetch, and bounded delivery. - -The models may share physical bytes only through an explicit reference contract. -Sharing never transfers namespace, publication, authorization, or reclamation -authority. - -Each HTTP model has an independent listener, request types, configuration, -authentication policy, admission budget, metrics, and lifecycle. Shared -utilities may manage buffers, credentials, errors, and shutdown, but cannot -reinterpret model semantics. - -The `crowdb-access-server` executable starts the S3 and Iceberg listeners -together by default. The container supervises one access process for both -ports. Explicit `s3` and `iceberg` commands are reserved for focused tests -and management operations. -The combined entry point signals the other listener when either service -returns, then waits for both services to drain their owned small-write pools -before exiting. An unexpected Iceberg recovery or GC worker exit fails the -Iceberg listener, so the combined process also stops the S3 listener. A -terminated small-write manager fails its owning listener; an empty pipeline -route set remains recoverable while the manager continues restarting pipelines. - -The listeners own separate `ChunkIoClient` instances and small-write pools. -New S3 chunks use type `S3`; new Iceberg file chunks use type `IcebergTable`. -Each protocol may override the legacy common small-write policy and select -its own large-write EC, memory, prefetch, and mirror settings. Each protocol -library constructs its own storage clients and chooses foreground writers; -the executable handles HTTP framing, listener startup, and process shutdown. -Iceberg file-record construction also lives in the Iceberg library, including -the format rule shared by ordinary uploads and multipart completion. -Iceberg GC uses a separate chunk client with Iceberg's small-write policy. -The Iceberg library owns its GC storage budget and file-block adapters. -After proving a payload range unreachable, GC hands its exact byte range to -ChunkDB. A successful deletion notification completes Access's responsibility; -Access does not poll physical reclamation or retain progress for DiskDB capacity -release. `ReclaimOutcome::Accepted` means the notification was accepted (or the -chunk is already absent), not that disk space is reusable. Physical range -retirement, layout grace and compaction belong to ChunkDB and DiskDB. -Each small-write policy also chooses its chunk capacity; each protocol's -large-write policy chooses its own maximum chunk size. The deployment profile -sets RPC workers and DiskIO connections independently of these data limits. - -## 5. Data paths - -The ordinary path streams bounded data through the Access Server over HTTP. It -is the universal path and remains available without specialized hardware. - -S3 and Iceberg PUT and multipart part uploads share an object-scoped transfer -driver. One task polls the protocol's body decoder and a bounded write -consumer; it can continue receiving while earlier strips are in flight. -Completed receive owners are offered to the writer before their logical -payload views enter a separate MD5 and optional SHA-256 worker. The writer -uses separately bounded held-buffer and in-flight strip windows, and -processes strip completion in submission order. Small objects use the same -handoff and digest completion but retain their protocol-owned shared -small-write pipeline after the handoff. Authentication, checksum declarations, -metadata publication, multipart authority, and cleanup remain with each -protocol. The [Iceberg upload-flow design](iceberge/design-crowdb-iceberg-upload-flow.md) -describes scheduling and stage measurements in detail. - -The Dataset native path embeds routing, retry, bounded planning, streaming, and -buffer ownership in the application. It resolves one immutable dataset -snapshot and distributes work directly to responsible CROWDB services. A -topology view routes work but never grants placement authority. - -Native S3 and Dataset access use the shared CROWDB runtime and RC SEND/RECV -transport with registered host buffers. This path avoids an Access Server payload -bounce while preserving metadata authority, publication and integrity rules. -Host-memory ownership and admission remain explicit. GPU destinations, cuObject -negotiation and one-sided client-memory transfer are outside the current contract. -See [Native RDMA access](s3/design-crowdb-access-s3-rdma.md) and -[RPC RDMA](../rpc/design-crowdb-rpc-rdma.md). - -## 6. Shared storage boundary - -Chunk-KV is the common metadata and ordered-index substrate. Chunk services are -the common byte, durability, placement, repair, and buffer-ownership substrate. -Each access model stores only its own logical authority and opaque references to -data below this boundary. - -Physical segments, disks, replicas, and erasure-coding shards remain storage -concerns. Native clients observe versioned topology for scheduling and routing, -but cannot publish or alter physical placement. - -## 7. Scale-out and security - -Access Server instances keep no authoritative namespace or operation outcome. -Durable state lives in CROWDB, so any healthy instance can serve a later request -or retry. Protocol-local caches are discardable hints. - -Trusted internal native clients share Access Server's trust domain and receive -explicit tenant and operation context from their embedding system. They do not -obtain the S3 secret database or verify SigV4. External HTTP authentication -remains in the HTTP frontend. Dataset HTTP capability contracts remain in the -Dataset design; native embedding does not depend on external-user capability -issuance. Topology and memory descriptors alone grant no access. - -All paths bound connections, metadata work, stream windows, retained buffers, -registered memory, retries, and background work independently of total object, -table, or dataset size. - -The process has a separate HTTP health listener configured by `HealthConfig` -(or `CROWDB_ACCESS_HEALTH_LISTEN`), defaulting to loopback port 9094. Its -`/_crowdb/health/live` and `/_crowdb/health/ready` routes are independent of the -S3 and Iceberg listeners. All three ports are distinct. Health serves no -user-data routes; S3 requests cannot bypass authentication through a health -path. Readiness observes the same storage dependencies and cleanup backlog -as the data dispatcher. Retained launch records identify the dedicated health -address; a missing address requires deployment reconciliation before restart. - -One Access background sampler runs every second and publishes an immutable -health snapshot through ArcSwap. It independently checks the routed Chunk-KV -bucket namespace, ChunkDB and online disks through the process's storage clients. -Metadata and chunk checks run concurrently with a two-second dependency budget; -hardware discovery, ChunkDB and disk headers share that chunk-check budget. A -2100 ms outer watchdog bounds the full round and cancels unfinished work. Only -one sampling round can run at a time; missed one-second ticks are skipped rather -than overlapping or replayed. A failed round publishes unavailable evidence and -logs its generation, configured budgets, elapsed time and dependency errors. -Sampling continues after every failure, and a later successful generation -recovers readiness without a restart. Failure never extends the probe budget, -adds a retry or changes the response to Ready. Physical checks read existing disk -bytes without creating objects, allocating chunks or repairing storage. -ChunkDB point checks use the validated service route and actual owner/storage -reads. They do not initiate an ordinary periodic full registry refresh before -the query; cold discovery and failed-route refresh retain the existing rules -and shared deadline. A cached route never substitutes for a current response. -Each physical check reads a 4096-byte disk header through the storage client's -bounded priority connection group. It retains the same authoritative disk route -and shares the chunk check's total deadline. Bulk response bytes on a normal -connection cannot delay that header through TCP response ordering. A rejected, -short or late header still fails the observation; the priority lane supplies no -additional retry or grace period. -Credential-cache freshness, listener state, native-pool pressure and cleanup -backlog are included in the same snapshot. Application requests only record -their own errors and metrics; their outcomes cannot overwrite health evidence. - -Health requests read the snapshot without dependency I/O or a new sampling task. -Responses retain the check timestamp, generation, age, exact flags and diagnostic. -New observations replace old failures or successes. Evidence older than three -seconds from the sampling start, or an absent first sample, reports Unknown and -HTTP 503; readers never extend its validity or clear it to Ready. Completion -does not renew the start timestamp. Consecutive slow successful rounds can leave -an expired-evidence interval before the next round completes; retain that 503 -and recover only when a new current generation is published. Process stop is an immediate lifecycle -fence even when the cached sample was healthy. The sampler is cancelled with the -listener. Recovery needs neither application traffic nor an Access restart. - -Web independently samples local monitor evidence, signed protocol reads and -service diagnostics in its backend. Its node-observation endpoint only returns -that bounded-age snapshot, including original operation identities and producer -timestamps. Status readers aggregate cached member evidence with current -committed intent, so stop/delete and ownership fences override older samples. -Frontend query failures do not mutate these backend health observations. - -## 8. Correctness invariants - -- **AS-I1 — First-class models:** S3, Iceberg, and Dataset each own their - namespace, publication, authorization, compatibility, and reclamation rules. -- **AS-I2 — HTTP boundary:** S3 and Iceberg are HTTP server protocols; Dataset - HTTP is the portable Dataset server surface. -- **AS-I3 — Native directness:** trusted native access reaches responsible - CROWDB services without an Access Server request or payload hop. -- **AS-I4 — Semantic equivalence:** Dataset HTTP, native CPU, and accelerated - GPU paths select the same immutable snapshot and logical data. -- **AS-I5 — Storage authority:** no Access Server module or native client owns - physical chunk placement, replicas, erasure-coding shards, or disks. -- **AS-I6 — Stateless server:** an authoritative operation survives loss of the - Access Server instance that began it. -- **AS-I7 — Bounded ownership:** retained state is bounded independently of - object, table, or dataset size. -- **AS-I8 — Authorized direct access:** topology, endpoints, and memory - descriptors are not capabilities by themselves. -- **AS-I9 — Completion lifetime:** every buffer and registration outlives all - socket, RPC, storage, NIC, and GPU operations that reference it. -- **AS-I10 — Protocol storage ownership:** S3 and Iceberg use distinct chunk - types and independently admitted foreground write pools; each library owns - its file or object authority and storage policy. -- **AS-I11 — Shared upload progress:** one body reader and one write consumer - own each S3 or Iceberg PUT or multipart part. Socket readiness and write - completion can each resume an idle transfer without changing protocol - publication authority. - -Coordination follows the semantic owner, rather than the Access Server process: - -- Dataset read admission is read-only. Clients retain continuation progress; - fetching, acknowledgements and snapshot activity create no reader records. - Existing write-side draft journals recover partial work and fence late writes - after cleanup. -- Dataset upload expiry scans upload authorities; creation does not enter a - global registry. Shared payload owner records prevent one snapshot's retirement - from reclaiming another snapshot's bytes. -- Iceberg streamed uploads use independent session bounds and per-part writes. - Catalog-wide credit records are retained only for legacy recovery. Ordinary - table commits use one head CAS; GC fences protect committed file reachability. -- Namespace deletion rejects live children. The associated parent admission and - recovery state preserves that strict contract and is not a reader lease. -- S3 multipart coordinates accounting and exact completion selection within one - upload. Ordinary object reads do not register persistent activity. -- A failed CAS observation supplies the compared key, revision and value for an - allowed retry. Unknown transport outcomes follow the operation's retry contract; - application preconditions must not silently move to a newer base. Credential - bootstrap publishes one encrypted final resource through create-if-absent; - comparison losers reuse that value, while older peers and busy/unknown replies - require a direct resource read. It has no persisted preparation record. - -## 9. Direction and risks - -The main architectural direction is to keep HTTP broadly compatible while -making Dataset native access the performance path for AI workloads. RDMA and -GPU-direct delivery should reduce movement, not introduce a second semantic or -storage authority. - -The main risks are compatibility drift, stale client topology, capability -containment, cross-service backpressure, native resource lifetime, and uneven -GPU-direct hardware support. Every accelerated path must have a bounded CPU -fallback and must prove the same selection, integrity, and completion semantics. diff --git a/doc/design/access-server/design-crowdb-native-sdk.md b/doc/design/access-server/design-crowdb-native-sdk.md deleted file mode 100644 index c2467194d..000000000 --- a/doc/design/access-server/design-crowdb-native-sdk.md +++ /dev/null @@ -1,185 +0,0 @@ - - - -# CROWDB - Design: Trusted native storage SDK - -The native SDK embeds internal storage access in the application. Its shared -runtime, topology and host-buffer delivery support typed S3 and Dataset frontends. - -Depends on: [Access Server](design-crowdb-access-server.md), -[RDMA RPC](../rpc/design-crowdb-rpc-rdma.md), -[Chunk I/O](../chunkio/design-crowdb-chunkio.md). - -Satisfies: Access Server invariants AS-I3, AS-I5, AS-I7 and AS-I9. - -## Table of contents - -1. [Trust and semantic ownership](#1-trust-and-semantic-ownership) -2. [Shared runtime](#2-shared-runtime) -3. [Topology and selection](#3-topology-and-selection) -4. [Data and completion](#4-data-and-completion) -5. [Budgets and shutdown](#5-budgets-and-shutdown) -6. [Transport and integration](#6-transport-and-integration) -7. [Correctness invariants](#7-correctness-invariants) - -## 1. Trust and semantic ownership - -The embedding process is a trusted internal cluster component in the Access -Server trust domain. It supplies tenant and operation context and reaches -management, Chunk-KV, Chunkdb and DiskIO directly. Native operations do not -verify SigV4, receive a user-secret database or call Access Server to authorize -an operation. Network access to internal services belongs to the embedding -system's deployment boundary. - -`S3Executor` owns shared S3 namespace, metadata selection, conditions, -publication and multipart orchestration. HTTP supplies parsing, authentication -and HTTP/XML encoding; `S3Client` supplies typed context and results. Both -preserve S3 generation, checksum, visibility and uncertain-publication rules. -Copy captures a source generation. Multipart completion binds exact immutable -part generations. An admitted independent-part writer may acknowledge a later -replacement without changing that selection. Each accepted Complete execution -publishes the frozen selection through an unconditional object PUT, including -retries after success; it can overwrite a later ordinary PUT. No completion -response is recorded. [S3 publication](s3/design-crowdb-access-s3.md#4-publication-and-lifecycle) -defines the shared multipart resource modes and completion rules. - -`DatasetClient` attaches `DatasetAuthority`, `DatasetUploads` and -`DatasetReads` to these same clients. Complete Snapshot/Split preparation takes -caller-owned immutable inputs and resource identities. Snapshot completion follows -final upload-owner claims; it stores no editing or evaluation progress. Mutable -head updates use explicit expected bases and preserve structured conflict -observations. Complete results remain readable by ID independently of head -selection. Unknown mutations retain their resource identity, full input and base; -the SDK never retries or rebases automatically. Read selection, worker partition, -resource CAS and cleanup remain in the Dataset authority. - -## 2. Shared runtime - -`ClientRuntime` owns or borrows control, metadata and data clients. Business -handles share transport groups, discovery, topology observations and resource -budgets. Compatible writer policies share one bounded small-writer pool; -incompatible policies use distinct pools while sharing the transport owners. -S3 and Dataset chunks retain their respective types; native embedding introduces no generic user-data -chunk type. - -`connect` creates owned groups. `adopt_clients` transfers shutdown -responsibility for existing groups. `from_clients` borrows them, so shutting -down that facade cannot stop its original owner's workers. These ownership -choices are explicit rather than inferred from an Arc reference count. - -## 3. Topology and selection - -`TopologyView` retains catalog generation, independently versioned disk routes, -service observations and actual transport configuration/counters. Sources have -separate observation ages and are not a globally atomic topology epoch. An -independent refresh with unknown observation time reports unknown freshness. -Catalog regression or invalid pages cannot replace a valid cached generation. - -`ClientRuntime::read_flow_metrics` snapshots existing cumulative layout, stream -credit, storage-wait and frame-decode counters on the shared chunk reader. Writer -policy groups share these counters. Reading them performs no RPC or reset; a -delta covers the reader's concurrent work, not one request. Resource status -reports current charges and lifetime high water separately from compatibility -copy bytes. These observations do not measure all copies or per-trial peaks. - -`ResolvedObject` binds one published object record to a tenant and metadata -authority. `object_locality` returns a bounded page of logical extents with -layout revisions, fragment roles and rack/node/disk observations. Inspection -does not dial storage, publish placement or replace routing checks. Execution -uses current routes and layout fencing while preserving the selected object's -identity. A retained old view remains inspectable but cannot pin execution to -an old owner or force a deleted/expired layout into a read. - -## 4. Data and completion - -`ObjectBody` is a lazy pull stream of `BufferLease` values. Each lease exposes -request/stream identity, sequence, logical offset, length, allocation capacity -and registration identity. Storage frame/control bytes are excluded from the -logical object stream. `OwnedPayload` retains the allocation through RPC, -DiskIO, ChunkIO, metrics wrappers and S3 adaptation. - -Successful EOF requires all applicable length and integrity checks. Delivered -metadata or a valid prefix is not full-read success. Terminal errors remain -sticky across polling; early body drop is cancellation. Cancelling one pending -pull preserves its position when the body itself remains live. - -Ordinary input retains its allocation or uses a bounded adapter. A unique -producer allocation can be filled directly in final storage frame slots, -frozen and consumed by `write_framed`. Submitted input is immutable and retained -through physical completion. Acceptance of an input buffer is distinct from -durable object publication; only successful finalization establishes success. -A lost publication reply retains uncertainty and cannot authorize a blind -replay with a new identity. - -## 5. Budgets and shutdown - -Admission is lock-free and bounded across requests, producers and caller-held -allocation capacity. Shared views charge physical capacity once until the final -owner releases it. An allocation's registration budget is separate from SDK -held bytes, posted receives, SQ work requests and writer windows. - -Held-capacity exhaustion pauses production without blocking RPC completion -workers. Releasing the final owner returns capacity and wakes waiters; deadlines, -cancellation and shutdown remain observable. Unknown external owners reserve -bounded capacity before a compatibility copy. Open small strips can retain -input for replica repair after operation completion until seal or pool drain. - -Shutdown closes admission, drains owned writer groups and joins owned workers. -All waiters observe one shutdown result; cancelling a waiter does not cancel -the shared cleanup. Caller-held leases retain storage after shutdown, and final -release never reposts to a destroyed transport. A host-to-GPU copy holds its -host lease until the copy completes, independently of stream completion. - -## 6. Transport and integration - -Metadata and data groups retain their own TCP/RDMA policies. Disabled RDMA uses -TCP; eligible setup failures follow the configured fallback policy. Accepted -or partial RDMA reads fail terminally through routing, mirror/EC recovery and -object delivery; they cannot append a TCP suffix or switch replicas to conceal -an interrupted transfer. Ambiguous writes preserve uncertainty. - -Eligible registered producers avoid staging; eligible receives transfer their -allocation without a second SDK payload copy. Framing/control decoding, -alignment and EC construction/reconstruction retain their required copies. -The base lease API exposes host memory and opaque ownership, not verbs objects, -GPU pointers or RDMA READ/WRITE authority. - -The independent Dataset HTTP distribution has no dependency on this storage/FFI -runtime. A Dataset facade attaches to the common runtime and delivery contracts -while retaining Dataset authority, Snapshot/Split binding and caller-owned -checkpoints. Buffer release, download completion and saving a processing -checkpoint remain separate events. `BatchBody` delivers field metadata and -lazy payload leases with per-part and whole-field integrity checks. Read -admission checks authoritative Snapshot and Split deletion state without -persistent reader registration, renewal or release. A durable deletion mark -rejects new reads and records a fixed twenty-minute cleanup grace; admitted -streams may finish, and overdue reads may fail after reclamation. There is no -persistent read session or remote ack/close. A continuation replays the same -batch on an independent client; only the caller saves the next position. - -The optional native Python extension owns a shared runtime and exposes retained -immutable leases. Slicing shares allocation ownership; conversion to Python -bytes explicitly copies. Pulls are serialized per batch with a bounded request -channel and no automatic payload prefetch. Runtime shutdown preserves already -delivered leases. Forked processes create new runtimes. Python control commands -reuse the Rust facade and shared authority schema without another storage path. - -## 7. Correctness invariants - -- **NS-I1 — Trusted embedding:** native tenant/context execute inside the - internal-service trust domain without a per-operation HTTP authentication hop. -- **NS-I2 — Shared semantics:** HTTP and native frontends use the same selection, - durability, publication, multipart and Dataset cleanup authorities. -- **NS-I3 — Non-authoritative observation:** topology inspection never changes - placement or bypasses owner, generation or layout validation. -- **NS-I4 — Verified completion:** only successful EOF establishes a full read; - a terminal failure cannot later become successful EOF. -- **NS-I5 — Final-owner accounting:** completion does not release caller-held - capacity; the allocation and its charge survive every dependent owner. -- **NS-I6 — Safe shutdown:** closed admission and drained workers do not - invalidate bytes still held by a caller or asynchronous copy. -- **NS-I7 — Bounded transport failure:** accepted RDMA failures never append - TCP data or replay an ambiguous mutation. -- **NS-I8 — Explicit Dataset progress:** fetching, verified EOF and final buffer - release cannot save progress; the client checkpoints the next continuation - only after successful processing. diff --git a/doc/design/access-server/iceberge/design-crowdb-iceberg-upload-flow.md b/doc/design/access-server/iceberge/design-crowdb-iceberg-upload-flow.md deleted file mode 100644 index 4dfbf222f..000000000 --- a/doc/design/access-server/iceberge/design-crowdb-iceberg-upload-flow.md +++ /dev/null @@ -1,395 +0,0 @@ - - - -# CROWDB - Design: Iceberg File Upload Flow - -This document defines native injection and the FileIO upload path used by Iceberg clients and records -measurements from the single-node profile. It covers file publication, not table -snapshot publication. - -Depends on: [Native Iceberg Storage](design-crowdb-iceberg.md) and -[Chunk I/O](../../chunkio/design-crowdb-chunkio.md). - -## Table of contents - -1. [Client and server flow](#1-client-and-server-flow) -2. [Large write ownership and scheduling](#2-large-write-ownership-and-scheduling) -3. [Integrity and durable publication](#3-integrity-and-durable-publication) -4. [Measured costs](#4-measured-costs) - -## 1. Client and server flow - -### Native injection - -Native injection owns both byte ingestion and the subsequent construction of -Iceberg file references: - -1. Finish durable IcebergTable chunk writes and seal an owner-bound directory - root. Even a one-leaf native tree includes its file identity in that root. -2. Form `f/v1/..parquet` under the immutable - `///` prefix. Bind the locator's - unique file identity, length, descriptor and exact filename by signature. -3. Durably select the descriptor in the UUIDv7 injection operation before - publishing one `file/` record with create-only CAS. Return the one - `s3://iceberg//` URL after publication. A retry uses - the same operation/file identity and returns the selected URL; conflicts - never overwrite published content. -4. Construct standard manifests using that URL, write manifest lists and - metadata JSON, then publish the table head by its normal atomic commit. - Failed table commits leave unreferenced durable files for the existing - safe cleanup flow; they do not change a previously returned URL. - -Direct HEAD/GET/range requests validate cached permissions and the locator, -then read native storage without Iceberg file-location or file-state KV lookup. -The chunk layer still performs its normal replica/EC layout resolution. -HEAD and conditional reads probe the physical directory root; GET reuses the -verified directory in its pull reader. Reclaimed or reused roots fail closed. -List and cleanup still use the single file record. Logical deletion removes -the file from list; its known direct URL may read retained bytes until physical -GC. Reclaimed identities cannot alias newly allocated bytes. - -The native injection entry is a signed PUT with -`crowdb-native-upload=` on a table-scoped parquet upload URI. It requires -an explicit SHA-256 payload hash and returns one JSON `location` after durable -selection and publication. The operation binds its principal, full table path, -readable stem, payload digest and retry deadline. A missing publication is -completed from the selected receipt after response loss; a changed or deleted -publication cannot be replayed. The listener advertises the single-request byte -bound. Native loader parts stay within it; normal SDK multipart retains its -chosen logical URL. - -The loader journals the stable upload URI before transfer and the returned -location before adding standard manifest references. Its report preserves the -upload URI for recovery; it does not add a file-location field to Iceberg metadata. - -### Standard SDK uploads - -SDK location providers choose logical filenames before PUT. These uploads -retain that filename and resolve the single file record for reads. They do -not require a pre-upload reservation API, a second SDK location field, or -post-upload rewriting of manifests and path-based delete references. - -For a catalog without native injection, the TPC loader hashes each local Parquet file through bounded reads, checks -that its unique target is absent, then sends a known-size S3 request using the -vended native credentials. Below 256 MiB it uses a single PUT; larger files -use bounded multipart parts aligned to complete frame payloads. The upload -finishes before the loader imports its files and commits the table. Other -Iceberg FileIO clients can select their own multipart thresholds; PyArrow -output streams may use multipart even for small files. - -1. An optional client existence probe publishes nothing. Direct PUT validates - the request body, completes the Chunk writer and publishes one immutable - location descriptor with create-only CAS. -2. Multipart starts an independent session. Each UploadPart authenticates and - validates its body, finishes its writer and persists that part's locations. - Streamed parts update their own key without a session write for each part. -3. CompleteMultipart freezes the selected parts and composes their locations - into one descriptor without copying payload. Location publication is - create-only. Recovery can resume interrupted completion. -4. A later table-head CAS publishes Iceberg metadata referencing the uploaded - files. File uploads themselves do not publish a table snapshot. - -Writer selection uses the decoded length of each HTTP request, when available. -Payloads below the small-object threshold can use the small-object writer for -one frame or the shared-object writer for a longer request. Other requests use -the large writer. The total logical file size alone does not select the writer. - -## 2. Large write ownership and scheduling - -The single-use Iceberg `WriteObject` owns the request body, bounds, file -identity, writer, digest pipe, measurements, and final publication action. S3 -PUT and `UploadPart` use the same transfer driver and checksum worker while -retaining S3 authorization and publication. Its transfer coroutine polls the -body receiver and write consumer with one task waker. Only -the receiver fetches the socket body. When it offers a prepared buffer, the -same task immediately polls the consumer. If both sides are pending, the task -yields; a Hyper body-read event or a write completion wakes it again. An idle -writer therefore resumes when a slow socket supplies the next buffer. - -The receiver assembles up to 1 MiB of payload from body frames. The native -receive path prepares frame headers and CRC32C on the received owner before -handoff. CRC32C excludes the chunk ID; after placement the writer fills that -field and writes the frame without an object-sized copy. The receiver offers -the owner to a bounded channel with four held-buffer slots. A full channel -pauses further body reads until the consumer removes an owner. The digest -worker receives borrowed payload views after the write offer, so checksum work -can overlap receive and DiskIO without controlling write backpressure. Small objects have a per-object writer owning inline MD5/SHA-256 state. The -native receiver allocates exact payload plus frame overhead, updates the digest -as socket payload views arrive, and hands one complete owner directly to that -writer without a per-object handoff channel. The shared small-write pipeline -sets destination Chunk IDs and slices the owner across strip boundaries without -payload copies. Each pipeline confirms readable cursor publication before its -next batch; queued arrivals naturally aggregate during that wait. Hash routing -is a hint and can select another pipe with capacity. Shared chunks default to -256 MiB, with 32-strip mirror groups refilled asynchronously at half consumption. -EC conversion is disabled in single-node mode; enabled multi-node conversion -can materialize a strip image to compute parity. HTTP header parsing retains -its prefetched-body copy and reports its count and bytes through one bandwidth -metric, `access.http.receive.prefix_copy.bw`. - -The large chunk writer prepares strips ahead of demand. For a known object -size, it batches up to the configured strip-prefetch limit and requests the -next batch when half of the current one has been consumed. Mirror strips are -submitted to independent tasks, with at most four strip writes in flight by -default. Later writes may finish first, but completion is consumed in strip -order. Each completed task keeps its buffer until every preceding strip has -committed. A failed mirror segment is replaced and replayed from those retained -bytes before its strip completes; later completions remain held meanwhile. -If replacement is exhausted, the writer drains submitted IO, reads the -committed prefix from the old chunk one strip at a time, and replays it with -the retained suffix into a new chunk. Frame footers receive the new chunk ID; -the CRC does not cover that field. The old chunk is deleted only after the new -copy is durable. Rotation attempts are bounded. -At a full write window the coroutine awaits the oldest completion; -the completed owner queue can still retain four prepared buffers. The -`large_parallel_strip_writes` and `large_held_buffers` settings are separate. - -The following invariants apply: - -- **I1 — One body reader.** No second task reads an object's HTTP body. -- **I2 — Bounded ownership.** Receive buffers remain owned until the writer and - digest have consumed their views; the write queue controls backpressure. -- **I3 — Ordered durability.** A later strip result cannot make an earlier - failed strip successful or release its replay buffer. Chunk sealing waits - for every submitted strip, replacement, and required fsync. -- **I4 — Event-driven progress.** Socket readiness and write completion wake - the suspended coroutine. The write path does not spin or poll a timer for - capacity. - -The upload metrics distinguish elapsed body-frame polling from actual body -waits. A body wait is counted only when the body future returns `Pending`; its -duration ends when that frame or EOF becomes ready. Immediate frames add no -wait count, while writer-capacity and write-flow pause time remain separate. -`queued_buffers_peak` is the highest number of owners waiting in the handoff -queue during one upload. `mirror_uncommitted_peak` counts submitted strip -writes that have not yet committed in order, including writes whose DiskIO task -already finished. The counters describe different stages and are not added. -`mirror_active_write_peak` counts mirror data writes actually inside DiskIO; -it excludes completed tasks waiting for an earlier strip. -The native allocator exports current and peak retained bytes. The chunk client -also counts repair-driven rotations, replayed bytes, and rotation time. - -## 3. Integrity and durable publication - -Large metadata JSON, manifest lists and manifests follow the same native -object durability rules. Each keeps one bounded file descriptor; complete -snapshot arrays and file bytes are not copied into TableHead or an unbounded -KV value. Public `metadata/` and `statistics/` prefixes remain available for -bounded listing without opening object payloads. Small inline metadata and -large chunk-backed metadata return identical canonical FileIO bytes. - -The per-object OpenSSL digest state computes MD5 over ordered logical payload -views. Small objects update it synchronously in the receive task; large objects -use a checksum worker. It also computes SHA-256 when a signed payload requires it. It never -hashes frame headers or footers. The writer fills placement-dependent chunk -IDs after the receiver has calculated placement-independent CRC32C. Digest -failure, declared-length mismatch, failed DiskIO, or seal failure prevents -file and part publication. An upload is published only after the body has -ended, the digest has been verified, all writes and fsyncs have completed, -and chunk locations have been sealed. Multipart completion and table commit -are later, distinct authority operations. - -## 4. Measured costs - -On 2026-10-01, a local single-node container received a 100-MiB generated -object through the same `CrowdbFileIO` output API as the loader. The object was -not registered in an Iceberg table. Two runs took 6.23 s and 5.90 s. The second -run's stage and server-counter deltas were: - -| Measurement | Result | -| ------------------------------------- | ----------: | -| Create output stream, including probe | 0.322 s | -| Thirteen 8-MiB client writes | 0.075 s | -| Close and finish multipart transfer | 5.507 s | -| End-to-end elapsed time | 5.904 s | -| Effective logical throughput | 16.9 MiB/s | -| Successful FileIO HTTP requests | 12 | -| Catalog GET operations | 144 | -| Catalog compare-exchange operations | 16 | -| Small-write completions | 0 | - -The 12 successful requests are consistent with creating a session, uploading -parts, and completing it. The existence probe returns not found and is not in -the success count. The summed server dispatch time was 31.61 s across requests; -multipart part requests overlap, so this sum is not wall-clock latency. - -The client writes returned in 75 ms because PyArrow buffers or schedules the -transfers. The 5.5-s `close()` wait is the dominant observed client stage. -The zero small-write delta rules out the small-object queue as the path for -this 100-MiB sample. Catalog GET and CAS counts show metadata work remains -per multipart session and part, rather than a single final table update. -These counters do not yet isolate network transfer, chunk allocation, DiskIO, -or sealing within the 5.5-s wait. - -A second container run with the checksum worker used one prepared 1-MiB block -100 times through the same FileIO output stream. The 100-MiB upload took -5.014 s: 0.034 s to open, 0.104 s in client writes, and 4.876 s in `close()`. -The FileIO path still uses multipart. This result shows that checksum work is -not the only cost in its close stage; it does not isolate the remaining -multipart, transport, or storage costs. - -### Small files - -The same container and API were used for three generated objects. Each used -three successful FileIO requests, 45 catalog GETs, and seven catalog CAS -operations, even though the payload sizes differed. - -| Object size | Open | Client write | Close | Total | Small-write completions | -| ----------- | ------: | -----------: | ------: | ------: | ----------------------: | -| 64 KiB | 0.345 s | <0.001 s | 1.242 s | 1.587 s | 1 | -| 1 MiB | 0.328 s | <0.001 s | 1.213 s | 1.541 s | 0 | -| 5 MiB | 0.344 s | 0.001 s | 1.498 s | 1.844 s | 0 | - -The 64-KiB transfer completed through the small-write queue. The 1-MiB and -5-MiB requests did not increment that counter; the available metrics do not -separate shared-object from large-writer completions. Fixed FileIO session, -part, and completion work is material for a small object. The table-level -commit is not included in these timings. - -### Focused direct PUT - -The single-node small-cluster fixture uses one mirror copy and null DiskIO. -Its producer yields 100 references to one prepared 1-MiB buffer, with -Content-MD5 computed before timing and `UNSIGNED-PAYLOAD` in the request. The -direct 100-MiB PUT completed in 335.9 ms. The upload observation was 315.4 ms, -including 224.0 ms in body-frame polling and waiting, 47.4 ms across 30 -writer-capacity waits, 23.1 ms across two strip-preparation waits, 23.0 ms in -writer finish, and 47.9 ms in metadata publication. The sum of 101 strip-write -durations was 558.0 ms and digest CPU time was 234.3 ms. These stages overlap -and must not be added to estimate wall-clock time. Body-frame time includes -Hyper delivery and coroutine scheduling, not just socket reads. - -The container's multipart FileIO timings above have different transport, -storage, and publication work from this focused direct PUT. A comparable -FileIO profile must record part size, concurrency, topology, durability, -software revision, and raw stage samples before attributing its close time -to a specific server stage. - -### Frame-aligned FileIO follow-up - -The loader's PyArrow S3 output stream split a 100-MiB file into ten multipart -parts. On the local single-node container, the first current-tree run took -4.668 s. Its non-native receive path handed the writer 1 MiB of logical bytes -at a time, and the writer pushed each 64-KiB physical frame separately. -Consequently the mirror-strip batch path never ran: the measured peak was one -active mirror write, zero queued full mirror strips, and 1,610 individual -strip-push completions. Aligning receive batches to 16 frame payloads alone -left the run at 4.638 s because the writer still split them into single-frame -pushes. - -After the large async writer grouped those 16 complete frames into one 1-MiB -physical strip push, the same FileIO path completed in 0.993 s: 0.109 s to -open, 0.034 s in client write calls, and 0.850 s in close. The server recorded -ten completed part uploads, 100 MiB of logical data, 110 strip-push -completions, and peaks of four queued full mirror strips and four active -DiskIO writes. This is one local sample, not a stable throughput distribution. -The loader now chooses a single PUT below 256 MiB; one 100-MiB direct loader -upload took 1.091 s including local SHA-256, and a 256-MiB four-part loader -upload took 3.042 s. These timings include different client preparation and -must not be compared as pure server write latency. - -The updated loader and image then completed a fresh TPC-H SF1 load in 24.94 s -and SF10 in 81.30 s. Those wall times include generation, upload, eight table -snapshot commits, and remote manifest/footer/sample-scan verification. SF10 -uploaded 3,635,609,132 data bytes; all eight tables and 86,586,082 rows were -verified. The access upload metrics recorded zero failed requests and a peak of -four active mirror writes. Previous runs on this host took about 50 s and -232 s, respectively. These single-run totals show a material end-to-end -improvement, but they do not isolate upload-only wall time or constitute a -repeatable benchmark distribution. - -### Native small-object profiles - -A 2026-10-02 comparison used the same local single-node, one-mirror, -1-MiB-strip null-DiskIO fixture. EC conversion and Iceberg GC were disabled. -Every measured group uploaded 128 distinct paths with Content-MD5, then read -back every file. Concurrency was one or 32; the 32-client groups followed -128 warmup uploads at that concurrency. The reference revision was cd5bede8; -the measured implementation (c27f6fce) uses exact native owners, inline checksums, -owner-view packing, 32-strip prefetch and bounded concurrent transport frames. -These are individual process runs, not a throughput distribution or physical -SSD benchmark. Sequential readback and background maintenance change pipeline -state between groups; keep that limitation when comparing workloads. - -**Reference** - -| Payload | Clients | Elapsed s | p50 ms | p95 ms | p99 ms | GET / CAS / scan | -| ------------ | ------- | --------- | -------- | -------- | -------- | ---------------- | -| 1KiB | 1 | 5.264 | 40.451 | 45.914 | 58.831 | 501 / 128 / 18 | -| 1KiB | 32 | 0.393 | 79.955 | 100.023 | 106.175 | 384 / 128 / 0 | -| 512KiB | 1 | 14.434 | 109.813 | 151.723 | 157.257 | 658 / 128 / 42 | -| 512KiB | 32 | 9.575 | 2242.789 | 3572.333 | 3905.036 | 581 / 128 / 30 | -| aligned64KiB | 1 | 11.569 | 88.975 | 104.206 | 154.668 | 618 / 128 / 36 | -| aligned64KiB | 32 | 1.083 | 209.547 | 306.681 | 337.257 | 403 / 128 / 3 | - -**Native owner implementation** - -| Payload | Clients | Elapsed s | p50 ms | p95 ms | p99 ms | GET / CAS / scan | -| ------------ | ------- | --------- | ------- | -------- | -------- | ---------------- | -| 1KiB | 1 | 5.268 | 40.650 | 45.195 | 51.810 | 483 / 128 / 15 | -| 1KiB | 32 | 0.421 | 85.970 | 103.340 | 112.044 | 384 / 128 / 0 | -| 512KiB | 1 | 11.665 | 88.193 | 117.547 | 125.175 | 609 / 128 / 33 | -| 512KiB | 32 | 4.337 | 956.139 | 1515.353 | 1716.117 | 464 / 128 / 12 | -| aligned64KiB | 1 | 9.606 | 71.214 | 118.007 | 125.484 | 560 / 128 / 27 | -| aligned64KiB | 32 | 0.986 | 191.480 | 298.624 | 317.061 | 403 / 128 / 3 | - -The 512-KiB 32-client sample is 2.21 times faster; the 1-KiB 32-client -sample is 7.1% slower, while its sequential sample is essentially unchanged. -The aligned 32-client sample improves by 9.0%. These samples establish neither -universal improvement nor a stable tiny-object regression distribution. -All 768 measured uploads and readbacks succeeded. Each group recorded 128 -small completions, zero large strips and 128 writer handoffs. Actual native -frame preparation was 128 frames for 1 KiB and aligned payloads, and 1,152 -frames for 512 KiB; all cases allocated exactly 128 receive owners. - -For 128 512-KiB uploads, socket-direct payload bytes plus accepted HTTP prefix -copies sum to 67,108,864 bytes. Prefix copies were about 927 KiB per group; -all 1-KiB payloads arrived in the parser's prefetched prefix, totaling 131,072 -copied bytes. These are the accepted HTTP parser copies, not strip coalescing. -Pointer-identity tests independently check owner views and frame layout. - -The 1-KiB concurrent group used one pipeline and eleven batches, with no -reservation or scale-out during timing. Summed writer-finish time was 1.685 s; -summed file-publication time was 5.851 s. These concurrent stage sums are not -wall time. File publication and transport latency remain material even after -payload copies and per-object digest tasks are removed. The common-path -location publication still performs exactly one CAS per file. Cluster-wide -GET and scan deltas include maintenance; the quiet 1-KiB window observed -384 GETs and 128 CAS operations for 128 files. - -A complete zero-copy aggregate can exceed the transport's 16-view descriptor -bound. DiskIO submits its disjoint ranges at bounded concurrent depth within -the same batch and drains outcomes before one fsync and cursor publication. -Waiting for each subdivision separately doubled tiny-object concurrent time -in a diagnostic run; bounded overlap removed that doubled regression. - -### Consolidated acceptance - -After the native owner implementation, consolidated acceptance passed the full -Iceberg library and default Access Server suites, nine native HTTP scenarios, -three native catalog/namespace recovery cases with the pinned PyIceberg client, -file storage restart, all nonignored GC control/capacity cases, and eighteen -real-process shared-writer cases. These cover cross-strip readback, mirror repair, -conversion, restart takeover and elasticity. Separate acceptance fixes preserve -visible commit retries in the existing TableHead CAS, reduce retained response -reservations, and terminate small pipelines after a pre-batch allocation failure. - -The final 5-MiB release profile recorded PUT 82 ms (3 GET, 1 CAS), UploadPart -69 ms (5 GET, 1 CAS) and GET 24 ms (4 GET, no CAS). PUT and UploadPart each -recorded zero small completions, six large strips and six writer feeds; PUT used -one Chunk location. This is another null-DiskIO sample, not an SSD result or a -stable latency distribution. The native path assembles socket reads into 1-MiB -receive buffers before issuing large writes. - -The final release loader run used TPC-H SF=0.01 with eight upload workers and -TPC-DS SF=0.01 with twenty-four workers against the same single-node fixture. -All eight TPC-H and twenty-four TPC-DS tables uploaded, committed and passed row -count verification, with no observed 503. The combined test completed in 61.94 s; -that time includes cluster startup, generation, uploads, commits and verification. -Run IDs were `ff518ccf7029449cb7fae2d7fc0491ae` and -`0c321c0991ae44b0b1c8b7182d46ac09`. It is a functional concurrency acceptance -sample rather than an upload-only throughput comparison. - -The ignored exhaustive crash matrix and additional Java/engine integration -profiles were not rerun in this acceptance batch. The suites above and the -recorded profiles establish the tested scope. diff --git a/doc/design/access-server/iceberge/design-crowdb-iceberg.md b/doc/design/access-server/iceberge/design-crowdb-iceberg.md deleted file mode 100644 index 3703d9880..000000000 --- a/doc/design/access-server/iceberge/design-crowdb-iceberg.md +++ /dev/null @@ -1,1291 +0,0 @@ - - - -# CROWDB - Design: Native Iceberg Storage - -Iceberg is CROWDB's HTTP catalog and table-storage access model. Catalog, -namespace, table, snapshot, commit, and file concepts are first-class CROWDB -storage authorities, not management metadata layered over S3. - -Depends on: [Access Server](../design-crowdb-access-server.md), -[Chunk I/O](../../chunkio/design-crowdb-chunkio.md), and -[Chunk-KV](../../chunkds/design-crowdb-chunk-kv.md). - -The backed-up [REST Catalog OpenAPI](iceberg-rest-catalog-open-api-1.11.0.yaml) -and [Table Specification](iceberg-table-spec-1.11.0.md) are normative. - -## Table of contents - -1. [Intent and boundary](#1-intent-and-boundary) -2. [Authority model](#2-authority-model) -3. [HTTP and FileIO surfaces](#3-http-and-fileio-surfaces) -4. [Commit and lifecycle](#4-commit-and-lifecycle) -5. [Compatibility](#5-compatibility) -6. [Relationship to other access models](#6-relationship-to-other-access-models) -7. [Correctness invariants](#7-correctness-invariants) - -## 1. Intent and boundary - -The Iceberg module lets standard Iceberg engines use CROWDB without first -modeling tables as ordinary S3 objects. It terminates an independent HTTP -listener and owns Iceberg request, authentication, error, compatibility, and -lifecycle behavior. - -The module does not become a query engine and does not own physical placement, -replication, erasure coding, repair, or disks. Server-side scan planning and -table processing are separate capabilities rather than requirements of the -core storage authority. - -## 2. Authority model - -Catalogs contain multipart namespaces; namespaces contain named tables; tables -select immutable Iceberg metadata generations; metadata and snapshots reach -immutable table files. Stable identities separate durable authority from -renameable names. - -Standard Iceberg metadata is the recoverable table state. CROWDB may maintain -derived indexes or projections for scale, but they are disposable and cannot -become a second table authority. - -Table ALL/REFS loads read and verify canonical storage, run the complete bounded -metadata parser, and recheck the namespace and selected head before responding. -ALL preserves exact canonical bytes; REFS filters snapshots from the same validated -content. Table loads do not read or write projection pages, validation receipts, -or other persistent cache state, including on repeated and conditional requests. -Optional generation-local JSON projections used by file-level selection remain -disposable derived data with canonical fallback; they cannot replace table -validation, participate in commit proofs, or become file/publication authority. - -Iceberg metadata stores bounded logical records and opaque data references. -Physical chunk placement and storage topology remain below the access boundary. - -### Identity directory and file authority - -The Iceberg key codec carries its storage family and layout version separately -from the directory payload. A table-owned payload begins with -`///`; it has no redundant -`iceberg`, `namespace` or `table` labels before those IDs. The storage namespace -is frozen at table creation, including staged creation. The default namespace -has a durable ID under the same rules as named namespaces. - -```text -/ -├── head -├── namespace/name/ -> namespace-id -└── / - ├── head - ├── name/ -> (storage-namespace-id, table-id) - └── / - ├── head - ├── file/ - └── projection// -``` - -Catalog and namespace control records have their own typed scopes. Namespace -names use the existing unambiguous multipart-name encoding. Table-name lookup -returns the complete storage identity; internal tasks carry it as well, so no -table-ID reverse index is needed. A move updates live name mappings and the -head's live identity/epoch while retaining the storage path. Deleting the old -namespace cannot cascade into moved tables, and IDs are never reassigned. - -Each file has one `file/` record. The suffix is also -its public object-key suffix. Its value contains FileId, canonical length, -format, integrity metadata, native descriptor/root and lifecycle state. FileId -is a value/locator binding, not another file index. One create-only CAS publishes -the complete record after durable bytes; exact-key conflicts cannot overwrite -selected content. Retries preserve the selected file identity and URL. Workflow -records exist only where recovery or multipart/GC progress requires them, not -as three mandatory records for every file. - -Service recovery journals are separate from that directory. Their existing -catalog control scopes remain explicitly versioned binary keys, even in a -native catalog. They are not file lookup indexes and cannot produce an object -listing. The service must discover abandoned work before a table head exists -and after a table has disappeared, without a table-ID reverse index: - -- Legacy creation, commit and drop journals and payload pages are keyed by - operation identity. Their selected heads carry the frozen storage identity. - New creation and staging keep complete input only within the request. - Name moves use bounded namespace occupancy instead - of request journals; existing rename journals remain recoverable. -- Multipart sessions and parts are keyed by upload identity/part number. Catalog - admission records apply only to legacy credited sessions; new streamed sessions - use independent per-session bounds. - Every session and part carries and validates the full storage identity. -- Write intents/fences track allocated but potentially unpublished native - extents, including failed creation. Catalog maintenance scans their bounded - control scopes; the value binds the full table/file owner and exact chunk root. -- GC tasks, candidates, claims, proof nodes and pages belong to the recovery - worker. A retirement task can cover several tables and adopts interrupted - purge work. Claims coordinate physical deletion across tasks/generations; - candidate values retain the canonical filename and full storage identity. - -These records are retained only for retry, fencing and reclamation. They do not -add another published file authority or participate in direct GET/HEAD. GC data -discovery chooses v1 or v2 from the persisted catalog layout; independent -service-control scans never merge the two file namespaces. Head, file and -projection records use the native table directory. Historical native injection -receipts remain decodable for reclamation; uploads create no request receipt. - -### Standard metadata and bounded scale - -TableHead selects an immutable standard metadata JSON file. Its `snapshots` -array contains snapshot records and its `refs` map selects branches/tags; -snapshot-log and metadata-log retain their standard meanings. Snapshot records -reference manifest-list files, which reference manifest files, which describe -data/delete files. Reading v1 also supports its direct `manifests` alternative. -The backed-up v1/v2/v3 specification determines fields, sequence inheritance, -schema/partition evolution, delete matching and row lineage. No authoritative -snapshot or manifest KV index substitutes for those standard file contents. - -All of these file kinds use the same single file-record model. Large metadata -JSON, manifests and manifest lists are native chunk-backed objects. KV values -and TableHead never embed an unbounded metadata document or snapshot array. -Small eligible metadata may use the bounded inline representation; larger -objects and large extent sets use chunk payloads and bounded directory roots. -Changing storage representation preserves the canonical bytes returned by -FileIO. Table publication selects a completely durable metadata generation by -CAS; a losing commit cannot expose a partially written generation. - -Metadata files use useful public relative prefixes such as -`metadata/.metadata.json`, `metadata/.avro` and -`statistics/.puffin`. These are defaults, not restrictions on safe -client-chosen paths. The optional generation-bound projections described above -support file-level selection; they are not used by table ALL/REFS loads or duplicate -file authorities. Parser, collection and response budgets remain explicit even for -chunk-backed JSON. -An unsupported oversize request fails clearly rather than silently truncating -snapshots. ALL and REFS keep their standard response semantics; internal -projection paging does not introduce private REST pagination fields. - -New catalogs use the identity-directory layout version. Existing legacy -catalogs retain their own decoder, keys, URLs and read/write behavior. Layout -selection is catalog-bound; relabeling or mixing key versions in one scan is -rejected. In-place migration is not implicit in decoding. - -### Catalog foundation - -One active root selects a random stable CatalogId and activation epoch. Display -rename updates its authority without moving descendant keys. System-scoped -management receipts, audit and retry bindings survive catalog replacement; -resource records and retained REST response bodies are catalog-scoped. - -Initialize, rename, capability activation and clear use bounded single-key CAS state machines, not a -global lock or a multi-key transaction. A root retains the operation identity -until its durable outcome and audit can be recovered by any instance. Clear -fences admission, records a maintenance observation after the durable fence, -publishes an empty replacement under maintenance, and persists the grace proof -before reopening admission. Completion uses persisted lease, request, delegated -access and clock-skew limits, never shorter restart configuration. Retired -authorities remain unreachable; bounded retired-catalog cleanup follows the -persisted retention and owner proofs. - -Format capability bits are durable catalog authority. Zero means no advertised -table format service; startup never rewrites or widens a legacy zero profile. -An authenticated management operation explicitly activates a validated profile -under the root fence. It preserves the catalog ID, activation epoch, table keys, -name generation and admission bounds, advances config generation, and may only -add support. A resumed operation replays its original profile and audit result. -Clear creates a new zero-profile catalog that requires separate activation. - -The baseline has no root lease. Each HTTP connection closes after five minutes -without network progress; active streamed file bodies and multipart completion -heartbeats extend the idle deadline. Request dispatch has a separate deadline -starting at connection acceptance; incomplete request headers close at that -deadline, while an active response remains governed by network idleness. REST -and FileIO admission reject configured request timeouts exceeding the persisted -catalog request bound. Newly initialized runtime catalogs use a five-minute -request bound; -new catalogs also persist a fifteen-minute delegated-access bound. Restart never -increases persisted bounds. Explicit catalog clear may expand them componentwise -under the maintenance fence and waits the resulting full grace before admission; -neither clear nor smaller restart settings can shorten existing bounds. -Listeners stop admission before bounded draining; -startup and periodic reconciliation resume interrupted management operations. - -Management and audit ledgers use 4096 deterministic fast-hash slots with -exact-identity overflow keys. An occupied slot routes a different identity to -its own durable key. Their retention and authority checks belong to the -management operation contract. Client write identities use UUIDv7 issuance -time with a 30-second future-clock allowance; requests without a client key -receive independent internal identities. - -REST namespace and table writes do not allocate a shared HTTP retry binding -or cache terminal HTTP results. Each request validates its current authority -and input. Existing business drivers bind operation identities to their stable -resources and validate their own request preconditions; direct expected-base -table updates retain their head CAS contract. A failed request without a -business operation can be submitted again after the resource state changes. -Legacy HTTP bindings, overflow keys and response payloads remain decodable -for retention cleanup, with no producer on new request paths. - -Immutable operation payloads use 32-KiB pages with a 2-MiB aggregate limit. Each -reference binds catalog, operation, content digest and total size; readers validate -every page and the complete digest. Business operation drivers use these -references for their existing inputs and results. Legacy large HTTP response -manifests remain readable by cleanup; independent requests do not create them. - -New namespace creation writes only its final name reservation and namespace -authority. Before publishing a nested child, it places that child's ID in the -parent's `creating_child` occupancy; the child authority is already complete. -Parent deletion must settle or cancel this occupancy before proving emptiness. -Publication clears it through an exact parent update, preserving concurrent -properties. Top-level creation needs no parent occupancy. Helpers resolve an -interrupted create from these resource facts, without an input or response -journal. Cancellation and retired-catalog cleanup permanently retire the same -child authority slot before releasing its exact name reservation, including an -absent slot. A delayed first authority write therefore cannot resurrect it. -Occupied parents use a format that older writers reject. - -Legacy namespace creation, property and drop journals occupy a separate key -scope from HTTP responses. Retained records bind the request and stable target, -immutable mutation snapshots, phase revisions and bounded child-probe cursors. -Recovery validates their phase transitions, snapshot identities and cursor ranges. -Fresh namespace writes do not create these journals or payloads. - -Authoritative namespace reads resolve each parent/name mapping against the -selected stable authority and full canonical identifier. Reservations, stale -epochs, missing targets and tombstones are not visible; corruption is an error. -Active-context checks bracket resolution so retirement cannot turn an old-domain -lookup into a response from the replacement catalog. - -Property updates apply the request's assignments and removals directly to the -current namespace authority through one conditional write. This advances the -property and mutation revisions once, preserving the name epoch and admission -fence. A definite conflict supplies the next authority; the request reapplies its -changes while preserving other properties. Deletion, changed identity or a -pending lifecycle operation blocks that update. Active-context validation -brackets resolution and checks completion. Retry work is bounded. - -Property requests persist no input, operation journal, marker, response payload -or HTTP retry binding. A request that leaves the properties unchanged performs -no write. A lost reply is returned as an error; the client can submit its changes -again. Each response describes that request's actual removed and missing keys, -so a repeated removal can move from removed to missing. Repeating assignments -can overwrite a concurrent assignment to the same property. Other properties -are retained. Old property journals remain readable by recovery, including the -holder-bound marker dispatcher. Namespace creation and drop likewise use -resource facts without an input or response journal. -The catalog configuration does not advertise a universal idempotency-key -lifetime, since independent property updates do not retain operation responses. - -Legacy namespace creation recovery installs a parent/name reservation before a -parent authority CAS. Its nested admission leaves a pending-operation marker -and advances only the mutation revision; a top-level admission conditionally -validates the active root without replacing its management operation. Helpers resolve uncertain parent -writes before allowing subsequent parent mutation. Definitive admission conflicts -may retry with fresh parent snapshots while retaining the name reservation. -Publication selects an initial authority and replaces the reservation with its -published mapping; a creation marker remains until the result is durable. Abort -records retain their exact failure outcome before conditional reservation cleanup. -Recursive creation helping shares one bounded phase budget. Root-admission -backend identities include the individual creation operation, so a different -creator cannot reuse a cached no-op CAS result from before its reservation. - -Namespace drop must reject a nonempty namespace. Before fencing the selected -namespace, deletion settles its child-creation occupancy, rename occupancy and -legacy pending markers. These resource facts protect child publication and name -moves while the parent is checked for emptiness. - -Each drop selects the current namespace binding once. An exact CAS changes a -Ready authority to ownerless Dropping, advancing the admission fence and -mutation revision; an existing Dropping authority can be helped without another -fence. Lifecycle tag 4 identifies this state and requires no pending operation, -child occupancy or rename; older decoders reject it. Dropping remains readable -and counts as a live child when an ancestor is being deleted. - -Deletion scans both direct-child index ranges through bounded pages, validating -range, ordering, continuations and child bindings. Reservations are helped and -stale mappings are conditionally removed; corruption blocks the empty proof. A -live namespace or table child restores the exact Dropping authority to Ready, -advancing its admission fence and mutation revision while preserving properties, -name epoch and property revision, and returns a nonempty error. Only completion -of both ranges for the same exact Dropping authority permits its CAS to -permanent RetiredNamespace at the same authority key. The retired slot prevents -delayed authority creation from resurrecting that namespace identity. - -Scan cursors and empty proofs exist only within the current invocation. A CAS -conflict supplies the next validated authority without a confirmation GET. A -changed Dropping generation discards both range cursors and the old empty proof -before rescanning; a Ready authority restored by another helper ends that drop -attempt. Unknown write results return an error immediately without a follow-up -read or write. An installed Dropping fence survives cancellation, timeout and -exhausted work budgets, so a later request or mapping-repair pass can restart a -complete proof. No input, phase, scan cursor or response record is persisted. -Active-catalog validation brackets selection, precedes each authority transition -and checks completion. - -Terminal cleanup conditionally deletes only the raw name mapping selected by -that invocation. It cannot delete a replacement NamespaceId at the same name. A -fresh client retry selects the current name again and may delete a recreated -empty namespace; a nonempty replacement still fails. A missing current name -returns absence rather than a cached terminal response. Legacy owner-bound -lifecycle states and journals retain their recovery and exact-target cleanup. - -Current property and rename-admission CAS conflicts with the same-key retired -slot report a conflict without another authority read. Retained legacy work -treats that slot as final parent absence and aborts work not admitted before -retirement. Cleanup still fences an absent table-head slot before removing its -exact reservation. Captured authority payloads keep their identity and owner -validation; a retired current parent does not make an invalid capture acceptable. - -Table-child probes resolve published mappings against the selected table head. -Ordinary table creation leaves only a reserved name and its final head slot. -Probes retire an absent slot before clearing that exact reservation, or settle -the matching Ready head and reject deletion as nonempty. A creator reserves its -name before its last Ready-parent check, so a concurrent drop cannot both miss -the reservation and permit a later first head. Retained staged operations and -legacy lifecycle journals remain recoverable through their exact markers. -Stable namespace identity prevents old state from becoming visible under a -recreated namespace. Name moves occupy both participating parent -authorities before reserving the destination; deletion helps settle or fence the -move before taking a fresh emptiness proof. Corrupt -table authority blocks the emptiness proof. -Each listener alternates bounded recovery of retained namespace journals with -mapping repair, using local page cursors, work budgets and a wall-clock deadline. -Mapping repair helps child-creation reservations and ownerless Dropping resources, -including a full new child proof after an interrupted deletion, then conditionally -removes bindings disproved by authoritative state. Catalog changes invalidate -local cursors; cancellation preserves resource facts. Corruption and unresolved -reservations never authorize deletion; no sweep physically removes file bytes. -The listener exposes authenticated namespace listing, load and exists routes. - -Namespace list pages scan bounded direct-child ranges and validate each published -mapping against its authority and canonical parent spelling. Reserved and stale -entries are omitted; corruption fails the page. HMAC-authenticated continuations -bind the catalog activation, stable parent identity, spelling, page size and last -scanned key. A stale-only page can therefore be empty while retaining a token. -Unpaginated lists build a complete in-memory spool before success headers, capped -independently at 2 MiB, 1024 results and 4096 scanned mappings. A shared -128 MiB byte budget admits response spools, reserving each operation's maximum -while constructing the response (2 MiB for namespace lists and 4 MiB for table -responses). Serialization shrinks the reservation to the retained buffer capacity; -the response releases it on cancellation or completion. Atomic admission rejects -excess work without waiting. The request -deadline bounds construction before success headers. Dispatch stops before that -deadline, reserving the smaller of 100 ms or 10% of the request timeout for -emitting a bounded error response. Header receipt does not restart this budget. -A stalled transport closes after the independent idle timeout. Cancellation -drops the spool permit. Completed -responses stream in 16-KiB frames. Absent page tokens request complete results; -empty page tokens begin paginated mode. Tokens use a domain-separated signing key -derived from the configured credentials so equally configured listeners interoperate. - -## 3. HTTP and FileIO surfaces - -The REST Catalog is the portable control surface. It exposes only capabilities -CROWDB implements with compliant Iceberg semantics. - -The listener classifies complete method/path pairs before domain mutation and -uses the same fixed route set for discovery and protocol metric labels. Request -measurements use bounded atomic counters and follow response bodies through -completion or cancellation; streamed file bytes are measured when emitted. -Protocol counters never use principal, table name, token or raw path as a label. -An authenticated management-credential-only `GET /_crowdb/metrics` exposes a -bounded snapshot, including while catalog storage is unavailable. This local -diagnostic is not an Iceberg REST endpoint and is absent from `/v1/config`. - -The catalog listener exposes authenticated config and namespace REST. An absent or -empty warehouse selects the sole active catalog; other selectors fail with -`NoSuchWarehouseException`. Its endpoint list advertises installed namespace and -table read/create/commit/lifecycle/credential routes only when the persisted -format profile permits them. A zero-profile catalog returns unavailable config -rather than publishing a misleading set of false overrides; table routes reject -until management activation. Selected table versions gate load, HEAD, credential -refresh, create and commit. Version upgrades require each intermediate edge, -including direct v1-to-v3 requests. FileIO bytes alone do not identify a table -file's semantic kind or grant format-version authority. File grants intersect -the principal role with the selected version: published tables require write -support for upload permission, while staged drafts require create support. -Runtime table routes require a persisted delegation bound of at least fifteen -minutes. Legacy catalogs below that bound retain foundation-only service; activation -requires an explicit clear with expanded bounds and a listener restart after the -maintenance grace. Namespace mutations, table creation and lifecycle operations -use retained UUIDv7 retry identities bound to route, input, principal and catalog -activation. Ordinary existing-table commits publish through one head CAS without -a separate durable retry-result ledger; a retry whose result is hidden by a later -generation may report an uncertain outcome. Server errors -remain retryable, never terminal ledger outcomes. Exhausting the configured request -deadline leaves durable recovery evidence; subsecond completion is not guaranteed. -Static bearer credentials -separate reader, writer, management and clear roles; this is not an OAuth token -issuer. All four credentials are required and distinct. Writer has a separate -namespace-write capability and no catalog management or clear privilege; reader, -manager and clearer do not inherit namespace-write rights. All four can read the -configuration endpoint and namespaces. Only writer may invoke namespace or table mutations. -Management commands are separate from the Iceberg REST listener. Operational -configuration follows the selected deployment profile and its startup inputs. - -Iceberg FileIO uses reserved S3-shaped locations so existing Iceberg clients can -address immutable metadata and data files. The shape is a compatibility -contract, not delegation to the general S3 authority. File publication, -immutability, authorization, and deletion remain under Iceberg control. -Public URIs use the fixed virtual bucket `iceberg`: -`s3://iceberg////`. -Native typed locations use lower-case fixed-width hexadecimal for all three -128-bit IDs. Legacy compatibility buckets retain their base32 catalog component. -Relative UTF-8 object keys preserve case, literal percent signs, plus -signs and repeated internal slashes; the whole object key is bounded to 1,024 -bytes. Dot traversal, leading slash, backslash, controls, query and fragment -delimiters are rejected rather than normalized. HTTP percent decoding belongs -only at the transport boundary, not in stored S3-shaped locations. - -Native `ListObjectsV2` retains this address model. A list request must specify -the complete `///` prefix or a -descendant prefix and a table-bound delegated list capability; -catalog-wide discovery and general S3 credentials are rejected. Reader and -writer FileIO credentials include listing; cleanup-only credentials do not. -Exact-object HEAD/GET/PUT remain independently usable. PyArrow missing-file -probes may issue listing to distinguish a missing object from a directory. - -Empty, catalog-only, namespace-only and foreign-table prefixes fail before a -scan. HeadBucket/GetBucketLocation expose only the configured virtual bucket -and region to a valid grant; ListBuckets returns AccessDenied, and bucket -creation/deletion is unsupported. The URI host determines the bucket. Arrow -FileSelector adds a trailing slash to its key prefix and uses `/` as delimiter -for nonrecursive listing; an exact HEAD miss may probe `/` without -widening scope. These behaviors follow the -[Arrow S3 implementation](https://raw.githubusercontent.com/apache/arrow/apache-arrow-25.0.0/cpp/src/arrow/filesystem/s3fs.cc). - -Listing enumerates selected, fully published immutable file-location records, -including files awaiting a table commit or later reclamation. It excludes -incomplete uploads, unselected draft/losing candidates and logical deletion -markers. It does not enumerate the files reachable from a current snapshot; -clients use Iceberg metadata and manifests for that selection. Names, lengths -and ETags come from the selected record. Since file records do not store a -wall-clock publication time, listing returns the fixed Unix epoch timestamp; -clients must not use it as a file-age or reclamation signal. - -Each page scans at most 256 records and 4 MiB, possibly returning fewer than -the requested maximum of 1,000 keys. The supported delimiter is `/`; URL -encoding, zero-key pages and table-scoped `start-after` are supported. XML is -bounded to 2 MiB. Unsupported or ambiguous selectors fail before scanning. -Pagination is a forward live traversal rather than a snapshot: unchanged -eligible sets have no omissions or duplicates, publication behind the cursor -is seen by a new traversal, deletion ahead removes that entry, and publication -ahead may appear. Common-prefix rollups advance past the entire subtree. -Authenticated continuation tokens bind the principal, credential nonce, -catalog epoch, table, prefix, delimiter and encoding. Their fixed expiry is -no later than the originating grant, and invalid or foreign tokens fail before -storage access. A credential refresh requires starting a new traversal. - -List operates directly on `file/` records in public-key order. For example, -the public prefix `/metadata/` maps to the KV prefix -`/file/metadata/`; filename prefixes narrow that same range. -Pagination uses encoded prefix bounds and the last examined key, including -filtered tombstones and sparse pages. It never materializes or sorts the whole -table, opens metadata objects, scans unrelated file prefixes or enumerates -snapshots inside JSON. Returned keys omit the internal `file/` component. -The same limits apply whether a table has ten or millions of metadata files. - -Native file records bind FileId to exact location, kind, format and canonical length. -Legacy tree and inline records also bind a whole-file SHA-256 digest; streamed -records bind a bounded array of complete Chunk locations and an HTTP ETag, and -rely on verified 64-KiB storage frames rather than a whole-file digest. Eligible -metadata stores at most 16 KiB inline; bounded LZ4 -compression considers at most 64 KiB original input, and decoding verifies the -canonical length and digest. Location vectors are validated for exact logical -coverage and bounded by the record limit. Hints are non-authoritative and out-of-bounds -hints are ignored. One exact-location key stores the complete file descriptor. -Publication uses one create-only CAS without a preliminary existence GET; equal -content is resolved from the CAS conflict value. Losing candidates cannot -overwrite the selected bytes. Legacy file-ID/mapping pairs remain readable for -recovery. Prefix scans of current records require no file-ID join. -An SDK upload supplies a path and bytes, not the eventual Iceberg data/delete -use. FileIO treats client-supplied bytes as opaque. It verifies the declared -transport checksum and durable frame writes before publishing the descriptor; -format interpretation belongs to the client or to a CROWDB component that -constructs those bytes. Commit validates the selected table metadata and request -requirements in memory; it does not traverse client manifests or perform -per-file reference checks. Format validators remain available to components -that explicitly construct or examine those files. -The isolated native HTTP surface exposes signed immutable object reads/writes, -multipart operations and separately authorized single/batch cleanup, without -a general S3 bucket authority. - -### Native injection and direct reads - -Native injection finishes chunk writes and its bounded descriptor, publishes -one file record, then returns one canonical URL before constructing manifests. -For parquet, the public filename is -`f/v1/..parquet`. The URL-safe, dot-free locator carries -a unique FileId, length, integrity information and complete stable native -location or chunk-resident directory root. Its versioned signature binds the -full table identity, readable stem and unsigned locator fields; the signature -bytes are excluded from that signing input. Repeated readable stems are harmless; -a full upload retry can return a new filename for equivalent contents. The -client retains the returned URL for its eventual metadata commit. Unreferenced -copies are discovered through file and physical range ownership facts. No -operation record selects or replays an upload response. Locator and stem -together must fit the object-key limit. - -Ordinary SDK writers choose logical filenames before S3 PUT and keep those -URLs. A PUT-time allocation does not change a client's chosen path. These -files resolve their one file record; no response field or manifest rewrite -converts them into direct locators. Metadata/Avro files may retain useful -logical prefixes regardless of their native object size. Position-delete and -deletion-vector path references always use the original canonical URL. - -Direct HEAD/GET/Range GET validate request permissions and locator authenticity -and read the native descriptor without Iceberg file-location or file-tombstone -KV queries. Chunk-layer layout resolution remains available for replica/EC -routing, cold reads, repair and migration. A KV-resolved opaque file token is not a direct -locator. Large files use bounded chunk-resident directory traversal, not an -unbounded URL. Native chunk routing preserves repair/failover without embedding -transient replica addresses in URLs. Reclaimed identities and reused extents -must never make an old locator return a different file's bytes. - -Native trees retain an owner-bound directory root even when they contain one -leaf. Its digest and encoded full file identity distinguish an identical-content -replacement at a reused extent. Direct request preflight checks that root before -HEAD or conditional success, and GET reuses its verified bytes in the pull reader. - -Authorization uses bounded validated grant/context caches and signature, -scope and expiry checks. An authorization cache miss may load authority; it -must not become a per-file location/state lookup. Expired entries are not -accepted during refresh failure. Cache entries bind principal, catalog epoch, -full storage identity and operation rights, with validity bounded by the grant -and applicable authority lease. Fresh credential issuance still checks live -authority. Persistent locators have no credential expiry; retain signing-key -verification material for their files' lifetime. A locator is not a bearer -permission. Cold/warm authorization costs and storage resolution counts are -measured separately. - -Native catalog authority persists the default locator verification/signing key, -independent of access tokens. Key version 1 is reserved for that catalog key; -renaming and capability activation preserve it. Additional signing versions are -configured through `iceberg.locator_signing`, with an active version and retained -verification keys. Configured version 1 must match the persisted catalog key. -Every serving node uses the same configured versions and retains them until the -corresponding files are physically reclaimed. A lock-free cache selects the -current catalog's key ring; clearing to a new catalog selects its new default key. -Native authority uses record envelope 2 so a legacy server cannot silently ignore -the storage layout. Legacy authorities retain envelope 1 and omitted layout -fields decode as the legacy layout. - -Credential refresh URLs carry the table ID and frozen storage namespace ID. -The listener resolves that immutable head directly, so refreshing an old endpoint -after a move or namespace recreation does not require a table-ID reverse index. - -Legacy chunk-backed files use bounded leaf blocks and immutable chunk-resident directory -pages, with at most 256 children per page and eight directory levels. Each page -binds its catalog, table and file identity, child heights and covered byte count. -The writer retains only one partial leaf and bounded per-level frontiers. Native -block completion waits for the readable chunk cursor before publishing a root. -Pull readers retain one leaf and its current verified leaf-directory page, -bounded to 32 KiB independently of file length. Directory reuse is reader-local, -bound to the exact immutable root, and never shared across files or generations. -Readers verify directory/leaf digests and read no future block until requested; -full-file reads also verify the canonical digest. Range -parsing accepts one contiguous interval and rejects multiple ranges explicitly. -The HTTP pull-body adapter adds shared response admission and 16-KiB frames. -Only body polling starts a storage read; cancellation drops the in-flight read -before releasing admission. Exact remaining-byte hints track delivery, and storage -errors terminate the body rather than returning a successful truncated stream. -Legacy writer checkpoint decoding remains only for reclaiming existing physical -frontiers. GC verifies owner identity, storage checksum, frontier heights and -complete byte coverage before deleting any child. Foreground writers do not -produce or restore byte checkpoints. Versioned digest state in legacy records is -trusted-storage metadata, never a client authentication assertion. -Legacy native block writes persist an exact physical-range ownership intent in the -catalog before DiskIO. A shared-writer callback receives the assigned location; -an uncertain catalog reply stops the physical batch without a confirmation GET. -Explicit retries reconcile the existing range fact. This ledger also covers -process loss before file publication and legacy checkpoint ranges. Reclamation waits for the chunk readable -cursor or terminal state to settle any unconfirmed physical write. -Staged-tree readers validate physical roots, byte lengths and digests without -assigning a semantic file kind or declaring an incomplete multipart fragment to -be a valid complete-format file. Published-file reads retain record validation. -The legacy tree assembly byte engine consumes complete selected parts in ordinal -order within one request. It preflights the entire selection against the -intersection of request, grant, service and upload file limits before physical -copying. The copier retains bounded read frames and writer frontiers, verifies -each full part digest, and emits no intermediate checkpoints or progress CAS. -One next frame read can overlap the current writer push; either failure cancels -the other future and poisons that request-local copier. A complete-input retry -starts a new copier, while unowned physical ranges remain discoverable by GC. -Multipart session/part models retain independent resource limits and validate -phase coherence: publishing requires complete candidate bytes, published outcomes -require a selected FileId, and abort retains completion evidence without claiming -publication. Their FlatBuffers envelopes bind session and part identities to -separate catalog key scopes, retaining only bounded checkpoint references and -current-part digest state. Unknown phases and invalid revisions fail closed. -New multipart sessions publish their own authority without a catalog-wide -credit reserve or release. Persisted per-session limits bound staged bytes and -part count. Global credit reserve/release/journal drivers have been removed, -including compatibility write hooks. New sessions reject legacy credit input; -existing session and policy formats remain decodable. Recovery never reads or -settles a global policy. A legacy pending reservation without a session is ignored: -session admission preceded any permitted part writes. A pending credit release -does not change the published FileId or part references and is not replayed. -Existing sessions still expire or finish from their own resource facts; obsolete -policy/payload metadata remains available to legacy metadata cleanup without -keeping accounting alive or creating another upload. -The legacy tree multipart repository reserves one part mutation in the session before -changing its part authority. A bounded before/after snapshot and monotonically -increasing revisions make the write and fence release recoverable across servers. -Counts and current staged bytes are reserved once at the session CAS. Abort cannot -bypass an unresolved mutation; stale helpers cannot restore an older part. Abort -retains parts and completion evidence rather than deleting physical storage. -Legacy completion freezes an ordered part-number/revision/digest selection in immutable -payload pages, then changes the session phase by CAS to fence part replacement. -Selections are independently bounded to 10,000 entries and 420,007 encoded bytes. -A client Complete request copies the entire bounded selection in one attempt and -holds the existing upload concurrency permit through assembly and sealing. Its -300-second response-body deadline cancels all pending work. A selection exceeding -the request copy limit fails explicitly before any physical copy. Existing byte -checkpoint sessions reject continuation with an instruction to abort and create -a new upload; their original frontiers remain available to expiry and GC. -Assembled bytes remain unexposed until semantic sealing and immutable location -publication. -Streamed UploadPart writes one part key by CAS without changing the session for -each part. Create caps part count by the reserved staged-byte ceiling divided -by the per-part byte ceiling. Complete freezes an ordered selection with each -part's exact location bytes, length and MD5 ETag in bounded immutable payload -pages. A later replacement cannot change that selection. Completion composes -logical offsets across the selected locations and derives the multipart ETag -from the ordered raw part MD5 values; it does not read part data or assemble a -new chunk. Session phase CAS freezes publication against later selections. The -final file descriptor becomes visible through the immutable location publication -protocol. -A successful completion transition passes its exact installed session to the next -step; a definite CAS conflict passes the returned session observation. The HTTP -driver does not reload the session to confirm successful freeze, publication -preparation or completion. Stream publication reuses this request's selection -after checking its count and digest against the frozen binding, without reading -the immutable selection pages again. An uncertain part or completion mutation -returns its error without a confirmation GET. A client can retry its complete -part transfer or Complete request; the existing frozen selection and publication -facts continue to protect physical ownership and GC. -A recovery page scans at most four session authorities and performs one pending -legacy part settlement, logical expiry or frozen publication per session. It -validates the complete scan page before mutations, rejects foreign continuations, -and never starts or resumes byte assembly. Completing uploads wait for a client -request or expiry. Expiry retains physical parts and legacy frontiers and cannot -bypass an unresolved part mutation or a publication fence. -Publication freezes a caller-validated sealed file record in immutable payload -pages before the publication phase CAS. Recovery replays that exact record through -the immutable file repository and persists the selected FileId. Equal preexisting -bytes retain their original identity. Only a proven incompatible immutable location -permits the terminal Conflicted phase; uncertain writes and context failures do not -become false aborts. Canonical format validation remains the seal caller's contract. -Each native listener schedules this resource sweep independently of namespace -recovery, with one bounded scan and no copy-observation pass. Context changes -reset the cursor. Each session uses the persisted catalog request deadline; -timeout defers only that session so later entries can progress. A separate outer -budget bounds page and context work. Empty and terminal-only sweeps perform two -context checks and the scan, with no policy GET or mutation. There is no global -credit recovery or background byte-copy continuation. - - -Multipart part listing uses one upload-scoped scan with at most 256 records per -page. Numeric markers preserve gaps and resume strictly after the returned part -number. Current-session checks bracket each scan; concurrent mutations invalidate -the page rather than mixing pending counters with old part records. Expired or -terminal sessions and malformed storage pages are not reported as successful lists. - -Native HTTP upload staging holds a bounded concurrency credit. Known small -bodies receive an owner sized to payload plus 34 bytes per 64-KiB storage frame. -A per-object small writer computes declared MD5/SHA-256 synchronously as socket -views arrive, then hands the complete framed owner to the shared pipeline once. -The worker assigns chunk IDs and slices immutable views across strip boundaries; -ordinary mirror writes do not copy payload. The HTTP parser's prefetched-body -copy is measured by one bandwidth counter with count and bytes. -Large and unknown-length bodies receive through bounded 1-MiB owners. Routing -uses the configured fraction of a strip's data capacity, not socket frame size. -Unknown-length bodies choose a dedicated chunk. Per-pipeline disk completion and -readable-cursor publication finish before the next batch; requests accumulating -while publication waits form the next batch. Hash selects a preferred pipeline, -with atomic alternate-route admission before backpressure. Shared chunks default -to 256 MiB with 32-strip groups, asynchronously refilled at half remaining. -Allocation appends after reserved capacity, including hidden strips, and runs -outside the existing chunk publication guard. Exact length, declared checksum and -durable completion precede file publication. Failed uploads leave unpublished -allocation ownership to lower-layer recovery and applicable cleanup tasks. -See the [upload flow](design-crowdb-iceberg-upload-flow.md) for transfer details. - -Metadata JSON structural validation uses a bounded pull-reader bridge and an -ignored-value parser rather than retaining the metadata graph. A separate scanner -bounds nesting and verifies raw UTF-8 before parser scratch can grow. Admission -caps blocking workers; cancellation keeps its permit until the worker exits. -The validating full-file reader checks the canonical digest in the same storage -pass for chunked JSON; no independent preliminary full-file read is required. -This structural check does not replace Iceberg schema or commit validation. - -Avro writer-schema binary layouts compile to bounded named-reference graphs. -Decoded block validation checks datum widths, UTF-8, collection byte counts, -union/enum indexes and exact record consumption without retaining datum graphs. -Independent graph, recursion and visited-value limits also bound zero-byte values. -The record reader compiles its container's schema once, decodes one bounded block -per pull and permanently stops after failure or cancelled reads. Reader-schema -resolution and Iceberg logical/manifest semantics remain separate checks. - -Parquet and Puffin container probes derive footer ranges from canonical framing, -ignoring stored hints even when those hints happen to be in bounds. Their reads -retain one bounded leaf and only fixed-size framing bytes, independent of the -advertised footer size. Puffin probing also checks footer-start magic and reserved -flags. Container framing does not validate footer contents or data semantics. -Puffin metadata parsing separately caps encoded and decoded footer payloads at -1 MiB and bounds blob, field and property collections. It accepts plain JSON or -one sized, checksum-verified LZ4 frame and rejects overlapping blob ranges. Footer -deletion-vector descriptors validate their reserved snapshot/sequence markers, -uncompressed storage, referenced file and cardinality; manifest checks require -exact offset/length and referenced-file/cardinality agreement. The deletion-vector -reader then streams Roaring array, bitset and run containers, validates their -directories and cardinalities, and checks the blob's framing and CRC-32. It retains -one bounded container directory, not the deleted-position set; byte and bitmap -limits independently bound work. Snapshot-wide uniqueness and referenced data-file row-count checks belong to -explicit selected-file validation; normal metadata commit does not run them. -ORC probing retains at most 255 postscript bytes, checks protobuf wire framing -and resolves footer/metadata spans without decoding stripe directories. It accepts -legacy header-only magic and skips bounded unknown protobuf fields. - -Avro OCF framing uses a bounded header map and pull-based encoded blocks. Header -bytes, metadata count, block bytes and records per block have independent caps; -negative map blocks must match their declared byte lengths. Sync markers and -canonical block integrity are verified before a block returns. Errors or cancelled -reads poison the cursor rather than resuming at an ambiguous record boundary. -Null and raw-deflate block decoding enforce an independent decoded-byte cap; -truncated compressed data or unused suffixes fail closed. This layer does not -resolve Avro schemas or validate Iceberg manifest fields. - -Manifest inheritance is a separate constant-state semantic layer. It distinguishes -the manifest version from the containing table version: v1 sequences default to -zero, while new snapshots can assign row IDs to older manifests. Only added files -inherit missing sequence numbers; explicit file ages are preserved. Unassigned -data files advance the row-ID cursor in manifest order, including existing files -after an upgrade; delete files cannot carry row IDs. Invalid entries and arithmetic -overflow leave the cursor unchanged. Avro decoding and commit admission are not -yet connected to this resolver. - -Delegation tokens carry catalog activation epoch, table, principal fingerprint, -nonce, exact operation set, issue/expiry times and independent request/file byte -limits. Domain-separated HMAC authenticates bounded claims and derives per-grant -S3 credential material without a mutable credential registry. Verification requires -the operation's authorized context: writes and credential issuance check Ready -authority, while direct reads may reuse the bounded validated permission cache -described above. Ordinary FileIO grants exclude DELETE; -explicit cleanup grants include the separate delete operation. These token -primitives feed native request-signature verification through a request-local -credential provider. Only the shared SigV4 algorithm is reused; general S3 -credentials and metadata are never consulted. Header and presigned requests have -bounded authentication input and reject duplicate authentication fields. Grant -expiry remains exact even when signature timestamps allow clock skew. Table -credential issuance requires a matching Ready catalog authority and rejects -lifetimes above its persisted delegated-access bound, independently of the -signer's configured maximum. A zero persisted delegation bound disables issuance. -Credential-issuance callers freshly authorize the root and exact live table or -client-selected unpublished storage identity; serialization performs no storage reads. -An unpublished identity requires a Ready namespace, current writer authorization, -an exact table-ID and format-version selector, and the matching catalog layout. -Its final head slot must still be absent. No original-owner or draft lookup is -required. Published heads and retired slots cannot be relabeled as unpublished. -Grants last at most fifteen minutes. Published readers receive -read-only grants; only writers receive upload and multipart rights. Responses -configure the native S3 origin and SDK credential-refresh endpoint without embedding -long-lived secrets or changing canonical metadata bytes. Table-load ETags include -the SDK configuration as well as the selected-generation metadata representation; -changed endpoints cannot be hidden by a metadata-only conditional response. -Routed operation checks and streamed -request/response limits already enforce signed scopes and server budgets. -A session token alone never authenticates a request. -The native path-style request parser preserves decoded object-key bytes and limits -operations to immutable object reads/writes, multipart subresources and -explicitly authorized cleanup. Unknown -query operations, duplicate parameters and general buckets fail closed. HTTP -DELETE identifies either an upload abort or separately authorized object cleanup; -batch cleanup uses the S3 DeleteObjects request and per-key response contract. -These request primitives are attached to the native listener. - -Writes and reads stream through bounded CROWDB storage clients. Delegated FileIO -access may move immutable ranges without an Access Server payload bounce, but -cannot overwrite published files or bypass its operation-specific authorization. - -## 4. Commit and lifecycle - -A composite table read checks CatalogContext at entry and before returning. -Inside that request it reuses context-bound namespace, table and file resolution -without repeating root reads. Admission resolves complete namespace ancestry: -namespace identities cannot move, and strict nonempty deletion protects every -ancestor while its child remains live. The final checks revalidate the selected -head generation, leaf mapping, authority identity/name epoch and lifecycle. -Property updates do not invalidate this binding; Dropping remains visible until -Tombstone. Public standalone repository methods retain their own context guards. -There is no cross-request permission cache or reader registration. The same -context permits capability additions only; catalog replacement advances its epoch. -Capability revocation or authority-only retirement would require revisiting this -request-local admission contract. - -A table commit validates requirements against one selected table generation, -constructs a complete new metadata state, writes immutable candidate files, and -atomically publishes one new table head. Concurrent commits either publish from -the generation they validated or fail for the client to reconcile; they never -merge implicitly. - -Ordinary commits prepare immutable metadata and publish with one TableHead CAS. -The Catalog config advertises `crowdb.guarded-maintenance=true` when table writes -are installed. Console maintenance uses the optional CrowDB extension -`x-crowdb-base-metadata-location` on an existing-table commit to bind the complete -reviewed metadata generation. A changed base conflicts; guarded `remove-snapshots` -also rejects every current or named-reference target on that exact base. The head -CAS prevents a concurrent Branch or Tag from being silently removed. This guard -does not change ordinary Iceberg update semantics and cannot accompany `assert-create`. -An exact successful request replay returns its published metadata. Prepared metadata -may still publish while its base remains current; when a newer generation conceals -that operation's metadata file, the result stays uncertain rather than being rebased. - -Existing-table commits do not write a phase journal, per-file committed state or post-publication -head settlement. No server instance is a table leader or lock owner. A current -head carries the operation-named metadata pointer and an optional 32-byte digest -binding principal, route and request bytes. A matching visible retry returns that -metadata; changed input conflicts and old heads without this binding remain -uncertain. After a later generation hides that result, -an old request may return uncertain rather than silently reapplying its update. - -Table creation retains its complete input, evaluated metadata and response only -inside the request. It writes the final metadata `FileRecord`, reserves the exact -namespace/name, checks that the same parent is Ready, creates the final table head -with an expected-absent CAS, and conditionally publishes the reserved name. This -order is required: a reservation precedes the last parent check, so namespace -deletion either sees it or prevents that check from succeeding. The head contains -no pending request identity. Cancellation discards request-local work; complete -client retries may create new unowned resources and never replay a stored response. -Response headroom is checked before any resource publication. - -A `RetiredTable` record occupies the same storage head key as a published table. -It binds the immutable catalog, TableId and optional storage namespace: 32 raw ID -bytes in the legacy layout, 48 in the native layout, plus a bounded envelope. It -contains no request, owner, schema, input, response, phase or continuation. It is a -permanent terminal resource fact and never returns to an absent or Ready slot. -Name resolution treats it as missing; credential issuance, initial publication -and physical write admission reject it. GC never interprets it as a live head. - -When namespace deletion or reservation cleanup encounters a reserved table whose -head is absent, it creates this retirement record by CAS before removing the -exact reservation. A conflict supplies the current head or retirement value; -cleanup uses that value without a confirmation GET. If a Ready head won, cleanup -can publish its matching reserved name and strict nonempty namespace deletion -fails. If retirement won, the delayed initial-head CAS fails permanently. Removing -only the reservation is unsafe: a delayed expected-absent head creation could -otherwise expose an object after the namespace's empty proof. Exact name bytes and -an input-bound reservation identity prevent delayed cleanup from removing a later table. -An uncertain CAS reply stops that request; a later full retry reconciles final -resource facts. - -For inline metadata, a successful immediate create performs four conditional -resource writes: metadata authority, name reservation, initial head and published -name. It creates no operation, input-page or response-page keys. Chunked metadata -adds only the existing per-range physical write intents. Ordinary creation does -not modify the parent namespace authority. Retiring an absent head adds one CAS -only on cancellation or GC; its retained metadata has fixed size independent of -table schema or request size. - -New-table requests may carry unique provisional field IDs starting at zero, -as produced by Spark. Creation assigns positive durable IDs before validating -the canonical schema and remaps identifier, partition, sort and nested-default -references consistently. Negative or duplicate provisional IDs remain invalid. -Existing-table commits and stored metadata still require their durable IDs. - -Staging evaluates metadata and returns its complete initialization material, -including a fresh UUID and storage identity, without writing any server record. -A repeated stage request produces independent material. The client retains the -chosen response and sends complete standard assert-create updates when ready. -The final request initializes an empty metadata builder with the supplied UUID, -location, schema, partitioning, sorting and properties, preserving field IDs -already used by client-prepared files. It evaluates initial metadata and -requirements, then follows the same resource publication order as immediate -creation without scanning staged data files or client manifest chains. - -An assert-create name reservation uses a domain-separated, length-delimited -digest of catalog activation, namespace identity, TableId, name, principal and -complete request bytes. It excludes request IDs and evaluation clocks, allowing -identical complete retries to use the exact existing reservation. Ordinary create -uses a fresh reservation identity. Competing input cannot adopt another reserved -resource. If the initial head already won, a retry only settles its matching name -and returns a conflict; clients load the published table to learn the result. -No response replay or server-owned draft is required. Unknown storage replies -stop the current request without a confirmation read. - -Bounded background scans rotate creation, legacy update and lifecycle journals, -four records per page. Independent continuations reset on catalog activation -changes. Legacy unbound drafts retain their existing expiry. Creation cleanup -never writes missing metadata or publishes a first head: it occupies an absent -head slot with permanent retirement, settles exact name/head markers from final -resource facts, and clears only the captured parent marker revision. After those -facts are durable, it conditionally deletes the old creation record. Unknown -writes stop cleanup. Physical allocation discovery remains in write intents. -Retired-catalog GC permanently retires the exact legacy head slot before releasing -that record's protection, whether the first head won or remained absent. It waits -for the catalog's existing retry and request grace; exact-head CAS conflicts stop -that step unless the returned value is already retired. Final file authorities -and per-range write intents remain independent discovery sources for physical -reclamation. An old nonterminal phase cannot retain the catalog indefinitely. -Legacy update and lifecycle recovery retain their own bounded compatibility paths. -New creators leave only resource facts for reservation cleanup and GC. - -Logical drop resolves the currently named table and conditionally tombstones its -exact head, then conditionally removes that table's name mapping. It writes no -request, input, phase or response journal. A conflicting in-flight head update -returns conflict without rebasing. A new DELETE retry resolves the name again -and may delete a table recreated at that name. Background cleanup of an older -table remains bound to its captured table ID and cannot remove the replacement. - -Same/cross-namespace rename stores no new input, phase, response or lifecycle -journal. Request admission validates the route and principal without reading an -old request journal or encoding a replay payload. Each affected parent holds one -bounded `NamespaceRename` occupancy containing the source mapping, target name, expected head generation/fence and a -fresh per-attempt reservation identity. Parents are occupied in stable ID order; -failure to acquire the second cancels and releases the first, without waiting on -another rename. Definite parent CAS conflicts use the observed authority directly, -preserving concurrent property changes. Ordinary readers do not acquire occupancy. - -The target name is reserved before one exact-head CAS switches the canonical -identifier and name epoch. Table identity, UUID, metadata generation, digest and -file location stay fixed. The destination is published before the captured source -mapping and parent occupancies are conditionally cleared. A namespace cannot enter -Dropping while occupied, including a same-parent move where a scan could otherwise -miss both names across the switch. Mapping visibility remains qualified by the -head, and the old name never becomes an alias. - -Recovery reads the head and name facts: a switched head completes the destination; -an unchanged head is first conditionally fenced to invalidate delayed publication, -then the reservation and occupancy can be removed. Conflicts use their returned -head. Namespace mapping sweeps and name-reservation/deletion helpers recover -abandoned occupancy. Every new attempt has a distinct reservation identity, even -when an HTTP request key is reused, preventing delayed cleanup of a canceled -attempt from deleting a later reservation. Cleanup never removes a recreated name -with a different identity. Existing lifecycle journals still finish through their -legacy recovery driver. - -Rename replies are generated directly from current resource state. After a -completed move the source is missing; repeating that request can return 404. -A recreated source with an occupied target conflicts. There is no historical -response replay or implicit rebasing of an in-flight head CAS. - -Drop tombstones the selected head without traversing snapshots or deleting files. -A purge request persists a `TablePurgeTask` containing the candidate tombstoned -head and activation epoch before publishing the tombstone, indexed by table, -generation and metadata file. Its initial marker time is zero. GC admits it only -after observing the exact tombstone and fixes a twenty-minute delay from that -observation; an existing admitted task never renews its deadline. Older markers -keep their stored times. If the deletion did not win, GC first fences the exact -Ready predecessor before removing the marker, blocking a delayed tombstone CAS. -Unknown writes stop immediately. Deferred or canceled markers do not keep an -otherwise idle scanner spinning. A drop without an explicit -purge request leaves the underlying files in place. Purge reclaims table-scoped -files after the delay and owner checks, without snapshot reachability scans -inside the dropped table. The REST -drop/rename routes require independent writer credentials and return empty success -responses; stale table names fail normal load, exists, commit and credential refresh. - -Drop and replacement remove logical authority first. Purge and retired-catalog -cleanup reclaim only their captured inactive identities after retention and -owner checks. Snapshot expiration alone does not delete published files. -Explicit single/batch cleanup scans retained references under the caller's -no-future-reference contract described below. - -The reclamation proof binds current and retained historical metadata to their -captured heads. Its immutable traversal stack and compressed binary file-ID index -use content-addressed payload pages. A task CAS publishes the pending stack and -mark root together; a missing page is an error, including during a nonmembership -query. Physical deletion runs only for a tombstoned table or retired catalog; -the worker rechecks inactive authority before sweeping. A Ready table may retain -unreachable files until drop or clear rather than interrupt reads or commits. -Unfinished table operations and active multipart work defer table purge. -Completed operation retry records and terminal multipart sessions do not extend -the 20-minute purge delay. Background task advancement is enabled by default; -it uses a separate storage client pool, one-step concurrency admission, bounded -KV and chunk request/byte budgets, and durable retry state. The -chunk byte budget covers payload reads and writes. Range retirement sends a -bounded control descriptor and consumes one chunk-operation request without -charging the referenced range as transferred payload. Large ranges therefore -remain eligible for notification under the same limits. These are logical -operation and payload budgets, not exact routed-RPC or wire-byte accounting. -enabled scheduler admits persisted table purge markers and completed catalog clears; -management may also start inactive tasks. Operators can disable the scheduler -or adjust validated resource limits. Discovery starts at startup. Each catalog -with unfinished discovery or runnable work continues after the retry-base delay -(one second by default, at least 100 ms and at most the configured scan interval). -One window still advances at most one worker step under its original I/O and time -budgets. A completed discovery round returns to the scan interval, 30 minutes by -default. A purge becomes eligible 20 minutes after the drop; discovery may add up -to one idle scan interval before its first step. A known task retry deadline can -wake its catalog sooner, without shortening retention or other resource deadlines. - -Scheduler discovery positions are discardable, per-catalog memory. Purge, -multipart admission, retired-catalog admission and task lanes rotate their first -turn before awaiting storage. Due catalogs rotate independently; one catalog's -failure does not delay another catalog's due work. A failed lane reports its -original error and leaves other lanes eligible. A validated raw row advances its local position before key -or value decoding and before any admission or worker action. An error, deadline -or cancellation therefore retains that observed position. An empty page closes -that lane's pass; actual task progress reopens its source lane when more work is -runnable. Terminal, paused and future-retry tasks do not keep a round busy. -Failed or cancelled lanes are rescanned with capped exponential backoff until a -complete error-free pass, so a long-lived corrupt row cannot cause a hot loop. -Multipart admission runs once per discovery round. Local scheduling retains at -most the 64 configured catalogs and the current catalog; retired identities remain -discoverable through durable system markers. Restart loses these hints and rescans the unchanged -durable markers, ledgers and tasks. These hints grant no deletion authority and -do not replace task proof, permanent retirement fences or retry state. - -Active multipart and deleted-file cleanup skips candidates whose fixed retention -deadline has not arrived, so later eligible files can advance in the same pass. -Those candidates and their deadlines remain unchanged. Complete denotes the end -of that eligibility pass; periodic admission reopens it to discover newly eligible -resources. Eligibility may therefore wait for the next scan interval and bounded -drain time. Reopened passes retain a monotonically increasing sweep round, so a -completed candidate contributes to deletion totals once. Inactive table purge -and retired-catalog cleanup retain their task-wide retention and fence rules. - -The dedicated read-only local scan retains validated catalog/scope/prefix bounds, -strict ordering, positive revisions and item/byte limits while allowing an -undecodable raw key as its continuation. It uses the native exclusive continuation -and an actual routed catalog generation, including for keys longer than the -Iceberg record format permits. Generic scans and persisted GC task continuations -keep their existing format validation. This path still uses the legacy scan -transport: its byte limit is checked after materialization. A transport failure -or an oversized response without an observed key cannot advance this position; -it remains an explicit error, with no claim of a wire memory bound or universal -poison-row recovery. - -Provisioned disk capacity is the allocation boundary for both foreground files -and GC durable workspace. A failed GC workspace write retains the last durable -continuation and defers retry; it never substitutes an incomplete proof or -authorizes deletion. Committed files remain readable when new chunk allocation -fails. Progress resumes after capacity is restored through the normal storage -flow. Shared-chunk ranges remain pending while range deletion is unsupported. - -Publication and logical-URL resolution check catalog and table or draft -authority without request-level pins. Direct reads instead use the validated -permission cache and authenticated locator. Physical reclamation observes the -configured retention interval. Logical deletion hides the file from listing; -its tombstone rejects logical-URL resolution but is not consulted by direct -HEAD/GET/Range GET. With a valid grant, an already known direct URL can still -read retained bytes until physical reclamation. Logical deletion is not an -instantaneous direct-URL revocation guarantee. Legacy live tasks are retired without -further deletion, releasing an owned table fence. Retained and deferred -candidates remain durable work for later inactive passes. - -An ordinary inline file in a table purge has no physical deletion frontier. -After checking retention, head generation, purge fence and immutable claim, one -bounded worker step conditionally deletes its exact file authority and persists -the completed candidate and task count. It does not persist an intermediate -Deleting checkpoint. Part, assembly and non-inline candidates retain their -destructive traversal checkpoints. An unknown mutation response stops the step; -a subsequent worker resumes from the durable file, candidate and task facts. - -An already authorized FileIO GET or HEAD can continue within the configured -retention interval after logical deletion. Native storage arbitrates direct -read/reclamation races, returning the original bytes or a storage error, never -bytes from a reused allocation. Logical-URL admission rejects a reclaiming -record. Uploads and new credentials require a Ready table. After logical drop, -direct URLs remain readable under still-valid grants while bytes are retained; -physical purge and grant expiry bound that access. - -Retired catalog recovery scans system retry and management ledgers before file -deletion and after the final file rescan. Pending or retained bindings stop the -pass; exact-identity overflow entries remain independent of occupied primary -slots. After files are reclaimed, the worker conditionally removes expired -bindings, audits, projections and non-GC catalog records while preserving the -active root's management operation. Multipart parts have their own durable tree -candidates. Assembly checkpoints have separate claims and a durable frontier-root -index. Abandoned frontiers are authenticated before traversal; a conflicted final -tree is traversed once instead of revisiting its shared frontier. Published -sessions reclaim only the checkpoint block, preserving the assembled data tree. -Unpublished server-generated initial metadata is identified by Metadata/JSON -resource type and the exact generation-one metadata filename bound to its FileId. -Its storage table must match the absent or retired final head slot; metadata for -another table cannot select that resource as its canonical head metadata. -The active resource sweep may discover this candidate when its final head is -absent or retired. Its first observation and existing safety retention remain -persisted in the GC candidate; restart never restarts or shortens that interval. -Immediately before reclaiming it, GC creates the same final-head retirement fact -if still absent. A Ready head returned by the CAS conflict preserves the file; -retirement winning prevents all delayed initial publishers. Only then may the -existing exact-range deletion cursor notify ChunkDB. Acceptance of DeleteRange is -the access server's completion boundary; it does not poll physical release. - -This proof does not authorize deleting data, manifests or inherited references -merely because their owner head is absent. Their existing final-resource reference -proof remains required. Physical ranges written before a final metadata authority -still retain their BlockIntent discovery facts. Both catalog cleanup layouts keep -retirement records permanently and allow those terminal facts in their final empty -proof; removing them would reopen a delayed expected-absent creation slot. - -Retired-catalog cleanup leaves a permanent retirement value at every discovered -table head key. A scanned head uses an exact-value CAS with the scan bytes. -Before removing a name mapping, cleanup reads its complete storage head key once -and retires the absent slot or observed head. This is independent of creation -journals, including a reserved name whose first head never won. One concurrent -slot change can be consumed directly from the CAS observation; further changes -leave the durable cleanup cursor unchanged. Unknown replies stop the step. A -restarted worker recognizes the retirement value before removing the exact -mapping. File authorities, range intents and GC candidates remain the physical -discovery chain. Table purge keeps its tombstone head; this cleanup cannot retire -a table in the current active catalog. - -An active catalog has a separate resource cleanup task for expired terminal parts, -assembly checkpoints, explicit deleted-file records and unpublished initial -metadata. It never infers missing snapshot references for other published files -or fences a Ready table. Candidate progress is -durable across restarts. After every part and checkpoint is reclaimed, it removes -the terminal session's payload pages and session record; pending part settlement -or unreleased multipart credit keeps that session available for recovery. -Streamed file and part candidates instead reclaim their exact Chunk location -ranges after retention. A published selected part remains owned by the immutable -file descriptor; part cleanup does not reclaim it a second time. Legacy tree allocations -abandoned before publication retain FileWriteIntent ownership. Native streamed -writes rely on Chunk-KV WAL and chunk allocation lifecycle rather than a -per-frame Iceberg intent record. Active-catalog cleanup does not reclaim an intent merely because a -file descriptor is absent: a multipart session in `Publishing` may still own -those blocks. Retired-catalog and table-purge passes can reclaim intents after -their authority and owner checks. The disk leak scanner alone does not establish -Iceberg ownership. -Each physical step rechecks the terminal session and retention. The checkpoint -block is deleted after its children, and session cleanup requires its completed -claim. Block intents are swept after tree candidates, preserving reachable owners -and any unfinished file or assembly cursor. An owner fence prevents publication -or new block writes once orphan deletion begins. Superseded intents of a reachable -owner are conservatively retained until that owner becomes unreachable. - -Final catalog cleanup verifies that all candidates are complete and that no owner -is paused or quarantined. A durable retirement marker selects the cleanup owner -and rejects stale GC mutations before bounded deletion of claims, candidates, -proof pages, write fences and old tasks. The retired authority, permanent table -retirement values, winning task result and retirement marker remain. Late -unfinished records stop cleanup. -Uncertain progress responses are resolved by reading durable state; an unfenced -live proof whose authority changed terminates without deleting files. - -## 5. Compatibility - -CROWDB covers the core Iceberg format semantics for v1, v2, and v3, including -reading, creating, writing, and the defined version upgrades. Mandatory behavior -for the selected format version is not weakened. Optional features are exposed -only when their complete semantics are enabled. - -REST wire types, Iceberg domain state, and CROWDB storage records remain -separate. Unknown or disabled requirements and updates fail before mutation. -The backed-up specifications decide behavior when implementations differ. - -### Executable conformance profile - -The official Java oracle is Apache Iceberg 1.11.0; the official Rust REST -client is 0.10.0. The declared selected-file profile uses Parquet data/deletes, -Avro manifests and Puffin deletion vectors/statistics. ORC bytes can be stored, -but selected ORC validation and compute-engine certification are separate work. - -- **v1/v2/v3 metadata and upgrades:** `official_java_metadata_roundtrips_without_rewriting`, - `official_catalog_creates_commits_upgrades_stages_and_refreshes_native_credentials`, - and the official create/update/snapshot fixtures compare canonical metadata. - `TestIcebergVersionRows` reads actual rows before and after adjacent upgrades, - reads historical snapshots, expires them logically, and reloads after restart. -- **Selected data and deletes:** client-generated fixtures exercise original - rows, equality-delete visibility and historical snapshots. Commit treats client - manifests as opaque references. `TestIcebergSelectedFiles` publishes mismatched - data/delete declarations in isolated tables and checks the selected head and - snapshot inventory; its valid table still verifies actual rows, equality deletes - and historical reads. Selected-use rejection probes are explicit validator - tests rather than a normal publication gate. `TestIcebergFileOperations` checks - that upload credentials lack cleanup permission (403 `AccessDenied`) and that - rejection preserves the immutable file. Canonical file - validators separately cover position deletes, v3 lineage, deletion vectors, - defaults, nested/variant types, integer encodings and nullable values. -- **Statistics:** `TestIcebergCatalogWrites` and the official partition-statistics - fixtures cover publication, replay, evolution and staged creation. - `TestIcebergCommitErrors` verifies supported partition-statistics metadata - publication, exact inventory reload and removal; a foreign file location - rejects the whole update batch and preserves the selected head. Metadata - acceptance is separate from explicit Parquet content validation. Historical - omissions follow the explicit compatibility rules described above. -- **Discovery and authorization:** - `discovery_uses_installed_routes_and_unsupported_paths_leave_no_record` and - `explicit_partial_activation_limits_discovery_and_table_admission` exercise - installed/disabled route and version combinations. HTTP namespace, table, - lifecycle, credentials and body tests cover roles, malformed requests, limits, - cancellation, unchanged authority on rejection and bounded metrics. -- **Faults and retirement:** `official_rust_client_observes_lost_create_reply_on_another_listener`, - `official_rust_client_lost_reply_survives_native_storage_restart`, and the - Java response-loss/retired-catalog fixtures exercise listener changes and - restart. The SDKs do not automatically replay a lost mutation POST with the - same key; direct HTTP fault tests prove the server's same-key replay contract. -- **Apache REST Compatibility Kit:** the unmodified 1.11.0 runner verifies six - supported cases: namespace create, basic table create, rename, drop, - missing-table drop and table list. This is not a full-kit pass. Other cases - assume register/views or local filesystem locations that the native authority - deliberately rejects. Custom SDK fixtures are not described as kit results. -- **Large logical files:** `tib_address_space_range_reads_keep_fixed_windows_and_small_authority` - and `tib_reclamation_progress_serializes_a_bounded_resumable_cursor` use a - 1 TiB logical tree with repeated immutable blocks. They prove bounded windows - and resumable state, not physical TiB capacity. Oversized declared metadata - is rejected before I/O at the configured metadata budget. - -## 6. Relationship to other access models - -General S3 and Iceberg share chunk, Chunk-KV, transport, credential, and safe -utility mechanisms, but not semantic authority. S3 bucket records, overwrite, -delete, listing, and lifecycle behavior cannot publish or alter an Iceberg -table. - -Dataset may consume data described by an Iceberg table through an explicit, -generation-bound reference. That does not transfer snapshot, commit, file, or -reclamation authority to Dataset. - -The native file endpoint exposes separately authorized S3 `DeleteObject` and -`DeleteObjects` cleanup. A cleanup credential is requested through the table -credentials endpoint with `cleanup=true`; ordinary FileIO credentials cannot -delete objects. The server scans retained table metadata and refuses a path -that is referenced or whose reference status is uncertain. Successful deletion -records a logical tombstone and removes list visibility. The background worker -discovers that final tombstone and durably claims its exact old file before -physical reclamation after 20 minutes. DELETE writes no GC task or candidate; -restart discovers the same DeletedFile. Direct URLs remain readable until their -bytes are reclaimed, subject to grant validity. Their FileIds/locators are never -reused. A new logical-path PUT can immediately reuse its path: if the worker has -not queued the old file, PUT first retains its exact resource in the existing -GC claim and candidate records, then conditionally replaces the observed -DeletedFile. An uncertain write stops without a confirmation read. Either the -deletion marker or the independent GC candidate retains the old reclamation -obligation across failure. Batch deletion returns individual errors inside -HTTP 200, so callers must inspect every item. - -> **Warning:** Before deleting, the client must stop and resolve every writer -> and retry that might still commit the path, including unpublished metadata. -> The server cannot see a future commit during its reference scan. If a client -> later commits a deleted path, the snapshot can reference missing data. Do not -> use cleanup credentials in ordinary FileIO. A reader retaining old metadata -> must finish before the 20-minute physical reclamation deadline. - -## 7. Correctness invariants - -- **ICE-I1 — Native authority:** catalog, namespace, table, snapshot, commit, - file, and reclamation semantics belong to Iceberg rather than general S3. -- **ICE-I2 — Standard recoverability:** published standard Iceberg metadata is - sufficient to recover table state; derived projections are not authoritative. -- **ICE-I3 — Atomic table publication:** a table head selects exactly one - complete immutable metadata generation. -- **ICE-I4 — Validated concurrency:** a commit publishes only from the table - generation against which its requirements were validated. -- **ICE-I5 — Immutable files:** a published Iceberg file is never overwritten. -- **ICE-I6 — Reachability before reclamation:** physical deletion cannot precede - proof that no protected Iceberg state reaches the file. -- **ICE-I7 — Spec compliance:** supported v1, v2, and v3 behavior preserves all - mandatory semantics of the selected format version. -- **ICE-I8 — Bounded operation:** table size and history do not determine one - Access Server request's retained memory or unbounded work. -- **ICE-I9 — Live file retention:** a Ready table's published file is not - reclaimed solely because no retained snapshot refers to it; active cleanup - handles expired multipart remnants and explicitly deleted files with durable - owner checks. -- **ICE-I10 — Stable identity directory:** rename and cross-namespace move - preserve storage namespace, table path and every published file URL. -- **ICE-I11 — Single file authority:** one filename-keyed record selects each - published file; list and lifecycle use that record without secondary file indexes. -- **ICE-I12 — Direct native read:** a direct URL supplies its immutable native - descriptor without Iceberg file-location or file-state KV queries; native - chunk layout resolution and repair retain their normal behavior. - Logical deletion removes list visibility; physical GC ends byte availability. -- **ICE-I13 — Bounded metadata scale:** large metadata bytes live in native - objects, descriptors/head remain bounded, prefix listing reads bounded KV - pages, and optional projections cannot replace standard metadata authority. diff --git a/doc/design/access-server/iceberge/iceberg-rest-catalog-open-api-1.11.0.yaml b/doc/design/access-server/iceberge/iceberg-rest-catalog-open-api-1.11.0.yaml deleted file mode 100644 index 06d13ec13..000000000 --- a/doc/design/access-server/iceberge/iceberg-rest-catalog-open-api-1.11.0.yaml +++ /dev/null @@ -1,5285 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. -# - ---- -openapi: 3.1.1 -info: - title: Apache Iceberg REST Catalog API - license: - name: Apache 2.0 - url: https://www.apache.org/licenses/LICENSE-2.0.html - version: 0.0.1 - description: - Defines the specification for the first version of the REST Catalog API. - Implementations should ideally support all Iceberg table spec versions. -servers: - - url: "{scheme}://{host}/{basePath}" - description: Server URL when the port can be inferred from the scheme - variables: - scheme: - description: The scheme of the URI, either http or https. - default: https - host: - description: The host address for the specified server - default: localhost - basePath: - description: Optional prefix to be appended to all routes - default: "" - - url: "{scheme}://{host}:{port}/{basePath}" - description: Generic base server URL, with all parts configurable - variables: - scheme: - description: The scheme of the URI, either http or https. - default: https - host: - description: The host address for the specified server - default: localhost - port: - description: The port used when addressing the host - default: "443" - basePath: - description: Optional prefix to be appended to all routes - default: "" -# All routes are currently configured using an Authorization header. -security: - - OAuth2: [catalog] - - BearerAuth: [] - -paths: - /v1/config: - - get: - tags: - - Configuration API - summary: List all catalog configuration settings - operationId: getConfig - parameters: - - name: warehouse - in: query - required: false - schema: - type: string - description: Warehouse location or identifier to request from the service - description: - " - All REST clients should first call this route to get catalog configuration - properties from the server to configure the catalog and its HTTP client. - Configuration from the server consists of two sets of key/value pairs. - - - defaults - properties that should be used as default configuration; applied before client configuration - - - overrides - properties that should be used to override client configuration; applied after defaults and client configuration - - - Catalog configuration is constructed by setting the defaults, then client- - provided configuration, and finally overrides. The final property set is then - used to configure the catalog. - - - For example, a default configuration property might set the size of the - client pool, which can be replaced with a client-specific setting. An - override might be used to set the warehouse location, which is stored - on the server rather than in client configuration. - - - Common catalog configuration settings are documented at - https://iceberg.apache.org/docs/latest/configuration/#catalog-properties - - - The catalog configuration also holds an optional `endpoints` field that contains information about the endpoints - supported by the server. If a server does not send the `endpoints` field, a default set of endpoints is assumed: - - - GET /v1/{prefix}/namespaces - - - POST /v1/{prefix}/namespaces - - - GET /v1/{prefix}/namespaces/{namespace} - - - DELETE /v1/{prefix}/namespaces/{namespace} - - - POST /v1/{prefix}/namespaces/{namespace}/properties - - - GET /v1/{prefix}/namespaces/{namespace}/tables - - - POST /v1/{prefix}/namespaces/{namespace}/tables - - - GET /v1/{prefix}/namespaces/{namespace}/tables/{table} - - - POST /v1/{prefix}/namespaces/{namespace}/tables/{table} - - - DELETE /v1/{prefix}/namespaces/{namespace}/tables/{table} - - - POST /v1/{prefix}/namespaces/{namespace}/register - - - POST /v1/{prefix}/namespaces/{namespace}/tables/{table}/metrics - - - POST /v1/{prefix}/tables/rename - - - POST /v1/{prefix}/transactions/commit - " - responses: - 200: - description: Server specified configuration values. - content: - application/json: - schema: - $ref: '#/components/schemas/CatalogConfig' - example: { - "overrides": { - "warehouse": "s3://bucket/warehouse/" - }, - "defaults": { - "clients": "4" - }, - "idempotency-key-lifetime": "PT30M", - "endpoints": [ - "GET /v1/{prefix}/namespaces/{namespace}", - "GET /v1/{prefix}/namespaces", - "POST /v1/{prefix}/namespaces", - "GET /v1/{prefix}/namespaces/{namespace}/tables/{table}", - "GET /v1/{prefix}/namespaces/{namespace}/views/{view}" - ] - } - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: Not Found - Warehouse provided in the `warehouse` query parameter is not found. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - NoSuchWarehouseExample: - $ref: '#/components/examples/NoSuchWarehouseError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - /v1/oauth/tokens: - - post: - tags: - - OAuth2 API - summary: Get a token using an OAuth2 flow (DEPRECATED for REMOVAL) - deprecated: true - operationId: getToken - description: - The `oauth/tokens` endpoint is **DEPRECATED for REMOVAL**. It is _not_ recommended to - implement this endpoint, unless you are fully aware of the potential security implications. - - All clients are encouraged to explicitly set the configuration property `oauth2-server-uri` - to the correct OAuth endpoint. - - Deprecated since Iceberg (Java) 1.6.0. The endpoint and related types will be removed from - this spec in Iceberg (Java) 2.0. - - See [Security improvements in the Iceberg REST specification](https://github.com/apache/iceberg/issues/10537) - - - Exchange credentials for a token using the OAuth2 client credentials flow or token exchange. - - - This endpoint is used for three purposes - - - 1. To exchange client credentials (client ID and secret) for an access token - This uses the client credentials flow. - - 2. To exchange a client token and an identity token for a more specific access token - This uses the token exchange flow. - - 3. To exchange an access token for one with the same claims and a refreshed expiration period - This uses the token exchange flow. - - - For example, a catalog client may be configured with client credentials from the OAuth2 - Authorization flow. This client would exchange its client ID and secret for an access token - using the client credentials request with this endpoint (1). Subsequent requests would then - use that access token. - - - Some clients may also handle sessions that have additional user context. These clients would - use the token exchange flow to exchange a user token (the "subject" token) from the session - for a more specific access token for that user, using the catalog's access token as the - "actor" token (2). The user ID token is the "subject" token and can be any token type - allowed by the OAuth2 token exchange flow, including a unsecured JWT token with a sub claim. - This request should use the catalog's bearer token in the "Authorization" header. - - - Clients may also use the token exchange flow to refresh a token that is about to expire by - sending a token exchange request (3). The request's "subject" token should be the expiring - token. This request should use the subject token in the "Authorization" header. - requestBody: - required: true - content: - application/x-www-form-urlencoded: - schema: - $ref: '#/components/schemas/OAuthTokenRequest' - responses: - 200: - $ref: '#/components/responses/OAuthTokenResponse' - 400: - $ref: '#/components/responses/OAuthErrorResponse' - 401: - $ref: '#/components/responses/OAuthErrorResponse' - 5XX: - $ref: '#/components/responses/OAuthErrorResponse' - - /v1/{prefix}/namespaces: - parameters: - - $ref: '#/components/parameters/prefix' - - get: - tags: - - Catalog API - summary: List namespaces, optionally providing a parent namespace to list underneath - description: - List all namespaces at a certain level, optionally starting from a given parent namespace. - If table accounting.tax.paid.info exists, using 'SELECT NAMESPACE IN accounting' would - translate into `GET /namespaces?parent=accounting` and must return a namespace, ["accounting", "tax"] only. - Using 'SELECT NAMESPACE IN accounting.tax' would - translate into `GET /namespaces?parent=accounting%1Ftax` and must return a namespace, ["accounting", "tax", "paid"]. - If `parent` is not provided, all top-level namespaces should be listed. - operationId: listNamespaces - parameters: - - $ref: '#/components/parameters/page-token' - - $ref: '#/components/parameters/page-size' - - name: parent - in: query - description: - An optional namespace, underneath which to list namespaces. - If not provided, all top-level namespaces should be listed. - For backward compatibility, empty string is treated as absent for now. - If parent is a multipart namespace, the parts must be separated by the namespace separator as - indicated via the /config override `namespace-separator`, which defaults to the unit separator `0x1F` byte (url encoded `%1F`). - To be compatible with older clients, servers must use both the advertised separator and `0x1F` as valid separators when decoding namespaces. - The `namespace-separator` should be provided in a url encoded form. - required: false - schema: - type: string - example: "accounting%1Ftax" - responses: - 200: - $ref: '#/components/responses/ListNamespacesResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: Not Found - Namespace provided in the `parent` query parameter is not found. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - NoSuchNamespaceExample: - $ref: '#/components/examples/NoSuchNamespaceError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - post: - tags: - - Catalog API - summary: Create a namespace - parameters: - - $ref: '#/components/parameters/idempotency-key' - description: - Create a namespace, with an optional set of properties. - The server might also add properties, such as `last_modified_time` etc. - operationId: createNamespace - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/CreateNamespaceRequest' - responses: - 200: - $ref: '#/components/responses/CreateNamespaceResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 406: - $ref: '#/components/responses/UnsupportedOperationResponse' - 409: - description: Conflict - The namespace already exists - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - NamespaceAlreadyExists: - $ref: '#/components/examples/NamespaceAlreadyExistsError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - /v1/{prefix}/namespaces/{namespace}: - parameters: - - $ref: '#/components/parameters/prefix' - - $ref: '#/components/parameters/namespace' - - get: - tags: - - Catalog API - summary: Load the metadata properties for a namespace - operationId: loadNamespaceMetadata - description: Return all stored metadata properties for a given namespace - responses: - 200: - $ref: '#/components/responses/GetNamespaceResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: Not Found - Namespace not found - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - NoSuchNamespaceExample: - $ref: '#/components/examples/NoSuchNamespaceError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - head: - tags: - - Catalog API - summary: Check if a namespace exists - operationId: namespaceExists - description: - Check if a namespace exists. The response does not contain a body. - responses: - 204: - description: Success, no content - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: Not Found - Namespace not found - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - NoSuchNamespaceExample: - $ref: '#/components/examples/NoSuchNamespaceError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - delete: - tags: - - Catalog API - summary: Drop a namespace from the catalog. Namespace must be empty. - operationId: dropNamespace - parameters: - - $ref: '#/components/parameters/idempotency-key' - responses: - 204: - description: Success, no content - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: Not Found - Namespace to delete does not exist. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - NoSuchNamespaceExample: - $ref: '#/components/examples/NoSuchNamespaceError' - 409: - description: Not Empty - Namespace to delete is not empty. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - NamespaceNotEmptyExample: - $ref: '#/components/examples/NamespaceNotEmptyError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - /v1/{prefix}/namespaces/{namespace}/properties: - parameters: - - $ref: '#/components/parameters/prefix' - - $ref: '#/components/parameters/namespace' - - post: - tags: - - Catalog API - summary: Set or remove properties on a namespace - operationId: updateProperties - parameters: - - $ref: '#/components/parameters/idempotency-key' - description: - Set and/or remove properties on a namespace. - The request body specifies a list of properties to remove and a map - of key value pairs to update. - - Properties that are not in the request are not modified or removed by this call. - - Server implementations are not required to support namespace properties. - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/UpdateNamespacePropertiesRequest' - examples: - UpdateAndRemoveProperties: - $ref: '#/components/examples/UpdateAndRemoveNamespacePropertiesRequest' - responses: - 200: - $ref: '#/components/responses/UpdateNamespacePropertiesResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: Not Found - Namespace not found - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - NamespaceNotFound: - $ref: '#/components/examples/NoSuchNamespaceError' - 406: - $ref: '#/components/responses/UnsupportedOperationResponse' - 422: - description: Unprocessable Entity - A property key was included in both `removals` and `updates` - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - UnprocessableEntityDuplicateKey: - $ref: '#/components/examples/UnprocessableEntityDuplicateKey' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - /v1/{prefix}/namespaces/{namespace}/tables: - parameters: - - $ref: '#/components/parameters/prefix' - - $ref: '#/components/parameters/namespace' - - get: - tags: - - Catalog API - summary: List all table identifiers underneath a given namespace - description: Return all table identifiers under this namespace - operationId: listTables - parameters: - - $ref: '#/components/parameters/page-token' - - $ref: '#/components/parameters/page-size' - responses: - 200: - $ref: '#/components/responses/ListTablesResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: Not Found - The namespace specified does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - NamespaceNotFound: - $ref: '#/components/examples/NoSuchNamespaceError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - post: - tags: - - Catalog API - summary: Create a table in the given namespace - description: - Create a table or start a create transaction, like atomic CTAS. - - - If `stage-create` is false, the table is created immediately. - - - If `stage-create` is true, the table is not created, but table metadata is initialized and returned. - The service should prepare as needed for a commit to the table commit endpoint to complete the create - transaction. The client uses the returned metadata to begin a transaction. To commit the transaction, - the client sends all create and subsequent changes to the table commit route. Changes from the table - create operation include changes like AddSchemaUpdate and SetCurrentSchemaUpdate that set the initial - table state. - operationId: createTable - parameters: - - $ref: '#/components/parameters/data-access' - - $ref: '#/components/parameters/idempotency-key' - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/CreateTableRequest' - responses: - 200: - $ref: '#/components/responses/CreateTableResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: Not Found - The namespace specified does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - NamespaceNotFound: - $ref: '#/components/examples/NoSuchNamespaceError' - 409: - description: Conflict - The identifier already exists as a table or view - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - TableAlreadyExists: - $ref: '#/components/examples/TableAlreadyExistsError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - /v1/{prefix}/namespaces/{namespace}/tables/{table}/plan: - parameters: - - $ref: '#/components/parameters/prefix' - - $ref: '#/components/parameters/namespace' - - $ref: '#/components/parameters/table' - - $ref: '#/components/parameters/data-access' - post: - tags: - - Catalog API - summary: Submit a scan for planning - description: > - Submits a scan for server-side planning. - - - Point-in-time scans are planned by passing snapshot-id to identify the - table snapshot to scan. Incremental scans are planned by passing both - start-snapshot-id and end-snapshot-id. Requests that include both point - in time config properties and incremental config properties are - invalid. If the request does not include either incremental or - point-in-time config properties, scan planning should produce a - point-in-time scan of the latest snapshot in the table's main branch. - - - Responses must include a valid status listed below. A "cancelled" status is considered invalid for this endpoint. - - - When "completed" the planning operation has produced plan tasks and - file scan tasks that must be returned in the response (not fetched - later by calling fetchPlanningResult) - - - When "submitted" the response must include a plan-id used to poll - fetchPlanningResult to fetch the planning result when it is ready - - - When "failed" the response must be a valid error response - - The response for a "completed" planning operation includes two types of - tasks (file scan tasks and plan tasks) and both may be included in the - response. Tasks must not be included for any other response status. - - - Responses that include a plan-id indicate that the service is holding - state or performing work for the client. - - - - Clients should use the plan-id to fetch results from - fetchPlanningResult when the response status is "submitted" - - - Clients should inform the service if planning results are no longer - needed by calling cancelPlanning. Cancellation is not necessary after - fetchScanTasks has been used to fetch scan tasks for each plan task. - operationId: planTableScan - parameters: - - $ref: '#/components/parameters/idempotency-key' - requestBody: - content: - application/json: - schema: - $ref: '#/components/schemas/PlanTableScanRequest' - responses: - 200: - $ref: '#/components/responses/PlanTableScanResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: - Not Found - - NoSuchTableException, the table does not exist - - NoSuchNamespaceException, the namespace does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - TableDoesNotExist: - $ref: '#/components/examples/NoSuchTableError' - NamespaceDoesNotExist: - $ref: '#/components/examples/NoSuchNamespaceError' - 406: - $ref: '#/components/responses/UnsupportedOperationResponse' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - /v1/{prefix}/namespaces/{namespace}/tables/{table}/plan/{plan-id}: - parameters: - - $ref: '#/components/parameters/prefix' - - $ref: '#/components/parameters/namespace' - - $ref: '#/components/parameters/table' - - $ref: '#/components/parameters/plan-id' - - $ref: '#/components/parameters/data-access' - - get: - tags: - - Catalog API - summary: Fetches the result of scan planning for a plan-id - operationId: fetchPlanningResult - description: > - Fetches the result of scan planning for a plan-id. - - - Responses must include a valid status - - - When "completed" the planning operation has produced plan-tasks and - file-scan-tasks that must be returned in the response - - - When "submitted" the planning operation has not completed; the client - should wait to call this endpoint again to fetch a completed response - - - When "failed" the response must be a valid error response - - - When "cancelled" the plan-id is invalid and should be discarded - - - The response for a "completed" planning operation includes two types of - tasks (file scan tasks and plan tasks) and both may be included in the - response. Tasks must not be included for any other response status. - responses: - 200: - $ref: '#/components/responses/FetchPlanningResultResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: - Not Found - - NoSuchPlanIdException, the plan-id does not exist - - NoSuchTableException, the table does not exist - - NoSuchNamespaceException, the namespace does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - PlanIdDoesNotExist: - $ref: '#/components/examples/NoSuchPlanIdError' - TableDoesNotExist: - $ref: '#/components/examples/NoSuchTableError' - NamespaceDoesNotExist: - $ref: '#/components/examples/NoSuchNamespaceError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - delete: - tags: - - Catalog API - summary: Cancels scan planning for a plan-id - operationId: cancelPlanning - parameters: - - $ref: '#/components/parameters/idempotency-key' - description: > - Cancels scan planning for a plan-id. - - - This notifies the service that it can release resources held for the - scan. Clients should cancel scans that are no longer needed, either - while the plan-id returns a "submitted" status or while there are - remaining plan tasks that have not been fetched. - - - Cancellation is not necessary when - - - Scan tasks for each plan task have been fetched using fetchScanTasks - - - A plan-id has produced a "failed" or "cancelled" status from - planTableScan or fetchPlanningResult - responses: - 204: - description: Success, no content - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: - Not Found - - NoSuchPlanIdException, the plan-id does not exist - - NoSuchTableException, the table does not exist - - NoSuchNamespaceException, the namespace does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - PlanIdDoesNotExist: - $ref: '#/components/examples/NoSuchPlanIdError' - TableDoesNotExist: - $ref: '#/components/examples/NoSuchTableError' - NamespaceDoesNotExist: - $ref: '#/components/examples/NoSuchNamespaceError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - - /v1/{prefix}/namespaces/{namespace}/tables/{table}/tasks: - parameters: - - $ref: '#/components/parameters/prefix' - - $ref: '#/components/parameters/namespace' - - $ref: '#/components/parameters/table' - - post: - tags: - - Catalog API - summary: Fetches result tasks for a plan task - operationId: fetchScanTasks - parameters: - - $ref: '#/components/parameters/idempotency-key' - description: Fetches result tasks for a plan task. - requestBody: - content: - application/json: - schema: - $ref: '#/components/schemas/FetchScanTasksRequest' - responses: - 200: - $ref: '#/components/responses/FetchScanTasksResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: - Not Found - - NoSuchPlanTaskException, the plan-task does not exist - - NoSuchTableException, the table does not exist - - NoSuchNamespaceException, the namespace does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - PlanTaskDoesNotExist: - $ref: '#/components/examples/NoSuchPlanTaskError' - TableDoesNotExist: - $ref: '#/components/examples/NoSuchTableError' - NamespaceDoesNotExist: - $ref: '#/components/examples/NoSuchNamespaceError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - /v1/{prefix}/namespaces/{namespace}/register: - parameters: - - $ref: '#/components/parameters/prefix' - - $ref: '#/components/parameters/namespace' - - post: - tags: - - Catalog API - summary: Register a table in the given namespace using given metadata file location - parameters: - - $ref: '#/components/parameters/data-access' - - $ref: '#/components/parameters/idempotency-key' - description: - Register a table using given metadata file location. - - operationId: registerTable - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/RegisterTableRequest' - responses: - 200: - $ref: '#/components/responses/LoadTableResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: Not Found - The namespace specified does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - NamespaceNotFound: - $ref: '#/components/examples/NoSuchNamespaceError' - 409: - description: Conflict - The identifier already exists as a table or view - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - TableAlreadyExists: - $ref: '#/components/examples/TableAlreadyExistsError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - /v1/{prefix}/namespaces/{namespace}/tables/{table}: - parameters: - - $ref: '#/components/parameters/prefix' - - $ref: '#/components/parameters/namespace' - - $ref: '#/components/parameters/table' - - get: - tags: - - Catalog API - summary: Load a table from the catalog - operationId: loadTable - description: - Load a table from the catalog. - - - The response contains both configuration and table metadata. The configuration, if non-empty is used - as additional configuration for the table that overrides catalog configuration. For example, this - configuration may change the FileIO implementation to be used for the table. - - - The response also contains the table's full metadata, matching the table metadata JSON file. - - - The catalog configuration may contain credentials that should be used for subsequent requests for the - table. The configuration key "token" is used to pass an access token to be used as a bearer token - for table requests. Otherwise, a token may be passed using a RFC 8693 token type as a configuration - key. For example, "urn:ietf:params:oauth:token-type:jwt=". - parameters: - - $ref: '#/components/parameters/data-access' - - name: If-None-Match - in: header - description: - An optional header that allows the server to return 304 (Not Modified) if the metadata - is current. The content is the value of the ETag received in a CreateTableResponse, - LoadTableResponse or CommitTableResponse. - required: false - schema: - type: string - - in: query - name: snapshots - description: - The snapshots to return in the body of the metadata via the `snapshots` field. Setting - the value to `all` would return the full set of snapshots currently valid for the table. - Setting the value to `refs` would load all snapshots referenced by branches or tags. - - Default if no param is provided is `all`. - required: false - schema: - type: string - enum: [ all, refs ] - - $ref: '#/components/parameters/referenced-by' - responses: - 200: - $ref: '#/components/responses/LoadTableResponse' - 304: - description: - Not Modified - Based on the content of the 'If-None-Match' header the table metadata has - not changed since. - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: - Not Found - NoSuchTableException, table to load does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - TableToLoadDoesNotExist: - $ref: '#/components/examples/NoSuchTableError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - post: - tags: - - Catalog API - summary: Commit updates to a table - operationId: updateTable - parameters: - - $ref: '#/components/parameters/idempotency-key' - description: - Commit updates to a table. - - - Commits have two parts, requirements and updates. Requirements are assertions that will be validated - before attempting to make and commit changes. For example, `assert-ref-snapshot-id` will check that a - named ref's snapshot ID has a certain value. - Server implementations are required to fail with a 400 status code - if any unknown updates or requirements are received. - - - Updates are changes to make to table metadata. For example, after asserting that the current main ref - is at the expected snapshot, a commit may add a new child snapshot and set the ref to the new - snapshot id. - - - Create table transactions that are started by createTable with `stage-create` set to true are - committed using this route. Transactions should include all changes to the table, including table - initialization, like AddSchemaUpdate and SetCurrentSchemaUpdate. The `assert-create` requirement is - used to ensure that the table was not created concurrently. - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/CommitTableRequest' - responses: - 200: - $ref: '#/components/responses/CommitTableResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: - Not Found - NoSuchTableException, table to load does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - TableToUpdateDoesNotExist: - $ref: '#/components/examples/NoSuchTableError' - 409: - description: - Conflict - CommitFailedException, one or more requirements failed. The client may retry. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 500: - description: - An unknown server-side problem occurred; the commit state is unknown. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - example: { - "error": { - "message": "Internal Server Error", - "type": "CommitStateUnknownException", - "code": 500 - } - } - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 502: - description: - A gateway or proxy received an invalid response from the upstream server; the commit state is unknown. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - example: { - "error": { - "message": "Invalid response from the upstream server", - "type": "CommitStateUnknownException", - "code": 502 - } - } - 504: - description: - A server-side gateway timeout occurred; the commit state is unknown. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - example: { - "error": { - "message": "Gateway timed out during commit", - "type": "CommitStateUnknownException", - "code": 504 - } - } - 5XX: - description: - A server-side problem that might not be addressable on the client. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - example: { - "error": { - "message": "Bad Gateway", - "type": "InternalServerError", - "code": 502 - } - } - - delete: - tags: - - Catalog API - summary: Drop a table from the catalog - operationId: dropTable - description: Remove a table from the catalog - parameters: - - $ref: '#/components/parameters/idempotency-key' - - name: purgeRequested - in: query - required: false - description: Whether the user requested to purge the underlying table's data and metadata - schema: - type: boolean - default: false - responses: - 204: - description: Success, no content - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: - Not Found - NoSuchTableException, Table to drop does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - TableToDeleteDoesNotExist: - $ref: '#/components/examples/NoSuchTableError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - head: - tags: - - Catalog API - summary: Check if a table exists - operationId: tableExists - description: - Check if a table exists within a given namespace. The response does not contain a body. - responses: - 204: - description: Success, no content - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: - Not Found - NoSuchTableException, Table not found - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - TableToLoadDoesNotExist: - $ref: '#/components/examples/NoSuchTableError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - /v1/{prefix}/namespaces/{namespace}/tables/{table}/credentials: - parameters: - - $ref: '#/components/parameters/prefix' - - $ref: '#/components/parameters/namespace' - - $ref: '#/components/parameters/table' - - get: - tags: - - Catalog API - summary: Load vended credentials for a table from the catalog - operationId: loadCredentials - parameters: - - name: planId - in: query - required: false - schema: - type: string - description: The plan ID that has been used for server-side scan planning - - $ref: '#/components/parameters/referenced-by' - description: Load vended credentials for a table from the catalog. - responses: - 200: - $ref: '#/components/responses/LoadCredentialsResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: - Not Found - NoSuchTableException, table to load credentials for does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - TableToLoadDoesNotExist: - $ref: '#/components/examples/NoSuchTableError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - /v1/{prefix}/namespaces/{namespace}/tables/{table}/sign: - parameters: - - $ref: '#/components/parameters/prefix' - - $ref: '#/components/parameters/namespace' - - $ref: '#/components/parameters/table' - - post: - tags: - - Catalog API - summary: Remotely signs requests to object storage - operationId: signRequest - requestBody: - description: The request to be signed - content: - application/json: - schema: - $ref: '#/components/schemas/RemoteSignRequest' - required: true - responses: - 200: - $ref: '#/components/responses/RemoteSignResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - /v1/{prefix}/tables/rename: - parameters: - - $ref: '#/components/parameters/prefix' - - post: - tags: - - Catalog API - summary: Rename a table from its current name to a new name - parameters: - - $ref: '#/components/parameters/idempotency-key' - description: - Rename a table from one identifier to another. It's valid to move a table - across namespaces, but the server implementation is not required to support it. - operationId: renameTable - requestBody: - description: Current table identifier to rename and new table identifier to rename to - content: - application/json: - schema: - $ref: '#/components/schemas/RenameTableRequest' - examples: - RenameTableSameNamespace: - $ref: '#/components/examples/RenameTableSameNamespace' - required: true - responses: - 204: - description: Success, no content - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: - Not Found - - NoSuchTableException, Table to rename does not exist - - NoSuchNamespaceException, The target namespace of the new table identifier does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - TableToRenameDoesNotExist: - $ref: '#/components/examples/NoSuchTableError' - NamespaceToRenameToDoesNotExist: - $ref: '#/components/examples/NoSuchNamespaceError' - 406: - $ref: '#/components/responses/UnsupportedOperationResponse' - 409: - description: Conflict - The target identifier to rename to already exists as a table or view - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - example: - $ref: '#/components/examples/TableAlreadyExistsError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - /v1/{prefix}/namespaces/{namespace}/tables/{table}/metrics: - parameters: - - $ref: '#/components/parameters/prefix' - - $ref: '#/components/parameters/namespace' - - $ref: '#/components/parameters/table' - - post: - tags: - - Catalog API - summary: Send a metrics report to this endpoint to be processed by the backend - operationId: reportMetrics - requestBody: - description: The request containing the metrics report to be sent - content: - application/json: - schema: - $ref: '#/components/schemas/ReportMetricsRequest' - required: true - responses: - 204: - description: Success, no content - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: - Not Found - NoSuchTableException, table to load does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - TableToLoadDoesNotExist: - $ref: '#/components/examples/NoSuchTableError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - /v1/{prefix}/transactions/commit: - parameters: - - $ref: '#/components/parameters/prefix' - - post: - tags: - - Catalog API - summary: Commit updates to multiple tables in an atomic operation - operationId: commitTransaction - parameters: - - $ref: '#/components/parameters/idempotency-key' - requestBody: - description: - Commit updates to multiple tables in an atomic operation - - - A commit for a single table consists of a table identifier with requirements and updates. - Requirements are assertions that will be validated before attempting to make and commit changes. - For example, `assert-ref-snapshot-id` will check that a named ref's snapshot ID has a certain value. - Server implementations are required to fail with a 400 status code - if any unknown updates or requirements are received. - - Updates are changes to make to table metadata. For example, after asserting that the current main ref - is at the expected snapshot, a commit may add a new child snapshot and set the ref to the new - snapshot id. - content: - application/json: - schema: - $ref: '#/components/schemas/CommitTransactionRequest' - required: true - responses: - 204: - description: Success, no content - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: - Not Found - NoSuchTableException, table to load does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - TableToUpdateDoesNotExist: - $ref: '#/components/examples/NoSuchTableError' - 409: - description: - Conflict - CommitFailedException, one or more requirements failed. The client may retry. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 500: - description: - An unknown server-side problem occurred; the commit state is unknown. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - example: { - "error": { - "message": "Internal Server Error", - "type": "CommitStateUnknownException", - "code": 500 - } - } - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 502: - description: - A gateway or proxy received an invalid response from the upstream server; the commit state is unknown. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - example: { - "error": { - "message": "Invalid response from the upstream server", - "type": "CommitStateUnknownException", - "code": 502 - } - } - 504: - description: - A server-side gateway timeout occurred; the commit state is unknown. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - example: { - "error": { - "message": "Gateway timed out during commit", - "type": "CommitStateUnknownException", - "code": 504 - } - } - 5XX: - description: - A server-side problem that might not be addressable on the client. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - example: { - "error": { - "message": "Bad Gateway", - "type": "InternalServerError", - "code": 502 - } - } - - /v1/{prefix}/namespaces/{namespace}/views: - parameters: - - $ref: '#/components/parameters/prefix' - - $ref: '#/components/parameters/namespace' - - get: - tags: - - Catalog API - summary: List all view identifiers underneath a given namespace - description: Return all view identifiers under this namespace - operationId: listViews - parameters: - - $ref: '#/components/parameters/page-token' - - $ref: '#/components/parameters/page-size' - responses: - 200: - $ref: '#/components/responses/ListTablesResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: Not Found - The namespace specified does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorModel' - examples: - NamespaceNotFound: - $ref: '#/components/examples/NoSuchNamespaceError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - post: - tags: - - Catalog API - summary: Create a view in the given namespace - description: - Create a view in the given namespace. - operationId: createView - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/CreateViewRequest' - responses: - 200: - $ref: '#/components/responses/LoadViewResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: Not Found - The namespace specified does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorModel' - examples: - NamespaceNotFound: - $ref: '#/components/examples/NoSuchNamespaceError' - 409: - description: Conflict - The identifier already exists as a table or view - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorModel' - examples: - ViewAlreadyExists: - $ref: '#/components/examples/ViewAlreadyExistsError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - /v1/{prefix}/namespaces/{namespace}/views/{view}: - parameters: - - $ref: '#/components/parameters/prefix' - - $ref: '#/components/parameters/namespace' - - $ref: '#/components/parameters/view' - - get: - tags: - - Catalog API - summary: Load a view from the catalog - operationId: loadView - description: - Load a view from the catalog. - - - The response contains both configuration and view metadata. The configuration, if non-empty is used - as additional configuration for the view that overrides catalog configuration. - - - The response also contains the view's full metadata, matching the view metadata JSON file. - - - The catalog configuration may contain credentials that should be used for subsequent requests for the - view. The configuration key "token" is used to pass an access token to be used as a bearer token - for view requests. Otherwise, a token may be passed using a RFC 8693 token type as a configuration - key. For example, "urn:ietf:params:oauth:token-type:jwt=". - parameters: - - $ref: '#/components/parameters/referenced-by' - responses: - 200: - $ref: '#/components/responses/LoadViewResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: - Not Found - NoSuchViewException, view to load does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorModel' - examples: - ViewToLoadDoesNotExist: - $ref: '#/components/examples/NoSuchViewError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - post: - tags: - - Catalog API - summary: Replace a view - operationId: replaceView - parameters: - - $ref: '#/components/parameters/idempotency-key' - description: - Commit updates to a view. - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/CommitViewRequest' - responses: - 200: - $ref: '#/components/responses/LoadViewResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: - Not Found - NoSuchViewException, view to load does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorModel' - examples: - ViewToUpdateDoesNotExist: - $ref: '#/components/examples/NoSuchViewError' - 409: - description: - Conflict - CommitFailedException. The client may retry. - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorModel' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 500: - description: - An unknown server-side problem occurred; the commit state is unknown. - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorModel' - example: { - "error": { - "message": "Internal Server Error", - "type": "CommitStateUnknownException", - "code": 500 - } - } - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 502: - description: - A gateway or proxy received an invalid response from the upstream server; the commit state is unknown. - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorModel' - example: { - "error": { - "message": "Invalid response from the upstream server", - "type": "CommitStateUnknownException", - "code": 502 - } - } - 504: - description: - A server-side gateway timeout occurred; the commit state is unknown. - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorModel' - example: { - "error": { - "message": "Gateway timed out during commit", - "type": "CommitStateUnknownException", - "code": 504 - } - } - 5XX: - description: - A server-side problem that might not be addressable on the client. - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorModel' - example: { - "error": { - "message": "Bad Gateway", - "type": "InternalServerError", - "code": 502 - } - } - - delete: - tags: - - Catalog API - summary: Drop a view from the catalog - operationId: dropView - description: Remove a view from the catalog - parameters: - - $ref: '#/components/parameters/idempotency-key' - responses: - 204: - description: Success, no content - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: - Not Found - NoSuchViewException, view to drop does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorModel' - examples: - ViewToDeleteDoesNotExist: - $ref: '#/components/examples/NoSuchViewError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - head: - tags: - - Catalog API - summary: Check if a view exists - operationId: viewExists - description: - Check if a view exists within a given namespace. This request does not return a response body. - responses: - 204: - description: Success, no content - 400: - description: Bad Request - 401: - description: Unauthorized - 404: - description: Not Found - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - /v1/{prefix}/views/rename: - parameters: - - $ref: '#/components/parameters/prefix' - - post: - tags: - - Catalog API - summary: Rename a view from its current name to a new name - description: - Rename a view from one identifier to another. It's valid to move a view - across namespaces, but the server implementation is not required to support it. - operationId: renameView - parameters: - - $ref: '#/components/parameters/idempotency-key' - requestBody: - description: Current view identifier to rename and new view identifier to rename to - content: - application/json: - schema: - $ref: '#/components/schemas/RenameTableRequest' - examples: - RenameViewSameNamespace: - $ref: '#/components/examples/RenameViewSameNamespace' - required: true - responses: - 204: - description: Success, no content - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: - Not Found - - NoSuchViewException, view to rename does not exist - - NoSuchNamespaceException, The target namespace of the new identifier does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorModel' - examples: - ViewToRenameDoesNotExist: - $ref: '#/components/examples/NoSuchViewError' - NamespaceToRenameToDoesNotExist: - $ref: '#/components/examples/NoSuchNamespaceError' - 406: - $ref: '#/components/responses/UnsupportedOperationResponse' - 409: - description: Conflict - The target identifier to rename to already exists as a table or view - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorModel' - example: - $ref: '#/components/examples/ViewAlreadyExistsError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - - /v1/{prefix}/namespaces/{namespace}/register-view: - parameters: - - $ref: '#/components/parameters/prefix' - - $ref: '#/components/parameters/namespace' - - post: - tags: - - Catalog API - summary: Register a view in the catalog - parameters: - - $ref: '#/components/parameters/idempotency-key' - description: - Register a view in the given namespace using given metadata file location. - - operationId: registerView - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/RegisterViewRequest' - responses: - 200: - $ref: '#/components/responses/LoadViewResponse' - 400: - $ref: '#/components/responses/BadRequestErrorResponse' - 401: - $ref: '#/components/responses/UnauthorizedResponse' - 403: - $ref: '#/components/responses/ForbiddenResponse' - 404: - description: Not Found - The namespace specified does not exist - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - NamespaceNotFound: - $ref: '#/components/examples/NoSuchNamespaceError' - 409: - description: Conflict - The identifier already exists as a table or view - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - examples: - ViewAlreadyExists: - $ref: '#/components/examples/ViewAlreadyExistsError' - 419: - $ref: '#/components/responses/AuthenticationTimeoutResponse' - 503: - $ref: '#/components/responses/ServiceUnavailableResponse' - 5XX: - $ref: '#/components/responses/ServerErrorResponse' - -components: - ####################################################### - # Common Parameter Definitions Used In Several Routes # - ####################################################### - parameters: - namespace: - name: namespace - in: path - required: true - description: - A namespace identifier as a single string. - Multipart namespace parts must be separated by the namespace separator as - indicated via the /config override `namespace-separator`, which defaults to the unit separator `0x1F` byte (url encoded `%1F`). - To be compatible with older clients, servers must use both the advertised separator and `0x1F` as valid separators when decoding namespaces. - The `namespace-separator` should be provided in a url encoded form. - schema: - type: string - examples: - singlepart_namespace: - value: "accounting" - multipart_namespace: - value: "accounting%1Ftax" - - prefix: - name: prefix - in: path - schema: - type: string - required: true - description: An optional prefix in the path - - table: - name: table - in: path - description: A table name - required: true - schema: - type: string - example: "sales" - - plan-id: - name: plan-id - in: path - description: ID used to track a planning request - required: true - schema: - type: string - - view: - name: view - in: path - description: A view name - required: true - schema: - type: string - example: "sales" - - data-access: - name: X-Iceberg-Access-Delegation - in: header - description: > - Optional signal to the server that the client supports delegated access - via a comma-separated list of access mechanisms. The server may choose - to supply access via any or none of the requested mechanisms. - - - Specific properties and handling for `vended-credentials` and `remote-signing` - are documented in the `LoadTableResult` schema section of this spec document. - - required: false - schema: - type: string - enum: - - vended-credentials - - remote-signing - style: simple - explode: false - example: "vended-credentials,remote-signing" - - page-token: - name: pageToken - in: query - required: false - schema: - $ref: '#/components/schemas/PageToken' - - page-size: - name: pageSize - in: query - description: - For servers that support pagination, this signals an upper bound of the number of results that a client will receive. - For servers that do not support pagination, clients may receive results larger than the indicated `pageSize`. - required: false - schema: - type: integer - minimum: 1 - - etag: - name: ETag - in: header - description: - Identifies a unique version of the table metadata. - - Implementations that support ETags should produce unique tags for responses that return - different metadata content but represent the same version of table metadata. For example, - the `snapshots` query parameter may result in different metadata representations depending - on whether `refs` or `all` is provided, therefore should have distinct ETags. - required: false - schema: - type: string - - referenced-by: - name: referenced-by - in: query - description: - A comma-separated list of fully qualified view names (namespace and view name) representing the view - reference chain when an entity (table or view) is loaded via a view. The list should be ordered with the outermost view - first, followed by any intermediate views it references, down to the view that directly references the entity. - For a simple case where a view directly references the entity, the list contains a single view identifier. - For nested views (a view referencing another view which references the entity), the list contains multiple - view identifiers representing the complete dependency chain. - - The view identifier is a composite string of the format {namespace}{separator}{viewName}. - The namespace-separator (defined in /config) acts as the delimiter. - When parsing, the last occurrence of this separator identifies the boundary between the namespace and the view name. - Multipart namespaces must follow the encoding rules of the parent parameter in the list namespaces endpoint. - - Multiple view identifiers are separated by commas. Servers should split the parameter value on comma characters - to parse individual view identifiers. If view names contain commas, they must be url-encoded as %2C. - - Example with multiple views (where prod%1Fanalytics is a nested namespace which has a quarterly_view which references to monthly_view - which then references the entity being loaded) - prod%1Fanalytics%1Fquarterly_view,prod%1Fanalytics%1Fmonthly_view - required: false - schema: - type: string - - idempotency-key: - name: Idempotency-Key - in: header - required: false - schema: - type: string - format: uuid - minLength: 36 - maxLength: 36 - example: "017F22E2-79B0-7CC3-98C4-DC0C0C07398F" - description: | - Optional client-provided idempotency key for safe request retries. - - When present, the server ensures no additional effects for requests that carry the same - Idempotency-Key. If a prior request with this key has been finalized, the server returns - an equivalent final response without re-running the operation. The response body may - reflect a newer state of the catalog than existed at the time of the commit. - - Finalization rules: - - Finalize & replay: 200, 201, 204, and deterministic terminal 4xx (including 409 - such as AlreadyExists, NamespaceNotEmpty, etc.) - - Do not finalize (not stored/replayed): 5xx - - Key Requirements: - - Key format: UUIDv7 in string form (RFC 9562). - - The idempotency key must be globally unique (no reuse across different operations). - - Catalogs SHOULD NOT expire keys before the end of the advertised token lifetime. - - If Idempotency-Key is used, clients MUST reuse the same key when retrying the same - logical operation and MUST generate a new key for a different operation. - - ############################## - # Application Schema Objects # - ############################## - schemas: - - ErrorModel: - type: object - description: JSON error payload returned in a response with further details on the error - required: - - message - - type - - code - properties: - message: - type: string - description: Human-readable error message - type: - type: string - description: Internal type definition of the error - example: NoSuchNamespaceException - code: - type: integer - minimum: 400 - maximum: 600 - description: HTTP response code - example: 404 - stack: - type: array - items: - type: string - - CatalogConfig: - type: object - description: Server-provided configuration for the catalog. - required: - - defaults - - overrides - properties: - overrides: - type: object - additionalProperties: - type: string - description: - Properties that should be used to override client configuration; applied after defaults and client configuration. - defaults: - type: object - additionalProperties: - type: string - description: - Properties that should be used as default configuration; applied before client configuration. - endpoints: - type: array - items: - type: string - description: A list of endpoints that the server supports. The format of each endpoint must be " ". - The HTTP verb and the resource path must be separated by a space character. - example: [ - "GET /v1/{prefix}/namespaces/{namespace}", - "GET /v1/{prefix}/namespaces", - "POST /v1/{prefix}/namespaces", - "GET /v1/{prefix}/namespaces/{namespace}/tables/{table}", - "GET /v1/{prefix}/namespaces/{namespace}/views/{view}" - ] - idempotency-key-lifetime: - type: string - format: duration - description: - Client reuse window for an Idempotency-Key (ISO-8601 duration, e.g., PT30M, PT24H). - Interpreted as the maximum time from the first submission using a key to the last retry - during which a client may reuse that key. Servers SHOULD accept retries for at least this - duration and MAY include a grace period to account for delays/clock skew. Clients SHOULD NOT - reuse an Idempotency-Key after this window elapses; they SHOULD generate a new key for any - subsequent attempt. Presence of this field indicates the server supports Idempotency-Key - semantics for mutation endpoints. If absent, clients MUST assume idempotency is not supported. - example: PT30M - - CreateNamespaceRequest: - type: object - required: - - namespace - properties: - namespace: - $ref: '#/components/schemas/Namespace' - properties: - type: object - description: Configured string to string map of properties for the namespace - example: { "owner": "Hank Bendickson" } - default: { } - additionalProperties: - type: string - - UpdateNamespacePropertiesRequest: - type: object - properties: - removals: - type: array - uniqueItems: true - items: - type: string - example: [ "department", "access_group" ] - updates: - type: object - example: { "owner": "Hank Bendickson" } - additionalProperties: - type: string - - RenameTableRequest: - type: object - required: - - source - - destination - properties: - source: - $ref: '#/components/schemas/TableIdentifier' - destination: - $ref: '#/components/schemas/TableIdentifier' - - Namespace: - description: Reference to one or more levels of a namespace - type: array - items: - type: string - example: [ "accounting", "tax" ] - - PageToken: - description: - An opaque token that allows clients to make use of pagination for list APIs - (e.g. ListTables). Clients may initiate the first paginated request by sending an empty - query parameter `pageToken` to the server. - - Servers that support pagination should identify the `pageToken` parameter and return a - `next-page-token` in the response if there are more results available. After the initial - request, the value of `next-page-token` from each response must be used as the `pageToken` - parameter value for the next request. The server must return `null` value for the - `next-page-token` in the last response. - - Servers that support pagination must return all results in a single response with the value - of `next-page-token` set to `null` if the query parameter `pageToken` is not set in the - request. - - Servers that do not support pagination should ignore the `pageToken` parameter and return - all results in a single response. The `next-page-token` must be omitted from the response. - - Clients must interpret either `null` or missing response value of `next-page-token` as - the end of the listing results. - - type: string - nullable: true - - TableIdentifier: - type: object - required: - - namespace - - name - properties: - namespace: - $ref: '#/components/schemas/Namespace' - name: - type: string - nullable: false - - PrimitiveType: - type: string - example: - - "long" - - "string" - - "fixed[16]" - - "decimal(10,2)" - - StructField: - type: object - required: - - id - - name - - type - - required - properties: - id: - type: integer - name: - type: string - type: - $ref: '#/components/schemas/Type' - required: - type: boolean - doc: - type: string - initial-default: - $ref: "#/components/schemas/PrimitiveTypeValue" - write-default: - $ref: "#/components/schemas/PrimitiveTypeValue" - - StructType: - type: object - required: - - type - - fields - properties: - type: - type: string - const: "struct" - fields: - type: array - items: - $ref: '#/components/schemas/StructField' - - ListType: - type: object - required: - - type - - element-id - - element - - element-required - properties: - type: - type: string - const: "list" - element-id: - type: integer - element: - $ref: '#/components/schemas/Type' - element-required: - type: boolean - - MapType: - type: object - required: - - type - - key-id - - key - - value-id - - value - - value-required - properties: - type: - type: string - const: "map" - key-id: - type: integer - key: - $ref: '#/components/schemas/Type' - value-id: - type: integer - value: - $ref: '#/components/schemas/Type' - value-required: - type: boolean - - Type: - oneOf: - - $ref: '#/components/schemas/PrimitiveType' - - $ref: '#/components/schemas/StructType' - - $ref: '#/components/schemas/ListType' - - $ref: '#/components/schemas/MapType' - - Schema: - allOf: - - $ref: '#/components/schemas/StructType' - - type: object - properties: - schema-id: - type: integer - readOnly: true - identifier-field-ids: - type: array - items: - type: integer - - Expression: - oneOf: - - $ref: '#/components/schemas/TrueExpression' - - $ref: '#/components/schemas/FalseExpression' - - $ref: '#/components/schemas/AndOrExpression' - - $ref: '#/components/schemas/NotExpression' - - $ref: '#/components/schemas/SetExpression' - - $ref: '#/components/schemas/LiteralExpression' - - $ref: '#/components/schemas/UnaryExpression' - - ExpressionType: - type: string - example: - - "true" - - "false" - - "eq" - - "and" - - "or" - - "not" - - "in" - - "not-in" - - "lt" - - "lt-eq" - - "gt" - - "gt-eq" - - "not-eq" - - "starts-with" - - "not-starts-with" - - "is-null" - - "not-null" - - "is-nan" - - "not-nan" - - TrueExpression: - type: object - required: - - type - properties: - type: - $ref: '#/components/schemas/ExpressionType' - const: "true" - - FalseExpression: - type: object - required: - - type - properties: - type: - $ref: '#/components/schemas/ExpressionType' - const: "false" - - AndOrExpression: - type: object - required: - - type - - left - - right - properties: - type: - $ref: '#/components/schemas/ExpressionType' - enum: ["and", "or"] - left: - $ref: '#/components/schemas/Expression' - right: - $ref: '#/components/schemas/Expression' - - NotExpression: - type: object - required: - - type - - child - properties: - type: - $ref: '#/components/schemas/ExpressionType' - const: "not" - child: - $ref: '#/components/schemas/Expression' - - UnaryExpression: - type: object - required: - - type - - term - properties: - type: - $ref: '#/components/schemas/ExpressionType' - enum: ["is-null", "not-null", "is-nan", "not-nan"] - term: - $ref: '#/components/schemas/Term' - - LiteralExpression: - type: object - required: - - type - - term - - value - properties: - type: - $ref: '#/components/schemas/ExpressionType' - enum: ["lt", "lt-eq", "gt", "gt-eq", "eq", "not-eq", "starts-with", "not-starts-with"] - term: - $ref: '#/components/schemas/Term' - value: - $ref: '#/components/schemas/PrimitiveTypeValue' - - SetExpression: - type: object - required: - - type - - term - - values - properties: - type: - $ref: '#/components/schemas/ExpressionType' - enum: ["in", "not-in"] - term: - $ref: '#/components/schemas/Term' - values: - type: array - items: - $ref: '#/components/schemas/PrimitiveTypeValue' - - Term: - oneOf: - - $ref: '#/components/schemas/Reference' - - $ref: '#/components/schemas/TransformTerm' - - Reference: - type: string - example: - - "column-name" - - TransformTerm: - type: object - required: - - type - - transform - - term - properties: - type: - type: string - const: "transform" - transform: - $ref: '#/components/schemas/Transform' - term: - $ref: '#/components/schemas/Reference' - - Transform: - type: string - example: - - "identity" - - "year" - - "month" - - "day" - - "hour" - - "bucket[256]" - - "truncate[16]" - - PartitionField: - type: object - required: - - source-id - - transform - - name - properties: - field-id: - type: integer - source-id: - type: integer - name: - type: string - transform: - $ref: '#/components/schemas/Transform' - - PartitionSpec: - type: object - required: - - fields - properties: - spec-id: - type: integer - readOnly: true - fields: - type: array - items: - $ref: '#/components/schemas/PartitionField' - - SortDirection: - type: string - enum: ["asc", "desc"] - - NullOrder: - type: string - enum: ["nulls-first", "nulls-last"] - - SortField: - type: object - required: - - source-id - - transform - - direction - - null-order - properties: - source-id: - type: integer - transform: - $ref: '#/components/schemas/Transform' - direction: - $ref: '#/components/schemas/SortDirection' - null-order: - $ref: '#/components/schemas/NullOrder' - - SortOrder: - type: object - required: - - order-id - - fields - properties: - order-id: - type: integer - readOnly: true - fields: - type: array - items: - $ref: '#/components/schemas/SortField' - - EncryptedKey: - type: object - required: - - key-id - - encrypted-key-metadata - properties: - key-id: - type: string - encrypted-key-metadata: - type: string - format: byte # for compatibility - contentEncoding: base64 - encrypted-by-id: - type: string - properties: - type: object - additionalProperties: - type: string - - Snapshot: - type: object - required: - - snapshot-id - - timestamp-ms - - manifest-list - - summary - properties: - snapshot-id: - type: integer - format: int64 - parent-snapshot-id: - type: integer - format: int64 - sequence-number: - type: integer - format: int64 - timestamp-ms: - type: integer - format: int64 - manifest-list: - type: string - description: Location of the snapshot's manifest list file - first-row-id: - type: integer - format: int64 - description: The first _row_id assigned to the first row in the first data file in the first manifest - added-rows: - type: integer - format: int64 - description: The upper bound of the number of rows with assigned row IDs - summary: - type: object - required: - - operation - properties: - operation: - type: string - enum: ["append", "replace", "overwrite", "delete"] - additionalProperties: - type: string - schema-id: - type: integer - - SnapshotReference: - type: object - required: - - type - - snapshot-id - properties: - type: - type: string - enum: ["tag", "branch"] - snapshot-id: - type: integer - format: int64 - max-ref-age-ms: - type: integer - format: int64 - max-snapshot-age-ms: - type: integer - format: int64 - min-snapshots-to-keep: - type: integer - - SnapshotReferences: - type: object - additionalProperties: - $ref: '#/components/schemas/SnapshotReference' - - SnapshotLog: - type: array - items: - type: object - required: - - snapshot-id - - timestamp-ms - properties: - snapshot-id: - type: integer - format: int64 - timestamp-ms: - type: integer - format: int64 - - MetadataLog: - type: array - items: - type: object - required: - - metadata-file - - timestamp-ms - properties: - metadata-file: - type: string - timestamp-ms: - type: integer - format: int64 - - TableMetadata: - type: object - required: - - format-version - - table-uuid - properties: - format-version: - type: integer - minimum: 1 - maximum: 3 - table-uuid: - type: string - location: - type: string - last-updated-ms: - type: integer - format: int64 - next-row-id: - type: integer - format: int64 - description: A long higher than all assigned row IDs; the next snapshot's first-row-id. - properties: - type: object - additionalProperties: - type: string - # schema tracking - schemas: - type: array - items: - $ref: '#/components/schemas/Schema' - current-schema-id: - type: integer - last-column-id: - type: integer - # partition spec tracking - partition-specs: - type: array - items: - $ref: '#/components/schemas/PartitionSpec' - default-spec-id: - type: integer - last-partition-id: - type: integer - # sort order tracking - sort-orders: - type: array - items: - $ref: '#/components/schemas/SortOrder' - default-sort-order-id: - type: integer - # encryption - encryption-keys: - type: array - items: - $ref: '#/components/schemas/EncryptedKey' - # snapshot tracking - snapshots: - type: array - items: - $ref: '#/components/schemas/Snapshot' - refs: - $ref: '#/components/schemas/SnapshotReferences' - current-snapshot-id: - type: integer - format: int64 - last-sequence-number: - type: integer - format: int64 - # logs - snapshot-log: - $ref: '#/components/schemas/SnapshotLog' - metadata-log: - $ref: '#/components/schemas/MetadataLog' - # statistics - statistics: - type: array - items: - $ref: '#/components/schemas/StatisticsFile' - partition-statistics: - type: array - items: - $ref: '#/components/schemas/PartitionStatisticsFile' - - SQLViewRepresentation: - type: object - required: - - type - - sql - - dialect - properties: - type: - type: string - sql: - type: string - dialect: - type: string - - ViewRepresentation: - oneOf: - - $ref: '#/components/schemas/SQLViewRepresentation' - - ViewHistoryEntry: - type: object - required: - - version-id - - timestamp-ms - properties: - version-id: - type: integer - timestamp-ms: - type: integer - format: int64 - - ViewVersion: - type: object - required: - - version-id - - timestamp-ms - - schema-id - - summary - - representations - - default-namespace - properties: - version-id: - type: integer - timestamp-ms: - type: integer - format: int64 - schema-id: - type: integer - description: Schema ID to set as current, or -1 to set last added schema - summary: - type: object - additionalProperties: - type: string - representations: - type: array - items: - $ref: '#/components/schemas/ViewRepresentation' - default-catalog: - type: string - default-namespace: - $ref: '#/components/schemas/Namespace' - - ViewMetadata: - type: object - required: - - view-uuid - - format-version - - location - - current-version-id - - versions - - version-log - - schemas - properties: - view-uuid: - type: string - format-version: - type: integer - minimum: 1 - maximum: 1 - location: - type: string - current-version-id: - type: integer - versions: - type: array - items: - $ref: '#/components/schemas/ViewVersion' - version-log: - type: array - items: - $ref: '#/components/schemas/ViewHistoryEntry' - schemas: - type: array - items: - $ref: '#/components/schemas/Schema' - properties: - type: object - additionalProperties: - type: string - - BaseUpdate: - discriminator: - propertyName: action - mapping: - assign-uuid: '#/components/schemas/AssignUUIDUpdate' - upgrade-format-version: '#/components/schemas/UpgradeFormatVersionUpdate' - add-schema: '#/components/schemas/AddSchemaUpdate' - set-current-schema: '#/components/schemas/SetCurrentSchemaUpdate' - add-spec: '#/components/schemas/AddPartitionSpecUpdate' - set-default-spec: '#/components/schemas/SetDefaultSpecUpdate' - add-sort-order: '#/components/schemas/AddSortOrderUpdate' - set-default-sort-order: '#/components/schemas/SetDefaultSortOrderUpdate' - add-snapshot: '#/components/schemas/AddSnapshotUpdate' - set-snapshot-ref: '#/components/schemas/SetSnapshotRefUpdate' - remove-snapshots: '#/components/schemas/RemoveSnapshotsUpdate' - remove-snapshot-ref: '#/components/schemas/RemoveSnapshotRefUpdate' - set-location: '#/components/schemas/SetLocationUpdate' - set-properties: '#/components/schemas/SetPropertiesUpdate' - remove-properties: '#/components/schemas/RemovePropertiesUpdate' - add-view-version: '#/components/schemas/AddViewVersionUpdate' - set-current-view-version: '#/components/schemas/SetCurrentViewVersionUpdate' - set-statistics: '#/components/schemas/SetStatisticsUpdate' - remove-statistics: '#/components/schemas/RemoveStatisticsUpdate' - set-partition-statistics: '#/components/schemas/SetPartitionStatisticsUpdate' - remove-partition-statistics: '#/components/schemas/RemovePartitionStatisticsUpdate' - remove-partition-specs: '#/components/schemas/RemovePartitionSpecsUpdate' - remove-schemas: '#/components/schemas/RemoveSchemasUpdate' - add-encryption-key: '#/components/schemas/AddEncryptionKeyUpdate' - remove-encryption-key: '#/components/schemas/RemoveEncryptionKeyUpdate' - type: object - required: - - action - properties: - action: - type: string - - AssignUUIDUpdate: - description: Assigning a UUID to a table/view should only be done when creating the table/view. It is not safe to re-assign the UUID if a table/view already has a UUID assigned - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - uuid - properties: - action: - type: string - const: "assign-uuid" - uuid: - type: string - - UpgradeFormatVersionUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - format-version - properties: - action: - type: string - const: "upgrade-format-version" - format-version: - type: integer - - AddSchemaUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - schema - properties: - action: - type: string - const: "add-schema" - schema: - $ref: '#/components/schemas/Schema' - last-column-id: - type: integer - deprecated: true - description: - This optional field is **DEPRECATED for REMOVAL** since it more safe to handle this internally, - and shouldn't be exposed to the clients. - - The highest assigned column ID for the table. This is used to ensure columns are always - assigned an unused ID when evolving schemas. When omitted, it will be computed on the server side. - - SetCurrentSchemaUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - schema-id - properties: - action: - type: string - const: "set-current-schema" - schema-id: - type: integer - description: Schema ID to set as current, or -1 to set last added schema - - AddPartitionSpecUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - spec - properties: - action: - type: string - const: "add-spec" - spec: - $ref: '#/components/schemas/PartitionSpec' - - SetDefaultSpecUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - spec-id - properties: - action: - type: string - const: "set-default-spec" - spec-id: - type: integer - description: Partition spec ID to set as the default, or -1 to set last added spec - - AddSortOrderUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - sort-order - properties: - action: - type: string - const: "add-sort-order" - sort-order: - $ref: '#/components/schemas/SortOrder' - - SetDefaultSortOrderUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - sort-order-id - properties: - action: - type: string - const: "set-default-sort-order" - sort-order-id: - type: integer - description: Sort order ID to set as the default, or -1 to set last added sort order - - AddSnapshotUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - snapshot - properties: - action: - type: string - const: "add-snapshot" - snapshot: - $ref: '#/components/schemas/Snapshot' - - SetSnapshotRefUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - - $ref: '#/components/schemas/SnapshotReference' - required: - - ref-name - properties: - action: - type: string - const: "set-snapshot-ref" - ref-name: - type: string - - RemoveSnapshotsUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - snapshot-ids - properties: - action: - type: string - const: "remove-snapshots" - snapshot-ids: - type: array - items: - type: integer - format: int64 - - RemoveSnapshotRefUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - ref-name - properties: - action: - type: string - const: "remove-snapshot-ref" - ref-name: - type: string - - SetLocationUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - location - properties: - action: - type: string - const: "set-location" - location: - type: string - - SetPropertiesUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - updates - properties: - action: - type: string - const: "set-properties" - updates: - type: object - additionalProperties: - type: string - - RemovePropertiesUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - removals - properties: - action: - type: string - const: "remove-properties" - removals: - type: array - items: - type: string - - AddViewVersionUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - view-version - properties: - action: - type: string - const: "add-view-version" - view-version: - $ref: '#/components/schemas/ViewVersion' - - SetCurrentViewVersionUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - view-version-id - properties: - action: - type: string - const: "set-current-view-version" - view-version-id: - type: integer - description: The view version id to set as current, or -1 to set last added view version id - - SetStatisticsUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - statistics - properties: - action: - type: string - const: "set-statistics" - snapshot-id: - type: integer - format: int64 - deprecated: true - description: - This optional field is **DEPRECATED for REMOVAL** since it contains redundant information. - Clients should use the `statistics.snapshot-id` field instead. - statistics: - $ref: '#/components/schemas/StatisticsFile' - - RemoveStatisticsUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - snapshot-id - properties: - action: - type: string - const: "remove-statistics" - snapshot-id: - type: integer - format: int64 - - SetPartitionStatisticsUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - partition-statistics - properties: - action: - type: string - const: "set-partition-statistics" - partition-statistics: - $ref: '#/components/schemas/PartitionStatisticsFile' - - RemovePartitionStatisticsUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - snapshot-id - properties: - action: - type: string - const: "remove-partition-statistics" - snapshot-id: - type: integer - format: int64 - - RemovePartitionSpecsUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - spec-ids - properties: - action: - type: string - const: "remove-partition-specs" - spec-ids: - type: array - items: - type: integer - - RemoveSchemasUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - schema-ids - properties: - action: - type: string - const: "remove-schemas" - schema-ids: - type: array - items: - type: integer - - AddEncryptionKeyUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - encryption-key - properties: - action: - type: string - const: "add-encryption-key" - encryption-key: - $ref: '#/components/schemas/EncryptedKey' - - RemoveEncryptionKeyUpdate: - allOf: - - $ref: '#/components/schemas/BaseUpdate' - required: - - key-id - properties: - action: - type: string - const: "remove-encryption-key" - key-id: - type: string - - TableUpdate: - anyOf: - - $ref: '#/components/schemas/AssignUUIDUpdate' - - $ref: '#/components/schemas/UpgradeFormatVersionUpdate' - - $ref: '#/components/schemas/AddSchemaUpdate' - - $ref: '#/components/schemas/SetCurrentSchemaUpdate' - - $ref: '#/components/schemas/AddPartitionSpecUpdate' - - $ref: '#/components/schemas/SetDefaultSpecUpdate' - - $ref: '#/components/schemas/AddSortOrderUpdate' - - $ref: '#/components/schemas/SetDefaultSortOrderUpdate' - - $ref: '#/components/schemas/AddSnapshotUpdate' - - $ref: '#/components/schemas/SetSnapshotRefUpdate' - - $ref: '#/components/schemas/RemoveSnapshotsUpdate' - - $ref: '#/components/schemas/RemoveSnapshotRefUpdate' - - $ref: '#/components/schemas/SetLocationUpdate' - - $ref: '#/components/schemas/SetPropertiesUpdate' - - $ref: '#/components/schemas/RemovePropertiesUpdate' - - $ref: '#/components/schemas/SetStatisticsUpdate' - - $ref: '#/components/schemas/RemoveStatisticsUpdate' - - $ref: '#/components/schemas/SetPartitionStatisticsUpdate' - - $ref: '#/components/schemas/RemovePartitionStatisticsUpdate' - - $ref: '#/components/schemas/RemovePartitionSpecsUpdate' - - $ref: '#/components/schemas/RemoveSchemasUpdate' - - $ref: '#/components/schemas/AddEncryptionKeyUpdate' - - $ref: '#/components/schemas/RemoveEncryptionKeyUpdate' - - ViewUpdate: - anyOf: - - $ref: '#/components/schemas/AssignUUIDUpdate' - - $ref: '#/components/schemas/UpgradeFormatVersionUpdate' - - $ref: '#/components/schemas/AddSchemaUpdate' - - $ref: '#/components/schemas/SetLocationUpdate' - - $ref: '#/components/schemas/SetPropertiesUpdate' - - $ref: '#/components/schemas/RemovePropertiesUpdate' - - $ref: '#/components/schemas/AddViewVersionUpdate' - - $ref: '#/components/schemas/SetCurrentViewVersionUpdate' - - TableRequirement: - type: object - discriminator: - propertyName: type - mapping: - assert-create: '#/components/schemas/AssertCreate' - assert-table-uuid: '#/components/schemas/AssertTableUUID' - assert-ref-snapshot-id: '#/components/schemas/AssertRefSnapshotId' - assert-last-assigned-field-id: '#/components/schemas/AssertLastAssignedFieldId' - assert-current-schema-id: '#/components/schemas/AssertCurrentSchemaId' - assert-last-assigned-partition-id: '#/components/schemas/AssertLastAssignedPartitionId' - assert-default-spec-id: '#/components/schemas/AssertDefaultSpecId' - assert-default-sort-order-id: '#/components/schemas/AssertDefaultSortOrderId' - properties: - type: - type: string - required: - - type - - AssertCreate: - type: object - description: The table must not already exist; used for create transactions - allOf: - - $ref: '#/components/schemas/TableRequirement' - required: - - type - properties: - type: - type: string - const: "assert-create" - - AssertTableUUID: - description: The table UUID must match the requirement's `uuid` - allOf: - - $ref: '#/components/schemas/TableRequirement' - required: - - type - - uuid - properties: - type: - type: string - const: "assert-table-uuid" - uuid: - type: string - - AssertRefSnapshotId: - description: | - The table branch or tag identified by the requirement's `ref` must reference the requirement's `snapshot-id`. - The `snapshot-id` field is required in this object, but in the case of a `null` - the ref must not already exist. - allOf: - - $ref: '#/components/schemas/TableRequirement' - required: - - ref - - snapshot-id - properties: - type: - type: string - const: "assert-ref-snapshot-id" - ref: - type: string - snapshot-id: - type: integer - format: int64 - nullable: true - - AssertLastAssignedFieldId: - description: - The table's last assigned column id must match the requirement's `last-assigned-field-id` - allOf: - - $ref: '#/components/schemas/TableRequirement' - required: - - last-assigned-field-id - properties: - type: - type: string - const: "assert-last-assigned-field-id" - last-assigned-field-id: - type: integer - - AssertCurrentSchemaId: - description: - The table's current schema id must match the requirement's `current-schema-id` - allOf: - - $ref: '#/components/schemas/TableRequirement' - required: - - current-schema-id - properties: - type: - type: string - const: "assert-current-schema-id" - current-schema-id: - type: integer - - AssertLastAssignedPartitionId: - description: - The table's last assigned partition id must match the requirement's `last-assigned-partition-id` - allOf: - - $ref: '#/components/schemas/TableRequirement' - required: - - last-assigned-partition-id - properties: - type: - type: string - const: "assert-last-assigned-partition-id" - last-assigned-partition-id: - type: integer - - AssertDefaultSpecId: - description: - The table's default spec id must match the requirement's `default-spec-id` - allOf: - - $ref: '#/components/schemas/TableRequirement' - required: - - default-spec-id - properties: - type: - type: string - const: "assert-default-spec-id" - default-spec-id: - type: integer - - AssertDefaultSortOrderId: - description: - The table's default sort order id must match the requirement's `default-sort-order-id` - allOf: - - $ref: '#/components/schemas/TableRequirement' - required: - - default-sort-order-id - properties: - type: - type: string - const: "assert-default-sort-order-id" - default-sort-order-id: - type: integer - - ViewRequirement: - type: object - discriminator: - propertyName: type - mapping: - assert-view-uuid: '#/components/schemas/AssertViewUUID' - oneOf: - - $ref: '#/components/schemas/AssertViewUUID' - - AssertViewUUID: - description: The view UUID must match the requirement's `uuid` - required: - - type - - uuid - properties: - type: - type: string - const: "assert-view-uuid" - uuid: - type: string - - StorageCredential: - type: object - required: - - prefix - - config - properties: - prefix: - type: string - description: Indicates a storage location prefix where the credential is relevant. Clients should choose the most - specific prefix (by selecting the longest prefix) if several credentials of the same type are available. - config: - type: object - additionalProperties: - type: string - - LoadCredentialsResponse: - type: object - required: - - storage-credentials - properties: - storage-credentials: - type: array - items: - $ref: '#/components/schemas/StorageCredential' - - LoadTableResult: - description: | - Result used when a table is successfully loaded. - - - The table metadata JSON is returned in the `metadata` field. The corresponding file location of table metadata should be returned in the `metadata-location` field, unless the metadata is not yet committed. For example, a create transaction may return metadata that is staged but not committed. - Clients can check whether metadata has changed by comparing metadata locations after the table has been created. - - - The `config` map returns table-specific configuration for the table's resources, including its HTTP client and FileIO. For example, config may contain a specific FileIO implementation class for the table depending on its underlying storage. - - - The following configurations should be respected by clients: - - ## General Configurations - - - `token`: Authorization bearer token to use for table requests if OAuth2 security is enabled - - `scan-planning-mode`: Communicates to clients the supported planning mode. Clients should use this value to fail fast if the supported scanning mode is not available on the client. Valid values: - - `client`: Clients MUST use client-side scan planning - - `server`: Clients MUST use server-side scan planning via the `planTableScan` endpoint - - ## AWS Configurations - - The following configurations should be respected when working with tables stored in AWS S3 - - `client.region`: region to configure client for making requests to AWS - - `s3.access-key-id`: id for credentials that provide access to the data in S3 - - `s3.secret-access-key`: secret for credentials that provide access to data in S3 - - `s3.session-token`: if present, this value should be used for as the session token - - `s3.remote-signing-enabled`: if `true` remote signing should be performed as described in the `RemoteSignRequest` schema section of this spec document. - - `s3.cross-region-access-enabled`: if `true`, S3 Cross-Region bucket access is enabled - - ## Storage Credentials - - Credentials for ADLS / GCS / S3 / ... are provided through the `storage-credentials` field. - Clients must first check whether the respective credentials exist in the `storage-credentials` field before checking the `config` for credentials. - - ## Remote Signing - - If remote signing for a specific storage provider is enabled, clients must respect the following configurations when creating a remote signer client: - - `signer.endpoint`: the remote signer endpoint. Required. Can either be a relative path (to be resolved against `signer.uri`) or an absolute URI. - - `signer.uri`: the base URI to resolve `signer.endpoint` against. Optional. Only meaningful if `signer.endpoint` is a relative path. Defaults to the catalog's base URI if not set. - type: object - required: - - metadata - properties: - metadata-location: - type: string - description: May be null if the table is staged as part of a transaction - nullable: true - metadata: - $ref: '#/components/schemas/TableMetadata' - config: - type: object - additionalProperties: - type: string - storage-credentials: - type: array - items: - $ref: '#/components/schemas/StorageCredential' - - ScanTasks: - type: object - description: > - Scan and planning tasks for server-side scan planning - - - - `plan-tasks` contains opaque units of planning work - - - `file-scan-tasks` contains a partial or complete list of table scan tasks - - - `delete-files` contains delete files referenced by file scan tasks - - - Each plan task must be passed to the fetchScanTasks endpoint to fetch - the file scan tasks for the plan task. - - - The list of delete files must contain all delete files referenced by - the file scan tasks. - properties: - delete-files: - description: Delete files referenced by file scan tasks - type: array - items: - $ref: '#/components/schemas/DeleteFile' - file-scan-tasks: - type: array - items: - $ref: '#/components/schemas/FileScanTask' - plan-tasks: - type: array - items: - $ref: '#/components/schemas/PlanTask' - - CompletedPlanningResult: - type: object - description: Completed server-side planning result - allOf: - - $ref: '#/components/schemas/ScanTasks' - - type: object - required: - - status - properties: - status: - $ref: '#/components/schemas/PlanStatus' - const: "completed" - storage-credentials: - type: array - description: - Storage credentials for accessing the files returned in the scan result. - - If the server returns storage credentials as part of the completed scan - planning response, the expectation is for the client to use these credentials - to read the files returned in the FileScanTasks as part of the scan result. - items: - $ref: '#/components/schemas/StorageCredential' - - CompletedPlanningWithIDResult: - type: object - allOf: - - $ref: '#/components/schemas/CompletedPlanningResult' - - type: object - required: - - plan-id - properties: - plan-id: - description: ID used to track a planning request - type: string - - FailedPlanningResult: - type: object - description: Failed server-side planning result - allOf: - - $ref: '#/components/schemas/IcebergErrorResponse' - - type: object - required: - - status - properties: - status: - $ref: '#/components/schemas/PlanStatus' - const: "failed" - - AsyncPlanningResult: - type: object - required: - - status - - plan-id - properties: - status: - $ref: '#/components/schemas/PlanStatus' - const: "submitted" - plan-id: - description: ID used to track a planning request - type: string - - EmptyPlanningResult: - type: object - description: Empty server-side planning result - required: - - status - properties: - status: - $ref: '#/components/schemas/PlanStatus' - enum: ["submitted", "cancelled"] - - PlanStatus: - description: Status of a server-side planning operation - type: string - enum: ["completed", "submitted", "cancelled", "failed"] - - FetchPlanningResult: - type: object - description: Result of server-side scan planning for fetchPlanningResult - discriminator: - propertyName: status - mapping: - completed: '#/components/schemas/CompletedPlanningResult' - submitted: '#/components/schemas/EmptyPlanningResult' - cancelled: '#/components/schemas/EmptyPlanningResult' - failed: '#/components/schemas/FailedPlanningResult' - oneOf: - - $ref: '#/components/schemas/CompletedPlanningResult' - - $ref: '#/components/schemas/FailedPlanningResult' - - $ref: '#/components/schemas/EmptyPlanningResult' - - PlanTableScanResult: - type: object - description: Result of server-side scan planning for planTableScan - discriminator: - propertyName: status - mapping: - completed: '#/components/schemas/CompletedPlanningWithIDResult' - submitted: '#/components/schemas/AsyncPlanningResult' - cancelled: '#/components/schemas/EmptyPlanningResult' - failed: '#/components/schemas/FailedPlanningResult' - oneOf: - - $ref: '#/components/schemas/CompletedPlanningWithIDResult' - - $ref: '#/components/schemas/FailedPlanningResult' - - $ref: '#/components/schemas/AsyncPlanningResult' - - $ref: '#/components/schemas/EmptyPlanningResult' - - FetchScanTasksResult: - type: object - description: Response schema for fetchScanTasks - allOf: - - $ref: '#/components/schemas/ScanTasks' - - CommitTableRequest: - type: object - required: - - requirements - - updates - properties: - identifier: - description: Table identifier to update; must be present for CommitTransactionRequest - $ref: '#/components/schemas/TableIdentifier' - requirements: - type: array - items: - $ref: '#/components/schemas/TableRequirement' - updates: - type: array - items: - $ref: '#/components/schemas/TableUpdate' - - CommitViewRequest: - type: object - required: - - updates - properties: - identifier: - description: View identifier to update - $ref: '#/components/schemas/TableIdentifier' - requirements: - type: array - items: - $ref: '#/components/schemas/ViewRequirement' - updates: - type: array - items: - $ref: '#/components/schemas/ViewUpdate' - - CommitTransactionRequest: - type: object - required: - - table-changes - properties: - table-changes: - type: array - items: - description: Table commit request; must provide an `identifier` - $ref: '#/components/schemas/CommitTableRequest' - - CreateTableRequest: - type: object - required: - - name - - schema - properties: - name: - type: string - location: - type: string - schema: - $ref: '#/components/schemas/Schema' - partition-spec: - $ref: '#/components/schemas/PartitionSpec' - write-order: - $ref: '#/components/schemas/SortOrder' - stage-create: - type: boolean - properties: - type: object - additionalProperties: - type: string - - RegisterTableRequest: - type: object - required: - - name - - metadata-location - properties: - name: - type: string - metadata-location: - type: string - overwrite: - description: Whether to overwrite table metadata if the table already exists - type: boolean - default: false - - CreateViewRequest: - type: object - required: - - name - - schema - - view-version - - properties - properties: - name: - type: string - location: - type: string - schema: - $ref: '#/components/schemas/Schema' - view-version: - $ref: '#/components/schemas/ViewVersion' - description: The view version to create, will replace the schema-id sent within the view-version with the id assigned to the provided schema - properties: - type: object - additionalProperties: - type: string - - LoadViewResult: - description: | - Result used when a view is successfully loaded. - - - The view metadata JSON is returned in the `metadata` field. The corresponding file location of view metadata is returned in the `metadata-location` field. - Clients can check whether metadata has changed by comparing metadata locations after the view has been created. - - The `config` map returns view-specific configuration for the view's resources. - - The following configurations should be respected by clients: - - ## General Configurations - - - `token`: Authorization bearer token to use for view requests if OAuth2 security is enabled - type: object - required: - - metadata-location - - metadata - properties: - metadata-location: - type: string - metadata: - $ref: '#/components/schemas/ViewMetadata' - config: - type: object - additionalProperties: - type: string - - RegisterViewRequest: - type: object - required: - - name - - metadata-location - properties: - name: - type: string - metadata-location: - type: string - - TokenType: - type: string - enum: - - urn:ietf:params:oauth:token-type:access_token - - urn:ietf:params:oauth:token-type:refresh_token - - urn:ietf:params:oauth:token-type:id_token - - urn:ietf:params:oauth:token-type:saml1 - - urn:ietf:params:oauth:token-type:saml2 - - urn:ietf:params:oauth:token-type:jwt - description: - Token type identifier, from RFC 8693 Section 3 - - - See https://datatracker.ietf.org/doc/html/rfc8693#section-3 - - OAuthClientCredentialsRequest: - deprecated: true - description: - The `oauth/tokens` endpoint and related schemas are **DEPRECATED for REMOVAL** from this - spec, see description of the endpoint. - - - OAuth2 client credentials request - - - See https://datatracker.ietf.org/doc/html/rfc6749#section-4.4 - type: object - required: - - grant_type - - client_id - - client_secret - properties: - grant_type: - type: string - enum: - - client_credentials - scope: - type: string - client_id: - type: string - description: - Client ID - - - This can be sent in the request body, but OAuth2 recommends sending it in - a Basic Authorization header. - client_secret: - type: string - description: - Client secret - - - This can be sent in the request body, but OAuth2 recommends sending it in - a Basic Authorization header. - - OAuthTokenExchangeRequest: - deprecated: true - description: - The `oauth/tokens` endpoint and related schemas are **DEPRECATED for REMOVAL** from this - spec, see description of the endpoint. - - - OAuth2 token exchange request - - - See https://datatracker.ietf.org/doc/html/rfc8693 - type: object - required: - - grant_type - - subject_token - - subject_token_type - properties: - grant_type: - type: string - enum: - - urn:ietf:params:oauth:grant-type:token-exchange - scope: - type: string - requested_token_type: - $ref: '#/components/schemas/TokenType' - subject_token: - type: string - description: Subject token for token exchange request - subject_token_type: - $ref: '#/components/schemas/TokenType' - actor_token: - type: string - description: Actor token for token exchange request - actor_token_type: - $ref: '#/components/schemas/TokenType' - - OAuthTokenRequest: - deprecated: true - description: - The `oauth/tokens` endpoint and related schemas are **DEPRECATED for REMOVAL** from this - spec, see description of the endpoint. - anyOf: - - $ref: '#/components/schemas/OAuthClientCredentialsRequest' - - $ref: '#/components/schemas/OAuthTokenExchangeRequest' - - CounterResult: - type: object - required: - - unit - - value - properties: - unit: - type: string - value: - type: integer - format: int64 - - TimerResult: - type: object - required: - - time-unit - - count - - total-duration - properties: - time-unit: - type: string - count: - type: integer - format: int64 - total-duration: - type: integer - format: int64 - - MetricResult: - anyOf: - - $ref: '#/components/schemas/CounterResult' - - $ref: '#/components/schemas/TimerResult' - - Metrics: - type: object - additionalProperties: - $ref: '#/components/schemas/MetricResult' - example: { - "metrics": { - "total-planning-duration": { - "count": 1, - "time-unit": "nanoseconds", - "total-duration": 2644235116 - }, - "result-data-files": { - "unit": "count", - "value": 1, - }, - "result-delete-files": { - "unit": "count", - "value": 0, - }, - "total-data-manifests": { - "unit": "count", - "value": 1, - }, - "total-delete-manifests": { - "unit": "count", - "value": 0, - }, - "scanned-data-manifests": { - "unit": "count", - "value": 1, - }, - "skipped-data-manifests": { - "unit": "count", - "value": 0, - }, - "total-file-size-bytes": { - "unit": "bytes", - "value": 10, - }, - "total-delete-file-size-bytes": { - "unit": "bytes", - "value": 0, - } - } - } - - ReportMetricsRequest: - anyOf: - - $ref: '#/components/schemas/ScanReport' - - $ref: '#/components/schemas/CommitReport' - required: - - report-type - properties: - report-type: - type: string - - ScanReport: - type: object - required: - - table-name - - snapshot-id - - filter - - schema-id - - projected-field-ids - - projected-field-names - - metrics - properties: - table-name: - type: string - snapshot-id: - type: integer - format: int64 - filter: - $ref: '#/components/schemas/Expression' - schema-id: - type: integer - projected-field-ids: - type: array - items: - type: integer - projected-field-names: - type: array - items: - type: string - metrics: - $ref: '#/components/schemas/Metrics' - metadata: - type: object - additionalProperties: - type: string - - CommitReport: - type: object - required: - - table-name - - snapshot-id - - sequence-number - - operation - - metrics - properties: - table-name: - type: string - snapshot-id: - type: integer - format: int64 - sequence-number: - type: integer - format: int64 - operation: - type: string - metrics: - $ref: '#/components/schemas/Metrics' - metadata: - type: object - additionalProperties: - type: string - - OAuthError: - deprecated: true - description: - The `oauth/tokens` endpoint and related schemas are **DEPRECATED for REMOVAL** from this - spec, see description of the endpoint. - type: object - required: - - error - properties: - error: - type: string - enum: - - invalid_request - - invalid_client - - invalid_grant - - unauthorized_client - - unsupported_grant_type - - invalid_scope - error_description: - type: string - error_uri: - type: string - - OAuthTokenResponse: - deprecated: true - description: - The `oauth/tokens` endpoint and related schemas are **DEPRECATED for REMOVAL** from this - spec, see description of the endpoint. - type: object - required: - - access_token - - token_type - properties: - access_token: - type: string - description: - The access token, for client credentials or token exchange - token_type: - type: string - enum: - - bearer - - mac - - N_A - description: - Access token type for client credentials or token exchange - - - See https://datatracker.ietf.org/doc/html/rfc6749#section-7.1 - expires_in: - type: integer - description: - Lifetime of the access token in seconds for client credentials or token exchange - issued_token_type: - $ref: '#/components/schemas/TokenType' - refresh_token: - type: string - description: Refresh token for client credentials or token exchange - scope: - type: string - description: Authorization scope for client credentials or token exchange - - IcebergErrorResponse: - description: JSON wrapper for all error responses (non-2xx) - type: object - required: - - error - properties: - error: - $ref: '#/components/schemas/ErrorModel' - additionalProperties: false - example: - { - "error": { - "message": "The server does not support this operation", - "type": "UnsupportedOperationException", - "code": 406 - } - } - - CreateNamespaceResponse: - type: object - required: - - namespace - properties: - namespace: - $ref: '#/components/schemas/Namespace' - properties: - type: object - additionalProperties: - type: string - description: - Properties stored on the namespace, if supported by the server. - example: { "owner": "Ralph", "created_at": "1452120468" } - default: { } - - GetNamespaceResponse: - type: object - required: - - namespace - properties: - namespace: - $ref: '#/components/schemas/Namespace' - properties: - type: object - description: - Properties stored on the namespace, if supported by the server. - If the server does not support namespace properties, it should return null for this field. - If namespace properties are supported, but none are set, it should return an empty object. - additionalProperties: - type: string - example: { "owner": "Ralph", 'transient_lastDdlTime': '1452120468' } - default: { } - nullable: true - - ListTablesResponse: - type: object - properties: - next-page-token: - $ref: '#/components/schemas/PageToken' - identifiers: - type: array - uniqueItems: true - items: - $ref: '#/components/schemas/TableIdentifier' - - ListNamespacesResponse: - type: object - properties: - next-page-token: - $ref: '#/components/schemas/PageToken' - namespaces: - type: array - uniqueItems: true - items: - $ref: '#/components/schemas/Namespace' - - UpdateNamespacePropertiesResponse: - type: object - required: - - updated - - removed - properties: - updated: - description: List of property keys that were added or updated - type: array - uniqueItems: true - items: - type: string - removed: - description: List of properties that were removed - type: array - items: - type: string - missing: - type: array - items: - type: string - description: - List of properties requested for removal that were not found - in the namespace's properties. Represents a partial success response. - Server's do not need to implement this. - nullable: true - - CommitTableResponse: - type: object - required: - - metadata-location - - metadata - properties: - metadata-location: - type: string - metadata: - $ref: '#/components/schemas/TableMetadata' - - StatisticsFile: - type: object - required: - - snapshot-id - - statistics-path - - file-size-in-bytes - - file-footer-size-in-bytes - - blob-metadata - properties: - snapshot-id: - type: integer - format: int64 - statistics-path: - type: string - file-size-in-bytes: - type: integer - format: int64 - file-footer-size-in-bytes: - type: integer - format: int64 - blob-metadata: - type: array - items: - $ref: '#/components/schemas/BlobMetadata' - - BlobMetadata: - type: object - required: - - type - - snapshot-id - - sequence-number - - fields - properties: - type: - type: string - snapshot-id: - type: integer - format: int64 - sequence-number: - type: integer - format: int64 - fields: - type: array - items: - type: integer - properties: - type: object - additionalProperties: - type: string - - PartitionStatisticsFile: - type: object - required: - - snapshot-id - - statistics-path - - file-size-in-bytes - properties: - snapshot-id: - type: integer - format: int64 - statistics-path: - type: string - file-size-in-bytes: - type: integer - format: int64 - - BooleanTypeValue: - type: boolean - example: true - - IntegerTypeValue: - type: integer - example: 42 - - LongTypeValue: - type: integer - format: int64 - example: 9223372036854775807 - - FloatTypeValue: - type: number - format: float - example: 3.14 - - DoubleTypeValue: - type: number - format: double - example: 123.456 - - DecimalTypeValue: - type: string - description: - "Decimal type values are serialized as strings. Decimals with a positive scale serialize as numeric plain - text, while decimals with a negative scale use scientific notation and the exponent will be equal to the - negated scale. For instance, a decimal with a positive scale is '123.4500', with zero scale is '2', - and with a negative scale is '2E+20'" - example: "123.4500" - - StringTypeValue: - type: string - example: "hello" - - UUIDTypeValue: - type: string - format: uuid - pattern: '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' - maxLength: 36 - minLength: 36 - description: - "UUID type values are serialized as a 36-character lowercase string in standard UUID format as specified - by RFC-4122" - example: "eb26bdb1-a1d8-4aa6-990e-da940875492c" - - DateTypeValue: - type: string - format: date - description: - "Date type values follow the 'YYYY-MM-DD' ISO-8601 standard date format" - example: "2007-12-03" - - TimeTypeValue: - type: string - description: - "Time type values follow the 'HH:MM:SS.ssssss' ISO-8601 format with microsecond precision" - example: "22:31:08.123456" - - TimestampTypeValue: - type: string - description: - "Timestamp type values follow the 'YYYY-MM-DDTHH:MM:SS.ssssss' ISO-8601 format with microsecond precision" - example: "2007-12-03T10:15:30.123456" - - TimestampTzTypeValue: - type: string - description: - "TimestampTz type values follow the 'YYYY-MM-DDTHH:MM:SS.ssssss+00:00' ISO-8601 format with microsecond precision, - and a timezone offset (+00:00 for UTC)" - example: "2007-12-03T10:15:30.123456+00:00" - - TimestampNanoTypeValue: - type: string - description: - "Timestamp_ns type values follow the 'YYYY-MM-DDTHH:MM:SS.sssssssss' ISO-8601 format with nanosecond precision" - example: "2007-12-03T10:15:30.123456789" - - TimestampTzNanoTypeValue: - type: string - description: - "Timestamp_ns type values follow the 'YYYY-MM-DDTHH:MM:SS.sssssssss+00:00' ISO-8601 format with nanosecond - precision, and a timezone offset (+00:00 for UTC)" - example: "2007-12-03T10:15:30.123456789+00:00" - - FixedTypeValue: - type: string - description: - "Fixed length type values are stored and serialized as an uppercase hexadecimal string - preserving the fixed length" - example: "78797A" - - BinaryTypeValue: - type: string - description: - "Binary type values are stored and serialized as an uppercase hexadecimal string" - example: "78797A" - - CountMap: - type: object - properties: - keys: - type: array - items: - $ref: '#/components/schemas/IntegerTypeValue' - description: "List of integer column ids for each corresponding value" - values: - type: array - items: - $ref: '#/components/schemas/LongTypeValue' - description: "List of Long values, matched to 'keys' by index" - example: - { - "keys": [ 1, 2 ], - "values": [ 100, 200 ] - } - - ValueMap: - type: object - properties: - keys: - type: array - items: - $ref: '#/components/schemas/IntegerTypeValue' - description: "List of integer column ids for each corresponding value" - values: - type: array - items: - $ref: '#/components/schemas/PrimitiveTypeValue' - description: "List of primitive type values, matched to 'keys' by index" - example: - { - "keys": [ 1, 2 ], - "values": [ 100, "test" ] - } - - PrimitiveTypeValue: - oneOf: - - $ref: '#/components/schemas/BooleanTypeValue' - - $ref: '#/components/schemas/IntegerTypeValue' - - $ref: '#/components/schemas/LongTypeValue' - - $ref: '#/components/schemas/FloatTypeValue' - - $ref: '#/components/schemas/DoubleTypeValue' - - $ref: '#/components/schemas/DecimalTypeValue' - - $ref: '#/components/schemas/StringTypeValue' - - $ref: '#/components/schemas/UUIDTypeValue' - - $ref: '#/components/schemas/DateTypeValue' - - $ref: '#/components/schemas/TimeTypeValue' - - $ref: '#/components/schemas/TimestampTypeValue' - - $ref: '#/components/schemas/TimestampTzTypeValue' - - $ref: '#/components/schemas/TimestampNanoTypeValue' - - $ref: '#/components/schemas/TimestampTzNanoTypeValue' - - $ref: '#/components/schemas/FixedTypeValue' - - $ref: '#/components/schemas/BinaryTypeValue' - - FileFormat: - type: string - enum: - - avro - - orc - - parquet - - puffin - - ContentFile: - discriminator: - propertyName: content - mapping: - data: '#/components/schemas/DataFile' - position-deletes: '#/components/schemas/PositionDeleteFile' - equality-deletes: '#/components/schemas/EqualityDeleteFile' - type: object - required: - - spec-id - - partition - - content - - file-path - - file-format - - file-size-in-bytes - - record-count - properties: - content: - type: string - file-path: - type: string - file-format: - $ref: '#/components/schemas/FileFormat' - spec-id: - type: integer - partition: - type: array - items: - $ref: '#/components/schemas/PrimitiveTypeValue' - description: - A list of partition field values ordered based on the fields of - the partition spec specified by the `spec-id` - example: [1, "bar"] - file-size-in-bytes: - type: integer - format: int64 - description: "Total file size in bytes" - record-count: - type: integer - format: int64 - description: "Number of records in the file" - key-metadata: - allOf: - - $ref: '#/components/schemas/BinaryTypeValue' - description: "Encryption key metadata blob" - split-offsets: - type: array - items: - type: integer - format: int64 - description: "List of splittable offsets" - sort-order-id: - type: integer - - DataFile: - allOf: - - $ref: '#/components/schemas/ContentFile' - type: object - required: - - content - properties: - content: - type: string - const: "data" - first-row-id: - type: integer - format: int64 - description: "The first row ID assigned to the first row in the data file" - column-sizes: - allOf: - - $ref: '#/components/schemas/CountMap' - description: "Map of column id to total count, including null and NaN" - value-counts: - allOf: - - $ref: '#/components/schemas/CountMap' - description: "Map of column id to null value count" - null-value-counts: - allOf: - - $ref: '#/components/schemas/CountMap' - description: "Map of column id to null value count" - nan-value-counts: - allOf: - - $ref: '#/components/schemas/CountMap' - description: "Map of column id to number of NaN values in the column" - lower-bounds: - allOf: - - $ref: '#/components/schemas/ValueMap' - description: "Map of column id to lower bound primitive type values" - upper-bounds: - allOf: - - $ref: '#/components/schemas/ValueMap' - description: "Map of column id to upper bound primitive type values" - - DeleteFile: - discriminator: - propertyName: content - mapping: - position-deletes: '#/components/schemas/PositionDeleteFile' - equality-deletes: '#/components/schemas/EqualityDeleteFile' - oneOf: - - $ref: '#/components/schemas/PositionDeleteFile' - - $ref: '#/components/schemas/EqualityDeleteFile' - - PositionDeleteFile: - allOf: - - $ref: '#/components/schemas/ContentFile' - required: - - content - properties: - content: - type: string - const: "position-deletes" - content-offset: - type: integer - format: int64 - description: Offset within the delete file of delete content - content-size-in-bytes: - type: integer - format: int64 - description: Length, in bytes, of the delete content; required if content-offset is present - - EqualityDeleteFile: - allOf: - - $ref: '#/components/schemas/ContentFile' - required: - - content - properties: - content: - type: string - const: "equality-deletes" - equality-ids: - type: array - items: - type: integer - description: "List of equality field IDs" - - PlanTableScanRequest: - type: object - properties: - snapshot-id: - description: - Identifier for the snapshot to scan in a point-in-time scan - type: integer - format: int64 - select: - description: List of selected schema fields - type: array - items: - $ref: '#/components/schemas/FieldName' - filter: - description: - Expression used to filter the table data - $ref: '#/components/schemas/Expression' - min-rows-requested: - description: - The minimum number of rows requested for the scan. This is used as a hint - to the server to not have to return more rows than necessary. It is not required - for the server to return that many rows since the scan may not produce that - many rows. The server can also return more rows than requested. - type: integer - format: int64 - case-sensitive: - description: Enables case sensitive field matching for filter and select - type: boolean - default: true - use-snapshot-schema: - description: - Whether to use the schema at the time the snapshot was written. - - When time travelling, the snapshot schema should be used (true). - When scanning a branch, the table schema should be used (false). - type: boolean - default: false - start-snapshot-id: - description: Starting snapshot ID for an incremental scan (exclusive) - type: integer - format: int64 - end-snapshot-id: - description: - Ending snapshot ID for an incremental scan (inclusive). - - Required when start-snapshot-id is specified. - type: integer - format: int64 - stats-fields: - description: - List of fields for which the service should send column stats. - type: array - items: - $ref: '#/components/schemas/FieldName' - - FieldName: - description: - A full field name (including parent field names), such as those passed - in APIs like Java `Schema#findField(String name)`. - - The nested field name follows these rules - - Nested struct fields are named by concatenating field names at each - struct level using dot (`.`) delimiter, e.g. - employer.contact_info.address.zip_code - - Nested fields in a map key are named using the keyword `key`, e.g. - employee_address_map.key.first_name - - Nested fields in a map value are named using the keyword `value`, - e.g. employee_address_map.value.zip_code - - Nested fields in a list are named using the keyword `element`, e.g. - employees.element.first_name - type: string - - FetchScanTasksRequest: - type: object - required: - - plan-task - properties: - plan-task: - $ref: '#/components/schemas/PlanTask' - - PlanTask: - description: - An opaque string provided by the REST server that represents a - unit of work to produce file scan tasks for scan planning. This allows - clients to fetch tasks across multiple requests to accommodate large result sets. - type: string - - FileScanTask: - type: object - required: - - data-file - properties: - data-file: - $ref: '#/components/schemas/DataFile' - delete-file-references: - description: A list of indices in the delete files array (0-based) - type: array - items: - type: integer - residual-filter: - description: - An optional filter to be applied to rows in this file scan task. - - If the residual is not present, the client must produce the - residual or use the original filter. - allOf: - - $ref: '#/components/schemas/Expression' - - MultiValuedMap: - description: A map of string keys where each key can map to multiple string values. - type: object - additionalProperties: - type: array - items: - type: string - - RemoteSignRequest: - description: The request to be signed remotely. - type: object - required: - - region - - uri - - method - - headers - properties: - region: - type: string - uri: - type: string - method: - type: string - enum: ["PUT", "GET", "HEAD", "POST", "DELETE", "PATCH", "OPTIONS"] - headers: - $ref: '#/components/schemas/MultiValuedMap' - properties: - type: object - additionalProperties: - type: string - body: - type: string - description: Optional body of the request to send to the signing API. This should only be populated - for requests where the body of the message contains content which must be validated before a request is - signed, such as the S3 DeleteObjects call. - provider: - type: string - description: The storage provider for which the request is to be signed. The provider should correspond to - the scheme used for a storage native URI. For example `s3` for AWS S3 paths. For backwards compatibility, - if this is not specified, the provider is assumed to be `s3`. - - RemoteSignResult: - description: The result of a remote request signing operation. - type: object - required: - - uri - - headers - properties: - uri: - type: string - headers: - $ref: '#/components/schemas/MultiValuedMap' - - ############################# - # Reusable Response Objects # - ############################# - responses: - - OAuthTokenResponse: - description: OAuth2 token response for client credentials or token exchange - content: - application/json: - schema: - $ref: '#/components/schemas/OAuthTokenResponse' - - OAuthErrorResponse: - description: OAuth2 error response - content: - application/json: - schema: - $ref: '#/components/schemas/OAuthError' - - BadRequestErrorResponse: - description: - Indicates a bad request error. It could be caused by an unexpected request - body format or other forms of request validation failure, such as invalid json. - Usually serves application/json content, although in some cases simple text/plain content might - be returned by the server's middleware. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - example: { - "error": { - "message": "Malformed request", - "type": "BadRequestException", - "code": 400 - } - } - - # Note that this is a representative example response for use as a shorthand in the spec. - # The fields `message` and `type` as indicated here are not presently prescriptive. - UnauthorizedResponse: - description: - Unauthorized. The REST Catalog SHOULD respond with the 401 UnauthorizedResponse when - the access token provided is expired, revoked, malformed, or invalid for other reasons. - The client MAY request a new access token and retry the request. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - example: { - "error": { - "message": "Not authorized to make this request", - "type": "NotAuthorizedException", - "code": 401 - } - } - - # Note that this is a representative example response for use as a shorthand in the spec. - # The fields `message` and `type` as indicated here are not presently prescriptive. - ForbiddenResponse: - description: Forbidden. Authenticated user does not have the necessary permissions. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - example: { - "error": { - "message": "Not authorized to make this request", - "type": "NotAuthorizedException", - "code": 403 - } - } - - # Note that this is a representative example response for use as a shorthand in the spec. - # The fields `message` and `type` as indicated here are not presently prescriptive. - UnsupportedOperationResponse: - description: Not Acceptable / Unsupported Operation. The server does not support this operation. - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorModel' - example: { - "error": { - "message": "The server does not support this operation", - "type": "UnsupportedOperationException", - "code": 406 - } - } - - IcebergErrorResponse: - description: JSON wrapper for all error responses (non-2xx) - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - example: { - "error": { - "message": "The server does not support this operation", - "type": "UnsupportedOperationException", - "code": 406 - } } - - CreateNamespaceResponse: - description: - Represents a successful call to create a namespace. - Returns the namespace created, as well as any properties that were stored for the namespace, - including those the server might have added. Implementations are not required to support namespace - properties. - content: - application/json: - schema: - $ref: '#/components/schemas/CreateNamespaceResponse' - example: { - "namespace": ["accounting", "tax"], - "properties": { "owner": "Ralph", "created_at": "1452120468" } - } - - GetNamespaceResponse: - description: - Returns a namespace, as well as any properties stored on the namespace if namespace properties - are supported by the server. - content: - application/json: - schema: - $ref: '#/components/schemas/GetNamespaceResponse' - - ListTablesResponse: - description: A list of table identifiers - content: - application/json: - schema: - $ref: '#/components/schemas/ListTablesResponse' - examples: - ListTablesResponseNonEmpty: - $ref: '#/components/examples/ListTablesNonEmptyExample' - ListTablesResponseEmpty: - $ref: '#/components/examples/ListTablesEmptyExample' - - ListNamespacesResponse: - description: A list of namespaces - content: - application/json: - schema: - $ref: '#/components/schemas/ListNamespacesResponse' - examples: - NonEmptyResponse: - $ref: '#/components/examples/ListNamespacesNonEmptyExample' - EmptyResponse: - $ref: '#/components/examples/ListNamespacesEmptyExample' - - AuthenticationTimeoutResponse: - description: - This is an optional status response type that the REST Catalog can issue when the - token has expired. The client MAY request a new access token and retry the request. - 401 UnauthorizedResponse SHOULD be preferred over this response type on token expiry. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - example: { - "error": { - "message": "Credentials have timed out", - "type": "AuthenticationTimeoutException", - "code": 419 - } - } - - ServiceUnavailableResponse: - description: - The service is not ready to handle the request, request could have been partially processed. - - The service may additionally send a Retry-After header to indicate when to retry, a non idempotent - request should only be retried by the client when the Retry-After header is present. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - example: { - "error": { - "message": "Slow down", - "type": "SlowDownException", - "code": 503 - } - } - - ServerErrorResponse: - description: - A server-side problem that might not be addressable from the client - side. Used for server 5xx errors without more specific documentation in - individual routes. - content: - application/json: - schema: - $ref: '#/components/schemas/IcebergErrorResponse' - example: { - "error": { - "message": "Internal Server Error", - "type": "InternalServerError", - "code": 500 - } - } - - UpdateNamespacePropertiesResponse: - description: JSON data response for a synchronous update properties request. - content: - application/json: - schema: - $ref: '#/components/schemas/UpdateNamespacePropertiesResponse' - example: { - "updated": [ "owner" ], - "removed": [ "foo" ], - "missing": [ "bar" ] - } - - CreateTableResponse: - description: Table metadata result after creating a table - content: - application/json: - schema: - $ref: '#/components/schemas/LoadTableResult' - headers: - etag: - $ref: '#/components/parameters/etag' - - PlanTableScanResponse: - description: Result of submitting a table scan to plan - content: - application/json: - schema: - $ref: '#/components/schemas/PlanTableScanResult' - - FetchPlanningResultResponse: - description: Result of fetching a submitted scan planning operation - content: - application/json: - schema: - $ref: '#/components/schemas/FetchPlanningResult' - - FetchScanTasksResponse: - description: Result of retrieving additional plan tasks and file scan tasks. - content: - application/json: - schema: - $ref: '#/components/schemas/FetchScanTasksResult' - - LoadTableResponse: - description: Table metadata result when loading a table - content: - application/json: - schema: - $ref: '#/components/schemas/LoadTableResult' - headers: - etag: - $ref: '#/components/parameters/etag' - - LoadViewResponse: - description: View metadata result when loading a view - content: - application/json: - schema: - $ref: '#/components/schemas/LoadViewResult' - - CommitTableResponse: - description: - Response used when a table is successfully updated. - - The table metadata JSON is returned in the metadata field. The corresponding file location of table metadata must be returned in the metadata-location field. Clients can check whether metadata has changed by comparing metadata locations. - content: - application/json: - schema: - $ref: '#/components/schemas/CommitTableResponse' - headers: - etag: - $ref: '#/components/parameters/etag' - - LoadCredentialsResponse: - description: Table credentials result when loading credentials for a table - content: - application/json: - schema: - $ref: '#/components/schemas/LoadCredentialsResponse' - - RemoteSignResponse: - description: The response containing signed & unsigned headers. The server will also send - a Cache-Control header, indicating whether the response can be cached (Cache-Control = ["private"]) - or not (Cache-Control = ["no-cache"]). - content: - application/json: - schema: - $ref: '#/components/schemas/RemoteSignResult' - - ####################################### - # Common examples of different values # - ####################################### - examples: - - ListTablesEmptyExample: - summary: An empty list for a namespace with no tables - value: { - "identifiers": [ ] - } - - ListNamespacesEmptyExample: - summary: An empty list of namespaces - value: { - "namespaces": [ ] - } - - ListNamespacesNonEmptyExample: - summary: A non-empty list of namespaces - value: { - "namespaces": [ - ["accounting", "tax"], - ["accounting", "credits"] - ] - } - - ListTablesNonEmptyExample: - summary: A non-empty list of table identifiers - value: { - "identifiers": [ - { "namespace": ["accounting", "tax"], "name": "paid" }, - { "namespace": ["accounting", "tax"], "name": "owed" } - ] - } - - MultipartNamespaceAsPathVariable: - summary: A multi-part namespace, as represented in a path parameter - value: "accounting%1Ftax" - - NamespaceAsPathVariable: - summary: A single part namespace, as represented in a path parameter - value: "accounting" - - NamespaceAlreadyExistsError: - summary: The requested namespace already exists - value: { - "error": { - "message": "The given namespace already exists", - "type": "AlreadyExistsException", - "code": 409 - } - } - - NoSuchPlanIdError: - summary: The plan id does not exist - value: { - "error": { - "message": "The plan id does not exist", - "type": "NoSuchPlanIdException", - "code": 404 - } - } - - NoSuchPlanTaskError: - summary: The plan task does not exist - value: { - "error": { - "message": "The plan task does not exist", - "type": "NoSuchPlanTaskException", - "code": 404 - } - } - - NoSuchTableError: - summary: The requested table does not exist - value: { - "error": { - "message": "The given table does not exist", - "type": "NoSuchTableException", - "code": 404 - } - } - - NoSuchViewError: - summary: The requested view does not exist - value: { - "error": { - "message": "The given view does not exist", - "type": "NoSuchViewException", - "code": 404 - } - } - - NoSuchWarehouseError: - summary: The requested warehouse does not exist - value: { - "error": { - "message": "The given warehouse does not exist", - "type": "NoSuchWarehouseException", - "code": 404 - } - } - - NoSuchNamespaceError: - summary: The requested namespace does not exist - value: { - "error": { - "message": "The given namespace does not exist", - "type": "NoSuchNamespaceException", - "code": 404 - } - } - - NamespaceNotEmptyError: - summary: The requested namespace is not empty - value: { - "error": { - "message": "The given namespace is not empty", - "type": "NamespaceNotEmptyException", - "code": 409 - } - } - - RenameTableSameNamespace: - summary: Rename a table in the same namespace - value: { - "source": { "namespace": ["accounting", "tax"], "name": "paid" }, - "destination": { "namespace": ["accounting", "tax"], "name": "owed" } - } - - RenameViewSameNamespace: - summary: Rename a view in the same namespace - value: { - "source": { "namespace": [ "accounting", "tax" ], "name": "paid-view" }, - "destination": { "namespace": [ "accounting", "tax" ], "name": "owed-view" } - } - - TableAlreadyExistsError: - summary: The requested table identifier already exists - value: { - "error": { - "message": "The requested table identifier already exists", - "type": "AlreadyExistsException", - "code": 409 - } - } - - ViewAlreadyExistsError: - summary: The requested view identifier already exists - value: { - "error": { - "message": "The requested view identifier already exists", - "type": "AlreadyExistsException", - "code": 409 - } - } - - # This is an example response and is not meant to be prescriptive regarding the message or type. - UnprocessableEntityDuplicateKey: - summary: - The request body either has the same key multiple times in what should be a map with unique keys - or the request body has keys in two or more fields which should be disjoint sets. - value: { - "error": { - "message": "The request cannot be processed as there is a key present multiple times", - "type": "UnprocessableEntityException", - "code": 422 - } - } - - UpdateAndRemoveNamespacePropertiesRequest: - summary: An update namespace properties request with both properties to remove and properties to upsert. - value: { - "removals": [ "foo", "bar" ], - "updates": { "owner": "Raoul" } - } - - securitySchemes: - OAuth2: - type: oauth2 - description: - This scheme is used for OAuth2 authorization. - - - For unauthorized requests, services should return an appropriate 401 or - 403 response. Implementations must not return altered success (200) - responses when a request is unauthenticated or unauthorized. - - If a separate authorization server is used, substitute the tokenUrl with - the full token path of the external authorization server, and use the - resulting token to access the resources defined in the spec. - flows: - clientCredentials: - tokenUrl: /v1/oauth/tokens - scopes: - catalog: Allows interacting with the Config and Catalog APIs - BearerAuth: - type: http - scheme: bearer diff --git a/doc/design/access-server/iceberge/iceberg-table-spec-1.11.0.md b/doc/design/access-server/iceberge/iceberg-table-spec-1.11.0.md deleted file mode 100644 index 94651da0f..000000000 --- a/doc/design/access-server/iceberge/iceberg-table-spec-1.11.0.md +++ /dev/null @@ -1,1858 +0,0 @@ ---- -title: "Spec" ---- - - -# Iceberg Table Spec - -This is a specification for the Iceberg table format that is designed to manage a large, slow-changing collection of files in a distributed file system or key-value store as a table. - -## Format Versioning - -Versions 1, 2 and 3 of the Iceberg spec are complete and adopted by the community. - -**Version 4 is under active development and has not been formally adopted.** - -The format version number is incremented when new features are added that will break forward-compatibility---that is, when older readers would not read newer table features correctly. Tables may continue to be written with an older version of the spec to ensure compatibility by not using features that are not yet implemented by processing engines. - -### Version 1: Analytic Data Tables - -Version 1 of the Iceberg spec defines how to manage large analytic tables using immutable file formats: Parquet, Avro, and ORC. - -All version 1 data and metadata files are valid after upgrading a table to version 2. [Appendix E](#version-2) documents how to default version 2 fields when reading version 1 metadata. - -### Version 2: Row-level Deletes - -Version 2 of the Iceberg spec adds row-level updates and deletes for analytic tables with immutable files. - -The primary change in version 2 adds delete files to encode rows that are deleted in existing data files. This version can be used to delete or replace individual rows in immutable data files without rewriting the files. - -In addition to row-level deletes, version 2 makes some requirements stricter for writers. The full set of changes are listed in [Appendix E](#version-2). - -### Version 3: Extended Types and Capabilities - -Version 3 of the Iceberg spec extends data types and existing metadata structures to add new capabilities: - -* New data types: nanosecond timestamp(tz), unknown, variant, geometry, geography -* Default value support for columns -* Multi-argument transforms for partitioning and sorting -* Row Lineage tracking -* Binary deletion vectors -* Table encryption keys - -The full set of changes are listed in [Appendix E](#version-3). - -## Goals - -* **Serializable isolation** -- Reads will be isolated from concurrent writes and always use a committed snapshot of a table’s data. Writes will support removing and adding files in a single operation and are never partially visible. Readers will not acquire locks. -* **Speed** -- Operations will use O(1) remote calls to plan the files for a scan and not O(n) where n grows with the size of the table, like the number of partitions or files. -* **Scale** -- Job planning will be handled primarily by clients and not bottleneck on a central metadata store. Metadata will include information needed for cost-based optimization. -* **Evolution** -- Tables will support full schema and partition spec evolution. Schema evolution supports safe column add, drop, reorder and rename, including in nested structures. -* **Dependable types** -- Tables will provide well-defined and dependable support for a core set of types. -* **Storage separation** -- Partitioning will be table configuration. Reads will be planned using predicates on data values, not partition values. Tables will support evolving partition schemes. -* **Formats** -- Underlying data file formats will support identical schema evolution rules and types. Both read-optimized and write-optimized formats will be available. - -## Overview - -![Iceberg snapshot structure](https://iceberg.apache.org/assets/images/iceberg-metadata.png) - -This table format tracks individual data files in a table instead of directories. This allows writers to create data files in-place and only adds files to the table in an explicit commit. - -Table state is maintained in metadata files. All changes to table state create a new metadata file and replace the old metadata with an atomic swap. The table metadata file tracks the table schema, partitioning config, custom properties, and snapshots of the table contents. A snapshot represents the state of a table at some time and is used to access the complete set of data files in the table. - -Data files in snapshots are tracked by one or more manifest files that contain a row for each data file in the table, the file's partition data, and its metrics. The data in a snapshot is the union of all files in its manifests. Manifest files are reused across snapshots to avoid rewriting metadata that is slow-changing. Manifests can track data files with any subset of a table and are not associated with partitions. - -The manifests that make up a snapshot are stored in a manifest list file. Each manifest list stores metadata about manifests, including partition stats and data file counts. These stats are used to avoid reading manifests that are not required for an operation. - -### Optimistic Concurrency - -An atomic swap of one table metadata file for another provides the basis for serializable isolation. Readers use the snapshot that was current when they load the table metadata and are not affected by changes until they refresh and pick up a new metadata location. - -Writers create table metadata files optimistically, assuming that the current version will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the table’s metadata file pointer from the base version to the new version. - -If the snapshot on which an update is based is no longer current, the writer must retry the update based on the new current version. Some operations support retry by re-applying metadata changes and committing, under well-defined conditions. For example, a change that rewrites files can be applied to a new table snapshot if all of the rewritten files are still in the table. - -The conditions required by a write to successfully commit determines the isolation level. Writers can select what to validate and can make different isolation guarantees. - -### Sequence Numbers - -The relative age of data and delete files relies on a sequence number that is assigned to every successful commit. When a snapshot is created for a commit, it is optimistically assigned the next sequence number, and it is written into the snapshot's metadata. If the commit fails and must be retried, the sequence number is reassigned and written into new snapshot metadata. - -All manifests, data files, and delete files created for a snapshot inherit the snapshot's sequence number. Manifest file metadata in the manifest list stores a manifest's sequence number. New data and metadata file entries are written with `null` in place of a sequence number, which is replaced with the manifest's sequence number at read time. When a data or delete file is written to a new manifest (as "existing"), the inherited sequence number is written to ensure it does not change after it is first inherited. - -Inheriting the sequence number from manifest metadata allows writing a new manifest once and reusing it in commit retries. To change a sequence number for a retry, only the manifest list must be rewritten -- which would be rewritten anyway with the latest set of manifests. - -### Row-level Deletes - -Row-level deletes are stored in delete files. - -There are two types of row-level deletes: - -* **Position deletes** -- Mark a row deleted by data file path and the row position in the data file. Position deletes are encoded in a [_position delete file_](#position-delete-files) (V2) or [_deletion vector_](#deletion-vectors) (V3 or above). - -* **Equality deletes** -- Mark a row deleted by one or more column values, like id = 5. Equality deletes are encoded in [_equality delete file_](#equality-delete-files). - -Like data files, delete files are tracked by partition. In general, a delete file must be applied to older data files with the same partition; see [Scan Planning](#scan-planning) for details. Column metrics can be used to determine whether a delete file's rows overlap the contents of a data file or a scan range. - -### File System Operations - -Iceberg only requires that file systems support the following operations: - -* **In-place write** -- Files are not moved or altered once they are written. -* **Seekable reads** -- Data file formats require seek support. -* **Deletes** -- Tables delete files that are no longer used. - -These requirements are compatible with object stores, like S3. - -Tables do not require random-access writes. Once written, data and metadata files are immutable until they are deleted. - -Tables do not require rename, except for tables that use atomic rename to implement the commit operation for new metadata files. - -## Specification - -#### Terms - -* **Schema** -- Names and types of fields in a table. -* **Partition spec** -- A definition of how partition values are derived from data fields. -* **Snapshot** -- The state of a table at some point in time, including the set of all data files. -* **Manifest list** -- A file that lists manifest files; one per snapshot. -* **Manifest** -- A file that lists data or delete files; a subset of a snapshot. -* **Data file** -- A file that contains rows of a table. -* **Delete file** -- A file that encodes rows of a table that are deleted by position or data values. - -#### Writer requirements - -Some tables in this spec have columns that specify requirements for tables by version. These requirements are intended for writers when adding metadata files (including manifests files and manifest lists) to a table with the given version. - -| Requirement | Write behavior | -|-------------|----------------| -| (blank) | The field should be omitted | -| _optional_ | The field can be written or omitted | -| _required_ | The field must be written | - -Readers should be more permissive because v1 metadata files are allowed in v2 tables (or later) so that tables can be upgraded to without rewriting the metadata tree. For manifest list and manifest files, this table shows the expected read behavior for later versions: - -| v1 | v2 | v2+ read behavior | -|------------|------------|-------------------| -| | _optional_ | Read the field as _optional_ | -| | _required_ | Read the field as _optional_; it may be missing in v1 files | -| _optional_ | | Ignore the field | -| _optional_ | _optional_ | Read the field as _optional_ | -| _optional_ | _required_ | Read the field as _optional_; it may be missing in v1 files | -| _required_ | | Ignore the field | -| _required_ | _optional_ | Read the field as _optional_ | -| _required_ | _required_ | Fill in a default or throw an exception if the field is missing | - -If a later version is not shown, the requirement for a version is not changed from the most recent version shown. For example, v3 uses the same requirements as v2 if a table shows only v1 and v2 requirements. - -Readers may be more strict for metadata JSON files because the JSON files are not reused and will always match the table version. Required fields that were not present in or were optional in prior versions may be handled as required fields. For example, a v2 table that is missing `last-sequence-number` can throw an exception. - -#### Writing data files - -All columns must be written to data files even if they introduce redundancy with metadata stored in manifest files (e.g. columns with identity partition transforms). Writing all columns provides a backup in case of corruption or bugs in the metadata layer. - -Writers are not allowed to commit files with a partition spec that contains a field with an unknown transform. - -### Schemas and Data Types - -A table's **schema** is a list of named columns. All data types are either primitives or nested types, which are maps, lists, or structs. A table schema is also a struct type. - -For the representations of these types in Avro, ORC, and Parquet file formats, see Appendix A. - -#### Nested Types - -A **`struct`** is a tuple of typed values. Each field in the tuple is named and has an integer id that is unique in the table schema. Each field can be either optional or required, meaning that values can (or cannot) be null. Fields may be any type. Fields may have an optional comment or doc string. Fields can have [default values](#default-values). - -A **`list`** is a collection of values with some element type. The element field has an integer id that is unique in the table schema. Elements can be either optional or required. Element types may be any type. - -A **`map`** is a collection of key-value pairs with a key type and a value type. Both the key field and value field each have an integer id that is unique in the table schema. Map keys are required and map values can be either optional or required. Both map keys and map values may be any type, including nested types. - -#### Semi-structured Types - -A **`variant`** is a value that stores semi-structured data. The structure and data types in a variant are not necessarily consistent across rows in a table or data file. The variant type and binary encoding are defined in the [Parquet project](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md), with support currently available for V1. Support for Variant is added in Iceberg v3. - -Variants are similar to JSON with a wider set of primitive values including date, timestamp, timestamptz, binary, and decimals. - -Variant values may contain nested types: - -1. An array is an ordered collection of variant values. -2. An object is a collection of fields that are a string key and a variant value. - -As a semi-structured type, there are important differences between variant and Iceberg's other types: - -1. Variant arrays are similar to lists, but may contain any variant value rather than a fixed element type. -2. Variant objects are similar to structs, but may contain variable fields identified by name and field values may be any variant value rather than a fixed field type. - -#### Primitive Types - -Supported primitive types are defined in the table below. Primitive types added after v1 have an "added by" version that is the first spec version in which the type is allowed. For example, nanosecond-precision timestamps are part of the v3 spec; using v3 types in v1 or v2 tables can break forward compatibility. - -| Added by version | Primitive type | Description | Requirements | -|------------------|--------------------|--------------------------------------------------------------------------|--------------------------------------------------| -| [v3](#version-3) | **`unknown`** | Default / null column type used when a more specific type is not known | Must be optional with `null` defaults; not stored in data files | -| | **`boolean`** | True or false | | -| | **`int`** | 32-bit signed integers | Can promote to `long` | -| | **`long`** | 64-bit signed integers | | -| | **`float`** | [32-bit IEEE 754](https://en.wikipedia.org/wiki/IEEE_754) floating point | Can promote to double | -| | **`double`** | [64-bit IEEE 754](https://en.wikipedia.org/wiki/IEEE_754) floating point | | -| | **`decimal(P,S)`** | Fixed-point decimal; precision P, scale S | Scale is fixed, precision must be 38 or less | -| | **`date`** | Calendar date without timezone or time | | -| | **`time`** | Time of day, microsecond precision, without date, timezone | | -| | **`timestamp`** | Timestamp, microsecond precision, without timezone | [1] | -| | **`timestamptz`** | Timestamp, microsecond precision, with timezone | [2] | -| [v3](#version-3) | **`timestamp_ns`** | Timestamp, nanosecond precision, without timezone | [1] | -| [v3](#version-3) | **`timestamptz_ns`** | Timestamp, nanosecond precision, with timezone | [2] | -| | **`string`** | Arbitrary-length character sequences | Encoded with UTF-8 [3] | -| | **`uuid`** | Universally unique identifiers | Should use 16-byte fixed | -| | **`fixed(L)`** | Fixed-length byte array of length L | | -| | **`binary`** | Arbitrary-length byte array | | -| [v3](#version-3) | **`geometry(C)`** | Geospatial features from [OGC – Simple feature access][1001]. Edge-interpolation is always linear/planar. See [Appendix G](#appendix-g-geospatial-notes). Parameterized by CRS C. If not specified, C is `OGC:CRS84`. | | -| [v3](#version-3) | **`geography(C, A)`** | Geospatial features from [OGC – Simple feature access][1001]. See [Appendix G](#appendix-g-geospatial-notes). Parameterized by CRS C and edge-interpolation algorithm A. If not specified, C is `OGC:CRS84` and A is `spherical`. | - -Notes: - -1. Timestamp values _without time zone_ represent a date and time of day regardless of zone: the time value is independent of zone adjustments (`2017-11-16 17:10:34` is always retrieved as `2017-11-16 17:10:34`). -2. Timestamp values _with time zone_ represent a point in time: values are stored as UTC and do not retain a source time zone (`2017-11-16 17:10:34 PST` is stored/retrieved as `2017-11-17 01:10:34 UTC` and these values are considered identical). -3. Character strings must be stored as UTF-8 encoded byte arrays. - -For details on how to serialize a schema to JSON, see Appendix C. - -[1001]: "OGC Simple feature access" - -##### CRS - -For `geometry` and `geography` types, the parameter C refers to the CRS (coordinate reference system), a mapping of how coordinates refer to locations on Earth. - -For `geometry` type, the CRS does not affect geometric calculations, which are always Cartesian. - -The default CRS value `OGC:CRS84` means that the objects must be stored in longitude, latitude based on the WGS84 datum. - -Custom CRS values can be specified by a string of the format `type:identifier`, where `type` is one of the following values: - -* `srid`: [Spatial reference identifier](https://en.wikipedia.org/wiki/Spatial_reference_system#Identifier), `identifier` is the SRID itself. -* `projjson`: [PROJJSON](https://proj.org/en/stable/specifications/projjson.html), `identifier` is the name of a table property where the projjson string is stored. - -For `geography` types, the custom CRS must be geographic, with longitudes bound by [-180, 180] and latitudes bound by [-90, 90]. - -##### Edge-Interpolation Algorithm - -For `geography` types, an additional parameter A specifies an algorithm for interpolating edges, and is one of the following values: - -* `spherical`: edges are interpolated as geodesics on a sphere. -* `vincenty`: [https://en.wikipedia.org/wiki/Vincenty%27s_formulae](https://en.wikipedia.org/wiki/Vincenty%27s_formulae) -* `thomas`: Thomas, Paul D. Spheroidal geodesics, reference systems, & local geometry. US Naval Oceanographic Office, 1970. -* `andoyer`: Thomas, Paul D. Mathematical models for navigation systems. US Naval Oceanographic Office, 1965. -* `karney`: [Karney, Charles FF. "Algorithms for geodesics." Journal of Geodesy 87 (2013): 43-55](https://link.springer.com/content/pdf/10.1007/s00190-012-0578-z.pdf), and [GeographicLib](https://geographiclib.sourceforge.io/) - -#### Default values - -Default values can be tracked for struct fields (both nested structs and the top-level schema's struct). There can be two defaults with a field: - -* `initial-default` is used to populate the field's value for all records that were written before the field was added to the schema -* `write-default` is used to populate the field's value for any records written after the field was added to the schema, if the writer does not supply the field's value - -The `initial-default` is set only when a field is added to an existing schema. The `write-default` is initially set to the same value as `initial-default` and can be changed through schema evolution. If either default is not set for an optional field, then the default value is null for compatibility with older spec versions. - -The `initial-default` and `write-default` produce SQL default value behavior, without rewriting data files. SQL default value behavior when a field is added handles all existing rows as though the rows were written with the new field's default value. Default value changes may only affect future records and all known fields are written into data files. Omitting a known field when writing a data file is never allowed. The write default for a field must be written if a field is not supplied to a write. If the write default for a required field is not set, the writer must fail. - -All columns of `unknown`, `variant`, `geometry`, and `geography` types must default to null. Non-null values for `initial-default` or `write-default` are invalid. - -Default values for the fields of a struct are tracked as `initial-default` and `write-default` at the field level. Default values for fields that are nested structs must not contain default values for the struct's fields (sub-fields). Sub-field defaults are tracked in sub-field's metadata. As a result, the default stored for a nested struct may be either null or a non-null struct with no field values. The effective default value is produced by setting each fields' default in a new struct. - -For example, a struct column `point` with fields `x` (default 0) and `y` (default 0) can be defaulted to `{"x": 0, "y": 0}` or `null`. A non-null default is stored by setting `initial-default` or `write-default` to an empty struct (`{}`) that will use field values set from each field's `initial-default` or `write-default`, respectively. - -| `point` default | `point.x` default | `point.y` default | Data value | Result value | -|-----------------|-------------------|-------------------|--------------|--------------| -| `null` | `0` | `0` | (missing) | `null` | -| `null` | `0` | `0` | `{"x": 3}` | `{"x": 3, "y": 0}` | -| `{}` | `0` | `0` | (missing) | `{"x": 0, "y": 0}` | -| `{}` | `0` | `0` | `{"y": -1}` | `{"x": 0, "y": -1}` | - -Default values are attributes of fields in schemas and serialized with fields in the JSON format. See [Appendix C](#appendix-c-json-serialization). - -#### Schema Evolution - -Schemas may be evolved by type promotion or adding, deleting, renaming, or reordering fields in structs (both nested structs and the top-level schema’s struct). - -Evolution applies changes to the table's current schema to produce a new schema that is identified by a unique schema ID, is added to the table's list of schemas, and is set as the table's current schema. - -Valid primitive type promotions are: - -| Primitive type | v1, v2 valid type promotions | v3+ valid type promotions | Requirements | -|------------------|------------------------------|------------------------------|--------------| -| `unknown` | | _any type_ | | -| `int` | `long` | `long` | | -| `date` | | `timestamp`, `timestamp_ns` | Promotion to `timestamptz` or `timestamptz_ns` is **not** allowed; values outside the promoted type's range must result in a runtime failure | -| `float` | `double` | `double` | | -| `decimal(P, S)` | `decimal(P', S)` if `P' > P` | `decimal(P', S)` if `P' > P` | Widen precision only | - -Iceberg's Avro manifest format does not store the type of lower and upper bounds, and type promotion does not rewrite existing bounds. For example, when a `float` is promoted to `double`, existing data file bounds are encoded as 4 little-endian bytes rather than 8 little-endian bytes for `double`. To correctly decode the value, the original type at the time the file was written must be inferred according to the following table: - -| Current type | Length of bounds | Inferred type at write time | -|------------------|------------------|-----------------------------| -| `long` | 4 bytes | `int` | -| `long` | 8 bytes | `long` | -| `double` | 4 bytes | `float` | -| `double` | 8 bytes | `double` | -| `timestamp` | 4 bytes | `date` | -| `timestamp` | 8 bytes | `timestamp` | -| `timestamp_ns` | 4 bytes | `date` | -| `timestamp_ns` | 8 bytes | `timestamp_ns` | -| `decimal(P, S)` | _any_ | `decimal(P', S)`; `P' <= P` | - -Type promotion is not allowed for a field that is referenced by `source-id` or `source-ids` of a partition field if the partition transform would produce a different value after promoting the type. For example, `bucket[N]` produces different hash values for `34` and `"34"` (2017239379 != -427558391) but the same value for `34` and `34L`; when an `int` field is the source for a bucket partition field, it may be promoted to `long` but not to `string`. This may happen for the following type promotion cases: - -* `date` to `timestamp` or `timestamp_ns` - -Any struct, including a top-level schema, can evolve through deleting fields, adding new fields, renaming existing fields, reordering existing fields, or promoting a primitive using the valid type promotions. Adding a new field assigns a new ID for that field and for any nested fields. Renaming an existing field must change the name, but not the field ID. Deleting a field removes it from the current schema. Field deletion cannot be rolled back unless the field was nullable or if the current snapshot has not changed. - -Grouping a subset of a struct’s fields into a nested struct is **not** allowed, nor is moving fields from a nested struct into its immediate parent struct (`struct ↔ struct>`). Evolving primitive types to structs is **not** allowed, nor is evolving a single-field struct to a primitive (`map ↔ map>`). - -Struct evolution requires the following rules for default values: - -* The `initial-default` must be set when a field is added and cannot change -* The `write-default` must be set when a field is added and may change -* When a required field is added, both defaults must be set to a non-null value -* When an optional field is added, the defaults may be null and should be explicitly set -* When a field that is a struct type is added, its default may only be null or a non-null struct with no field values. Default values for fields must be stored in field metadata. -* If a field value is missing from a struct's `initial-default`, the field's `initial-default` must be used for the field -* If a field value is missing from a struct's `write-default`, the field's `write-default` must be used for the field - -##### Column Projection - -Columns in Iceberg data files are selected by field id. The table schema's column names and order may change after a data file is written, and projection must be done using field ids. - -Values for field ids which are not present in a data file must be resolved according the following rules: - -* Return the value from partition metadata if an [Identity Transform](#partition-transforms) exists for the field and the partition value is present in the `partition` struct on `data_file` object in the manifest. This allows for metadata only migrations of Hive tables. -* Use `schema.name-mapping.default` metadata to map field id to columns without field id as described below and use the column if it is present. -* Return the default value if it has a defined `initial-default` (See [Default values](#default-values) section for more details). -* Return `null` in all other cases. - -For example, a file may be written with schema `1: a int, 2: b string, 3: c double` and read using projection schema `3: measurement, 2: name, 4: a`. This must select file columns `c` (renamed to `measurement`), `b` (now called `name`), and a column of `null` values called `a`; in that order. - -Tables may also define a property `schema.name-mapping.default` with a JSON name mapping containing a list of field mapping objects. These mappings provide fallback field ids to be used when a data file does not contain field id information. Each object should contain - -* `names`: A required list of 0 or more names for a field. -* `field-id`: An optional Iceberg field ID used when a field's name is present in `names` -* `fields`: An optional list of field mappings for child field of structs, maps, and lists. - -Field mapping fields are constrained by the following rules: - -* A name may contain `.` but this refers to a literal name, not a nested field. For example, `a.b` refers to a field named `a.b`, not child field `b` of field `a`. -* Each child field should be defined with their own field mapping under `fields`. -* Multiple values for `names` may be mapped to a single field ID to support cases where a field may have different names in different data files. For example, all Avro field aliases should be listed in `names`. -* Fields which exist only in the Iceberg schema and not in imported data files may use an empty `names` list. -* Fields that exist in imported files but not in the Iceberg schema may omit `field-id`. -* List types should contain a mapping in `fields` for `element`. -* Map types should contain mappings in `fields` for `key` and `value`. -* Struct types should contain mappings in `fields` for their child fields. - -For details on serialization, see [Appendix C](#name-mapping-serialization). - -#### Identifier Field IDs - -A schema can optionally track the set of primitive fields that identify rows in a table, using the property `identifier-field-ids` (see JSON encoding in Appendix C). - -Two rows are the "same"---that is, the rows represent the same entity---if the identifier fields are equal. However, uniqueness of rows by this identifier is not guaranteed or required by Iceberg and it is the responsibility of processing engines or data providers to enforce. - -Identifier fields may be nested in structs but cannot be nested within maps or lists. Float, double, and optional fields cannot be used as identifier fields and a nested field cannot be used as an identifier field if it is nested in an optional struct, to avoid null values in identifiers. - -#### Reserved Field IDs - -Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALUE - 200`). This id range is reserved for metadata columns that can be used in user data schemas, like the `_file` column that holds the file path in which a row was stored. - -The set of metadata columns is: - -| Field id, name | Type | Description | -|----------------------------------|---------------|--------------------------------------------------------------------------------------------------------| -| **`2147483646 _file`** | `string` | Path of the file in which a row is stored | -| **`2147483645 _pos`** | `long` | Ordinal position of a row in the source data file, starting at `0` | -| **`2147483644 _deleted`** | `boolean` | Whether the row has been deleted | -| **`2147483643 _spec_id`** | `int` | Spec ID used to track the file containing a row | -| **`2147483642 _partition`** | `struct` | Partition to which a row belongs | -| **`2147483546 file_path`** | `string` | Path of a file, used in position-based delete files | -| **`2147483545 pos`** | `long` | Ordinal position of a row, used in position-based delete files | -| **`2147483544 row`** | `struct<...>` | Deleted row values, used in position-based delete files | -| **`2147483543 _change_type`** | `string` | The record type in the changelog (INSERT, DELETE, UPDATE_BEFORE, or UPDATE_AFTER) | -| **`2147483542 _change_ordinal`** | `int` | The order of the change | -| **`2147483541 _commit_snapshot_id`** | `long` | The snapshot ID in which the change occurred | -| **`2147483540 _row_id`** | `long` | A unique long assigned for row lineage, see [Row Lineage](#row-lineage) | -| **`2147483539 _last_updated_sequence_number`** | `long` | The sequence number which last updated this row, see [Row Lineage](#row-lineage) | - -#### Row Lineage - -In v3 and later, an Iceberg table must track row lineage fields for all newly created rows. Engines must maintain the `next-row-id` table field and the following row-level fields when writing data files: - -* `_row_id` a unique long identifier for every row within the table. The value is assigned via inheritance when a row is first added to the table. -* `_last_updated_sequence_number` the sequence number of the commit that last updated a row. The value is inherited when a row is first added or modified. - -These fields are assigned and updated by inheritance because the commit sequence number and starting row ID are not assigned until the snapshot is successfully committed. Inheritance is used to allow writing data and manifest files before values are known so that it is not necessary to rewrite data and manifest files when an optimistic commit is retried. - -Row lineage does not track lineage for rows updated via [Equality Deletes](#equality-delete-files), because engines using equality deletes avoid reading existing data before writing changes and can't provide the original row ID for the new rows. These updates are always treated as if the existing row was completely removed and a unique new row was added. - -##### Row lineage assignment - -When a row is added or modified, the `_last_updated_sequence_number` field is set to `null` so that it is inherited when reading. Similarly, the `_row_id` field for an added row is set to `null` and assigned when reading. - -A data file with only new rows for the table may omit the `_last_updated_sequence_number` and `_row_id`. If the columns are missing, readers should treat both columns as if they exist and are set to null for all rows. - -On read, if `_last_updated_sequence_number` is `null` it is assigned the `sequence_number` of the data file's manifest entry. The data sequence number of a data file is documented in [Sequence Number Inheritance](#sequence-number-inheritance). - -When `null`, a row's `_row_id` field is assigned to the `first_row_id` from its containing data file plus the row position in that data file (`_pos`). A data file's `first_row_id` field is assigned using inheritance and is documented in [First Row ID Inheritance](#first-row-id-inheritance). A manifest's `first_row_id` is assigned when writing the manifest list for a snapshot and is documented in [First Row ID Assignment](#first-row-id-assignment). A snapshot's `first-row-id` is set to the table's `next-row-id` and is documented in [Snapshot Row IDs](#snapshot-row-ids). - -When an existing row is moved to a different data file for any reason, writers should write `_row_id` and `_last_updated_sequence_number` according to the following rules: - -1. The row's existing non-null `_row_id` must be copied into the new data file -2. If the write has modified the row, the `_last_updated_sequence_number` field must be set to `null` (so that the modification's sequence number replaces the current value) -3. If the write has not modified the row, the existing non-null `_last_updated_sequence_number` value must be copied to the new data file - -Engines may model operations as deleting/inserting rows or as modifications to rows that preserve row ids. - -##### Row lineage example - -This example demonstrates how `_row_id` and `_last_updated_sequence_number` are assigned for a snapshot. This starts with a table with a `next-row-id` of 1000. - -Writing a new append snapshot creates snapshot metadata with `first-row-id` assigned to the table's `next-row-id`: - -```json -{ - "operation": "append", - "first-row-id": 1000, - ... -} -``` - -The snapshot's manifest list will contain existing manifests, plus new manifests that are each assigned a `first_row_id` based on the `added_rows_count` and `existing_rows_count` of preceding new manifests: - -| `manifest_path` | `added_rows_count` | `existing_rows_count` | `first_row_id` | -|-----------------|--------------------|-----------------------|--------------------| -| ... | ... | ... | ... | -| existing | 75 | 0 | 925 | -| added1 | 100 | 25 | 1000 | -| added2 | 0 | 100 | 1125 | -| added3 | 125 | 25 | 1225 | - -The existing manifests are written with the `first_row_id` assigned when the manifests were added to the table. - -The first added manifest, `added1`, is assigned the same `first_row_id` as the snapshot and each of the remaining added manifests are assigned a `first_row_id` based on the number of rows in preceding manifests that were assigned a `first_row_id`. - -Note that the second file, `added2`, changes the `first_row_id` of the next manifest even though it contains no added data files because any data file without a `first_row_id` could be assigned one, even if it has existing status. This is optional if the writer knows that existing data files in the manifest have assigned `first_row_id` values. - -Within `added1`, the first added manifest, each data file's `first_row_id` follows a similar pattern: - -| `status` | `file_path` | `record_count` | `first_row_id` | -|----------|-------------|----------------|----------------| -| EXISTING | data1 | 25 | 800 | -| ADDED | data2 | 50 | null (1000) | -| ADDED | data3 | 50 | null (1050) | - -The `first_row_id` of the EXISTING file `data1` was already assigned, so the file metadata was copied into manifest `added1`. - -Files `data2` and `data3` are written with `null` for `first_row_id` and are assigned `first_row_id` at read time based on the manifest's `first_row_id` and the `record_count` of previous files without `first_row_id` in this manifest: (1,000 + 0) and (1,000 + 50). - -The snapshot then populates the total number of `added-rows` based on the sum of all added rows in the manifests: 100 (50 + 50) - -When the new snapshot is committed, the table's `next-row-id` must also be updated (even if the new snapshot is not in the main branch). Because 375 rows were in data files in manifests that were assigned a `first_row_id` (`added1` 100+25, `added2` 0+100, `added3` 125+25) the new value is 1,000 + 375 = 1,375. - -##### Row Lineage for Upgraded Tables - -When a table is upgraded to v3, its `next-row-id` is initialized to 0 and existing snapshots are not modified (that is, `first-row-id` remains unset or null). For such snapshots without `first-row-id`, `first_row_id` values for data files and data manifests are null, and values for `_row_id` are read as null for all rows. When `first_row_id` is null, inherited row ID values are also null. - -Snapshots that are created after upgrading to v3 must set the snapshot's `first-row-id` and assign row IDs to existing and added files in the snapshot. When writing the manifest list, all data manifests must be assigned a `first_row_id`, which assigns a `first_row_id` to all data files via inheritance. - -Note that: - -* Snapshots created before upgrading to v3 do not have row IDs. -* After upgrading, new snapshots in different branches will assign disjoint ID ranges to existing data files, based on the table's `next-row-id` when the snapshot is committed. For a data file in multiple branches, a writer may write the `first_row_id` from another branch or may assign a new `first_row_id` to the data file (to avoid large metadata rewrites). -* Existing rows will inherit `_last_updated_sequence_number` from their containing data file. - -### Partitioning - -Data files are stored in manifests with a tuple of partition values that are used in scans to filter out files that cannot contain records that match the scan’s filter predicate. Partition values for a data file must be the same for all records stored in the data file. (Manifests store data files from any partition, as long as the partition spec is the same for the data files.) - -Tables are configured with a **partition spec** that defines how to produce a tuple of partition values from a record. A partition spec has a list of fields that consist of: - -* A **source column id** or a list of **source column ids** from the table’s schema -* A **partition field id** that is used to identify a partition field and is unique within a partition spec. In v2 table metadata, it is unique across all partition specs. -* A **transform** that is applied to the source column(s) to produce a partition value -* A **partition name** - -The source columns, selected by ids, must be a primitive type and cannot be contained in a map or list, but may be nested in a struct. For details on how to serialize a partition spec to JSON, see Appendix C. - -Partition specs capture the transform from table data to partition values. This is used to transform predicates to partition predicates, in addition to transforming data values. Deriving partition predicates from column predicates on the table data is used to separate the logical queries from physical storage: the partitioning can change and the correct partition filters are always derived from column predicates. This simplifies queries because users don’t have to supply both logical predicates and partition predicates. For more information, see Scan Planning below. - -Partition fields that use an unknown transform can be read by ignoring the partition field for the purpose of filtering data files during scan planning. In v1 and v2, readers should ignore fields with unknown transforms while reading; this behavior is required in v3. Writers are not allowed to commit data using a partition spec that contains a field with an unknown transform. - -Two partition specs are considered equivalent with each other if they have the same number of fields and for each corresponding field, the fields have the same source column IDs, transform definition and partition name. Writers must not create a new partition spec if there already exists a compatible partition spec defined in the table. - -Partition field IDs must be reused if an existing partition spec contains an equivalent field. - -#### Partition Transforms - -| Transform name | Description | Source types | Result type | -|-------------------|--------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|-------------| -| **`identity`** | Source value, unmodified | Any except for `geometry`, `geography`, and `variant` | Source type | -| **`bucket[N]`** | Hash of value, mod `N` (see below) | `int`, `long`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`, `string`, `uuid`, `fixed`, `binary` | `int` | -| **`truncate[W]`** | Value truncated to width `W` (see below) | `int`, `long`, `decimal`, `string`, `binary` | Source type | -| **`year`** | Extract a date or timestamp year, as years from 1970 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` | `int` | -| **`month`** | Extract a date or timestamp month, as months from 1970-01-01 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` | `int` | -| **`day`** | Extract a date or timestamp day, as days from 1970-01-01 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` | `int` | -| **`hour`** | Extract a timestamp hour, as hours from 1970-01-01 00:00:00 | `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` | `int` | -| **`void`** | Always produces `null` | Any | Source type or `int` | - -All transforms must return `null` for a `null` input value. - -The `void` transform may be used to replace the transform in an existing partition field so that the field is effectively dropped in v1 tables. See partition evolution below. - -#### Bucket Transform Details - -Bucket partition transforms use a 32-bit hash of the source value. The 32-bit hash implementation is the 32-bit Murmur3 hash, x86 variant, seeded with 0. - -Transforms are parameterized by a number of buckets [1], `N`. The hash mod `N` must produce a positive value by first discarding the sign bit of the hash value. In pseudo-code, the function is: - -``` - def bucket_N(x) = (murmur3_x86_32_hash(x) & Integer.MAX_VALUE) % N -``` - -Notes: - -1. Changing the number of buckets as a table grows is possible by evolving the partition spec. - -For hash function details by type, see Appendix B. - -#### Truncate Transform Details - -| **Type** | **Config** | **Truncate specification** | **Examples** | -|---------------|-----------------------|------------------------------------------------------------------|----------------------------------| -| **`int`** | `W`, width | `v - (v % W)` remainders must be positive [1] | `W=10`: `1` ￫ `0`, `-1` ￫ `-10` | -| **`long`** | `W`, width | `v - (v % W)` remainders must be positive [1] | `W=10`: `1` ￫ `0`, `-1` ￫ `-10` | -| **`decimal`** | `W`, width (no scale) | `scaled_W = decimal(W, scale(v))` `v - (v % scaled_W)` [1, 2] | `W=50`, `s=2`: `10.65` ￫ `10.50` | -| **`string`** | `L`, length | Substring of length `L`: `v.substring(0, L)` [3] | `L=3`: `iceberg` ￫ `ice` | -| **`binary`** | `L`, length | Sub array of length `L`: `v.subarray(0, L)` [4] | `L=3`: `\x01\x02\x03\x04\x05` ￫ `\x01\x02\x03` | - -Notes: - -1. The remainder, `v % W`, must be positive. For languages where `%` can produce negative values, the correct truncate function is: `v - (((v % W) + W) % W)` -2. The width, `W`, used to truncate decimal values is applied using the scale of the decimal column to avoid additional (and potentially conflicting) parameters. -3. Strings are truncated to a valid UTF-8 string with no more than `L` code points. -4. In contrast to strings, binary values do not have an assumed encoding and are truncated to `L` bytes. - -#### Partition Evolution - -Table partitioning can be evolved by adding, removing, renaming, or reordering partition spec fields. - -Changing a partition spec produces a new spec identified by a unique spec ID that is added to the table's list of partition specs and may be set as the table's default spec. - -When evolving a spec, changes should not cause partition field IDs to change because the partition field IDs are used as the partition tuple field IDs in manifest files. - -In v2, partition field IDs must be explicitly tracked for each partition field. New IDs are assigned based on the last assigned partition ID in table metadata. - -In v1, partition field IDs were not tracked, but were assigned sequentially starting at 1000 in the reference implementation. This assignment caused problems when reading metadata tables based on manifest files from multiple specs because partition fields with the same ID may contain different data types. For compatibility with old versions, the following rules are recommended for partition evolution in v1 tables: - -1. Do not reorder partition fields -2. Do not drop partition fields; instead replace the field's transform with the `void` transform -3. Only add partition fields at the end of the previous partition spec - -### Sorting - -Users can sort their data within partitions by columns to gain performance. The information on how the data is sorted can be declared per data or delete file, by a **sort order**. - -A sort order is defined by a sort order id and a list of sort fields. The order of the sort fields within the list defines the order in which the sort is applied to the data. Each sort field consists of: - -* A **source column id** or a list of **source column ids** from the table's schema -* A **transform** that is used to produce values to be sorted on from the source column(s). This is the same transform as described in [partition transforms](#partition-transforms). -* A **sort direction**, that can only be either `asc` or `desc` -* A **null order** that describes the order of null values when sorted. Can only be either `nulls-first` or `nulls-last` - -For details on how to serialize a sort order to JSON, see Appendix C. - -Order id `0` is reserved for the unsorted order. - -Sorting floating-point numbers should produce the following behavior: `-NaN` < `-Infinity` < `-value` < `-0` < `0` < `value` < `Infinity` < `NaN`. This aligns with the implementation of Java floating-point types comparisons. - -A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. - -### Manifests - -A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot. - -A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. - -A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata. - -A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest. - -A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata: - -=== "v1 - v3" - | v1 | v2 and v3 | Key | Value | - |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------| - | _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written | - | _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string | - | _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) | - | _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string | - | _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string | - | | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" | - -The schema of a manifest file is defined by the `manifest_entry` struct, described in the following section. - -#### Manifest Entry Fields - -The `manifest_entry` struct consists of the following fields: - -=== "v1 - v3" - | v1 | v2 and v3 | Field id, name | Type | Description | - | ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------| - | _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. | - | _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. | - | | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). | - | | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). | - | _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... | - -The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields. - -When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added). - -When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1]. - -Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list. -The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file. -The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number. -The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly. - -Notes: - -1. Technically, data files can be deleted when the last snapshot that contains the file as “live” data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots. -2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available. - -##### Data File Fields - -The `data_file` struct consists of the following fields: - -=== "v1 - v3" - | v1 | v2 | v3 | Field id, name | Type | Description | - | ---------- |------------|------------|-----------------------------------|-----------------------------------------------------------------------------|-------------| - | | _required_ | _required_ | **`134 content`** | `int` with meaning: `0: DATA`, `1: POSITION DELETES`, `2: EQUALITY DELETES` | Type of content stored by the data file: data, equality deletes, or position deletes (all v1 files are data files) | - | _required_ | _required_ | _required_ | **`100 file_path`** | `string` | Full URI for the file with FS scheme | - | _required_ | _required_ | _required_ | **`101 file_format`** | `string` | String file format name, `avro`, `orc`, `parquet`, or `puffin` | - | _required_ | _required_ | _required_ | **`102 partition`** | `struct<...>` | Partition data tuple, schema based on the partition spec output using partition field ids for the struct field ids | - | _required_ | _required_ | _required_ | **`103 record_count`** | `long` | Number of records in this file, or the cardinality of a deletion vector | - | _required_ | _required_ | _required_ | **`104 file_size_in_bytes`** | `long` | Total file size in bytes | - | _required_ | | | ~~**`105 block_size_in_bytes`**~~ | `long` | **Deprecated. Always write a default in v1. Do not write in v2 or v3.** | - | _optional_ | | | ~~**`106 file_ordinal`**~~ | `int` | **Deprecated. Do not write.** | - | _optional_ | | | ~~**`107 sort_columns`**~~ | `list<112: int>` | **Deprecated. Do not write.** | - | _optional_ | _optional_ | _optional_ | **`108 column_sizes`** | `map<117: int, 118: long>` | Map from column id to the total size on disk of all regions that store the column. **Does not include bytes necessary to read other columns, like footers.** Leave null for row-oriented formats (Avro) | - | _optional_ | _optional_ | _optional_ | **`109 value_counts`** | `map<119: int, 120: long>` | Map from column id to number of values in the column (including null and NaN values) | - | _optional_ | _optional_ | _optional_ | **`110 null_value_counts`** | `map<121: int, 122: long>` | Map from column id to number of null values in the column | - | _optional_ | _optional_ | _optional_ | **`137 nan_value_counts`** | `map<138: int, 139: long>` | Map from column id to number of NaN values in the column | - | _optional_ | _optional_ | | ~~**`111 distinct_counts`**~~ | `map<123: int, 124: long>` | **Deprecated. Do not write.** | - | _optional_ | _optional_ | _optional_ | **`125 lower_bounds`** | `map<126: int, 127: binary>` | Map from column id to lower bound in the column serialized as binary [1]. Each value must be less than or equal to all non-null, non-NaN values in the column for the file [2] | - | _optional_ | _optional_ | _optional_ | **`128 upper_bounds`** | `map<129: int, 130: binary>` | Map from column id to upper bound in the column serialized as binary [1]. Each value must be greater than or equal to all non-null, non-Nan values in the column for the file [2] | - | _optional_ | _optional_ | _optional_ | **`131 key_metadata`** | `binary` | Implementation-specific key metadata for encryption | - | _optional_ | _optional_ | _optional_ | **`132 split_offsets`** | `list<133: long>` | Split offsets for the data file. For example, all row group offsets in a Parquet file. Must be sorted ascending | - | | _optional_ | _optional_ | **`135 equality_ids`** | `list<136: int>` | Field ids used to determine row equality in equality delete files. Required when `content=2` and should be null otherwise. Fields with ids listed in this column must be present in the delete file | - | _optional_ | _optional_ | _optional_ | **`140 sort_order_id`** | `int` | ID representing sort order for this file [3]. | - | | | _optional_ | **`142 first_row_id`** | `long` | The `_row_id` for the first row in the data file. See [First Row ID Inheritance](#first-row-id-inheritance) | - | | _optional_ | _optional_ | **`143 referenced_data_file`** | `string` | Fully qualified location (URI with FS scheme) of a data file that all deletes reference [4] | - | | | _optional_ | **`144 content_offset`** | `long` | The offset in the file where the content starts [5] | - | | | _optional_ | **`145 content_size_in_bytes`** | `long` | The length of a referenced content stored in the file; required if `content_offset` is present [5] | - -The `partition` struct stores the tuple of partition values for each file. Its type is derived from the partition fields of the partition spec used to write the manifest file. In v2, the partition struct's field ids must match the ids from the partition spec. - -The column metrics maps are used when filtering to select both data and delete files. For delete files, the metrics must store bounds and counts for all deleted rows, or must be omitted. Storing metrics for deleted rows ensures that the values can be used during job planning to find delete files that must be merged during a scan. - -Notes: - -1. Single-value serialization for lower and upper bounds is detailed in Appendix D. -2. For `float` and `double`, the value `-0.0` must precede `+0.0`, as in the IEEE 754 `totalOrder` predicate. NaNs are not permitted as lower or upper bounds. -3. If sort order ID is missing or unknown, then the order is assumed to be unsorted. Only data files and equality delete files should be written with a non-null order id. [Position deletes](#position-delete-files) are required to be sorted by file and position, not a table order, and should set sort order id to null. Readers must ignore sort order id for position delete files. -4. Position delete metadata can use `referenced_data_file` when all deletes tracked by the entry are in a single data file. Setting the referenced file is required for deletion vectors. -5. The `content_offset` and `content_size_in_bytes` fields are used to reference a specific blob for direct access to a deletion vector. For deletion vectors, these values are required and must exactly match the `offset` and `length` stored in the Puffin footer for the deletion vector blob. -6. The following field ids are reserved on `data_file`: 141. - -###### Bounds for Variant, Geometry, and Geography - -For Variant, values in the `lower_bounds` and `upper_bounds` maps store serialized Variant objects that contain lower or upper bounds respectively. The object keys for the bound-variants are normalized JSON path expressions that uniquely identify a field. The object values are primitive Variant representations of the lower or upper bound for that field. Including bounds for any field is optional and upper and lower bounds must have the same Variant type. - -Bounds for a field must be accurate for all non-null values of the field in a data file. Bounds for values within arrays must be accurate for all values in the array. Bounds must not be written to describe values with mixed Variant types (other than null). For example, a **measurement** field that contains int64 and null values may have bounds, but if the field also contained a string value such as **n/a** or **0** then the field may not have bounds. - -The Variant bounds objects are serialized by concatenating the [Variant encoding](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) of the metadata (containing the normalized field paths) and the bounds object. -Field paths follow the JSON path format to use normalized path, such as `$['location']['latitude']` or `$['user.name']`. The special path `$` represents bounds for the variant root, indicating that the variant data consists of uniform primitive types, such as strings. - -Examples of valid field paths using normalized JSON path format are: - -* `$` -- the root of the Variant value -* `$['event_type']` -- the `event_type` field in a Variant object -* `$['user.name']` -- the `"user.name"` field in a Variant object -* `$['location']['latitude']` -- the `latitude` field nested within a `location` object -* `$['tags']` -- the `tags` array -* `$['addresses']['zip']` -- the `zip` field in an `addresses` array that contains objects - -For `geometry` and `geography` types, `lower_bounds` and `upper_bounds` are both points of the following coordinates X, Y, Z, and M (see Appendix G) which are the lower / upper bound of all objects in the file. - -For `geography` only, xmin (X value of `lower_bounds`) may be greater than xmax (X value of `upper_bounds`), in which case an object in this bounding box may match if it contains an X such that x >= xmin OR x <= xmax. In geographic terminology, the concepts of xmin, xmax, ymin, and ymax are also known as westernmost, easternmost, southernmost and northernmost, respectively. These points are further restricted to the canonical ranges of [-180..180] for X and [-90..90] for Y. - -When calculating upper and lower bounds for `geometry` and `geography`, null or NaN values in a coordinate dimension are skipped; for example, POINT (1 NaN) contributes a value to X but no values to Y, Z, or M dimension bounds. If a dimension has only null or NaN values, that dimension is omitted from the bounding box. If either the X or Y dimension is missing then the bounding box itself is not produced. - -#### Sequence Number Inheritance - -Manifests track the sequence number when a data or delete file was added to the table. - -When adding a new file, its data and file sequence numbers are set to `null` because the snapshot's sequence number is not assigned until the snapshot is successfully committed. When reading, sequence numbers are inherited by replacing `null` with the manifest's sequence number from the manifest list. -It is also possible to add a new file with data that logically belongs to an older sequence number. In that case, the data sequence number must be provided explicitly and not inherited. However, the file sequence number must be always assigned when the snapshot is successfully committed. - -When writing an existing file to a new manifest or marking an existing file as deleted, the data and file sequence numbers must be non-null and set to the original values that were either inherited or provided at the commit time. - -Inheriting sequence numbers through the metadata tree allows writing a new manifest without a known sequence number, so that a manifest can be written once and reused in commit retries. To change a sequence number for a retry, only the manifest list must be rewritten. - -When reading v1 manifests with no sequence number column, sequence numbers for all files must default to 0. - -#### First Row ID Inheritance - -When adding a new data file, its `first_row_id` field is set to `null` because it is not assigned until the snapshot is successfully committed. - -When reading, the `first_row_id` is assigned by replacing `null` with the manifest's `first_row_id` plus the sum of `record_count` for all data files that preceded the file in the manifest that also had a null `first_row_id`. - -The inherited value of `first_row_id` must be written into data file metadata when creating existing and deleted entries. The value of `first_row_id` for delete files is always `null`. - -Any null (unassigned) `first_row_id` must be assigned via inheritance, even if the data file is existing. This ensures that row IDs are assigned to existing data files in upgraded tables in the first commit after upgrading to v3. - -### Snapshots - -A snapshot consists of the following fields: -=== "v1 - v3" - | v1 | v2 | v3 | Field | Description | - | ---------- | ---------- |------------|------------------------------|-------------| - | _required_ | _required_ | _required_ | **`snapshot-id`** | A unique long ID | - | _optional_ | _optional_ | _optional_ | **`parent-snapshot-id`** | The snapshot ID of the snapshot's parent. Omitted for any snapshot with no parent | - | | _required_ | _required_ | **`sequence-number`** | A monotonically increasing long that tracks the order of changes to a table | - | _required_ | _required_ | _required_ | **`timestamp-ms`** | A timestamp when the snapshot was created, used for garbage collection and table inspection | - | _optional_ | _required_ | _required_ | **`manifest-list`** | The location of a manifest list for this snapshot that tracks manifest files with additional metadata | - | _optional_ | | | **`manifests`** | A list of manifest file locations. Must be omitted if `manifest-list` is present | - | _optional_ | _required_ | _required_ | **`summary`** | A string map that summarizes the snapshot changes, including `operation` as a _required_ field (see below) | - | _optional_ | _optional_ | _optional_ | **`schema-id`** | ID of the table's current schema when the snapshot was created | - | | | _required_ | **`first-row-id`** | The first `_row_id` assigned to the first row in the first data file in the first manifest, see [Row Lineage](#row-lineage) | - | | | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) | - | | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata | - -The snapshot summary's `operation` field is used by some operations, like snapshot expiration, to skip processing certain snapshots. Possible `operation` values are: - -* `append` -- Only data files were added and no files were removed. -* `replace` -- Data and delete files were added and removed without changing table data; i.e., compaction, changing the data file format, or relocating data files. -* `overwrite` -- Data and delete files were added and removed in a logical overwrite operation. -* `delete` -- Data files were removed and their contents logically deleted and/or delete files were added to delete rows. - -For other optional snapshot summary fields, see [Appendix F](#optional-snapshot-summary-fields). - -Data and delete files for a snapshot can be stored in more than one manifest. This enables: - -* Appends can add a new manifest to minimize the amount of data written, instead of adding new records by rewriting and appending to an existing manifest. (This is called a “fast append”.) -* Tables can use multiple partition specs. A table’s partition configuration can evolve if, for example, its data volume changes. Each manifest uses a single partition spec, and queries do not need to change because partition filters are derived from data predicates. -* Large tables can be split across multiple manifests so that implementations can parallelize job planning or reduce the cost of rewriting a manifest. - -Manifests for a snapshot are tracked by a manifest list. - -Valid snapshots are stored as a list in table metadata. For serialization, see Appendix C. - -#### Snapshot Row IDs - -A snapshot's `first-row-id` is assigned to the table's current `next-row-id` on each commit attempt. If a commit is retried, the `first-row-id` must be reassigned based on the table's current `next-row-id`. The `first-row-id` field is required even if a commit does not assign any ID space. - -The snapshot's `first-row-id` is the starting `first_row_id` assigned to manifests in the snapshot's manifest list. - -The snapshot's `added-rows` captures the upper bound of the number of rows with assigned row IDs. -It can be used safely to increment the table's `next-row-id` during a commit. -It can be more than the number of rows added in this snapshot and include some existing rows, -see [Row Lineage Example](#row-lineage-example). - -### Manifest Lists - -Snapshots are embedded in table metadata, but the list of manifests for a snapshot are stored in a separate manifest list file. - -A new manifest list is written for each attempt to commit a snapshot because the list of manifests always changes to produce a new snapshot. When a manifest list is written, the (optimistic) sequence number of the snapshot is written for all new manifest files tracked by the list. - -A manifest list includes summary metadata that can be used to avoid scanning all of the manifests in a snapshot when planning a table scan. This includes the number of added, existing, and deleted files, and a summary of values for each field of the partition spec used to write the manifest. - -A manifest list is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection. - -Manifest list files store `manifest_file`, a struct with the following fields: - -=== "v1 - v3" - | v1 | v2 | v3 | Field id, name | Type | Description | - | ---------- | ---------- |------------|-------------------------------------|---------------------------------------------|-------------| - | _required_ | _required_ | _required_ | **`500 manifest_path`** | `string` | Location of the manifest file | - | _required_ | _required_ | _required_ | **`501 manifest_length`** | `long` | Length of the manifest file in bytes | - | _required_ | _required_ | _required_ | **`502 partition_spec_id`** | `int` | ID of a partition spec used to write the manifest; must be listed in table metadata `partition-specs` | - | | _required_ | _required_ | **`517 content`** | `int` with meaning: `0: data`, `1: deletes` | The type of files tracked by the manifest, either data or delete files; 0 for all v1 manifests | - | | _required_ | _required_ | **`515 sequence_number`** | `long` | The sequence number when the manifest was added to the table; use 0 when reading v1 manifest lists | - | | _required_ | _required_ | **`516 min_sequence_number`** | `long` | The minimum data sequence number of all live data or delete files in the manifest; use 0 when reading v1 manifest lists | - | _required_ | _required_ | _required_ | **`503 added_snapshot_id`** | `long` | ID of the snapshot where the manifest file was added | - | _optional_ | _required_ | _required_ | **`504 added_files_count`** | `int` | Number of entries in the manifest that have status `ADDED` (1), when `null` this is assumed to be non-zero | - | _optional_ | _required_ | _required_ | **`505 existing_files_count`** | `int` | Number of entries in the manifest that have status `EXISTING` (0), when `null` this is assumed to be non-zero | - | _optional_ | _required_ | _required_ | **`506 deleted_files_count`** | `int` | Number of entries in the manifest that have status `DELETED` (2), when `null` this is assumed to be non-zero | - | _optional_ | _required_ | _required_ | **`512 added_rows_count`** | `long` | Number of rows in all of files in the manifest that have status `ADDED`, when `null` this is assumed to be non-zero | - | _optional_ | _required_ | _required_ | **`513 existing_rows_count`** | `long` | Number of rows in all of files in the manifest that have status `EXISTING`, when `null` this is assumed to be non-zero | - | _optional_ | _required_ | _required_ | **`514 deleted_rows_count`** | `long` | Number of rows in all of files in the manifest that have status `DELETED`, when `null` this is assumed to be non-zero | - | _optional_ | _optional_ | _optional_ | **`507 partitions`** | `list<508: field_summary>` **(see below)** | A list of field summaries for each partition field in the spec. Each field in the list corresponds to a field in the manifest file’s partition spec. | - | _optional_ | _optional_ | _optional_ | **`519 key_metadata`** | `binary` | Implementation-specific key metadata for encryption | - | | | _optional_ | **`520 first_row_id`** | `long` | The starting `_row_id` to assign to rows added by `ADDED` data files [First Row ID Assignment](#first-row-id-assignment) | - -`field_summary` is a struct with the following fields: -=== "v1 - v3" - | v1 | v2 and v3 | Field id, name | Type | Description | - | ---------- | ---------- |-------------------------|---------------|-------------| - | _required_ | _required_ | **`509 contains_null`** | `boolean` | Whether the manifest contains at least one partition with a null value for the field | - | _optional_ | _optional_ | **`518 contains_nan`** | `boolean` | Whether the manifest contains at least one partition with a NaN value for the field | - | _optional_ | _optional_ | **`510 lower_bound`** | `bytes` [1] | Lower bound for the non-null, non-NaN values in the partition field, or null if all values are null or NaN [2] | - | _optional_ | _optional_ | **`511 upper_bound`** | `bytes` [1] | Upper bound for the non-null, non-NaN values in the partition field, or null if all values are null or NaN [2] | - -Notes: - -1. Lower and upper bounds are serialized to bytes using the single-object serialization in Appendix D. The type of used to encode the value is the type of the partition field data. -2. If -0.0 is a value of the partition field, the `lower_bound` must not be +0.0, and if +0.0 is a value of the partition field, the `upper_bound` must not be -0.0. - -#### First Row ID Assignment - -The `first_row_id` for existing manifests must be preserved when writing a new manifest list. The value of `first_row_id` for delete manifests is always `null`. The `first_row_id` is only assigned for data manifests that do not have a `first_row_id`. Assignment must account for data files that will be assigned `first_row_id` values when the manifest is read. - -The first manifest without a `first_row_id` is assigned a value that is greater than or equal to the `first_row_id` of the snapshot. Subsequent manifests without a `first_row_id` are assigned one based on the previous manifest to be assigned a `first_row_id`. Each assigned `first_row_id` must increase by the row count of all files that will be assigned a `first_row_id` via inheritance in the last assigned manifest. That is, each `first_row_id` must be greater than or equal to the last assigned `first_row_id` plus the total record count of data files with a null `first_row_id` in the last assigned manifest. - -A simple and valid approach is to estimate the number of rows in data files that will be assigned a `first_row_id` using the manifest's `added_rows_count` and `existing_rows_count`: `first_row_id = last_assigned.first_row_id + last_assigned.added_rows_count + last_assigned.existing_rows_count`. - -### Scan Planning - -Scans are planned by reading the manifest files for the current snapshot. Deleted entries in data and delete manifests (those marked with status "DELETED") are not used in a scan. - -Manifests that contain no matching files, determined using either file counts or partition summaries, may be skipped. - -For each manifest, scan predicates, which filter data rows, are converted to partition predicates, which filter partition tuples. These partition predicates are used to select relevant data files, delete files, and deletion vector metadata. Conversion uses the partition spec that was used to write the manifest file regardless of the current partition spec. - -Scan predicates are converted to partition predicates using an _inclusive projection_: if a scan predicate matches a row, then the partition predicate must match that row’s partition. This is called _inclusive_ [1] because rows that do not match the scan predicate may be included in the scan by the partition predicate. - -For example, an `events` table with a timestamp column named `ts` that is partitioned by `ts_day=day(ts)` is queried by users with ranges over the timestamp column: `ts > X`. The inclusive projection is `ts_day >= day(X)`, which is used to select files that may have matching rows. Note that, in most cases, timestamps just before `X` will be included in the scan because the file contains rows that match the predicate and rows that do not match the predicate. - -The inclusive projection for an unknown partition transform is _true_ because the partition field is ignored and not used in filtering. - -Scan predicates are also used to filter data and delete files using column bounds and counts that are stored by field id in manifests. The same filter logic can be used for both data and delete files because both store metrics of the rows either inserted or deleted. If metrics show that a delete file has no rows that match a scan predicate, it may be ignored just as a data file would be ignored [2]. - -Data files that match the query filter must be read by the scan. - -Note that for any snapshot, all file paths marked with "ADDED" or "EXISTING" may appear at most once across all manifest files in the snapshot. If a file path appears more than once, the results of the scan are undefined. Reader implementations may raise an error in this case, but are not required to do so. - -Delete files and deletion vector metadata that match the filters must be applied to data files at read time, limited by the following scope rules. - -* A deletion vector must be applied to a data file when all of the following are true: - - The data file's `file_path` is equal to the deletion vector's `referenced_data_file` - - The data file's data sequence number is _less than or equal to_ the deletion vector's data sequence number - - The data file's partition (both spec and partition values) is equal [4] to the deletion vector's partition -* A _position_ delete file must be applied to a data file when all of the following are true: - - The data file's `file_path` is equal to the delete file's `referenced_data_file` if it is non-null - - The data file's data sequence number is _less than or equal to_ the delete file's data sequence number - - The data file's partition (both spec and partition values) is equal [4] to the delete file's partition - - There is no deletion vector that must be applied to the data file (when added, such a vector must contain all deletes from existing position delete files) -* An _equality_ delete file must be applied to a data file when all of the following are true: - - The data file's data sequence number is _strictly less than_ the delete's data sequence number - - The data file's partition (both spec id and partition values) is equal [4] to the delete file's partition _or_ the delete file's partition spec is unpartitioned - -In general, deletes are applied only to data files that are older and in the same partition, except for two special cases: - -* Equality delete files stored with an unpartitioned spec are applied as global deletes. Otherwise, delete files do not apply to files in other partitions. -* Position deletes (vectors and files) must be applied to data files from the same commit, when the data and delete file data sequence numbers are equal. This allows deleting rows that were added in the same commit. - -Notes: - -1. An alternative, *strict projection*, creates a partition predicate that will match a file if all of the rows in the file must match the scan predicate. These projections are used to calculate the residual predicates for each file in a scan. -2. For example, if `file_a` has rows with `id` between 1 and 10 and a delete file contains rows with `id` between 1 and 4, a scan for `id = 9` may ignore the delete file because none of the deletes can match a row that will be selected. -3. Floating point partition values are considered equal if their IEEE 754 floating-point "single format" bit layout are equal with NaNs normalized to have only the most significant mantissa bit set (the equivalent of calling `Float.floatToIntBits` or `Double.doubleToLongBits` in Java). The Avro specification requires all floating point values to be encoded in this format. -4. Unknown partition transforms do not affect partition equality. Although partition fields with unknown transforms are ignored for filtering, the result of an unknown transform is still used when testing whether partition values are equal. - -### Snapshot References - -Iceberg tables keep track of branches and tags using snapshot references. -Tags are labels for individual snapshots. Branches are mutable named references that can be updated by committing a new snapshot as the branch's referenced snapshot using the [Commit Conflict Resolution and Retry](#commit-conflict-resolution-and-retry) procedures. - -The snapshot reference object records all the information of a reference including snapshot ID, reference type and [Snapshot Retention Policy](#snapshot-retention-policy). - -=== "v1 - v3" - | v1 | v2 and v3 | Field name | Type | Description | - | ---------- | ---------- | ---------------------------- | --------- | ----------- | - | _required_ | _required_ | **`snapshot-id`** | `long` | A reference's snapshot ID. The tagged snapshot or latest snapshot of a branch. | - | _required_ | _required_ | **`type`** | `string` | Type of the reference, `tag` or `branch` | - | _optional_ | _optional_ | **`min-snapshots-to-keep`** | `int` | For `branch` type only, a positive number for the minimum number of snapshots to keep in a branch while expiring snapshots. Defaults to table property `history.expire.min-snapshots-to-keep`. | - | _optional_ | _optional_ | **`max-snapshot-age-ms`** | `long` | For `branch` type only, a positive number for the max age of snapshots to keep when expiring, including the latest snapshot. Defaults to table property `history.expire.max-snapshot-age-ms`. | - | _optional_ | _optional_ | **`max-ref-age-ms`** | `long` | For snapshot references except the `main` branch, a positive number for the max age of the snapshot reference to keep while expiring snapshots. Defaults to table property `history.expire.max-ref-age-ms`. The `main` branch never expires. | - -Valid snapshot references are stored as the values of the `refs` map in table metadata. For serialization, see Appendix C. - -### Snapshot Retention Policy - -Table snapshots expire and are removed from metadata to allow removed or replaced data files to be physically deleted. -The snapshot expiration procedure removes snapshots from table metadata and applies the table's retention policy. -Retention policy can be configured both globally and on snapshot reference through properties `min-snapshots-to-keep`, `max-snapshot-age-ms` and `max-ref-age-ms`. - -When expiring snapshots, retention policies in table and snapshot references are evaluated in the following way: - -1. Start with an empty set of snapshots to retain -2. Remove any refs (other than main) where the referenced snapshot is older than `max-ref-age-ms` -3. For each branch and tag, add the referenced snapshot to the retained set -4. For each branch, add its ancestors to the retained set until: - 1. The snapshot is older than `max-snapshot-age-ms`, AND - 2. The snapshot is not one of the first `min-snapshots-to-keep` in the branch (including the branch's referenced snapshot) -5. Expire any snapshot not in the set of snapshots to retain. - -### Table Metadata - -Table metadata is stored as JSON. Each table metadata change creates a new table metadata file that is committed by an atomic operation. This operation is used to ensure that a new version of table metadata replaces the version on which it was based. This produces a linear history of table versions and ensures that concurrent writes are not lost. - -The atomic operation used to commit metadata depends on how tables are tracked and is not standardized by this spec. See the sections below for examples. - -#### Table Metadata Fields - -Table metadata consists of the following fields: - -=== "v1 - v3" - | v1 | v2 | v3 | Field | Description | - | ---------- | ---------- |------------|-----------------------------| ------------| - | _required_ | _required_ | _required_ | **`format-version`** | An integer version number for the format. Implementations must throw an exception if a table’s version is higher than the supported version. | - | _optional_ | _required_ | _required_ | **`table-uuid`** | A UUID that identifies the table, generated when the table is created. Implementations must throw an exception if a table’s UUID does not match the expected UUID after refreshing metadata. | - | _required_ | _required_ | _required_ | **`location`** | The table’s base location. This is used by writers to determine where to store data files, manifest files, and table metadata files. | - | | _required_ | _required_ | **`last-sequence-number`** | The table’s highest assigned sequence number, a monotonically increasing long that tracks the order of snapshots in a table. | - | _required_ | _required_ | _required_ | **`last-updated-ms`** | Timestamp in milliseconds from the unix epoch when the table was last updated. Each table metadata file should update this field just before writing. | - | _required_ | _required_ | _required_ | **`last-column-id`** | An integer; the highest assigned column ID for the table. This is used to ensure columns are always assigned an unused ID when evolving schemas. | - | _required_ | | | **`schema`** | The table’s current schema. (**Deprecated**: use `schemas` and `current-schema-id` instead) | - | _optional_ | _required_ | _required_ | **`schemas`** | A list of schemas, stored as objects with `schema-id`. | - | _optional_ | _required_ | _required_ | **`current-schema-id`** | ID of the table’s current schema. | - | _required_ | | | **`partition-spec`** | The table’s current partition spec, stored as only fields. (**Deprecated**: use `partition-specs` and `default-spec-id` instead) Note that this is used by writers to partition data, but is not used when reading because reads use the specs stored in manifest files. | - | _optional_ | _required_ | _required_ | **`partition-specs`** | A list of partition specs, stored as full partition spec objects. | - | _optional_ | _required_ | _required_ | **`default-spec-id`** | ID of the "current" spec that writers should use by default. | - | _optional_ | _required_ | _required_ | **`last-partition-id`** | An integer; the highest assigned partition field ID across all partition specs for the table. This is used to ensure partition fields are always assigned an unused ID when evolving specs. | - | _optional_ | _optional_ | _optional_ | **`properties`** | A string to string map of table properties. This is used to control settings that affect reading and writing and is not intended to be used for arbitrary metadata. For example, `commit.retry.num-retries` is used to control the number of commit retries. | - | _optional_ | _optional_ | _optional_ | **`current-snapshot-id`** | `long` ID of the current table snapshot; must be the same as the current ID of the `main` branch in `refs`. | - | _optional_ | _optional_ | _optional_ | **`snapshots`** | A list of valid snapshots. Valid snapshots are snapshots for which all data files exist in the file system. A data file must not be deleted from the file system until the last snapshot in which it was listed is garbage collected. | - | _optional_ | _optional_ | _optional_ | **`snapshot-log`** | A list (optional) of timestamp and snapshot ID pairs that encodes changes to the current snapshot for the table. Each time the current-snapshot-id is changed, a new entry should be added with the last-updated-ms and the new current-snapshot-id. When snapshots are expired from the list of valid snapshots, all entries before a snapshot that has expired should be removed. | - | _optional_ | _optional_ | _optional_ | **`metadata-log`** | A list (optional) of timestamp and metadata file location pairs that encodes changes to the previous metadata files for the table. Each time a new metadata file is created, a new entry of the previous metadata file location should be added to the list. Tables can be configured to remove oldest metadata log entries and keep a fixed-size log of the most recent entries after a commit. | - | _optional_ | _required_ | _required_ | **`sort-orders`** | A list of sort orders, stored as full sort order objects. | - | _optional_ | _required_ | _required_ | **`default-sort-order-id`** | Default sort order id of the table. Note that this could be used by writers, but is not used when reading because reads use the specs stored in manifest files. | - | | _optional_ | _optional_ | **`refs`** | A map of snapshot references. The map keys are the unique snapshot reference names in the table, and the map values are snapshot reference objects. There is always a `main` branch reference pointing to the `current-snapshot-id` even if the `refs` map is null. | - | _optional_ | _optional_ | _optional_ | **`statistics`** | A list (optional) of [table statistics](#table-statistics). | - | _optional_ | _optional_ | _optional_ | **`partition-statistics`** | A list (optional) of [partition statistics](#partition-statistics). | - | | | _required_ | **`next-row-id`** | A `long` higher than all assigned row IDs; the next snapshot’s `first-row-id`. See [Row Lineage](#row-lineage). | - | | | _optional_ | **`encryption-keys`** | A list (optional) of [encryption keys](#encryption-keys) used for table encryption. | - -For serialization details, see Appendix C. - -When a new snapshot is added, the table's `next-row-id` should be increased by the sum of `record_count` for all data files that will be assigned a `first_row_id` via inheritance in the snapshot. The `next-row-id` must always be higher than any assigned row ID in the table. - -A simple and valid approach is estimate of the number of rows in data files that will be assigned a `first_row_id` using the manifests' `added_rows_count` and `existing_rows_count`. Using the last assigned manifest, this is `next-row-id = last_assigned.first_row_id + last_assigned.added_rows_count + last_assigned.existing_rows_count`. - -#### Table Statistics - -Table statistics files are valid [Puffin files](puffin-spec.md). Statistics are informational. A reader can choose to -ignore statistics information. Statistics support is not required to read the table correctly. A table can contain -many statistics files associated with different table snapshots. - -Statistics files metadata within `statistics` table metadata field is a struct with the following fields: - -=== "v1 - v3" - | v1 | v2 and v3 | Field name | Type | Description | - | ---------- | ---------- |---------------------------------|-----------------------|-------------| - | _required_ | _required_ | **`snapshot-id`** | `long` | ID of the Iceberg table's snapshot the statistics file is associated with. | - | _required_ | _required_ | **`statistics-path`** | `string` | Path of the statistics file. See [Puffin file format](puffin-spec.md). | - | _required_ | _required_ | **`file-size-in-bytes`** | `long` | Size of the statistics file. | - | _required_ | _required_ | **`file-footer-size-in-bytes`** | `long` | Total size of the statistics file's footer (not the footer payload size). See [Puffin file format](puffin-spec.md) for footer definition. | - | _optional_ | _optional_ | **`key-metadata`** | | Base64-encoded implementation-specific key metadata for encryption. | - | _required_ | _required_ | **`blob-metadata`** | `list` (see below) | A list of the blob metadata for statistics contained in the file with structure described below. | - -Blob metadata is a struct with the following fields: - -=== "v1 - v3" - | v1 | v2 and v3 | Field name | Type | Description | - | ---------- | ---------- |-----------------------|-----------------------|-------------| - | _required_ | _required_ | **`type`** | `string` | Type of the blob. Matches Blob type in the Puffin file. | - | _required_ | _required_ | **`snapshot-id`** | `long` | ID of the Iceberg table's snapshot the blob was computed from. | - | _required_ | _required_ | **`sequence-number`** | `long` | Sequence number of the Iceberg table's snapshot the blob was computed from. | - | _required_ | _required_ | **`fields`** | `list` | Ordered list of fields, given by field ID, on which the statistic was calculated. | - | _optional_ | _optional_ | **`properties`** | `map` | Additional properties associated with the statistic. Subset of Blob properties in the Puffin file. | - -#### Partition Statistics - -Partition statistics files are based on [partition statistics file spec](#partition-statistics-file). -Partition statistics are not required for reading or planning and readers may ignore them. -Each table snapshot may be associated with at most one partition statistics file. -A writer can optionally write the partition statistics file during each write operation, or it can also be computed on demand. -Partition statistics file must be registered in the table metadata file to be considered as a valid statistics file for the reader. - -`partition-statistics` field of table metadata is an optional list of structs with the following fields: - -=== "v1 - v3" - | v1 | v2 | v3 | Field name | Type | Description | - | ---------- | ---------- |------------|--------------------------|----------|-------------| - | _required_ | _required_ | _required_ | **`snapshot-id`** | `long` | ID of the Iceberg table's snapshot the partition statistics file is associated with. | - | _required_ | _required_ | _required_ | **`statistics-path`** | `string` | Path of the partition statistics file. See [Partition statistics file](#partition-statistics-file). | - | _required_ | _required_ | _required_ | **`file-size-in-bytes`** | `long` | Size of the partition statistics file. | - -##### Partition Statistics File - -Statistics information for each unique partition tuple is stored as a row in any of the data file format of the table (for example, Parquet or ORC). -These rows must be sorted (in ascending manner with NULL FIRST) by `partition` field to optimize filtering rows while scanning. - -The schema of the partition statistics file is as follows: - -=== "v1 - v3" - | v1 | v2 | v3 | Field id, name | Type | Description | - | ---------- | ---------- |------------|------------------------------------------|--------------|-------------| - | _required_ | _required_ | _required_ | **`1 partition`** | `struct<..>` | Partition data tuple, schema based on the unified partition type considering all specs in a table | - | _required_ | _required_ | _required_ | **`2 spec_id`** | `int` | Partition spec id | - | _required_ | _required_ | _required_ | **`3 data_record_count`** | `long` | Count of records in data files | - | _required_ | _required_ | _required_ | **`4 data_file_count`** | `int` | Count of data files | - | _required_ | _required_ | _required_ | **`5 total_data_file_size_in_bytes`** | `long` | Total size of data files in bytes | - | _optional_ | _optional_ | _required_ | **`6 position_delete_record_count`** | `long` | Count of position deletes across position delete files and deletion vectors | - | _optional_ | _optional_ | _required_ | **`7 position_delete_file_count`** | `int` | Count of position delete files ignoring deletion vectors | - | | | _required_ | **`13 dv_count`** | `int` | Count of deletion vectors | - | _optional_ | _optional_ | _required_ | **`8 equality_delete_record_count`** | `long` | Count of records in equality delete files | - | _optional_ | _optional_ | _required_ | **`9 equality_delete_file_count`** | `int` | Count of equality delete files | - | _optional_ | _optional_ | _optional_ | **`10 total_record_count`** | `long` | Accurate count of records in a partition after applying deletes if any | - | _optional_ | _optional_ | _optional_ | **`11 last_updated_at`** | `long` | Timestamp in milliseconds from the unix epoch when the partition was last updated | - | _optional_ | _optional_ | _optional_ | **`12 last_updated_snapshot_id`** | `long` | ID of snapshot that last updated this partition | - -Note that partition data tuple's schema is based on the partition spec output using partition field ids for the struct field ids. -The unified partition type is a struct containing all fields that have ever been a part of any spec in the table -and sorted by the field ids in ascending order. -In other words, the struct fields represent a union of all known partition fields sorted in ascending order by the field ids. - -For example, - -1. `spec#0` has two fields `{field#1, field#2}` -and then the table has evolved into `spec#1` which has three fields `{field#1, field#2, field#3}`. -The unified partition type looks like `Struct`. - -2. `spec#0` has two fields `{field#1, field#2}` -and then the table has evolved into `spec#1` which has just one field `{field#2}`. -The unified partition type looks like `Struct`. - -When a v2 table is upgraded to v3 or later, the `position_delete_record_count` field must account for all position deletes, including those from remaining v2 position delete files and any deletion vectors added after the upgrade. - -Calculating `total_record_count` for a table with equality deletes or v2 position delete files requires reading data. In such cases, implementations may omit this field and must write `NULL`, indicating that the exact record count in a partition is unknown. -If a table has no deletes or only deletion vectors, implementations are encouraged to populate `total_record_count` using metadata in manifests. - -#### Encryption Keys - -Keys used for table encryption can be tracked in table metadata as a list named `encryption-keys`. The schema of each key is a struct with the following fields: -=== "v1 - v3" - | v1 | v2 | v3 | Field name | Type | Description | - |----|----|------------|-------------------------------|-----------------------|-------------| - | | | _required_ | **`key-id`** | `string` | ID of the encryption key | - | | | _required_ | **`encrypted-key-metadata`** | `string` | Encrypted key and metadata, base64 encoded [1] | - | | | _optional_ | **`encrypted-by-id`** | `string` | Optional ID of the key used to encrypt or wrap `key-metadata` | - | | | _optional_ | **`properties`** | `map` | A string to string map of additional metadata used by the table's encryption scheme | - -Notes: - -1. The format of encrypted key metadata is determined by the table's encryption scheme and can be a wrapped format specific to the table's KMS provider. - -### Commit Conflict Resolution and Retry - -When two commits happen at the same time and are based on the same version, only one commit will succeed. In most cases, the failed commit can be applied to the new current version of table metadata and retried. Updates verify the conditions under which they can be applied to a new version and retry if those conditions are met. - -* Append operations have no requirements and can always be applied. -* Replace operations must verify that the files that will be deleted are still in the table. Examples of replace operations include format changes (replace an Avro file with a Parquet file) and compactions (several files are replaced with a single file that contains the same rows). -* Delete operations must verify that specific files to delete are still in the table. Delete operations based on expressions can always be applied (e.g., where timestamp < X). -* Table schema updates and partition spec changes must validate that the schema has not changed between the base version and the current version. - -#### File System Tables - -_Note: This file system based scheme to commit a metadata file is **deprecated** and will be removed in version 4 of this spec. The scheme is **unsafe** in object stores and local file systems._ - -An atomic swap can be implemented using atomic rename in file systems that support it, like HDFS [1]. - -Each version of table metadata is stored in a metadata folder under the table’s base location using a file naming scheme that includes a version number, `V`: `v.metadata.json`. To commit a new metadata version, `V+1`, the writer performs the following steps: - -1. Read the current table metadata version `V`. -2. Create new table metadata based on version `V`. -3. Write the new table metadata to a unique file: `.metadata.json`. -4. Rename the unique file to the well-known file for version `V`: `v.metadata.json`. - 1. If the rename succeeds, the commit succeeded and `V+1` is the table’s current version - 2. If the rename fails, go back to step 1. - -Notes: - -1. The file system table scheme is implemented in [HadoopTableOperations](../javadoc/{{ icebergVersion }}/org/apache/iceberg/hadoop/HadoopTableOperations.html). - -#### Metastore Tables - -The atomic swap needed to commit new versions of table metadata can be implemented by storing a pointer in a metastore or database that is updated with a check-and-put operation [1]. The check-and-put validates that the version of the table that a write is based on is still current and then makes the new metadata from the write the current version. - -Each version of table metadata is stored in a metadata folder under the table’s base location using a naming scheme that includes a version and UUID: `-.metadata.json`. To commit a new metadata version, `V+1`, the writer performs the following steps: - -1. Create a new table metadata file based on the current metadata. -2. Write the new table metadata to a unique file: `-.metadata.json`. -3. Request that the metastore swap the table’s metadata pointer from the location of `V` to the location of `V+1`. - 1. If the swap succeeds, the commit succeeded. `V` was still the latest metadata version and the metadata file for `V+1` is now the current metadata. - 2. If the swap fails, another writer has already created `V+1`. The current writer goes back to step 1. - -Notes: - -1. The metastore table scheme is partly implemented in [BaseMetastoreTableOperations](../javadoc/{{ icebergVersion }}/org/apache/iceberg/BaseMetastoreTableOperations.html). - -### Delete Formats - -This section details how to encode row-level deletes in Iceberg delete files. Row-level deletes are added by v2 and are not supported in v1. Deletion vectors are added in v3 and are not supported in v2 or earlier. Position delete files must not be added to v3 tables, but existing position delete files are valid. - -There are different formats for encoding row-level deletes: - -* Deletion vectors (DVs) identify deleted rows within a single referenced data file by position in a bitmap -* Position delete files identify deleted rows by file location and row position (**deprecated** in v3) -* Equality delete files identify deleted rows by the value of one or more columns - -Deletion vectors are a binary representation of deletes for a single data file that is more efficient at execution time than position delete files. Unlike equality or position delete files, there can be at most one deletion vector for a given data file in a snapshot. Writers must ensure that there is at most one deletion vector per data file and must merge new deletes with existing vectors or position delete files. -When removing a data file, writers must also remove any deletion vector that applies to that data file from delete manifests. Writers are not required to rewrite Puffin files that contain the removed deletion vectors. - -Row-level delete files (both equality and position delete files) are valid Iceberg data files: files must use valid Iceberg formats, schemas, and column projection. It is recommended that these delete files are written using the table's default file format. - -Row-level delete files and deletion vectors are tracked by manifests. A separate set of manifests is used for delete files and DVs, but the same manifest schema is used for both data and delete manifests. Deletion vectors are tracked individually by file location, offset, and length within the containing file. Deletion vector metadata must include the referenced data file. - -Both position and equality delete files allow encoding deleted row values with a delete. This can be used to reconstruct a stream of changes to a table. - -#### Deletion Vectors - -Deletion vectors identify deleted rows of a file by encoding deleted positions in a bitmap. A set bit at position P indicates that the row at position P is deleted. - -These vectors are stored using the `deletion-vector-v1` blob definition from the [Puffin spec][puffin-spec]. - -Deletion vectors support positive 64-bit positions, but are optimized for cases where most positions fit in 32 bits by using a collection of 32-bit Roaring bitmaps. 64-bit positions are divided into a 32-bit "key" using the most significant 4 bytes and a 32-bit sub-position using the least significant 4 bytes. For each key in the set of positions, a 32-bit Roaring bitmap is maintained to store a set of 32-bit sub-positions for that key. - -To test whether a certain position is set, its most significant 4 bytes (the key) are used to find a 32-bit bitmap and the least significant 4 bytes (the sub-position) are tested for inclusion in the bitmap. If a bitmap is not found for the key, then it is not set. - -Delete manifests track deletion vectors individually by the containing file location (`file_path`), starting offset of the DV blob (`content_offset`), and total length of the blob (`content_size_in_bytes`). Multiple deletion vectors can be stored in the same file. There are no restrictions on the data files that can be referenced by deletion vectors in the same Puffin file. - -At most one deletion vector is allowed per data file in a snapshot. If a DV is written for a data file, it must replace all previously written position delete files so that when a DV is present, readers can safely ignore matching position delete files. - -[puffin-spec]: https://iceberg.apache.org/puffin-spec/ - -#### Position Delete Files - -Position-based delete files identify deleted rows by file and position in one or more data files, and may optionally contain the deleted row. - -_Note: Position delete files are **deprecated** in v3. Existing position deletes must be written to delete vectors when updating the position deletes for a data file._ - -A data row is deleted if there is an entry in a position delete file for the row's file and position in the data file, starting at 0. - -Position-based delete files store `file_position_delete`, a struct with the following fields: - -| Field id, name | Type | Description | -|-----------------------------|----------------------------|-------------| -| **`2147483546 file_path`** | `string` | Full URI of a data file with FS scheme. This must match the `file_path` of the target data file in a manifest entry | -| **`2147483545 pos`** | `long` | Ordinal position of a deleted row in the target data file identified by `file_path`, starting at `0` | -| **`2147483544 row`** | `required struct<...>` [1] | Deleted row values. Omit the column when not storing deleted rows. | - -1. When present in the delete file, `row` is required because all delete entries must include the row values. - -When the deleted row column is present, its schema may be any subset of the table schema and must use field ids matching the table. - -To ensure the accuracy of statistics, all delete entries must include row values, or the column must be omitted (this is why the column type is `required`). - -The rows in the delete file must be sorted by `file_path` then `pos` to optimize filtering rows while scanning. - -* Sorting by `file_path` allows filter pushdown by file in columnar storage formats. -* Sorting by `pos` allows filtering rows while scanning, to avoid keeping deletes in memory. - -#### Equality Delete Files - -Equality delete files identify deleted rows in a collection of data files by one or more column values, and may optionally contain additional columns of the deleted row. - -Equality delete files store any subset of a table's columns and use the table's field ids. The _delete columns_ are the columns of the delete file used to match data rows. Delete columns are identified by id in the delete file [metadata column `equality_ids`](#manifests). The column restrictions for columns used in equality delete files are the same as those for [identifier fields](#identifier-field-ids) with the exception that optional columns and columns nested under optional structs are allowed (if a parent struct column is null it implies the leaf column is null). - -A data row is deleted if its values are equal to all delete columns for any row in an equality delete file that applies to the row's data file (see [`Scan Planning`](#scan-planning)). - -Each row of the delete file produces one equality predicate that matches any row where the delete columns are equal. Multiple columns can be thought of as an `AND` of equality predicates. A `null` value in a delete column matches a row if the row's value is `null`, equivalent to `col IS NULL`. - -For example, a table with the following data: - -```text - 1: id | 2: category | 3: name --------|-------------|--------- - 1 | marsupial | Koala - 2 | toy | Teddy - 3 | NULL | Grizzly - 4 | NULL | Polar -``` - -The delete `id = 3` could be written as either of the following equality delete files: - -```text -equality_ids=[1] - - 1: id -------- - 3 -``` - -```text -equality_ids=[1] - - 1: id | 2: category | 3: name --------|-------------|--------- - 3 | NULL | Grizzly -``` - -The delete `id = 4 AND category IS NULL` could be written as the following equality delete file: - -```text -equality_ids=[1, 2] - - 1: id | 2: category | 3: name --------|-------------|--------- - 4 | NULL | Polar -``` - -If a delete column in an equality delete file is later dropped from the table, it must still be used when applying the equality deletes. If a column was added to a table and later used as a delete column in an equality delete file, the column value is read for older data files using normal projection rules (defaults to `null`). - -#### Delete File Stats - -Manifests hold the same statistics for delete files and data files. For delete files, the metrics describe the values that were deleted. - -## Appendix A: Format-specific Requirements - -### Avro - -**Data Type Mappings** - -Values should be stored in Avro using the Avro types and logical type annotations in the table below. - -Optional fields, array elements, and map values must be wrapped in an Avro `union` with `null`. This is the only union type allowed in Iceberg data files. - -Optional fields without an Iceberg default must set the Avro field default value to null. Fields with a non-null Iceberg default must convert the default to an equivalent Avro default. - -Maps with non-string keys must use an array representation with the `map` logical type. The array representation or Avro’s map type may be used for maps with string keys. - -|Type|Avro type|Notes| -|--- |--- |--- | -|**`unknown`**|`null` or omitted|| -|**`boolean`**|`boolean`|| -|**`int`**|`int`|| -|**`long`**|`long`|| -|**`float`**|`float`|| -|**`double`**|`double`|| -|**`decimal(P,S)`**|`{ "type": "fixed",`
  `"size": minBytesRequired(P),`
  `"logicalType": "decimal",`
  `"precision": P,`
  `"scale": S }`|Stored as fixed using the minimum number of bytes for the given precision.| -|**`date`**|`{ "type": "int",`
  `"logicalType": "date" }`|Stores days from 1970-01-01.| -|**`time`**|`{ "type": "long",`
  `"logicalType": "time-micros" }`|Stores microseconds from midnight.| -|**`timestamp`** | `{ "type": "long",`
  `"logicalType": "timestamp-micros",`
  `"adjust-to-utc": false }` | Stores microseconds from 1970-01-01 00:00:00.000000. [1] | -|**`timestamptz`** | `{ "type": "long",`
  `"logicalType": "timestamp-micros",`
  `"adjust-to-utc": true }` | Stores microseconds from 1970-01-01 00:00:00.000000 UTC. [1] | -|**`timestamp_ns`** | `{ "type": "long",`
  `"logicalType": "timestamp-nanos",`
  `"adjust-to-utc": false }` | Stores nanoseconds from 1970-01-01 00:00:00.000000000. [1], [2] | -|**`timestamptz_ns`** | `{ "type": "long",`
  `"logicalType": "timestamp-nanos",`
  `"adjust-to-utc": true }` | Stores nanoseconds from 1970-01-01 00:00:00.000000000 UTC. [1], [2] | -|**`string`**|`string`|| -|**`uuid`**|`{ "type": "fixed",`
  `"size": 16,`
  `"logicalType": "uuid" }`|| -|**`fixed(L)`**|`{ "type": "fixed",`
  `"size": L }`|| -|**`binary`**|`bytes`|| -|**`struct`**|`record`|| -|**`list`**|`array`|| -|**`map`**|`array` of key-value records, or `map` when keys are strings (optional).|Array storage must use logical type name `map` and must store elements that are 2-field records. The first field is a non-null key and the second field is the value.| -|**`variant`**|`record` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs and the fields are accessed through names. |Shredding is not supported in Avro.| -|**`geometry`**|`bytes`|WKB format, see [Appendix G](#appendix-g-geospatial-notes)| -|**`geography`**|`bytes`|WKB format, see [Appendix G](#appendix-g-geospatial-notes)| - -Notes: - -1. Avro type annotation `adjust-to-utc` is an Iceberg convention; default value is `false` if not present. -2. Avro logical type `timestamp-nanos` is an Iceberg convention; the Avro specification does not define this type. - -**Field IDs** - -Iceberg struct, list, and map types identify nested types by ID. When writing data to Avro files, these IDs must be stored in the Avro schema to support ID-based column pruning. - -IDs are stored as JSON integers in the following locations: - -|ID|Avro schema location|Property|Example| -|--- |--- |--- |--- | -|**Struct field**|Record field object|`field-id`|`{ "type": "record", ...`
  `"fields": [`
    `{ "name": "l",`
      `"type": ["null", "long"],`
      `"default": null,`
      `"field-id": 8 }`
  `] }`| -|**List element**|Array schema object|`element-id`|`{ "type": "array",`
  `"items": "int",`
  `"element-id": 9 }`| -|**String map key**|Map schema object|`key-id`|`{ "type": "map",`
  `"values": "int",`
  `"key-id": 10,`
  `"value-id": 11 }`| -|**String map value**|Map schema object|`value-id`|| -|**Map key, value**|Key, value fields in the element record.|`field-id`|`{ "type": "array",`
  `"logicalType": "map",`
  `"items": {`
    `"type": "record",`
    `"name": "k12_v13",`
    `"fields": [`
      `{ "name": "key",`
        `"type": "int",`
        `"field-id": 12 },`
      `{ "name": "value",`
        `"type": "string",`
        `"field-id": 13 }`
    `] } }`| - -Note that the string map case is for maps where the key type is a string. Using Avro’s map type in this case is optional. Maps with string keys may be stored as arrays. - -### Parquet - -**Data Type Mappings** - -Values should be stored in Parquet using the types and logical type annotations in the table below. Column IDs are required to be stored as [field IDs](http://github.com/apache/parquet-format/blob/40699d05bd24181de6b1457babbee2c16dce3803/src/main/thrift/parquet.thrift#L459) on the parquet schema. - -Lists must use the [3-level representation](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists). - -| Type | Parquet physical type | Logical type | Notes | -|--------------------|----------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------|----------------------------------------------------------------| -| **`unknown`** | None | | Omit from data files | -| **`boolean`** | `boolean` | | | -| **`int`** | `int32` | | | -| **`long`** | `int64` | | | -| **`float`** | `float` | | | -| **`double`** | `double` | | | -| **`decimal(P,S)`** | `P <= 9`: `int32`,
`P <= 18`: `int64`,
`fixed` otherwise | `DECIMAL(P,S)` | Fixed must use the minimum number of bytes that can store `P`. | -| **`date`** | `int32` | `DATE` | Stores days from 1970-01-01. | -| **`time`** | `int64` | `TIME_MICROS` with `adjustToUtc=false` | Stores microseconds from midnight. | -| **`timestamp`** | `int64` | `TIMESTAMP_MICROS` with `adjustToUtc=false` | Stores microseconds from 1970-01-01 00:00:00.000000. | -| **`timestamptz`** | `int64` | `TIMESTAMP_MICROS` with `adjustToUtc=true` | Stores microseconds from 1970-01-01 00:00:00.000000 UTC. | -| **`timestamp_ns`** | `int64` | `TIMESTAMP_NANOS` with `adjustToUtc=false` | Stores nanoseconds from 1970-01-01 00:00:00.000000000. | -| **`timestamptz_ns`** | `int64` | `TIMESTAMP_NANOS` with `adjustToUtc=true` | Stores nanoseconds from 1970-01-01 00:00:00.000000000 UTC. | -| **`string`** | `binary` | `UTF8` | Encoding must be UTF-8. | -| **`uuid`** | `fixed_len_byte_array[16]` | `UUID` | | -| **`fixed(L)`** | `fixed_len_byte_array[L]` | | | -| **`binary`** | `binary` | | | -| **`struct`** | `group` | | | -| **`list`** | `3-level list` | `LIST` | See Parquet docs for 3-level representation. | -| **`map`** | `3-level map` | `MAP` | See Parquet docs for 3-level representation. | -| **`variant`** | `group` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs and the fields are accessed through names. | `VARIANT` | See Parquet docs for [Variant encoding](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) and [Variant shredding encoding](https://github.com/apache/parquet-format/blob/master/VariantShredding.md). | -| **`geometry`** | `binary` | `GEOMETRY` | WKB format, see [Appendix G](#appendix-g-geospatial-notes). | -| **`geography`** | `binary` | `GEOGRAPHY` | WKB format, see [Appendix G](#appendix-g-geospatial-notes). | - -When reading an `unknown` column, any corresponding column must be ignored and replaced with `null` values. - -### ORC - -**Data Type Mappings** - -| Type | ORC type | ORC type attributes | Notes | -|--------------------|---------------------|------------------------------------------------------|-----------------------------------------------------------------------------------------| -| **`unknown`** | None | | Omit from data files | -| **`boolean`** | `boolean` | | | -| **`int`** | `int` | | ORC `tinyint` and `smallint` would also map to **`int`**. | -| **`long`** | `long` | | | -| **`float`** | `float` | | | -| **`double`** | `double` | | | -| **`decimal(P,S)`** | `decimal` | | | -| **`date`** | `date` | | | -| **`time`** | `long` | `iceberg.long-type`=`TIME` | Stores microseconds from midnight. | -| **`timestamp`** | `timestamp` | `iceberg.timestamp-unit`=`MICROS` | Stores microseconds from 2015-01-01 00:00:00.000000. [1], [2] | -| **`timestamptz`** | `timestamp_instant` | `iceberg.timestamp-unit`=`MICROS` | Stores microseconds from 2015-01-01 00:00:00.000000 UTC. [1], [2] | -| **`timestamp_ns`** | `timestamp` | `iceberg.timestamp-unit`=`NANOS` | Stores nanoseconds from 2015-01-01 00:00:00.000000000. [1] | -| **`timestamptz_ns`** | `timestamp_instant` | `iceberg.timestamp-unit`=`NANOS` | Stores nanoseconds from 2015-01-01 00:00:00.000000000 UTC. [1] | -| **`string`** | `string` | | ORC `varchar` and `char` would also map to **`string`**. | -| **`uuid`** | `binary` | `iceberg.binary-type`=`UUID` | | -| **`fixed(L)`** | `binary` | `iceberg.binary-type`=`FIXED` & `iceberg.length`=`L` | The length would not be checked by the ORC reader and should be checked by the adapter. | -| **`binary`** | `binary` | | | -| **`struct`** | `struct` | | | -| **`list`** | `array` | | | -| **`map`** | `map` | | | -| **`variant`** | `struct` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs. | `iceberg.struct-type`=`VARIANT` | Shredding is not supported in ORC. | -| **`geometry`** | `binary` | `iceberg.binary-type`=`GEOMETRY` | WKB format, see [Appendix G](#appendix-g-geospatial-notes). | -| **`geography`** | `binary` | `iceberg.binary-type`=`GEOGRAPHY` | WKB format, see [Appendix G](#appendix-g-geospatial-notes). | - -Notes: - -1. ORC's [TimestampColumnVector](https://orc.apache.org/api/hive-storage-api/org/apache/hadoop/hive/ql/exec/vector/TimestampColumnVector.html) consists of a time field (milliseconds since epoch) and a nanos field (nanoseconds within the second). Hence the milliseconds within the second are reported twice; once in the time field and again in the nanos field. The read adapter should only use milliseconds within the second from one of these fields. The write adapter should also report milliseconds within the second twice; once in the time field and again in the nanos field. ORC writer is expected to correctly consider millis information from one of the fields. More details at https://issues.apache.org/jira/browse/ORC-546 -2. ORC `timestamp` and `timestamp_instant` values store nanosecond precision. Iceberg ORC writers for Iceberg types `timestamp` and `timestamptz` **must** truncate nanoseconds to microseconds. `iceberg.timestamp-unit` is assumed to be `MICROS` if not present. - -One of the interesting challenges with this is how to map Iceberg’s schema evolution (id based) on to ORC’s (name based). In theory, we could use Iceberg’s column ids as the column and field names, but that would be inconvenient. - -The column IDs must be stored in ORC type attributes using the key `iceberg.id`, and `iceberg.required` to store `"true"` if the Iceberg column is required, otherwise it will be optional. - -Iceberg would build the desired reader schema with their schema evolution rules and pass that down to the ORC reader, which would then use its schema evolution to map that to the writer’s schema. Basically, Iceberg would need to change the names of columns and fields to get the desired mapping. - -|Iceberg writer|ORC writer|Iceberg reader|ORC reader| -|--- |--- |--- |--- | -|`struct`|`struct`|`struct`|`struct`| -|`struct>`|`struct>`|`struct>`|`struct>`| - -## Appendix B: 32-bit Hash Requirements - -The 32-bit hash implementation is 32-bit Murmur3 hash, x86 variant, seeded with 0. - -| Primitive type | Hash specification | Test value | -|--------------------|-------------------------------------------|--------------------------------------------| -| **`int`** | `hashLong(long(v))` [1] | `34` ￫ `2017239379` | -| **`long`** | `hashBytes(littleEndianBytes(v))` | `34L` ￫ `2017239379` | -| **`decimal(P,S)`** | `hashBytes(minBigEndian(unscaled(v)))`[2] | `14.20` ￫ `-500754589` | -| **`date`** | `hashInt(daysFromUnixEpoch(v))` | `2017-11-16` ￫ `-653330422` | -| **`time`** | `hashLong(microsecsFromMidnight(v))` | `22:31:08` ￫ `-662762989` | -| **`timestamp`** | `hashLong(microsecsFromUnixEpoch(v))` | `2017-11-16T22:31:08` ￫ `-2047944441`
`2017-11-16T22:31:08.000001` ￫ `-1207196810` | -| **`timestamptz`** | `hashLong(microsecsFromUnixEpoch(v))` | `2017-11-16T14:31:08-08:00` ￫ `-2047944441`
`2017-11-16T14:31:08.000001-08:00` ￫ `-1207196810` | -| **`timestamp_ns`** | `hashLong(microsecsFromUnixEpoch(v))` [3] | `2017-11-16T22:31:08` ￫ `-2047944441`
`2017-11-16T22:31:08.000001001` ￫ `-1207196810` | -| **`timestamptz_ns`** | `hashLong(microsecsFromUnixEpoch(v))` [3]| `2017-11-16T14:31:08-08:00` ￫ `-2047944441`
`2017-11-16T14:31:08.000001001-08:00` ￫ `-1207196810` | -| **`string`** | `hashBytes(utf8Bytes(v))` | `iceberg` ￫ `1210000089` | -| **`uuid`** | `hashBytes(uuidBytes(v))` [4] | `f79c3e09-677c-4bbd-a479-3f349cb785e7` ￫ `1488055340` | -| **`fixed(L)`** | `hashBytes(v)` | `00 01 02 03` ￫ `-188683207` | -| **`binary`** | `hashBytes(v)` | `00 01 02 03` ￫ `-188683207` | - -The types below are not currently valid for bucketing, and so are not hashed. However, if that changes and a hash value is needed, the following table shall apply: - -| Primitive type | Hash specification | Test value | -|--------------------|-------------------------------------------|--------------------------------------------| -| **`unknown`** | always `null` | | -| **`boolean`** | `false: hashInt(0)`, `true: hashInt(1)` | `true` ￫ `1392991556` | -| **`float`** | `hashLong(doubleToLongBits(double(v))` [5]| `1.0F` ￫ `-142385009`, `0.0F` ￫ `1669671676`, `-0.0F` ￫ `1669671676` | -| **`double`** | `hashLong(doubleToLongBits(v))` [5]| `1.0D` ￫ `-142385009`, `0.0D` ￫ `1669671676`, `-0.0D` ￫ `1669671676` | - -A 32-bit hash is not defined for `variant` because there are multiple representations for equivalent values. - -Notes: - -1. Integer and long hash results must be identical for all integer values. This ensures that schema evolution does not change bucket partition values if integer types are promoted. -2. Decimal values are hashed using the minimum number of bytes required to hold the unscaled value as a two’s complement big-endian; this representation does not include padding bytes required for storage in a fixed-length array. -Hash results are not dependent on decimal scale, which is part of the type, not the data value. -3. Nanosecond timestamps must be converted to microsecond precision before hashing to ensure timestamps have the same hash value. -4. UUIDs are encoded using big endian. The test UUID for the example above is: `f79c3e09-677c-4bbd-a479-3f349cb785e7`. This UUID encoded as a byte array is: -`F7 9C 3E 09 67 7C 4B BD A4 79 3F 34 9C B7 85 E7` -5. `doubleToLongBits` must give the IEEE 754 compliant bit representation of the double value. All `NaN` bit patterns must be canonicalized to `0x7ff8000000000000L`. Negative zero (`-0.0`) must be canonicalized to positive zero (`0.0`). Float hash values are the result of hashing the float cast to double to ensure that schema evolution does not change hash values if float types are promoted. - -## Appendix C: JSON serialization - -### Schemas - -Schemas are serialized as a JSON object with the same fields as a struct in the table below, and the following additional fields: - -| v1 | v2 |Field|JSON representation|Example| -| ---------- | ---------- |--- |--- |--- | -| _optional_ | _required_ |**`schema-id`**|`JSON int`|`0`| -| _optional_ | _optional_ |**`identifier-field-ids`**|`JSON list of ints`|`[1, 2]`| - -Types are serialized according to this table: - -|Type|JSON representation|Example| -|--- |--- |--- | -|**`unknown`**|`JSON string: "unknown"`|`"unknown"`| -|**`boolean`**|`JSON string: "boolean"`|`"boolean"`| -|**`int`**|`JSON string: "int"`|`"int"`| -|**`long`**|`JSON string: "long"`|`"long"`| -|**`float`**|`JSON string: "float"`|`"float"`| -|**`double`**|`JSON string: "double"`|`"double"`| -|**`date`**|`JSON string: "date"`|`"date"`| -|**`time`**|`JSON string: "time"`|`"time"`| -|**`timestamp, microseconds, without zone`**|`JSON string: "timestamp"`|`"timestamp"`| -|**`timestamp, microseconds, with zone`**|`JSON string: "timestamptz"`|`"timestamptz"`| -|**`timestamp, nanoseconds, without zone`**|`JSON string: "timestamp_ns"`|`"timestamp_ns"`| -|**`timestamp, nanoseconds, with zone`**|`JSON string: "timestamptz_ns"`|`"timestamptz_ns"`| -|**`string`**|`JSON string: "string"`|`"string"`| -|**`uuid`**|`JSON string: "uuid"`|`"uuid"`| -|**`fixed(L)`**|`JSON string: "fixed[]"`|`"fixed[16]"`| -|**`binary`**|`JSON string: "binary"`|`"binary"`| -|**`decimal(P, S)`**|`JSON string: "decimal(

,)"`|`"decimal(9,2)"`,
`"decimal(9, 2)"`| -|**`struct`**|`JSON object: {`
  `"type": "struct",`
  `"fields": [ {`
    `"id": ,`
    `"name": ,`
    `"required": ,`
    `"type": ,`
    `"doc": ,`
    `"initial-default": ,`
    `"write-default": `
    `}, ...`
  `] }`|`{`
  `"type": "struct",`
  `"fields": [ {`
    `"id": 1,`
    `"name": "id",`
    `"required": true,`
    `"type": "uuid",`
    `"initial-default": "0db3e2a8-9d1d-42b9-aa7b-74ebe558dceb",`
    `"write-default": "ec5911be-b0a7-458c-8438-c9a3e53cffae"`
  `}, {`
    `"id": 2,`
    `"name": "data",`
    `"required": false,`
    `"type": {`
      `"type": "list",`
      `...`
    `}`
  `} ]`
`}`| -|**`list`**|`JSON object: {`
  `"type": "list",`
  `"element-id": ,`
  `"element-required": `
  `"element": `
`}`|`{`
  `"type": "list",`
  `"element-id": 3,`
  `"element-required": true,`
  `"element": "string"`
`}`| -|**`map`**|`JSON object: {`
  `"type": "map",`
  `"key-id": ,`
  `"key": ,`
  `"value-id": ,`
  `"value-required": `
  `"value": `
`}`|`{`
  `"type": "map",`
  `"key-id": 4,`
  `"key": "string",`
  `"value-id": 5,`
  `"value-required": false,`
  `"value": "double"`
`}`| -| **`variant`**| `JSON string: "variant"`|`"variant"`| -| **`geometry(C)`** |`JSON string: "geometry()"`|`"geometry(srid:4326)"`| -| **`geography(C, A)`** |`JSON string: "geography(,)"`|`"geography(srid:4326,spherical)"`| - -Note that default values are serialized using the JSON single-value serialization in [Appendix D](#appendix-d-single-value-serialization). - -### Partition Specs - -Partition specs are serialized as a JSON object with the following fields: - -|Field|JSON representation|Example| -|--- |--- |--- | -|**`spec-id`**|`JSON int`|`0`| -|**`fields`**|`JSON list: [`
  `,`
  `...`
`]`|`[ {`
  `"source-id": 4,`
  `"field-id": 1000,`
  `"name": "ts_day",`
  `"transform": "day"`
`}, {`
  `"source-id": 1,`
  `"field-id": 1001,`
  `"name": "id_bucket",`
  `"transform": "bucket[16]"`
`} ]`| - -Each partition field in `fields` is stored as a JSON object with the following properties. - -| V1 | V2 | V3 | Field | JSON representation | Example | -|----------|----------|----------|------------------|---------------------|--------------| -| required | required | optional | **`source-id`** | `JSON int` | 1 | -| | | optional | **`source-ids`** | `JSON list of ints` | `[1,2]` | -| | required | required | **`field-id`** | `JSON int` | 1000 | -| required | required | required | **`name`** | `JSON string` | `id_bucket` | -| required | required | required | **`transform`** | `JSON string` | `bucket[16]` | - -Notes: - -1. For partition fields with a transform with a single argument, only `source-id` is written. In case of a multi-argument transform, only `source-ids` is written. - -Supported partition transforms are listed below. - -|Transform or Field|JSON representation|Example| -|--- |--- |--- | -|**`identity`**|`JSON string: "identity"`|`"identity"`| -|**`bucket[N]`**|`JSON string: "bucket[]"`|`"bucket[16]"`| -|**`truncate[W]`**|`JSON string: "truncate[]"`|`"truncate[20]"`| -|**`year`**|`JSON string: "year"`|`"year"`| -|**`month`**|`JSON string: "month"`|`"month"`| -|**`day`**|`JSON string: "day"`|`"day"`| -|**`hour`**|`JSON string: "hour"`|`"hour"`| - -In some cases partition specs are stored using only the field list instead of the object format that includes the spec ID, like the deprecated `partition-spec` field in table metadata. The object format should be used unless otherwise noted in this spec. - -The `field-id` property was added for each partition field in v2. In v1, the reference implementation assigned field ids sequentially in each spec starting at 1,000. See Partition Evolution for more details. - -Older versions of the reference implementation can read tables with transforms unknown to it, ignoring them. But other implementations may break if they encounter unknown transforms. All v3 readers are required to read tables with unknown transforms, ignoring them. Writers should not write using partition specs that use unknown transforms. - -### Sort Orders - -Sort orders are serialized as a list of JSON objects, each of which contains the following fields: - -|Field|JSON representation|Example| -|--- |--- |--- | -|**`order-id`**|`JSON int`|`1`| -|**`fields`**|`JSON list: [`
  `,`
  `...`
`]`|`[ {`
  `"transform": "identity",`
  `"source-id": 2,`
  `"direction": "asc",`
  `"null-order": "nulls-first"`
  `}, {`
  `"transform": "bucket[4]",`
  `"source-id": 3,`
  `"direction": "desc",`
  `"null-order": "nulls-last"`
`} ]`| - -Each sort field in the fields list is stored as an object with the following properties: - -| V1 | V2 | V3 | Field | JSON representation | Example | -|----------|----------|----------|------------------|---------------------|-------------| -| required | required | required | **`transform`** | `JSON string` | `bucket[4]` | -| required | required | optional | **`source-id`** | `JSON int` | 1 | -| | | optional | **`source-ids`** | `JSON list of ints` | `[1,2]` | -| required | required | required | **`direction`** | `JSON string` | `asc` | -| required | required | required | **`null-order`** | `JSON string` | `nulls-last`| - -Notes: - -1. For sort fields with a transform with a single argument, only `source-id` is written. In case of a multi-argument transform, only `source-ids` is written. - -Older versions of the reference implementation can read tables with transforms unknown to it, ignoring them. But other implementations may break if they encounter unknown transforms. All v3 readers are required to read tables with unknown transforms, ignoring them. - -The following table describes the possible values for the some of the field within sort field: - -|Field|JSON representation|Possible values| -|--- |--- |--- | -|**`direction`**|`JSON string`|`"asc", "desc"`| -|**`null-order`**|`JSON string`|`"nulls-first", "nulls-last"`| - -### Table Metadata and Snapshots - -Table metadata is serialized as a JSON object according to the following table. Snapshots are not serialized separately. Instead, they are stored in the table metadata JSON. - -A metadata JSON file may be compressed with [GZIP](https://datatracker.ietf.org/doc/html/rfc1952). - -|Metadata field|JSON representation|Example| -|--- |--- |--- | -|**`format-version`**|`JSON int`|`1`| -|**`table-uuid`**|`JSON string`|`"fb072c92-a02b-11e9-ae9c-1bb7bc9eca94"`| -|**`location`**|`JSON string`|`"s3://b/wh/data.db/table"`| -|**`last-updated-ms`**|`JSON long`|`1515100955770`| -|**`last-column-id`**|`JSON int`|`22`| -|**`schema`**|`JSON schema (object)`|`See above, read schemas instead`| -|**`schemas`**|`JSON schemas (list of objects)`|`See above`| -|**`current-schema-id`**|`JSON int`|`0`| -|**`partition-spec`**|`JSON partition fields (list)`|`See above, read partition-specs instead`| -|**`partition-specs`**|`JSON partition specs (list of objects)`|`See above`| -|**`default-spec-id`**|`JSON int`|`0`| -|**`last-partition-id`**|`JSON int`|`1000`| -|**`properties`**|`JSON object: {`
  `"": "",`
  `...`
`}`|`{`
  `"write.format.default": "avro",`
  `"commit.retry.num-retries": "4"`
`}`| -|**`current-snapshot-id`**|`JSON long`|`3051729675574597004`| -|**`snapshots`**|`JSON list of objects: [ {`
  `"snapshot-id": ,`
  `"timestamp-ms": ,`
  `"summary": {`
    `"operation": ,`
    `... },`
  `"manifest-list": "",`
  `"schema-id": ""`
  `},`
  `...`
`]`|`[ {`
  `"snapshot-id": 3051729675574597004,`
  `"timestamp-ms": 1515100955770,`
  `"summary": {`
    `"operation": "append"`
  `},`
  `"manifest-list": "s3://b/wh/.../s1.avro"`
  `"schema-id": 0`
`} ]`| -|**`snapshot-log`**|`JSON list of objects: [`
  `{`
  `"snapshot-id": ,`
  `"timestamp-ms":`
  `},`
  `...`
`]`|`[ {`
  `"snapshot-id": 30517296...,`
  `"timestamp-ms": 1515100...`
`} ]`| -|**`metadata-log`**|`JSON list of objects: [`
  `{`
  `"metadata-file": ,`
  `"timestamp-ms":`
  `},`
  `...`
`]`|`[ {`
  `"metadata-file": "s3://bucket/.../v1.json",`
  `"timestamp-ms": 1515100...`
`} ]` | -|**`sort-orders`**|`JSON sort orders (list of sort field object)`|`See above`| -|**`default-sort-order-id`**|`JSON int`|`0`| -|**`refs`**|`JSON map with string key and object value:`
`{`
  `"": {`
  `"snapshot-id": ,`
  `"type": ,`
  `"max-ref-age-ms": ,`
  `...`
  `}`
  `...`
`}`|`{`
  `"test": {`
  `"snapshot-id": 123456789000,`
  `"type": "tag",`
  `"max-ref-age-ms": 10000000`
  `}`
`}`| -|**`encryption-keys`**|`JSON list of encryption key objects`|`[ {"key-id": "5f819b", "key-metadata": "aWNlYmVyZwo="} ]`| - -### Name Mapping Serialization - -Name mapping is serialized as a list of field mapping JSON Objects which are serialized as follows - -|Field mapping field|JSON representation|Example| -|--- |--- |--- | -|**`names`**|`JSON list of strings`|`["latitude", "lat"]`| -|**`field-id`**|`JSON int`|`1`| -|**`fields`**|`JSON field mappings (list of objects)`|`[{`
  `"field-id": 4,`
  `"names": ["latitude", "lat"]`
`}, {`
  `"field-id": 5,`
  `"names": ["longitude", "long"]`
`}]`| - -Example -```json -[ { "field-id": 1, "names": ["id", "record_id"] }, - { "field-id": 2, "names": ["data"] }, - { "field-id": 3, "names": ["location"], "fields": [ - { "field-id": 4, "names": ["latitude", "lat"] }, - { "field-id": 5, "names": ["longitude", "long"] } - ] } ] -``` - -## Appendix D: Single-value serialization - -### Binary single-value serialization - -This serialization scheme is for storing single values as individual binary values. - -| Type | Binary serialization | -|------------------------------|--------------------------------------------------------------------------------------------------------------| -| **`unknown`** | Not supported | -| **`boolean`** | `0x00` for false, non-zero byte for true | -| **`int`** | Stored as 4-byte little-endian | -| **`long`** | Stored as 8-byte little-endian | -| **`float`** | Stored as 4-byte little-endian | -| **`double`** | Stored as 8-byte little-endian | -| **`date`** | Stores days from the 1970-01-01 in an 4-byte little-endian int | -| **`time`** | Stores microseconds from midnight in an 8-byte little-endian long | -| **`timestamp`** | Stores microseconds from 1970-01-01 00:00:00.000000 in an 8-byte little-endian long | -| **`timestamptz`** | Stores microseconds from 1970-01-01 00:00:00.000000 UTC in an 8-byte little-endian long | -| **`timestamp_ns`** | Stores nanoseconds from 1970-01-01 00:00:00.000000000 in an 8-byte little-endian long | -| **`timestamptz_ns`** | Stores nanoseconds from 1970-01-01 00:00:00.000000000 UTC in an 8-byte little-endian long | -| **`string`** | UTF-8 bytes (without length) | -| **`uuid`** | 16-byte big-endian value, see example in Appendix B | -| **`fixed(L)`** | Binary value | -| **`binary`** | Binary value (without length) | -| **`decimal(P, S)`** | Stores unscaled value as two’s-complement big-endian binary, using the minimum number of bytes for the value | -| **`struct`** | Not supported | -| **`list`** | Not supported | -| **`map`** | Not supported | -| **`variant`** | Not supported | -| **`geometry`** | WKB format, see [Appendix G](#appendix-g-geospatial-notes) | -| **`geography`** | WKB format, see [Appendix G](#appendix-g-geospatial-notes) | - -### Bound serialization - -The binary single-value serialization can be used to store the lower and upper bounds maps of manifest files, except as specified by the following table. - -| Type | Binary serialization | -|------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **`geometry`** | A single point, encoded as a x:y:z:m concatenation of its 8-byte little-endian IEEE 754 coordinate values. x and y are mandatory. This becomes x:y if z and m are both unset, x:y:z if only m is unset, and x:y:NaN:m if only z is unset. | -| **`geography`** | A single point, encoded as a x:y:z:m concatenation of its 8-byte little-endian IEEE 754 coordinate values. x and y are mandatory. This becomes x:y if z and m are both unset, x:y:z if only m is unset, and x:y:NaN:m if only z is unset. | -| **`variant`** | A serialized Variant of encoded v1 metadata concatenated with an encoded variant object. Object keys are normalized JSON paths to identify fields; values are lower or upper bound values. | - -### JSON single-value serialization - - Single values are serialized as JSON by type according to the following table: - -| Type | JSON representation | Example | Description | -| ------------------ | ----------------------------------------- | ------------------------------------------ | -- | -| **`boolean`** | **`JSON boolean`** | `true` | | -| **`int`** | **`JSON int`** | `34` | | -| **`long`** | **`JSON long`** | `34` | | -| **`float`** | **`JSON number`** | `1.0` | | -| **`double`** | **`JSON number`** | `1.0` | | -| **`decimal(P,S)`** | **`JSON string`** | `"14.20"`, `"2E+20"` | Stores the string representation of the decimal value, specifically, for values with a positive scale, the number of digits to the right of the decimal point is used to indicate scale, for values with a negative scale, the scientific notation is used and the exponent must equal the negated scale | -| **`date`** | **`JSON string`** | `"2017-11-16"` | Stores ISO-8601 standard date | -| **`time`** | **`JSON string`** | `"22:31:08.123456"` | Stores ISO-8601 standard time with microsecond precision | -| **`timestamp`** | **`JSON string`** | `"2017-11-16T22:31:08.123456"` | Stores ISO-8601 standard timestamp with microsecond precision; must not include a zone offset | -| **`timestamptz`** | **`JSON string`** | `"2017-11-16T22:31:08.123456+00:00"` | Stores ISO-8601 standard timestamp with microsecond precision; must include a zone offset and it must be '+00:00' | -| **`timestamp_ns`** | **`JSON string`** | `"2017-11-16T22:31:08.123456789"` | Stores ISO-8601 standard timestamp with nanosecond precision; must not include a zone offset | -| **`timestamptz_ns`** | **`JSON string`** | `"2017-11-16T22:31:08.123456789+00:00"` | Stores ISO-8601 standard timestamp with nanosecond precision; must include a zone offset and it must be '+00:00' | -| **`string`** | **`JSON string`** | `"iceberg"` | | -| **`uuid`** | **`JSON string`** | `"f79c3e09-677c-4bbd-a479-3f349cb785e7"` | Stores the lowercase uuid string | -| **`fixed(L)`** | **`JSON string`** | `"000102ff"` | Stored as a hexadecimal string | -| **`binary`** | **`JSON string`** | `"000102ff"` | Stored as a hexadecimal string | -| **`struct`** | **`JSON object by field ID`** | `{"1": 1, "2": "bar"}` | Stores struct fields using the field ID as the JSON field name; field values are stored using this JSON single-value format | -| **`list`** | **`JSON array of values`** | `[1, 2, 3]` | Stores a JSON array of values that are serialized using this JSON single-value format | -| **`map`** | **`JSON object of key and value arrays`** | `{ "keys": ["a", "b"], "values": [1, 2] }` | Stores arrays of keys and values; individual keys and values are serialized using this JSON single-value format | -| **`geometry`** | **`JSON string`** | `POINT (30 10)` | Stored using WKT representation, see [Appendix G](#appendix-g-geospatial-notes) | -| **`geography`** | **`JSON string`** | `POINT (30 10)` | Stored using WKT representation, see [Appendix G](#appendix-g-geospatial-notes) | - -## Appendix E: Format version changes - -### Version 3 - -Default values are added to struct fields in v3. - -* The `write-default` is a forward-compatible change because it is only used at write time. Old writers will fail because the field is missing. -* Tables with `initial-default` will be read correctly by older readers if `initial-default` is always null for optional fields. Otherwise, old readers will default optional columns with null. Old readers will fail to read required fields which are populated by `initial-default` because that default is not supported. - -Types `variant`, `geometry`, `geography`, `unknown`, `timestamp_ns`, and `timestamptz_ns` are added in v3. - -All readers are required to read tables with unknown partition transforms, ignoring the unsupported partition fields when filtering. - -Writing v3 metadata: - -* Partition Field and Sort Field JSON: - * `source-ids` was added and must be written in the case of a multi-argument transform. - * `source-id` must be written in the case of single-argument transforms. - -Row-level delete changes: - -* Deletion vectors are added in v3, stored using the Puffin `deletion-vector-v1` blob type -* Manifests are updated to track deletion vectors: - * `referenced_data_file` was added and can be used for both deletion vectors (required) and v2 position delete files that contain deletes for only one data file (optional) - * `content_offset` was added and must match the deletion vector blob's offset in a Puffin file - * `content_size_in_bytes` was added and must match the deletion vector blob's length in a Puffin file -* Deletion vectors are maintained synchronously: Writers must merge DVs (and older position delete files) to ensure there is at most one DV per data file - * Readers can safely ignore position delete files if there is a DV for a data file -* Writers are not allowed to add new position delete files to v3 tables -* Existing position delete files are valid in tables that have been upgraded from v2 - * These position delete files must be merged into the DV for a data file when one is created - * Position delete files that contain deletes for more than one data file need to be kept in table metadata until all deletes are replaced by DVs - -Row lineage changes: - -* Writers must set the table's `next-row-id` and use the existing `next-row-id` as the `first-row-id` when creating new snapshots - * When a table is upgraded to v3, `next_row_id` should be initialized to 0 - * When committing a new snapshot `next-row-id` must be incremented by at least the number of newly assigned row ids in the snapshot - * It is recommended to increment `next-row-id` by the total `added_rows_count` and `existing_rows_count` of all manifests assigned a `first_row_id` -* Writers must assign a `first_row_id` to new data manifests when writing a manifest list - * When writing a new manifest list each `first_row_id` must be incremented by at least the number of newly assigned row ids in the manifest - * It is recommended to increment `first_row_id` by a manifest's `added_rows_count` and `existing_rows_count` -* When writing a manifest, new data files must be written with a null `first_row_id` so that the value is assigned at read time based on the manifest's `first_row_id` -* When a manifest has a non-null `first_row_id`, readers must assign a `first_row_id` to any data file that has a missing or null value in that manifest - * Readers must increment `first_row_id` by the data file's `record_count` -* When writing an existing data file into a new manifest, its `first_row_id` must be written into the manifest -* When a data file has a non-null `first_row_id`, readers must: - * Replace any null or missing `_row_id` with the data file's `first_row_id` plus the row's `_pos` - * Replace any null or missing `_last_updated_sequence_number` to the data file's `data_sequence_number` - * Read any non-null `_row_id` or `_last_updated_sequence_number` without modification -* When a data file has a null `first_row_id`, readers must produce null for `_row_id` and `_last_updated_sequence_number` -* When writing an existing row into a new data file, writers must write `_row_id` and `_last_updated_sequence_number` if they are non-null - -Encryption changes: - -* Encryption keys are tracked by table metadata `encryption-keys` -* The encryption key used for a snapshot is specified by `key-id` - -### Version 2 - -Writing v1 metadata: - -* Table metadata field `last-sequence-number` should not be written -* Snapshot field `sequence-number` should not be written -* Manifest list field `sequence-number` should not be written -* Manifest list field `min-sequence-number` should not be written -* Manifest list field `content` must be 0 (data) or omitted -* Manifest entry field `sequence_number` should not be written -* Manifest entry field `file_sequence_number` should not be written -* Data file field `content` must be 0 (data) or omitted - -Reading v1 metadata for v2: - -* Table metadata field `last-sequence-number` must default to 0 -* Snapshot field `sequence-number` must default to 0 -* Manifest list field `sequence-number` must default to 0 -* Manifest list field `min-sequence-number` must default to 0 -* Manifest list field `content` must default to 0 (data) -* Manifest entry field `sequence_number` must default to 0 -* Manifest entry field `file_sequence_number` must default to 0 -* Data file field `content` must default to 0 (data) - -Writing v2 metadata: - -* Table metadata JSON: - * `last-sequence-number` was added and is required; default to 0 when reading v1 metadata - * `table-uuid` is now required - * `current-schema-id` is now required - * `schemas` is now required - * `partition-specs` is now required - * `default-spec-id` is now required - * `last-partition-id` is now required - * `sort-orders` is now required - * `default-sort-order-id` is now required - * `schema` is no longer required and should be omitted; use `schemas` and `current-schema-id` instead - * `partition-spec` is no longer required and should be omitted; use `partition-specs` and `default-spec-id` instead -* Snapshot JSON: - * `sequence-number` was added and is required; default to 0 when reading v1 metadata - * `manifest-list` is now required - * `manifests` is no longer required and should be omitted; always use `manifest-list` instead -* Manifest list `manifest_file`: - * `content` was added and is required; 0=data, 1=deletes; default to 0 when reading v1 manifest lists - * `sequence_number` was added and is required - * `min_sequence_number` was added and is required - * `added_files_count` is now required - * `existing_files_count` is now required - * `deleted_files_count` is now required - * `added_rows_count` is now required - * `existing_rows_count` is now required - * `deleted_rows_count` is now required -* Manifest key-value metadata: - * `schema-id` is now required - * `partition-spec-id` is now required - * `format-version` is now required - * `content` was added and is required (must be "data" or "deletes") -* Manifest `manifest_entry`: - * `snapshot_id` is now optional to support inheritance - * `sequence_number` was added and is optional, to support inheritance - * `file_sequence_number` was added and is optional, to support inheritance -* Manifest `data_file`: - * `content` was added and is required; 0=data, 1=position deletes, 2=equality deletes; default to 0 when reading v1 manifests - * `equality_ids` was added, to be used for equality deletes only - * `block_size_in_bytes` was removed (breaks v1 reader compatibility) - * `file_ordinal` was removed - * `sort_columns` was removed - -Note that these requirements apply when writing data to a v2 table. Tables that are upgraded from v1 may contain metadata that does not follow these requirements. Implementations should remain backward-compatible with v1 metadata requirements. - -## Appendix F: Implementation Notes - -This section covers topics not required by the specification but recommendations for systems implementing the Iceberg specification to help maintain a uniform experience. - -### Point in Time Reads (Time Travel) - -Iceberg supports two types of histories for tables. A history of previous "current snapshots" stored in ["snapshot-log" table metadata](#table-metadata-fields) and [parent-child lineage stored in "snapshots"](#table-metadata-fields). These two histories -might indicate different snapshot IDs for a specific timestamp. The discrepancies can be caused by a variety of table operations (e.g. updating the `current-snapshot-id` can be used to set the snapshot of a table to any arbitrary snapshot, which might have a lineage derived from a table branch or no lineage at all). - -When processing point in time queries implementations should use "snapshot-log" metadata to lookup the table state at the given point in time. This ensures time-travel queries reflect the state of the table at the provided timestamp. For example a SQL query like `SELECT * FROM prod.db.table TIMESTAMP AS OF '1986-10-26 01:21:00Z';` would find the snapshot of the Iceberg table just prior to '1986-10-26 01:21:00 UTC' in the snapshot logs and use the metadata from that snapshot to perform the scan of the table. If no snapshot exists prior to the timestamp given or "snapshot-log" is not populated (it is an optional field), then systems should raise an informative error message about the missing metadata. - -### Optional Snapshot Summary Fields - -Snapshot summary can include metrics fields to track numeric stats of the snapshot (see [Metrics](#metrics)) and operational details (see [Other Fields](#other-fields)). The value of these fields should be of string type (e.g., `"120"`). - -#### Metrics - -| Field | Description | -|-------------------------------------|--------------------------------------------------------------------------------------------------| -| **`added-data-files`** | Number of data files added in the snapshot | -| **`deleted-data-files`** | Number of data files deleted in the snapshot | -| **`total-data-files`** | Total number of live data files in the snapshot | -| **`added-delete-files`** | Number of positional/equality delete files and deletion vectors added in the snapshot | -| **`added-equality-delete-files`** | Number of equality delete files added in the snapshot | -| **`removed-equality-delete-files`** | Number of equality delete files removed in the snapshot | -| **`added-position-delete-files`** | Number of position delete files added in the snapshot | -| **`removed-position-delete-files`** | Number of position delete files removed in the snapshot | -| **`added-dvs`** | Number of deletion vectors added in the snapshot | -| **`removed-dvs`** | Number of deletion vectors removed in the snapshot | -| **`removed-delete-files`** | Number of positional/equality delete files and deletion vectors removed in the snapshot | -| **`total-delete-files`** | Total number of live positional/equality delete files and deletion vectors in the snapshot | -| **`added-records`** | Number of records added in the snapshot | -| **`deleted-records`** | Number of records deleted in the snapshot | -| **`total-records`** | Total number of records in the snapshot | -| **`added-files-size`** | The size of files added in the snapshot | -| **`removed-files-size`** | The size of files removed in the snapshot | -| **`total-files-size`** | Total size of live files in the snapshot | -| **`added-position-deletes`** | Number of position delete records added in the snapshot | -| **`removed-position-deletes`** | Number of position delete records removed in the snapshot | -| **`total-position-deletes`** | Total number of position delete records in the snapshot | -| **`added-equality-deletes`** | Number of equality delete records added in the snapshot | -| **`removed-equality-deletes`** | Number of equality delete records removed in the snapshot | -| **`total-equality-deletes`** | Total number of equality delete records in the snapshot | -| **`deleted-duplicate-files`** | Number of duplicate files deleted (duplicates are files recorded more than once in the manifest) | -| **`changed-partition-count`** | Number of partitions with files added or removed in the snapshot | -| **`manifests-created`** | Number of manifest files created in the snapshot | -| **`manifests-kept`** | Number of manifest files kept in the snapshot | -| **`manifests-replaced`** | Number of manifest files replaced in the snapshot | -| **`entries-processed`** | Number of manifest entries processed in the snapshot | - -#### Other Fields - -| Field | Example | Description | -|--------------------------|------------|-----------------------------------------------------------------| -| **`wap.id`** | "12345678" | The Write-Audit-Publish id of a staged snapshot | -| **`published-wap-id`** | "12345678" | The Write-Audit-Publish id of a snapshot already been published | -| **`source-snapshot-id`** | "12345678" | The original id of a cherry-picked snapshot | -| **`engine-name`** | "spark" | Name of the engine that created the snapshot | -| **`engine-version`** | "3.5.4" | Version of the engine that created the snapshot | - -### Assignment of Snapshot IDs and `current-snapshot-id` - -Writers should produce positive values for snapshot ids in a manner that minimizes the probability of id collisions and should verify the id does not conflict with existing snapshots. Producing snapshot ids based on timestamps alone is not recommended as it increases the potential for collisions. - -The reference Java implementation uses a type 4 uuid and XORs the 8 most significant bytes with the 8 least significant bytes then ANDs with the maximum long value to arrive at a pseudo-random snapshot id with a low probability of collision. - -Java writes `-1` for "no current snapshot" with V1 and V2 tables and considers this equivalent to omitted or `null`. This has never been formalized in the spec, but for compatibility, other implementations can accept `-1` as `null`. Java will no longer write `-1` and will use `null` for "no current snapshot" for all tables with a version greater than or equal to V3. - -### Naming for GZIP compressed Metadata JSON files - -Some implementations require that GZIP compressed files have the suffix `.gz.metadata.json` to be read correctly. The Java reference implementation can additionally read GZIP compressed files with the suffix `metadata.json.gz`. - -### Position Delete Files with Row Data - -Although the spec allows for including the deleted row itself (in addition to the path and position of the row in the data file) in v2 position delete files, writing the row is optional and no implementation currently writes it. The ability to write and read the row is supported in the Java implementation but is deprecated in version 1.11.0. - -## Appendix G: Geospatial Notes - -The Geometry and Geography class hierarchy and its Well-known text (WKT) and Well-known binary (WKB) serializations (ISO supporting XY, XYZ, XYM, XYZM) are defined by [OpenGIS Implementation Specification for Geographic information – Simple feature access – Part 1: Common architecture](https://portal.ogc.org/files/?artifact_id=25355), from [OGC (Open Geospatial Consortium)](https://www.ogc.org/standard/sfa/). - -Points are always defined by the coordinates X, Y, Z (optional), and M (optional), in this order. X is the longitude/easting, Y is the latitude/northing, and Z is usually the height, or elevation. M is a fourth optional dimension, for example a linear reference value (e.g., highway milepost value), a timestamp, or some other value as defined by the CRS. - -The version of the OGC standard first used here is 1.2.1, but future versions may also be used if the WKB representation remains wire-compatible. diff --git a/doc/design/access-server/s3/design-crowdb-access-s3-rdma.md b/doc/design/access-server/s3/design-crowdb-access-s3-rdma.md deleted file mode 100644 index 6c047f69e..000000000 --- a/doc/design/access-server/s3/design-crowdb-access-s3-rdma.md +++ /dev/null @@ -1,67 +0,0 @@ - - - -# CROWDB - Design: Native S3 and Dataset RDMA Access - -Native S3 and Dataset clients use CROWDB's shared RPC transport for direct -storage access. The HTTP S3 endpoint retains its protocol and authorization -contract; it does not negotiate an alternate cuObject data plane. - -Depends on: [Access Server](../design-crowdb-access-server.md), -[S3](design-crowdb-access-s3.md), -[Dataset](../dataset/design-crowdb-access-dataset.md), and -[RPC RDMA](../../rpc/design-crowdb-rpc-rdma.md). - -Satisfies: bounded native storage access without an Access Server payload bounce. - -## Table of contents - -1. [Access paths](#1-access-paths) -2. [Transport and buffers](#2-transport-and-buffers) -3. [Correctness and scope](#3-correctness-and-scope) - -## 1. Access paths - -```text -HTTP client -> Access Server -> shared metadata/chunk services - -Trusted native S3 / Dataset client - -> shared ClientRuntime - -> metadata authority and immutable content resolution - -> ChunkIO / DiskIO RPC or stream - -> registered host buffers over RC SEND/RECV -``` - -Native clients resolve immutable object generations or Dataset Snapshots and -use the existing metadata, publication, durability, integrity and reclamation -contracts. Transport selection changes byte delivery, not logical ownership. -Native clients are trusted storage participants; HTTP authorization is not -implicitly inherited by direct native access. - -## 2. Transport and buffers - -RDMA uses RC SEND/RECV, RDMA CM connections, registered host-buffer arenas, -bounded completion polling and explicit buffer ownership. Endpoint capability -preflight and fallback follow the shared RPC transport contract. An established -connection does not switch transports in the middle of an operation. Callers -retain request identities and treat interrupted mutations as uncertain. - -The native SDK owns request and byte admission. Returned host-memory leases -remain valid until released, including after runtime shutdown. Dataset fields -and S3 streaming bodies preserve their existing integrity checks and bounded -lifetime. Device selection, class-specific frame capacities, diagnostics and -hardware verification are defined by the RPC design and native SDK operations -instructions. - -## 3. Correctness and scope - -- I1: Transport selection never changes immutable generation/Snapshot binding, - exact offsets, integrity validation, publication or durable completion. -- I2: Buffer ownership persists through asynchronous completion and consumer use. -- I3: Partial or uncertain transfers are not reported as complete logical results. -- I4: Native admission and the shared RPC fallback policy bound resource use; - failed transfers do not silently splice data from another transport. -- Host-memory RC transfer is the supported path. cuObject/DC negotiation, - one-sided client-memory READ/WRITE, GPU pointer registration and direct GPU - destinations are outside this contract. Their absence is not a missing step - in native host-buffer RDMA support. diff --git a/doc/design/access-server/s3/design-crowdb-access-s3.md b/doc/design/access-server/s3/design-crowdb-access-s3.md deleted file mode 100644 index b48c004e7..000000000 --- a/doc/design/access-server/s3/design-crowdb-access-s3.md +++ /dev/null @@ -1,345 +0,0 @@ - - - -# CROWDB - Design: Access Server S3 - -S3 is CROWDB's HTTP object-storage access model. It provides the core S3 -behavior needed by existing tools while retaining CROWDB's bounded streaming, -atomic publication, and distributed storage properties. - -Depends on: [Access Server](../design-crowdb-access-server.md), -[Chunk I/O](../../chunkio/design-crowdb-chunkio.md), and -[Chunk-KV](../../chunkds/design-crowdb-chunk-kv.md). - -Satisfies: S3-compatible access without making S3 the universal abstraction for -Iceberg, Dataset, or CROWDB internals. - -## Table of contents - -1. [Intent and boundary](#1-intent-and-boundary) -2. [Authority model](#2-authority-model) -3. [HTTP data path](#3-http-data-path) -4. [Publication and lifecycle](#4-publication-and-lifecycle) -5. [Relationship to other access models](#5-relationship-to-other-access-models) -6. [Correctness invariants](#6-correctness-invariants) - -## 1. Intent and boundary - -The S3 module terminates an independent HTTP listener and owns S3 request, -authentication, error, namespace, and compatibility behavior. Its supported -surface is deliberately smaller than the complete AWS product surface. An -unsupported operation returns a stable S3-shaped error and does not mutate -state. - -The module does not own physical data placement, replication, erasure coding, -repair, or disk lifecycle. The HTTP implementation library is not part of the -S3 contract. - -## 2. Authority model - -S3 bucket and object records are first-class CROWDB metadata. Bucket names map -to stable identities, while object keys form an ordered namespace within a -bucket identity. Object records select complete immutable data generations and -carry the logical information needed for S3 reads and integrity checks. - -Bucket creation first attempts create-if-absent. A returned active mapping is -the result; a tombstone is replaced with a fresh bucket identity through CAS. -Definite conflicts reuse the returned mapping directly, with bounded attempts -and no confirmation GET. Missing observations permit create-if-absent again. -Transport errors propagate without treating an unknown result as a conflict. - -S3 is authoritative for bucket and object visibility, overwrite behavior, -listing position, multipart publication, deletion, and S3 credentials. It -stores opaque data references rather than physical storage topology. - -The current listener uses one configured tenant for all accepted credentials. -Signature verification authenticates a key but does not pass a user/grant -identity to operations. Accepted keys therefore share that listener namespace; -per-user bucket ACLs and IAM policies are not part of the installed surface. -Namespace isolation currently means the configured tenant/bucket identity -boundaries, not isolation between multiple accepted users on one listener. -Presigned request expiry is enforced independently of the allowed future-clock -skew; clock tolerance cannot extend an issued URL's lifetime. - -User bootstrap reads the bounded final credential inventory to preserve existing -keys and reject already ambiguous or disabled users. If absent, it derives a -stable public access-key identifier from the user and conditionally creates one -final encrypted credential. The secret and encryption nonce remain random; -the public identifier is not an authentication secret. A collision cannot replace -another user: the losing caller reads the exact target and validates its user, -ciphertext and enabled state. Successful creation returns its known token without -another inventory scan. There is no persistent bootstrap preparation or response. -Group-0 CAS currently supplies only a conflicting revision, so that exceptional -path needs one authoritative GET. An absent value after an uncertain write remains -an error for the caller to retry. - -Explicit token issuance can still create multiple credentials for a user; -bootstrap does not provide an atomic uniqueness guarantee against that independent -administrative operation. Existing duplicates fail the inventory check. Unused -legacy preparation records are neither authentication data nor a source for -restoring a deleted or disabled final credential. Deleting the final credential -and invoking bootstrap creates a fresh random secret; ordinary lookup never -creates data. Credential snapshots continue to load only final encrypted records. - -## 3. HTTP data path - -PUT and multipart upload stream HTTP bodies into bounded CROWDB writers. GET -streams owner-backed CROWDB data into an HTTP response. Backpressure bounds -memory and storage work independently of object size; slow peers cannot create -unbounded buffering. - -Uploads use the shared bounded `UploadBody` decoder. It verifies supported -CRC32, CRC32C, CRC64NVME, SHA1 and SHA256 headers or declared AWS trailers, -alongside Content-MD5 and signed payload hashes. Streaming SigV4 first verifies -the request seed, then every signed chunk and signed trailer. Unsigned chunks -require a verified declared trailer. Encoded and decoded lengths, frame size, -terminal chunks and trailing bytes are checked before publication; malformed -or corrupt bodies cannot replace a selected object. Checksum calculation on -requests is supported; arbitrary response checksum negotiation is outside the -installed surface. - -Presigned PUT and UploadPart also accept those checksum values and their SDK -algorithm declaration in the signed query. Authentication verifies the original -URI first; normalized checksum fields then enter the same upload verifier. -Duplicate query fields, header/query overlap and declarations on other -operations are rejected before publication. Signed query values cannot bypass -body integrity checks. - -The ordinary HTTP path is always available. An optional direct data plane may -move an authenticated object range between DiskIO and registered client memory -without relaying payload through Access Server memory. HTTP remains the control -and compatibility surface, and acceleration cannot change S3 semantics. - -## 4. Publication and lifecycle - -An object becomes visible only after its bytes and integrity state are complete. -Publication selects the complete object generation atomically. A failed or -losing upload remains invisible and is reclaimed asynchronously. - -Reads retain one published generation for the operation. Overwrite selects a -new complete generation; it never mutates bytes observed by an existing read. -Delete removes logical visibility before physical reclamation. Bucket deletion -cannot expose objects from an earlier bucket identity if the name is later -reused. - -Listings are ordered and continuation-safe within their documented consistency -model. Continuation state is opaque and bound to the original request scope. -Bucket paths accept a single trailing slash. ListObjectsV2 supports URL encoding -of XML-incompatible keys and selected prefix/delimiter fields. ListBuckets -reports the Unix epoch as a stable CreationDate placeholder because bucket -records do not retain creation timestamps. -Bucket listings skip object and multipart rows sharing the metadata interval only -after validating their complete key and value identities; unknown or corrupt rows fail. - -User metadata is accepted through x-amz-meta-* headers on ordinary PUT, -multipart initiation and CopyObject. Nonempty HTTP token names are normalized -to lowercase; values are opaque printable ASCII strings. Combined key/value -bytes, excluding the header prefix, are limited to 2 KiB. Duplicate names, -invalid values and unsupported operation placement are rejected before writes. -The complete attribute map is published with the object's immutable generation -and returned on HEAD, GET and ranged GET. A new upload without metadata clears -the previous generation's attributes. Multipart sessions persist initiation -attributes across part replacement and recovery and publish them on completion. -The session schema has a distinct version; old-version data migration is outside -the supported contract. - -STANDARD is the sole storage class reported by listings and accepted explicitly -on PUT, multipart initiation and CopyObject. Other classes are rejected; this -selector does not alter the configured storage policy or introduce tiering. - -Server-side CopyObject and UploadPartCopy resolve source and destination through -the configured S3 tenant. Copy selects one immutable source record before -returning response headers; subsequent source overwrite or logical deletion -does not select new bytes. The source is streamed through bounded readers and -writers, respecting writer capacity, and publication uses the ordinary object -or multipart-part fences. No source reference is shared with the destination. -Copy never deletes the source. Unpublished candidates use ordinary reclamation. - -CopyObject supports at most 5 GiB and copies the entire payload. COPY preserves -Content-Type and user metadata; REPLACE selects the supplied Content-Type -or application/octet-stream and replaces the entire user metadata map, including -clearing it when omitted. A self-copy requires REPLACE. -Cache/disposition/encoding/language/expiry metadata, version selectors, tags, -encryption, storage-class changes and destination conditions are unsupported -and rejected. Source ETag/date conditions apply to the captured generation; -their failure returns PreconditionFailed. Copy selectors must be signed. -The optional SDK x-id query marker must match CopyObject or UploadPartCopy and -occur at most once; it does not change operation selection. - -UploadPartCopy supports complete objects and explicit inclusive byte ranges -from source objects larger than 5 MiB, with the session's normal part bounds. -It replaces a part using the durable generation-selection protocol below. -Neither interrupted copy path publishes partial bytes. After request validation -and source selection, the HTTP 200 response carries whitespace keepalives and -ends with CopyObjectResult, CopyPartResult, or an embedded Error. Clients must -consume and validate the whole response. The body owns the copy future, cancels -it on disconnect, and caps execution at 300 seconds. A retry after response loss -can select a newer source generation; copy is not an exactly-once operation. - -Destination mutation conditions (`If-Match`, `If-None-Match` and date conditions) -are rejected before dispatch. Object GET/HEAD and copy-source conditions remain -supported. Ordinary object publication and deletion do not acquire a preliminary -object read or conditional mutation. - -DeleteObjects validates its entire request before issuing a mutation: at most -1,000 nonempty UTF-8/XML keys of 1,024 bytes each, a 2 MiB body, and no versions -or conditional object selectors. Content-MD5 is supported; verified CRC32 may -replace it for the current SDK serializer. This is an explicit integrity -compatibility extension to the general-bucket MD5 rule. Every supplied supported -checksum and signed payload hash is verified; unknown integrity extensions are -rejected. Neither missing integrity nor malformed XML can delete any key. - -A valid batch uses the same logical deletion as DeleteObject, sequentially in -input order. Each occurrence of a duplicate key gets its own result, and absent -keys are successes. Failures do not undo preceding successes; Quiet omits only -success entries. After validation, the response streams the XML opening, each -ordered result, and the closing tag. Quiet successes send XML whitespace to -maintain read progress. The response body owns the deletion stream; dropping -it cancels remaining work. Batch deletion is -not a transaction or an exactly-once operation: retries converge for keys with -no intervening writes, but can delete a new unversioned PUT after response loss. -Physical reclamation remains independent of this request. - -Multipart uploads keep a durable `MultipartSessionRecord`, current part pointers, -and immutable `MultipartPartRecord` generations under one upload prefix. The -upload resource binds admission bounds, expiration and the completion selection. -Replacing a part number retains its previous generation as reference evidence; -expiry does not remove that evidence. - -Independent-part uploads use the `S3MS\x05` resource format. They require the -checked product of `max_parts` and `max_part_bytes` to fit within -`max_staged_bytes`, so every valid set of current parts fits the staged-byte -bound without a shared accounting update. The ordinary bounds are 10,000 parts, -5 GiB per part, a 5 TiB completed object and a staged bound of 10,000 times -5 GiB. Completion separately enforces the object-size bound. These bounds do not -count retained superseded generations as current parts. Format 5 rejects pending -reservations and tighter or overflowing bounds. Codecs that do not support this -format reject it; resource CAS cannot change between independent and strict modes. - -`MultipartRepository::put_stream_part` admits an open upload, loads its current -part pointer, persists an immutable generation, then conditionally replaces that -part's pointer. It does not reserve or clear a shared upload mutation, and distinct -part numbers share no accounting CAS. An initially matching length and raw MD5 -return the retained part directly after admission. Once a writer enters the -mutation path, successful acknowledgement also observes the upload authority: -an observed abort or invalid open admission rejects it. This check reuses the -request's supplied admission timestamp and does not fence a later abort. Failed -acknowledgement does not remove persisted part evidence. - -An admitted part writer may acknowledge B after Complete freezes A for the same -part number. The frozen selection still references A's exact -generation and digest; publication and retries use A even when the current pointer -selects B or is absent. A request first admitted after the freeze is rejected. -Completing B requires a new upload identity. - -Strict uploads retain the reservation protocol and exact staged-byte accounting. -They include decoded `S3MS\x03` and `S3MS\x04` resources and uploads whose bounds -require a tighter quota or cannot establish the checked product bound. Their writer stores the immutable generation, -reserves the pointer update with an upload CAS, publishes the pointer, then clears -the reservation. A later request can finish an interrupted reservation; Complete -settles it before freezing. Publication also rechecks the selected current pointers. -Ordinary bounds do not upgrade a decoded strict resource to independent mode. - -In both modes, an unused immutable candidate does not block a part number: -matching length and raw MD5 reuse its retained generation and locations; -different contents select the next unused generation. Generation numbers increase -monotonically and can have gaps. Definite pointer and upload CAS conflicts consume -the returned observation rather than issuing a confirmation GET. Part progress -carries that observation into the next iteration, retaining already-written bytes. -Its five-second local work deadline returns a retryable busy result under sustained -contention. An uncertain mutation returns without a confirmation GET after the -routed client's retry budget; a later request retries the full input against the -retained resource facts. - -A bound upload is loaded through its authoritative key and validated against -bucket, object and upload identity; direct lookup does not read the listing index. -ListUploads and expiry scanning validate that index when enumerating resources. -Exact observed strict-format bytes are retained only in memory for conditional -updates; they do not add a response or recovery record. - -Completion validates ordered part numbers, raw MD5 values, minimum nonfinal size, -current generations and aggregate object length. `freeze_completion` binds the -selected numbers, revisions and digests under the upload-resource CAS. -`publish_completion` validates those immutable generation identities and digests, -composes chunk locations with adjusted logical offsets, and publishes one object -record through an unconditional object-key PUT. Independent publication does not -reload current part pointers. It never reads or concatenates part payload bytes. -The multipart ETag is the hexadecimal MD5 of the selected raw part MD5 values in -order, followed by the part count suffix. - -Every accepted Complete execution performs the object PUT, including retries after -an earlier success; it may overwrite a later ordinary PUT to the same name. The frozen selection is -the reference authority, not a recorded response. No predecessor, request digest, -cached response or publication-success transition is persisted. Abort and expiry -mark open uploads terminal while retaining their listing entries, pointers and -immutable generations. Publishing uploads remain bound, retaining the selected -generation references. Expiry reuses the authority loaded through the validated -listing entry and consumes the abort CAS observation without another confirmation -read. An uncertain mutation remains an error; later scanning rechecks the durable -resource state. - -The background expiry worker owns discardable tenant and upload key positions -for one configured tenant. Each two-second step scans one raw tenant metadata -key or at most 20 upload-index entries, with a four-MiB aggregate result budget. The legacy -scan transport materializes values before applying that aggregation limit; it -does not enforce the budget on wire responses or native value materialization. -The first sixteen bytes after the tenant prefix supply a BucketId hint; the -name mapping and its payload are not discovery authority. An old bucket remains -discoverable after deletion or same-name recreation. The worker then scans that -identity's existing listing index, validating each canonical index and upload -authority before expiry. Upload creation persists the index before its authority. -Long name keys may supply harmless extra hints; short keys advance by their -observed raw position. There is no bucket-count cutoff. - -Selecting a hint also saves the inclusive successor of the entire bucket -prefix. Finishing, failing or cancelling its index scan can therefore move to -later identities without scanning retained object, part and generation rows -one by one. A later complete round retries discarded hints. Successful partial -index progress advances before its point operation, so a corrupt row or -uncertain abort cannot pin subsequent uploads. A process restart scans the -configured tenant again without loading a persistent progress record. - -Bucket discovery requests one raw row at a time. If a materialized first row -exceeds the aggregate budget, the client returns only its validated key. The -worker retains its identity hint for the next window and reports an oversized-row -error, preserving the original metadata for later rounds. A short key instead -advances its raw position. A transport failure that -provides no validated key cannot be skipped safely and remains retryable. -Obtaining discovery keys without materializing their values requires a separate -key-only or native bounded-scan capability. - -Workers continue after 50 milliseconds while a round has more work, wait -15 seconds between complete rounds, and defer failures for 500 milliseconds. -These local cursors authorize no physical deletion. Physical range discovery, -reference checks and retirement fences remain separate reclamation duties; -marking an upload terminal does not report its physical storage reclaimed. - -## 5. Relationship to other access models - -Iceberg is not implemented as special objects in the S3 namespace. Its catalog, -table, file, commit, and reclamation records belong to the Iceberg authority, -even when an Iceberg client uses S3-shaped file locations. - -Dataset is not an S3 extension. A Dataset may explicitly refer to immutable -bytes associated with S3, but Dataset selection, batching, streaming, and -native topology access remain Dataset semantics. - -## 6. Correctness invariants - -- **S3-I1 — Atomic visibility:** a reader observes absence or one complete - published object generation, never partial upload bytes. -- **S3-I2 — Stable read:** one HEAD or GET retains one object generation for - the entire operation. -- **S3-I3 — Bounded streaming:** object size does not determine Access Server - memory consumption. -- **S3-I4 — Namespace isolation:** credentials, bucket identity, object key, - and continuation state cannot escape their authorized scope. -- **S3-I5 — Reclamation after invisibility:** physical deletion follows logical - removal and cannot restore visibility. -- **S3-I6 — Independent authority:** S3 records cannot publish, mutate, or - reclaim Iceberg or Dataset authority. -- **S3-I7 — Transport equivalence:** ordinary HTTP and accelerated transfer - produce the same S3 range, integrity, publication, and error outcome. -- **S3-I8 — Multipart selection fence:** a part pointer cannot advance after - completion freezes its selection. An interrupted pointer reservation is - settled before completion or abort proceeds. diff --git a/doc/design/chunkdb/chunkdb-allocate-flow-analysis.md b/doc/design/chunkdb/chunkdb-allocate-flow-analysis.md deleted file mode 100644 index cd4ca4a9a..000000000 --- a/doc/design/chunkdb/chunkdb-allocate-flow-analysis.md +++ /dev/null @@ -1,189 +0,0 @@ - - - -# ChunkDB Allocation Flow Analysis - -## Scope - -This analysis measures one-strip EC 8+4 chunk allocation on three co-located -logical nodes. Each node runs KV, DiskDB, and ChunkDB; the fixture has three -racks and three DiskDB data groups backed by three-replica KV groups. One -successful ChunkDB request allocates and commits twelve physical blocks. - -The implementation keeps strips sequential. Within one strip it groups blocks -by data group, sends one DiskDB allocation request for four blocks to each of -the three nodes, and runs those requests concurrently. After they return, -ChunkDB persists one Active chunk record and returns success. Three grouped -block-commit requests run asynchronously after publication. - -## Current Allocation Flow - -For one EC 8+4 strip, the success path is: - -1. ChunkDB selects twelve placements and groups them into three data-group - requests. Each request asks one DiskDB server for four blocks on four - different disks. The three requests run concurrently. -2. Each DiskDB server claims four bitmap ranges, constructs four - `BusyBlockValue` records with `state=Ok` and `commit_state=Tentative`, and - sends one four-key batch write to that data group's KV leader. After the KV - write succeeds, DiskDB caches the tentative values by `allocation_ts` and - returns the four segments to ChunkDB. -3. After all DiskDB responses return, ChunkDB constructs one Active chunk - containing the twelve segments and writes that chunk record to its KV - group. This is the allocation publication and durability boundary. -4. Once the Active chunk write succeeds, ChunkDB starts three asynchronous - commit RPCs, refreshes its chunk cache, and returns the Active chunk to the - client. The client does not wait for block commit. -5. Each DiskDB commit validates the physical identity, owner, and allocation - timestamp. A tentative-cache hit supplies the value without a KV read; a - miss reads it from KV. DiskDB changes `commit_state` to Committed, writes - all changed busy-block values in one batch, and removes exact cache entries - after success. An already Committed value is idempotent success. - -At client success, all twelve busy blocks and the referencing Active chunk are -durable. The blocks may remain Tentative until background commit completes. A -future reconciliation scanner can resolve that crash window by asking -ChunkDB whether the Active chunk references the same allocation incarnation, -then committing or freeing the block. - -## Environment and Results - -- Host: AMD Ryzen 9 5950X, 16 cores / 32 threads, 62 GiB RAM. -- Duration: 20 seconds per row. -- Capacity: four 4-TiB logical disks per DiskDB; 256-GiB zones. -- KV inflight/coalescing: 32/32. -- Command: `pixi run -- bash tools/benchmark/bench-chunkdb-regression.sh`. - -| Workload | Groups | Threads | Strips | EC | Client conn | ChunkDB conn | DiskDB conn | KV conn | Workers | Chunk/s | Block/s | p50 us | p99 us | Errors | Space | -|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---| -| allocate | 3 | 1 | 1 | 8+4 | 2 | 2 | 2 | 2 | 2 | 799 | 9,588 | 1,275 | 1,561 | 0 | exact | -| allocate | 3 | 16 | 1 | 8+4 | 2 | 2 | 2 | 2 | 2 | 8,800 | 105,600 | 1,719 | 3,481 | 0 | exact | -| allocate | 3 | 128 | 1 | 8+4 | 4 | 4 | 4 | 4 | 4 | 11,914 | 142,968 | 9,945 | 22,922 | 0 | exact | -| allocate | 3 | 256 | 1 | 8+4 | 4 | 4 | 4 | 4 | 4 | 12,685 | 152,220 | 18,427 | 49,050 | 0 | exact | -| allocate | 3 | 512 | 1 | 8+4 | 4 | 4 | 4 | 4 | 4 | 12,734 | 152,808 | 37,131 | 91,641 | 0 | exact | - -All rows stopped at the deadline and retained three KV, DiskDB, and ChunkDB -metrics/RPC log pairs plus the CLI pair. Clean artifacts are: - -- `bench-log/chunkdb-r98-final2-20260905-223329` (all rows). -- `bench-log/chunkdb-r98-common-latency-20260905-224613` (selected row). -- `bench-log/chunkdb-r98-confirm-2-20260905-224959` (confirmation). -- `bench-log/chunkdb-r98-confirm-3-20260905-225127` (confirmation). - -The selected 256-thread confirmation samples are 12,707, 12,460, and -12,634 chunks/s. Their median is 12,634 chunks/s and the maximum deviation is -1.4%, within the 10% stability bound. Every sample has zero errors and exact -space accounting. - -An R149 validation run on an Intel Core i9-7960X (16 cores / 32 threads) -confirmed the cache-assisted batched commit path after removing per-block -BusyBlock CAS. These numbers are a same-host regression sentinel and are not -directly comparable with the AMD baseline above. - -| Threads | Chunk/s | Block/s | Average us | p50 us | p99 us | Errors | Space | -|--------:|--------:|--------:|-----------:|-------:|-------:|-------:|:------| -| 1 | 238 | 2,856 | 4,195 | 4,188 | 5,408 | 0 | exact | -| 16 | 7,015 | 84,180 | 2,279 | 2,194 | 3,773 | 0 | exact | -| 128 | 9,795 | 117,540 | 13,046 | 12,647 | 22,392 | 0 | exact | -| 256 | 10,441 | 125,292 | 24,474 | 23,307 | 47,543 | 0 | exact | -| 512 | 9,702 | 116,424 | 52,647 | 48,765 | 112,289 | 0 | exact | - -The retained artifact is -`bench-log/chunkdb-regression-20260913-132220`. A focused same-host comparison -measured 151 chunk/s at one thread and 150 chunk/s at 16 threads with -per-block CAS, including a 4.24-second p99 at 16 threads. Restoring one -ordinary batched commit write per data group reached 249 and 6,920 chunk/s in -the focused run and removed the seconds-scale tail. - -## Bottleneck - -The peak is 12,734 chunks/s, or 152,808 physical blocks/s, at 512 threads. -Increasing from 256 to 512 threads adds only 0.4% throughput while p99 grows -from 49 ms to 92 ms, so 256 threads remains the better latency/throughput -operating point. - -DiskDB retains each freshly persisted tentative `BusyBlockValue` in a bounded -per-data-group cache keyed by `allocation_ts`. Commit validates the cached -physical identity and owner and builds one batch write without a preceding -read. A miss after restart or eviction falls back to KV. The historical -512-thread run reported 3,153,276 cache hits, exactly 262,773 chunks times -twelve blocks, and zero misses. - -At 256 threads, weighted ChunkDB metrics show: - -- grouped DiskDB allocation: 9.48 ms; -- the single Active metadata KV persistence: 5.08 ms; -- complete request: 18.93 ms; -- asynchronous block commit: 9.30 ms, outside response latency; -- placement: below 0.02 ms. - -Each DiskDB server receives exactly one four-block allocation RPC and one -four-block commit RPC per chunk. The earlier flow sent four one-block -allocation RPCs to each node and synchronously waited for commit. Grouping and -asynchronous commit raised the 256-thread row from 8,737 to 12,860 chunks/s -(47.2%) and reduced p50 from 28 ms to 19 ms. - -The response-path bottleneck is now the grouped DiskDB allocation round. At -512 threads it grows to 26.16 ms, while the single metadata write is 6.62 ms; -the asynchronous commit averages 23.72 ms. These concurrent writes share the -same KV/Paxos groups, so commit still contributes background queue pressure -even though callers no longer wait for it. Future DiskDB reconciliation must -resolve tentative blocks after a ChunkDB crash by checking whether the Active -chunk still references the exact allocation incarnation, then committing or -freeing it. - -### Allocation latency attribution - -At 256 threads with four connections and four RPC workers, one three-node -allocation round averages 9.48 ms. The average DiskDB `AllocateBlocks` request -is 6.02 ms, including 4.99 ms in its KV batch write. Bitmap selection is -0.003 ms and response construction is 0.007 ms. Therefore DiskDB's local -allocator is not the bottleneck: about 83% of server request latency is the -KV/Paxos write. The remaining 3.46 ms between average server latency and the -client round includes transport queueing and waiting for the slowest of three -parallel responses. - -A 256-thread connection/worker sweep produced: - -| Connections/workers | Chunk/s | Allocation round | DiskDB request | DiskDB KV persist | Request latency | Result | -|---:|---:|---:|---:|---:|---:|---| -| 2 | 14,129 | 10.46 ms | 7.26 ms | 6.45 ms | 16.98 ms | exact, zero errors | -| 4 | 12,860 | 9.48 ms | 6.02 ms | 4.99 ms | 18.93 ms | exact, zero errors | -| 8 | 11,252 | 10.49 ms | 6.67 ms | 5.45 ms | 21.39 ms | initial run failed accounting | - -Four connections/workers minimize the isolated allocation round, but two -produce better end-to-end throughput because the single metadata write falls -from 5.08 ms to 3.92 ms despite a slower allocation round. Eight add queue -pressure rather than throughput. The first 8/8 run exposed a benchmark result -parser bug: its `busy_delta` lookup could match the suffix of -`expected_busy_delta` and incorrectly print `exact`. The parser now requires a -field boundary. A 120-second diagnostic run also identified the apparent -accounting error as verification overload: two of 256 parallel chunk queries -timed out, so their 24 valid blocks were omitted from expected space. Chunk -verification is now limited to 64 parallel queries and prints the exact failed -ID and reason. -Two subsequent 60-second stress runs were clean: 10,933 chunks/s at 256 -threads and 10,882 chunks/s at 512 threads, both with zero errors and exact -space. After limiting verification concurrency, the final 20-second 256-thread -8/8 run reached 11,571 chunks/s with zero errors and exact space. This -indicates shared KV/Paxos contention, not insufficient -ChunkDB-to-DiskDB connections or workers. - -### KV load - -Each successful EC 8+4 chunk issues three four-key DiskDB allocation writes, -one single-key Active chunk write, and three four-key asynchronous commit -writes. The total is seven logical KV write requests and 25 key mutations per -chunk. Therefore the canonical 512-thread result of 12,734 chunks/s represents -89,138 logical KV requests/s and 318,350 key mutations/s. The latter is above -the direct KV write benchmark peak of 264,130 single-key writes/s; grouped -DiskDB writes and KV coalescing make the request and Paxos-round rates much -lower than the key-mutation rate. - -## Metrics - -Every ChunkDB RPC exports `request.{method}.lh`, `.inflight.g`, and `.errors.c`. -Allocation also exports placement, DiskDB round/call/retry, asynchronous block -commit/error, record-build, Active KV persistence, rollback, strip/block -count, and inflight metrics. CLI, ChunkDB, DiskDB, KV, and crowdb-rpc logs -share the timestamped regression root. diff --git a/doc/design/chunkdb/design-crowdb-chunkdb-mirror-to-ec.md b/doc/design/chunkdb/design-crowdb-chunkdb-mirror-to-ec.md deleted file mode 100644 index 622d8671a..000000000 --- a/doc/design/chunkdb/design-crowdb-chunkdb-mirror-to-ec.md +++ /dev/null @@ -1,278 +0,0 @@ - - - -# CROWDB - Design: Mirror-to-EC Conversion and Chunk Tasks - -This document defines synchronous small-write EC conversion and the persistent -chunkdb task framework used for recovery and background conversion. - -Depends on: [chunkdb](design-crowdb-chunkdb.md), -[chunkdb RPC](design-crowdb-chunkdb-rpc.md), and -[small-object writer](../chunkio/design-crowdb-chunkio-small-object-writer.md). - -The read-side producer for repair tasks is the -[chunk object reader](../chunkio/design-crowdb-chunkio-reader.md). - -Satisfies: durable space reduction for shared mirror chunks and reusable -crash-recoverable chunk maintenance tasks. - -## Table of Contents - -- [1. Scope](#1-scope) -- [2. Persistent Chunk Tasks](#2-persistent-chunk-tasks) -- [3. Foreground Conversion](#3-foreground-conversion) -- [4. Background Conversion](#4-background-conversion) -- [5. Publication and Cleanup](#5-publication-and-cleanup) -- [6. Crash Correctness](#6-crash-correctness) -- [7. Policy and Observability](#7-policy-and-observability) -- [8. Test Contract](#8-test-contract) - -## 1. Scope - -Eight adjacent closed one-MiB mirror strips can become one 8+4 EC strip whose -eight data shards preserve the mirror images and whose four parity shards are -generated by isa-l. The small-write client performs this work without rereading -mirrors when memory and I/O are available. A persistent chunkdb task provides -takeover after client failure and supports scanner-created background work. - -Incomplete groups remain mirrored. Conversion preserves logical capacity, -`chunk_offset`, `sealed_length`, and the monotonic `next_strip_sequence`. - -## 2. Persistent Chunk Tasks - -### 2.1 Keys - -Each task has one canonical record and at most one runnable or leased index. -Integers use big-endian encoding so index scans retain scheduling order. - -```text -ChunkTaskKey = magic | 0x000D | partition-id:16 | kind:u16 | task-id:16 -ReadyChunkTaskKey = magic | 0x000E | priority-inverse:u8 | - eligible-at-ms:u64 | partition-id:16 | kind:u16 | - task-id:16 -LeasedChunkTaskKey = magic | 0x000F | lease-deadline-ms:u64 | - partition-id:16 | kind:u16 | task-id:16 -``` - -The target `ChunkId` is the partition ID for chunk-scoped tasks. Canonical and -index mutations route to the same KV group and are written in one batch. Index -values are empty and rebuildable; the canonical value is task truth. - -`task-id` is a deterministic hash of task kind and its deduplication identity. -Mirror conversion uses the first mirror sequence and EC scheme. Its operation -ID is also deterministic, allowing publication retries to identify an already -installed replacement. - -`RepairStrip` uses the chunk as partition, the task kind as the dispatch -namespace, and a deterministic task ID derived from strip sequence plus the -sorted failed-segment identities. Its versioned payload contains the chunk ID, -strip sequence, and failed segments. The canonical envelope retains source -revision, estimated full-rebuild I/O bytes, retry state, claim generation, and -the deterministic publication operation ID. Other task types keep their own -payload schema while sharing the envelope, indexes, leases, and transitions. - -### 2.2 Value and Handler Dispatch - -`ChunkTaskValue` is a verified, versioned FlatBuffer envelope containing stable -identity, kind and kind version, state, priority, revision, source revision, -retry eligibility, attempt budget, queue-cost estimate, claim owner and -generation, lease deadline, bounded error detail, and an opaque typed payload. - -States are `Pending`, `Running`, `RetryWait`, `Completed`, `Failed`, and -`Cancelled`. `TaskHandler` dispatch is keyed by kind and checks the payload -version before execution. New task kinds can therefore add their own payload -without changing the scheduler. Replica repair, EC rebuild, orphan cleanup, -and placement verification can use the same envelope. - -### 2.3 Claim and Recovery - -`TaskManager` admits deterministic tasks, claims ready records, renews leases, -and persists complete, retry, or failure outcomes. `TaskScanner` wakes on -admission and also performs a periodic safety scan. It first requeues expired -leases, then claims no more ready tasks than `TaskExecutor` has available -slots. The executor heartbeats while a handler runs. - -The configured concurrency is a fixed upper bound. Retry eligibility and lease -deadlines schedule work; elapsed idle time does not scale worker count. -Conversion and repair handlers also enforce independent local concurrency -semaphores. Repair has the highest ready priority because it restores data -redundancy, while conversion only reclaims space. - -### 2.4 Read Repair Admission and Execution - -Readers report only verified frame parse or checksum failures. ChunkDB first -marks the exact committed DiskDB BusyBlock incarnation `Corrupt`, then records -its segment in `ChunkStrip.unavailable_segments` and immediately admits -`RepairStrip`. Network and unknown I/O errors do not mark or admit repair. -A bounded rotating metadata scan also admits one -task per strip and failure set. An existing pending, running, or retry-wait -task deduplicates admission. A terminal task is revived at a higher revision -when its marker remains, preventing a stale completion or failure from -suppressing necessary repair. - -`RepairStripTaskHandler` re-queries current metadata and treats the marker as -authority. It reads one full healthy mirror or full EC shards, records any new -I/O failures before retrying, reconstructs unavailable shards, and allocates -replacement segments excluding every disk already used by the strip. Mirror -placement excludes survivor nodes. EC replacement instead preserves the -current safe `code_num` per-node limit, or the balanced ceiling of an -explicitly unsafe layout. A disabled-by-default test escape hatch may relax -only node anti-affinity, never same-disk exclusion. - -Each replacement is fully written and fsynced before exact revision-and-range -fenced publication. The publication removes repaired markers and adds retired -segments to the normal layout-validity cleanup intent. A changed layout causes -re-query and retry. Repair tasks have unlimited attempts and use delayed retry -rather than tight polling when memory, placement, parity, or I/O is temporarily -unavailable. - -The read-triggered ad-hoc manager joins routed requests by chunk, strip, and -failed segment incarnation. It uses an in-memory shared future, at most 32 -concurrent jobs, and a 512-MiB decode/result budget by default; it never owns -a second durable task or replacement target. It may return reconstructed -bytes after the task writes and fsyncs its checkpointed target, while the task -continues fenced publication and confirmation. Restart discards the future, -not the task or target checkpoint. Stale revisions and healed incarnations -cannot join or allocate. - -## 3. Foreground Conversion - -Each small-write pipeline owns one special reservation group containing eight -mirror candidate sets with the configured copy count and four parity segments. -The group is chosen from one placement snapshot and is hidden from -`Chunk.strips` until individual mirrors are confirmed. A group is not started -when the current chunk cannot -contain eight remaining full strips. - -After a mirror strip is durable, its existing one-MiB shadow updates four -persistent parity accumulators and is released. The eighth update completes -parity without rereading mirrors and without retaining eight data images. - -A pipeline holding a partial group is not a scale-in candidate. Scale-in first -requires every queue to be empty and every pipeline to be non-busy, then removes -an individual pipeline that does not own partial conversion state. Queue bytes -and object count remain the only scale-out inputs. - -After shard eight, the client writes and fsyncs only the four parity segments, -then requests fenced publication. ChunkDB re-evaluates every healthy survivor -combination against current topology, selects the minimum-concentration safe -8+4 layout, atomically replaces the eight mirrors, and records the sixteen -redundant replicas for grace-period cleanup. A fast-path parity or publication -error does not revoke acknowledged mirrors; it admits the existing durable -`MirrorToEc` task for takeover. - -An early one-through-seven-strip tail cancels unused mirror reservations and -all parity segments while retaining every confirmed mirror strip. - -## 4. Background Conversion - -`ConversionCoordinator` scans chunk metadata in bounded rotating pages and -admits complete compatible mirror groups. A group must have equal nonzero unit -geometry, contiguous offsets, and an immutable closed boundary. Sealed chunks -also obey the configured age policy; manual single-chunk triggers bypass age -and count policy but not geometry, closure, or chunk-state checks. - -`MirrorToEcTaskHandler` reads one healthy replica for each mirror, falling back -across replicas on routing or DiskIO errors. It generates parity, writes and -fsyncs all replacement shards, and invokes the same fenced publication used by -the client. Disk-to-DiskIO routes are atomically published and periodically -refreshed from hardware and service-registry state. - -Automatic scans leave incomplete tails mirrored and continue across task -failures. Manual single and batch triggers are available through crowdb-rpc and -the `/convert_chunk` and `/convert_all` HTTP endpoints. - -A separate bounded, rotating reservation scan closes the crash window before a -completed foreground group has admitted its task. Complete groups enter the -same deterministic task identity, making repeated scans idempotent. Expired -incomplete groups are cancelled; consumed blocks are reclaimed only when the -reservation persisted the planned write cursors that prove generation-fenced -DiskIO addressing. Legacy records without that proof remain allocated. - -## 5. Publication and Cleanup - -The special reservation allocator creates all twenty-eight blocks with -all-or-rollback semantics. DiskDB requests are aggregated per DiskDB group and -use one persisted request per group. Conversion requests permit repeated -anti-affinity passes: every pass visits distinct disks before any disk is reused, -while the first twelve EC roles are validated to fit on distinct disks. This -avoids both one RPC per block and an oversized request failing solely because -the mirror candidates outnumber disks. No reservation record makes a block -readable. - -`replace_chunk_strip_range` validates the expected revision and exact old -range. Its KV transaction commits replacement segments, publishes the EC strip, -and records retired mirror segments as one atomic metadata change. Retired -segments remain allocated for the layout-validity grace period; reconciliation -then frees only segments absent from the current layout. - -On a definite pre-publication failure, the task first clears its durable -replacement reference. DiskDB's tentative-block scanner reclaims the now -unreferenced allocation. An ambiguous KV response retains the allocation until -recovery determines whether its checkpoint or publication committed. - -## 6. Crash Correctness - -- **C1 — Layout authority:** readers use only committed `Chunk.strips`; task - state and tentative blocks never select readable data. -- **C2 — Mirror preservation:** every mirror remains referenced until every EC - shard is written and fsynced. -- **C3 — Atomic visibility:** fenced range replacement is the only EC - publication point. -- **C4 — Ambiguity retains data:** an ambiguous checkpoint or publication never - frees a possibly referenced replacement. -- **C5 — Replayable retirement:** replacement and cleanup intent commit - together, and cleanup uses set difference against the current layout. -- **C6 — Duplicate safety:** deterministic identity and revision/range fencing - permit at most one layout publication even if claims overlap. -- **C7 — Append independence:** replacement preserves capacity and - `next_strip_sequence`, so later appends cannot duplicate or reorder strips. - -A takeover rewrites all shards and fsyncs them; it does not trust partial-write -progress. If publication is already visible anywhere in the current layout, -the handler treats the operation as installed. If allocation succeeded but no -task checkpoint references it, the allocation remains tentative garbage and is -eventually reclaimed by DiskDB. This can temporarily consume space but cannot -make corrupt data visible. - -## 7. Policy and Observability - -`ConversionConfig` defaults to disabled automatic scanning, 8+4 EC, one-hour -minimum seal age, eight mirror strips, four concurrent handlers, 50 MiB/s -background I/O, a 30-second scan interval, and a 30-second task lease. - -The bandwidth limiter atomically reserves read-plus-write byte time and permits -one conversion-group burst. Time controls only background bandwidth and retry -or lease scheduling; it does not control small-write pipeline scale-in/out. - -`ConversionMetrics` exposes attempts started/completed/failed, bytes read and -written, retired mirror segments, active handlers, and peak active handlers. -`GET /conversion_metrics` returns a nonblocking snapshot. - -`RepairConfig` defaults to enabled, four concurrent handlers, a 64-MiB shared -full-shard memory budget, and a one-second marker scan. The memory semaphore is -reserved before shard reads. A repair larger than the configured budget stays -durably retryable. `RepairMetrics` exposes admissions, attempts, repaired -segments and bytes, active handlers, and current/limit/peak memory through -`GET /repair_metrics`. - -## 8. Test Contract - -Core coverage uses real KV, chunkdb, diskdb, DiskIO, and routed clients. It -verifies foreground no-reread conversion, data and parity bytes, missing-shard -decode, active-prefix append, client-I/O failure takeover, manual and automatic -triggers, multiple groups plus mirror tail, bounded task dispatch, cleanup -reconciliation, and process crash/restart recovery. Mock-based tests remain -auxiliary for deterministic error injection and unit-level queue behavior. - -Reservation coverage verifies hidden reserve/consume state, idempotent -confirmation and cancellation, eight joint mirror candidate sets plus parity, -every early tail from one through seven, and reconstruction from the published -survivor/parity layout. - -Read-repair coverage additionally verifies same-offset partial EC recovery, -minimum surviving-shard reads, mirror fallback and total loss, exact partial -ranges, stream termination, incomplete-parity behavior, durable marker -admission across process restart, full-shard replacement, task revival, -memory/metrics bounds, and continued writes to an active chunk whose revision -was advanced by a reader. diff --git a/doc/design/chunkdb/design-crowdb-chunkdb-range-binding.md b/doc/design/chunkdb/design-crowdb-chunkdb-range-binding.md deleted file mode 100644 index dc32b404e..000000000 --- a/doc/design/chunkdb/design-crowdb-chunkdb-range-binding.md +++ /dev/null @@ -1,306 +0,0 @@ - - - -# CROWDB - Design: ChunkDB Slot Routing - -Depends on: [ChunkDB architecture](design-crowdb-chunkdb.md), -[KV group 0](../kv/design-crowdb-kv-group0.md). -Satisfies: ChunkDB stateless execution and explicit nonzero persistence. - -ChunkDB uses independent service and storage maps over the same fixed hash -space. This document covers their publication, request routing, task authority, -and fixed/dynamic service ownership policies. - -## Table of Contents - -- [1. Slot identity and invariants](#1-slot-identity-and-invariants) -- [2. Control-plane records](#2-control-plane-records) -- [3. Publication and startup](#3-publication-and-startup) -- [4. Client and storage routing](#4-client-and-storage-routing) -- [5. Maintenance authority](#5-maintenance-authority) -- [6. Costs and operating limits](#6-costs-and-operating-limits) -- [7. Data-group execution fences](#7-data-group-execution-fences) -- [8. Durable handoff records](#8-durable-handoff-records) -- [9. Submission epoch policy](#9-submission-epoch-policy) - -## 1. Slot identity and invariants - -`ChunkSlot::for_chunk` hashes the canonical 16-byte big-endian ChunkId with -XXH64 seed zero and takes modulo 1024. Slots are 0..1023. The hash includes -both ID words and the concrete purpose prefix. Neither server count nor group -count participates in this calculation. - -- **I1 — Independent owners:** each slot has one service instance and one - selected nonzero `(store_id, group_id)` destination. A service bitmap can - span several storage groups, and a storage group can serve several instances. -- **I2 — Complete publication:** each map covers all slots exactly once, with - no overlaps, missing owners, malformed bitmaps, or mixed generations. -- **I3 — Chunk-local atomicity:** chunk, canonical task, maintenance indexes, - and reservation records route by the owning ChunkId. Their byte-key order - never selects a group. Conditional multi-record writes remain group-local. -- **I4 — Fail closed:** missing routes, unavailable selected groups, invalid - types, and zero-slot service owners cannot become group-0 or any-owner fallbacks. -- **I5 — Fixed authority:** under fixed policy, refreshing an initialized map may not change its - generation or ownership. Endpoint changes do not change either slot map. - -These are hash partitions, distinct from chunk-kv ordered key ranges. The -maps do not place payload bytes in KV; payload remains on DiskIO/DiskDB. - -## 2. Control-plane records - -Group 0 contains two independent map namespaces: - -- `/chunkdb/slot_service/`: one service binding per instance, - including an all-zero bitmap for an instance owning no slots. -- `/chunkdb/slot_storage//`: one storage binding per - explicitly selected nonzero group. -- `/chunkdb/slot_head/service` and `/chunkdb/slot_head/storage`: independent - heads containing layout version, generation, and owner count. - -Each binding contains owner identity, generation, and a 1024-bit bitmap. -JSON encoding adds metadata overhead; the raw bitmap is exactly 128 bytes. -These routing maps have no per-slot or per-contiguous-range records. Service -endpoints are registered separately and can refresh after same-instance restart. - -## 3. Publication and startup - -`ChunkSlotMapClient` validates the full bootstrap and publishes each map's -head and owner records in one head-conditional group-0 batch. A retry reads and -reconciles the committed result; a conflicting initializer fails. Both maps -must exist before ChunkDB accepts allocations, even though the maps have -independent publication boundaries. - -Map reads use fixed-cutoff pagination and recheck the head revision before -publishing a validated immutable cache. Legacy range bindings, orphan records, -and existing group-0 chunk/task/reservation state prevent unsupported bootstrap. -They are preserved for an explicit conversion rather than silently rewritten. - -Startup requires an explicit instance identity and validated storage/service -maps. Optional bootstrap configuration supplies the eligible groups and -service instances. The deterministic bootstrap uses balanced service slot bands -and interleaves slots among storage groups; this assignment is initial policy, -not the definition of a slot. Readiness requires initialized maps and valid -storage routing. An instance with zero service slots is ready but rejects work -outside its empty bitmap; local disks are not a service-readiness requirement. -HTTP readiness reads the independently sampled ownership result without issuing -Group-0 requests. One worker refreshes the complete map every second, with a -three-second read deadline and no overlapping refreshes. Evidence ages from the -start of that authority read. Pending, failed or older-than-three-second evidence -returns unavailable; HTTP readers cannot renew it. This observation does not -replace per-slot execution admission or weaken map validation. -The fixed-layout monitor audits ownership and does not reassign -slots when a heartbeat expires. Legacy assignment writers reject updates. - -Console automatic service plans deploy the selected ChunkDB cohort before -starting Chunk-KV journal bootstrap. A serial deployment must not await journal -allocation while another published ChunkDB owner still needs to start. - -## 4. Client and storage routing - -Ordinary Rust clients load only the service map and service discovery. -`RangeBindingClient` retains an immutable map and endpoint snapshot. Existing -chunk operations resolve the ChunkId slot; generated allocation selects a live, -nonempty service owner, whose server generates an ID within its owned bitmap -using bounded retries. Caller-supplied IDs are checked before side effects. - -`RangeGuard` admits only the installed instance bitmap. `BindingCache` builds a -constant-time storage lookup from the independent storage map. Neither hot -routing path introduces a mutex. Cold restart loads durable records from remote -KV; local locks, caches, and in-flight operations are not persistence authority. - -Native tree clients use a retained service resolver with asynchronously refreshed -immutable snapshots. Per-call route leases retain old connections through -completion, and the page store retains resolver/DiskIO owners for its lifetime. -Native RPC completion pools are separate from ordinary Rust RPC pools. -Explicit `NotMyRange` permits bounded refresh and retry. Unknown allocation or -mutation outcomes return to the caller without blind resubmission; read-only -queries can refresh and retry. No original-owner fallback is supported. - -Readiness sends real point queries through the initialized service route without -first running the ordinary periodic Group0 registry/map refresh. Cold discovery -still requires a validated map; ownership rejection or transport failure retains -the existing read-only refresh/retry path and caller deadline. The receiving -owner validates admission and reads its actual KV storage. This keeps a bounded -probe independent of an unrelated full registry refresh without caching a -successful result or changing ordinary requests' freshness rules. - -The owner's KV transport isolates Get responses from background scans with one -additional connection per endpoint and the same RPC workers and completion -budget. The point request still reads current storage and retains its original -read mode, error handling and deadline. A partially received catalog page must -not hold the receiving owner's point-read response behind that page's bytes. - -## 5. Maintenance authority - -One process hosts separate system and user-data lifecycle/task runtimes. -Wal, BtreePage, and PageIndex are system purposes; Stream, S3, IcebergTable, -and Dataset are user-data purposes. Each runtime has its own scan scope, claims, queue, -execution capacity, and recovery. Shared transport, codecs, IO, and algorithms -do not create shared task authority. - -Ready, lease, and finalize indexes encode domain and slot before scheduling -fields. Scans select owned slot runs in each eligible destination and paginate -before applying result limits; eligible work is merged by its scheduling order. -Canonical tasks retain the owning chunk ID. Stores reject foreign slots and -domains at read, admission, claim, and publication boundaries. Execution verifies -the durable claim before handler side effects; claim generations and writer -epochs fence delayed completion after same-owner restart. - -Chunk and reservation metadata retain their existing ChunkId-ordered keys. -Scoped scans pass the domain's half-open purpose-byte bounds to KV before -transferring values: system types 1 through 3 and user-data types 4 through 7. -Continuation stays within those bounds; ownership filtering and fixed-cutoff -pagination remain in effect. Growing an unrelated domain must not add pages -to the scoped read. Unscoped inventory retains the complete keyspace. - -The reservation budget is apportioned by service slot share and split equally -between system and user-data runtimes. Execution limits are independently -reserved; shared reservation gauges aggregate domain contributions. Domains can -share a physical Paxos group and therefore still share its availability limits. - -## 6. Costs and operating limits - -For S service owners and G storage destinations, binding records total S + G, -plus two heads. Raw bitmaps total `128 * (S + G)` bytes. With three instances -and three groups this is six owner records and 768 raw bitmap bytes, plus heads -and encoding overhead. Each initial map requires one conditional consensus -batch; allocation does not rewrite a binding record. - -Each loaded map builds a 1024-entry lookup. Hot route lookup is constant-time -and allocation-free. Map reload costs scale with owner records and paginate -in bounded batches. Maintenance scans scale with owned contiguous slot runs -and selected destinations, rather than allocating a worker or tree per slot. -Arbitrary fragmented bitmaps are supported but increase scan requests; balanced -bootstrap bands reduce that cost. Slot balance does not guarantee byte or load -balance when a few chunks dominate traffic. - -Under fixed policy, topology is fixed while serving. Same-owner restart, endpoint refresh, and -Paxos replica failover preserve both maps. Adding/removing service owners or -storage groups, remapping slots, and legacy conversion require an external -fenced handoff/migration protocol and are currently rejected. Service changes -need task-authority handoff; storage changes must move complete chunk/task/ -index/reservation state before ownership cutover. Neither operation implicitly -moves DiskIO payload. Chunk-kv is not a ChunkDB metadata backend; its ordinary -batch API does not supply the required conditional multi-record atomicity. - -## 7. Data-group execution fences - -KV supports a separate execution fence at `/chunkdb/ownership-fence/` -inside the selected data group. The slot segment uses canonical decimal in -0..1023; slot fence operations reject group zero. This namespace is independent -of routing-map records in group 0 and of DiskDB's ownership namespace. -Fixed-policy ChunkDB does not initialize or -use these execution fences. - -`ChunkSlotAuthority` identifies an instance, a nonzero 128-bit process -incarnation and a nonzero per-slot generation. A same-ID process restart has -a distinct incarnation. The canonical comparison value has exactly 33 bytes: -version 1, big-endian instance ID, incarnation bytes and big-endian generation. -Unknown versions, zero identity fields and malformed keys are rejected. A -routing-map generation and a slot-authority generation are separate values. - -`batch_write_owned` compares the applied authority without mutating its fence -record. Business-record revision CAS remains independent and must be supplied -for conditional chunk/task transitions. Ordinary Put, Delete and Batch cannot -mutate either reserved ownership namespace. A conditional ownership batch -may change only the reserved key used as its revision precondition; slot -fences cannot be deleted or replaced by malformed authority values. - -Owner writes use concurrent atomic admission for one fence key and leader -tenure. A fence CAS closes that admission, drains previously admitted work -through apply, and then publishes the changed value. Other slots retain their -own admissions. Cancelling the caller does not release a pending proposal; -an unresolved proposal blocks handover and readiness until the leader's -recovery barrier resolves it. Topology replacement preserves the admission -state, while a new leader tenure requires its own recovery barrier. Delayed -writes retaining the old comparison value fail after fence publication. - -## 8. Durable handoff records - -Group zero stores one service handoff cohort at `/chunkdb/slot_handoff/service`. -`ChunkServiceHandoff` validates its source service generation, unchanged storage -generation, distinct moved slots, previous and target incarnations, selected -data groups and per-slot fence receipts. Target authority advances exactly one -generation; a missing previous authority is reserved for bootstrap without -admitted writers. Transfer and receipt ordering is canonical. - -The phases are Prepare, Fence, Publish and Activate. A cohort is reserved with -revision CAS on the complete service-map head; its unchanged head value and -the prepared record are written atomically. Advancing the head revision keeps -concurrent controllers from reserving different cohorts against the same source. -An unfinished cohort is resumed rather than replaced. - -Data-group fencing requires a persisted Fence-phase snapshot. An already -applied target identity is reconciled using its revision instead of issuing -another CAS. Otherwise the source value must match the recorded previous -identity. Unknown outcomes are reread through the KV recovery barrier, without -choosing another target or epoch. Before saving a new receipt, the client checks -the target authority and applied revision in its selected data group. - -Progress updates CAS the observed handoff revision. They cannot change cohort -identity, remove receipts or return to an earlier phase. Decoding rejects -duplicate slots, conflicting receipts and Publish/Activate records without -complete fence coverage. Publish cannot be saved as a standalone progress -update: it belongs in the same transaction as complete routing and authority -publication. Fixed-policy execution does not initiate handoff or activate -dynamic assignments. - -## 9. Submission epoch policy - -Dynamic service ownership is selected explicitly with `service_ownership = -"dynamic"`; fixed deployments keep their existing policy. Console deployment -uses `dynamic_ownership` to select the policy when initializing a new layout. -An initialized fixed deployment cannot silently switch policy. Storage groups -and DiskIO payload placement are unaffected by service ownership changes. - -- **I6 — Slot-local admission:** each execution captures its slot owner and - ownership epoch. Immediately before each client KV submission, RangeGuard - compares those values against its current immutable snapshot. Updating slot - 10 does not invalidate an execution on unchanged slot 5. Global publication - generation controls snapshot freshness; it is not a slot ownership epoch. -- **I7 — Epoch monotonicity:** owner change, regrant and same-ID restart advance - the affected durable slot epochs. Returning to a previous owner never reuses - an epoch. Incarnation remains diagnostic data in the existing authority - encoding, with no independent admission comparison. It cannot replace durable - epoch advancement. -- **I8 — Submission boundary:** entering client KV accepts that submission. - Accepted work may finish after ownership changes. The race between the final - local check and submission is intentional. No strict exclusion of old writes - after cutover is promised; existing business-record CAS governs conflicts. - -The complete service and authority maps share a publication generation. -Authority bitmaps are keyed by the canonical hexadecimal authority encoding -under `/chunkdb/slot_authority/`; their head is `/chunkdb/slot_head/authority`. -Publication CASes the service head and writes both complete tables and heads -atomically. Readers validate coverage, paired owner IDs and matching generations -and recheck the service-head revision. Epoch rollback and owner changes that -reuse an epoch are rejected. Lost publication replies reconcile the already -published map instead of incrementing epochs again. - -RPC dispatch captures authority before spawning asynchronous execution. Task -claims retain their capture through queueing, handler work, renewals and -completion. Maintenance operations capture the snapshot for their scan/work -iteration. Cache entries carry the captured owner/epoch, so a delayed old fill -cannot supply metadata to an execution under a newer epoch. Submission checks -use local constant-time lookup without network requests, new locks, sleeps or -old-write draining. The data-group fence/handoff APIs described above are -separate primitives and are not used by this policy; DiskDB fencing is preserved. - -An ownership rejection before submission is a routing signal. Clients refresh -the complete service snapshot and reexecute on its owner, including the same -process under a newer epoch. An old execution never adopts a newer epoch while -retaining its old state. The Rust RPC client keeps the same request correlation ID through explicit -ownership retries and refreshes immediately, under one ten-second operation -budget. Business operation IDs and revision CAS remain unchanged. Unknown -submitted transport outcomes return `OutcomeUnknown` for mutation callers to -reconcile; they never enter the ownership retry path. Read-only operations may -refresh and retry. - -The version-3 Group 0 monitor balances compatible prepared services advertised -under `/srv/chunkdb-epoch-v1/`. It uses the persisted descriptor's heartbeat and -failure grace bounds, stable instance-ID tie-breaking, a one-slot count -hysteresis and a 64-slot movement budget per tick. Missing observations retain -a durable first-suspected timestamp across monitor replacement; recovery clears -that timestamp so a later failure gets a fresh grace period. Only the exact -Group 0 leader tenure can publish a complete revision-conditional batch. The -fixed version-2 monitor continues to audit without reassigning slots. diff --git a/doc/design/chunkdb/design-crowdb-chunkdb-rpc.md b/doc/design/chunkdb/design-crowdb-chunkdb-rpc.md deleted file mode 100644 index b8896359f..000000000 --- a/doc/design/chunkdb/design-crowdb-chunkdb-rpc.md +++ /dev/null @@ -1,111 +0,0 @@ - - - -# CROWDB - Design: ChunkdbService RPC - -Depends on: [RPC transport](../rpc/design-crowdb-rpc.md), -[ChunkDB lifecycle](design-crowdb-chunkdb.md). -Satisfies: ChunkDB control-plane transport, routing and mutation fences. - -ChunkDB uses the crowdb-rpc FlatBuffer transport for lifecycle, placement, -reservation, replacement, conversion and repair operations. Clients and services -use one current schema. Unknown messages and malformed requests fail explicitly. - -## Table of Contents - -- [1. Transport and routing](#1-transport-and-routing) -- [2. Wire records](#2-wire-records) -- [3. Mutation contracts](#3-mutation-contracts) -- [4. Dispatch and responses](#4-dispatch-and-responses) -- [5. Errors and retry](#5-errors-and-retry) - -## 1. Transport and routing - -- `ChunkdbClient` requires an `Arc` at construction. - `RangeBindingClient` resolves a chunk's service-slot owner from Group 0. - `ServiceRegistryClient` supplies live endpoints; endpoint cache refresh does - not change ownership. -- `ChunkdbRpcTransport` uses an immutable `ConnectionPoolIndex`. Cold connection - creation happens outside the index, followed by compare-and-swap installation. - Retry invalidates only the selected connection generation. -- `ChunkdbRpcService` dispatches to the independently scoped system/user-data - lifecycle and task runtimes. Each mutation checks current range authority. -- The RPC listener defaults to port 9961; HTTP health and metrics default to - 9972. The configured RPC address must parse as a socket address. - -## 2. Wire records - -- Requests start with `id` and `rpc_create_nano`, followed by operation fields. - Responses carry request identity, `ret_code`, optional `error_msg`, diagnostic - `range_start`/`range_end`, and typed result data. -- Message identities are defined by `FBMsgType`. Generated Rust/C++ definitions - derive from the same schema; native producers use the same request contract. -- `FBChunkType` requires explicit Wal, BtreePage, PageIndex, Stream, S3 or - IcebergTable purpose. Reserved zero is invalid. Chunk-ID purpose, stored type - and owner attribution must agree. Stream/Wal carry their logical Stream name; - other types have an empty owner key. -- `FBChunk` includes writer epoch, exact acknowledged byte cursor, lease deadline, - physical strips, coarse capacity/sealed geometry, cleanup intents and the last - replacement operation. `FBChunkStrip` identifies sequence, logical interval, - unit geometry, mirror/EC segments and unavailable segment identities. -- Segment identity includes disk, zone, unit range, allocation incarnation and - owning chunk. Mirror and EC bodies carry their explicit physical geometry. -- `QueryChunkResponse` also returns reader-layout validity duration. - `ListChunksResponse` carries a bounded chunk page and continuation identity. -- Response `Ref` wrappers parse one root and expose typed borrowed accessors. - Invalid buffers remain invalid; callers check validity and return codes before - accepting data. - -## 3. Mutation contracts - -- Allocation specifies purpose, writer epoch, lease and applicable owner key. - The active chunk and its finalization task are created atomically. Allocation - retries retain operation identity rather than create unrelated chunks. -- Append and strip reservation extend physical layout under their current fences. - Reservations remain hidden until confirmation. A consumed reservation prevents - conflicting seal transitions. -- `AdvanceChunkWrite` carries expected metadata revision, writer epoch, exact - byte cursor and optional closed-strip sequence. It cannot move progress backward - or beyond capacity. -- `SealChunkRequest.seal_bytes` is the exact effective byte boundary, including - zero for an empty seal. KiB geometry does not authorize extra readable bytes. - Native clients validate the returned identity, epoch, exact cursor and state. -- `ReplaceChunkStripRange` carries expected revision, exact old strips, replacement - strips and a stable operation ID. The lifecycle validates geometry and ownership, - commits new segments before publication, and retains retired segments through - cleanup grace. One-strip changes use this same fenced range contract. -- Replacement allocation and discard manage tentative segments. Conversion and - repair publish through their fenced lifecycle transitions and durable tasks. -- Delete publishes a tombstone with durable cleanup work before freeing blocks. - Whole-chunk retries are idempotent. `DeleteChunkRange` returns `Unimplemented` - until shared-object range reclamation is supported; overlap alone cannot free - a strip containing other live objects. - -## 4. Dispatch and responses - -- Each registered message has a request metric guard and typed handler. Dispatch - validates its FlatBuffer and required identities before spawning lifecycle work. -- Handlers execute on the service runtime and retain captured ownership checks - around external operations. Responses are submitted through the RPC server. -- Successful responses contain current typed state. Lifecycle errors map to a - typed return code; `NotMyRange` includes routing diagnostics for the client. -- Layout replacement, conversion and relocation share the `layout_mutation` - request metric family. Allocate, append, reservation, cursor, query, seal, - deletion, listing and ad-hoc recovery retain their own families. -- Startup validates topology and service/storage slot maps, constructs separate - domain runtimes, registers RPC handlers and starts discovery refresh plus HTTP - observability. Shutdown stops background work and transport through the runtime. - -## 5. Errors and retry - -- Invalid arguments and strip indices map to `InvalidArgument` or - `StripIndexOutOfRange`. Missing/existing chunks map to `NotFound`/`AlreadyExists`. -- Invalid lifecycle transitions map to `FailedPrecondition`; revision/operation - conflicts map to `Aborted`. Storage/allocation failures map to `Internal`. -- Lock admission failures map to `Unavailable`. Ownership failures return - `NotMyRange`, causing binding refresh and routing to the current owner. -- Client retries are bounded and apply only to transient failures. Typed fence - failures and malformed responses do not become successes. Mutations that may - have completed retain their operation identity and reconcile durable state. -- A transport response alone does not establish Stream writer ownership or a - synchronized DiskIO write; those contracts belong to their respective layers. diff --git a/doc/design/chunkdb/design-crowdb-chunkdb.md b/doc/design/chunkdb/design-crowdb-chunkdb.md deleted file mode 100644 index f8d04e00c..000000000 --- a/doc/design/chunkdb/design-crowdb-chunkdb.md +++ /dev/null @@ -1,1294 +0,0 @@ - - - -# CROWDB - Design: chunkdb (Overview) - -This is the root design document for the chunkdb component area. It -defines **what chunkdb is**, **why key choices were made**, and **how the -component is structured**. Field-level details live in the proto files -and Rust source; this doc covers decisions and architecture only. - ---- - -## Table of Contents - -- [1. Overview](#1-overview) -- [2. Non-Goals (Design Envelope)](#2-non-goals-design-envelope) -- [3. Key Design Decisions](#3-key-design-decisions) - - [3.1 Chunk as logical abstraction over disk blocks](#31-chunk-as-logical-abstraction-over-disk-blocks) - - [3.2 Strip as atomic redundancy unit](#32-strip-as-atomic-redundancy-unit) - - [3.3 Rack/node-aware placement for fault tolerance](#33-racknode-aware-placement-for-fault-tolerance) - - [3.4 Topology cache with group-0 integration](#34-topology-cache-with-group-0-integration) -- [3.4a Watch/notify for real-time updates](#34a-watchnotify-for-real-time-updates) - - [3.5 EC at strip level via isa-l (crowdb-common module)](#35-ec-at-strip-level-via-isa-l-crowdb-common-module) - - [3.6 Stateless with KV persistence](#36-stateless-with-kv-persistence) - - [3.7 Common protocol crate; crowdb-rpc](#37-common-protocol-crate-crowdb-rpc) - - [3.8 Proto types used directly; no Rust type duplication](#38-proto-types-used-directly-no-rust-type-duplication) - - [3.9 Chunk types for different use cases](#39-chunk-types-for-different-use-cases) -- [4. Architecture Overview](#4-architecture-overview) -- [5. Data Model](#5-data-model) - - [5.1 Disk block](#51-disk-block) - - [5.2 Strip](#52-strip) - - [5.3 Chunk](#53-chunk) - - [5.4 Chunk ID structure](#54-chunk-id-structure) - - [5.4a Logical hash bucket system](#54a-logical-hash-bucket-system) - - [5.4b Request handling during migration](#54b-request-handling-during-migration) - - [5.5 Chunk types](#55-chunk-types) -- [6. Topology Management](#6-topology-management) -- [7. Placement Strategy](#7-placement-strategy) - - [7.1 Mirror placement](#71-mirror-placement) - - [7.2 EC placement](#72-ec-placement) - - [7.3 Physical validation and degraded-placement repair](#73-physical-validation-and-degraded-placement-repair) - - [7.4 Active cross-domain rebalance](#74-active-cross-domain-rebalance) -- [8. Allocation Flow](#8-allocation-flow) -- [9. Chunk Lifecycle](#9-chunk-lifecycle) -- [10. Per-Chunk-ID Lifecycle Lock + Chunk Cache](#10-per-chunk-id-lifecycle-lock--chunk-cache) - - [10.1 ChunkLockMap](#101-chunklockmap) - - [10.2 Lock policy + cache hint](#102-lock-policy--cache-hint) - - [10.3 ChunkGuard](#103-chunkguard) - - [10.4 Chunk cache](#104-chunk-cache) - - [10.5 Sweep task](#105-sweep-task) - - [10.6 LifecycleHandler integration](#106-lifecyclehandler-integration) - - [10.7 Error variants + service mapping](#107-error-variants--service-mapping) - - [10.8 Metrics + HTTP endpoints](#108-metrics--http-endpoints) - - [10.9 Edge cases](#109-edge-cases) -- [11. EC Encoding/Decoding](#11-ec-encodingdecoding) -- [12. Crate Layout](#12-crate-layout) -- [13. Concurrency Model](#13-concurrency-model) -- [14. Configuration](#14-configuration) -- [15. Implementation Scope](#15-implementation-scope) -- [16. References](#16-references) - -## 1. Overview - -chunkdb is a **distributed chunk metadata service** that runs on top of -CROWDB's KV cluster and diskdb. It is a **lightweight, stateless server**: -a chunkdb instance manages chunk metadata, orchestrates block allocation -from diskdb, and persists all durable state to CROWDB KV. It holds no -state that cannot be reconstructed from KV — on crash or restart it -rebuilds in-memory structures from the KV records and group-0 metadata. - -Multiple chunkdb instances run across a cluster for fast chunk -allocation and lifecycle management. chunkdb does rack/node-aware -placement for fault tolerance, supports both mirror and EC strip types, -and manages chunk state transitions (allocate → seal → delete). All -state changes are durably persisted to CROWDB KV before being acknowledged -to callers. - -chunkdb manages chunk metadata and orchestrates background maintenance I/O -through DiskIO clients. Foreground callers write allocated blocks themselves -and tell chunkdb when chunks are sealed or deleted. - -ChunkDB conversion and EC repair share one `ConversionDiskIo` policy adapter -over the routed semantic `crowdb-diskio-client`. The adapter selects the -priority lane and expresses reads, buffered writes, and fsync barriers against -checked segment targets. It does not parse DiskIO endpoints, join topology, -construct RPC servers or connections, refresh connection vectors, or decode -wire return codes. Those responsibilities and their immutable generations are -owned by the DiskIO client; ChunkDB retains conversion/reconstruction policy -and decides when a completed write group requires a durability barrier. - -**Language:** Rust. **Runtime:** tokio (async everywhere). - -**Core goals:** -- **Fast allocation** — rack/node-aware placement with parallel block - allocation via diskdb, rollback on partial failure. -- **Fault tolerance** — mirror strips across distinct racks, EC strips - with rack-aware placement, configurable safe/unsafe modes. -- **Durability via CROWDB KV** — the paxos journal is the sole durable - store; chunkdb has no local WAL and is stateless on disk. -- **Crash safety via KV replay** — chunk metadata in KV is the source of - truth; in-memory structures (topology cache, allocator state) are rebuilt - on restart. -- **Flexible EC** — support multiple EC schemes simultaneously, selected - per-strip based on cluster size and fault tolerance requirements. - -**Design philosophy:** "chunkdb is a thin, stateless client of crowdb-kv and -diskdb." All consensus, replication, and durability are delegated to -crowdb-kv; block allocation is delegated to diskdb. chunkdb's job is chunk -lifecycle management, placement policy, and EC coordination, nothing more. - -## 2. Non-Goals (Design Envelope) - -- **No local block engine.** chunkdb does not own devices or implement block - reads and writes. Maintenance handlers route I/O through DiskIO services. -- **No local WAL.** CROWDB KV's WAL is the sole durability mechanism. -- **No consensus code.** chunkdb is a client of crowdb-kv; all interaction - uses the existing crowdb-kv API. -- **No GC operations in v1.** Garbage collection (reclaim, collapse, merge) - is deferred to a future requirement. Chunks are allocated and deleted as - whole units in v1. -- **No EC strip type restrictions.** Chunk type (Wal/BtreePage/PageIndex/Stream/S3/IcebergTable) - is independent of strip type (mirror/EC). Any chunk can use either strip type - based on configuration. v1 supports both mirror and EC strips for all chunk types. -- **No native SMR / zoned-namespace SSD support (v1).** Start with - conventional HDD/SSD via diskdb; advanced disk features are deferred. - -## 3. Key Design Decisions - -### 3.1 Chunk as logical abstraction over disk blocks - -A **chunk** is a logical container for strips, which in turn contain -physical disk blocks. This abstraction layer allows upper-layer services -to work with chunks without worrying about physical placement details. -chunkdb handles the mapping from logical chunk offsets to physical disk -blocks across multiple nodes and racks. - -**Rationale:** Separates logical data management from physical storage -layout. Upper layers (object store, file system) work with chunks; chunkdb -handles the complexity of block allocation, placement, and fault tolerance. - -### 3.2 Strip as atomic redundancy unit - -A **strip** is the atomic unit of redundancy. Every strip is either a -**mirror strip** (replicated across multiple nodes) or an **EC strip** -(erasure-coded with data + parity blocks). Strip-level operations -(allocation, EC encoding, recovery) are atomic. A strip is either fully -allocated or not at all. - -**Rationale:** Simplifies recovery and placement logic. When a disk fails, -recovery operates at strip level (rebuild lost replica or EC parity), -not at individual block level. - -### 3.3 Rack/node-aware placement for fault tolerance - -Placement decisions are rack-aware and node-aware to survive rack failures -and node failures. Mirror strips place replicas on distinct racks when -possible. EC strips distribute data+parity blocks across multiple racks, -with per-node block limits to ensure no single node failure exceeds the -code_num threshold. - -**Rationale:** Rack failures are more common than individual disk failures. -Rack-aware placement ensures the system can survive rack-level failures -without data loss. - -### 3.4 Topology cache with group-0 integration - -chunkdb maintains a **TopologyCache** with site/rack/node/disk-group -hierarchy, refreshed periodically from group-0 via `HardwareClient`. This -cache is used for placement decisions and is point-in-time consistent for -each allocation operation. - -**Rationale:** Placement decisions need up-to-date cluster topology. -Querying group-0 for every allocation would be too slow. Caching with -periodic refresh provides a good balance between freshness and performance. - -### 3.4a Watch/notify for real-time updates - -chunkdb uses crowdb-kv's watch/notify mechanism for real-time topology updates -in addition to periodic refresh: - -- **Watch registration**: chunkdb registers for group-0 watch on disk-group - and node status keys via the crowdb-kv watch/notify client. -- **Immediate updates**: A notification coalesces into a complete group-0 - topology rebuild. The immutable snapshot is published with one `ArcSwap` - operation, so allocation never observes a mixture of generations. -- **Fallback mechanism**: Periodic refresh (30s default) continues as a fallback - for missed notifications and consistency verification. -- **Placement impact**: Real-time updates enable placement decisions to reflect - current cluster state, avoiding placement on failed/degraded resources. - -**Rationale**: Watch/notify provides sub-second responsiveness to critical -cluster state changes (disk failures, maintenance) while periodic refresh -ensures eventual consistency and recovery from missed notifications. - -### 3.5 EC at strip level via isa-l (crowdb-common module) - -Erasure coding is performed at the strip level using the **isa-l** library -(Intel Storage Acceleration Library). A Rust FFI wrapper in `crowdb-common` -provides safe interfaces to isa-l's encode/decode functions. EC encoding -happens when strips are allocated; EC decoding happens during recovery. - -**Rationale:** isa-l is the industry standard for high-performance EC. -Strip-level EC aligns with the strip-as-atomic-redundancy-unit design. -The FFI wrapper isolates unsafe C code from the rest of the Rust codebase. - -### 3.6 Stateless with KV persistence - -chunkdb is stateless on disk. All chunk metadata and maintenance state is persisted to selected nonzero -direct Paxos KV groups by ChunkId slot. System and user-data lifecycle/task -runtimes have separate authority, queues, and execution capacity. -On restart, chunkdb rebuilds in-memory structures (topology cache, -allocator state) by replaying KV records and fetching fresh topology from -group-0. - -**Rationale:** Simplifies operations (no local data to manage), enables -crash recovery without local WAL, and relies on CROWDB KV's durability -guarantees. Restart under the same logical instance reloads durable state. -Changing ownership requires a fenced handoff; adding storage groups requires -metadata migration. - -### 3.7 Common protocol crate; crowdb-rpc - -Protocol definitions live in `lib/crowdb-protocol/src/fbs/chunkdb_*.fbs` -and are shared between chunkdb server and chunkdb client. Communication -uses crowdb-rpc (flatbuffer transport), the same engine as the KV -consensus hot path. See sub-design -[`design-crowdb-chunkdb-rpc.md`](design-crowdb-chunkdb-rpc.md). - -**Rationale:** Single source of truth for protocol definitions. crowdb-rpc -uses the same engine as the KV consensus hot path for lower latency. - -### 3.8 Proto types used directly; no Rust type duplication - -Rust code uses the proto-generated types directly from `crowdb-protocol`. -No separate Rust struct definitions mirror the proto types. Conversion -between proto and domain types happens only at boundaries (e.g., KV -persistence). - -**Rationale:** Reduces duplication and drift. Proto is the canonical -definition; Rust code works with proto types directly. - -### 3.9 Chunk types for different use cases - -Six concrete chunk types are supported. Repo chunk is the collective -discussion term for user data, not an implementation type: -- **WAL chunk**: Write-ahead log entries. -- **BTree page chunk**: B-tree page storage for the crowdb-tree engine. -- **Page index chunk**: Persistent tree page mapping payload. -- **Stream chunk**: Non-system business stream data. -- **S3 chunk**: S3 object data. -- **Iceberg table chunk**: Iceberg immutable file data. - -**Rationale:** Different storage components have different redundancy and -performance requirements. Chunk types allow optimization for each component's -needs. Chunk type is independent of strip type. Any chunk type can use -either mirror or EC strips based on configuration and access patterns. - -## 4. Architecture Overview - -``` -┌─────────────────────────────────────────────────────────────────┐ -│ chunkdb Server │ -├─────────────────────────────────────────────────────────────────┤ -│ crowdb-rpc Service (ChunkdbService) │ -│ ├── AllocateChunk │ -│ ├── SealChunk │ -│ ├── DeleteChunk │ -│ ├── QueryChunk │ -│ └── ListChunk │ -├─────────────────────────────────────────────────────────────────┤ -│ Lifecycle Layer │ -│ ├── allocate_chunk() │ -│ ├── seal_chunk() │ -│ └── delete_chunk() │ -├─────────────────────────────────────────────────────────────────┤ -│ Allocation Layer │ -│ ├── ChunkAllocator (orchestrate strip allocation) │ -│ ├── MirrorStripAllocator │ -│ └── ECStripAllocator │ -├─────────────────────────────────────────────────────────────────┤ -│ Placement Layer │ -│ ├── RackNodeSelector │ -│ ├── MirrorPlacement (distinct racks) │ -│ └── ECPlacement (rack-aware, safe/unsafe) │ -├─────────────────────────────────────────────────────────────────┤ -│ Topology Layer │ -│ ├── TopologyCache (site/rack/node/disk-group) │ -│ ├── TopologyRefresh (periodic from group-0) │ -│ └── WatchNotify (real-time updates from group-0) │ -├─────────────────────────────────────────────────────────────────┤ -│ Storage Layer │ -│ ├── KV Persistence (chunk metadata) │ -│ └── DiskdbClient Pool (block allocation/free) │ -├─────────────────────────────────────────────────────────────────┤ -│ EC Layer │ -│ └── crowdb-common EC (isa-l FFI wrapper) │ -└─────────────────────────────────────────────────────────────────┘ - │ │ │ - ▼ ▼ ▼ - ┌─────────┐ ┌─────────┐ ┌─────────┐ - │ crowdb-kv │ │ diskdb │ │ group-0 │ - │ (KV) │ │ │ │Hardware │ - └─────────┘ └─────────┘ └─────────┘ -``` - -**Key components:** - -- **crowdb-rpc Service**: External API for chunk operations. -- **Lifecycle Layer**: Implements chunk state machine transitions. -- **Allocation Layer**: Orchestrates strip allocation with rollback on failure. -- **Placement Layer**: Rack/node-aware placement algorithms. -- **Topology Layer**: Caches and refreshes cluster topology from group-0. -- **Storage Layer**: KV persistence for chunk metadata, diskdb client pool. -- **EC Layer**: isa-l wrapper for strip-level EC operations. - -## 5. Data Model - -### 5.1 Disk block - -A **disk block** is the smallest fixed-size physical storage unit managed -by diskdb. The default size is **1 MB** (configurable, range 1 MB – 2 MB). -Disk blocks are identified by `Segment { node_id, disk_id, zone_index, -zone_offset, size, tag }` (from diskdb proto). - -### 5.2 Strip - -Current Strip metadata has one physical `unit_kb` shared by all fragments. -Chunk placement therefore requires a uniform allocation unit across active -disks. Independent DiskDB groups may use different units, but Console rejects -ChunkDB/Chunk-KV deployment against mixed active units before spawning a -process. Runtime topology records incompatible geometry and rejects normal, -replacement and conversion allocation before any DiskDB fragment request. -It must not silently reuse the first disk's unit for other groups. - -A **strip** is the atomic redundancy unit. Two strip types: - -**Mirror Strip**: A configured number of disk allocation units replicated -across N nodes (configurable copy count, production default 2). Each segment -is a full copy on a different node. Data capacity = unit_count × unit_size. - -**EC Strip**: `data_num` data segments + `code_num` parity segments, -distributed across nodes under the failure-domain placement rule. Data -capacity = `data_num × unit_count × unit_size`. For example: -- 6+3 EC with 1 MB blocks → 6 MB data capacity, 9 MB total. -- 8+4 EC with 1 MB blocks → 8 MB data capacity, 12 MB total. - -Each strip tracks: -- **Usage bitmap**: Per-strip bitmap tracking which sub-block ranges are - written (1 bit per granularity, e.g., 16 KB). -- **EC state** (EC strips only): `NotStarted` → `Complete`. -- **Block state**: Per-block health (Good, Suspicious, Bad). -- **Placement intent and assessment**: The selected `rack_first` or - `node_first` priority and the creation-time maximum fragment count for each - rack, node, and physical disk. The assessment records whether each domain - satisfies the strip loss budget. EC strips that violate the node or disk - loss budget carry a durable placement-repair marker. Rack diversity is a - preference: missing rack protection alone does not require repair. Existing - markers are cleared after physical reassessment confirms node and disk - protection; the rack-protection assessment remains truthful. - -### 5.3 Chunk - -A **chunk** is a container for strips. Chunk properties: -- **128-bit ID**: Chunk type (8 bits) + Timestamp (48 bits) + Randomness (72 bits). -- **State**: `Init` → `Active` → `Sealed` → `Deleted`. -- **Type**: Wal, BtreePage, PageIndex, Stream, S3, or IcebergTable. - The ID's high-byte prefix and the stored type must agree. Retired type-0 - records are rejected; legacy conversion is unsupported. -- **Capacity**: Total data capacity across all strips. -- **Write granularity**: Minimum write alignment (e.g., 4 KB). -- **Strips**: Ordered list of strips (mirror or EC). -- **Next strip sequence**: Monotonic identity consumed by append and never - derived from the current vector length. -- **Cleanup intents**: Retired segment sets and their earliest safe reuse time. -- **Last strip replacement**: Stable operation identity for idempotent retry. -- **Logical-to-physical mapping**: Encoded offset arrays for GC (deferred). - -Each strip may also identify replicas known unavailable. Readers can avoid -those segment identities while background recovery is pending. - -Chunk size is variable, determined by the total size of its constituent -strips (1 MB – 4 GB range). - -### 5.4 Chunk ID structure - -The 128-bit chunk ID uses a custom format optimized for chunkdb: - -| Bits | Width | Field | Description | -|---------|--------|----------|--------------------------------------| -| 0–7 | 8 bits | Chunk Type | Chunk type identifier (0-255) | -| 8–55 | 48 bits| Timestamp | Milliseconds since Unix epoch | -| 56–127 | 72 bits| Randomness | Cryptographically secure random | - -**Layout benefits:** -- **Chunk type prefix**: First byte enables efficient filtering and scanning by chunk type -- **Time ordering**: 48-bit timestamp provides natural ordering for sequential operations -- **Global uniqueness**: 48-bit timestamp + 72-bit randomness provides sufficient uniqueness -- **Compact size**: 128 bits total, same as standard UUID - -**Generation**: Chunk IDs are generated using `getrandom` for cryptographically secure randomness combined with system timestamp. The generation function will be added to `crowdb-common` for reuse across components. - -The first byte has an explicit supported purpose (see §5.5). Value zero is -retired and cannot be allocated or silently reclassified. - -### 5.4a Logical Hash Bucket System - -The fixed logical hash space contains 1024 slots. XXH64 seed zero hashes the -canonical big-endian 128-bit ChunkId, modulo 1024. Group/server counts never -change this calculation. Group 0 holds one 128-byte bitmap per service instance -and one per selected nonzero storage group, under independent generation heads. - -Clients resolve slots to ChunkDB instances; ChunkDB independently resolves -slots to direct Paxos KV groups. All canonical chunk/task/index/reservation -records follow the owning ChunkId route and preserve group-local conditional -atomic writes. Payload remains on DiskIO. A service owner can span groups. - -See [fixed slot routing](design-crowdb-chunkdb-range-binding.md) for publication, -cache costs, task-domain scope, startup validation, and immutable assignments. - -### 5.4b Request Handling During Migration - -Live service handoff, storage-group add/remove/remap, and legacy conversion are -unsupported. Initialized maps reject ownership changes. Same-owner restart and -endpoint refresh preserve authority. A storage ownership change requires complete -chunk/task/index/reservation migration and a fenced cutover before bitmap changes; -a service ownership change requires task-authority handoff without metadata copy. -There is no steady-state group-0 destination or old-owner mutation fallback. - -### 5.5 Chunk types - -| Type | Chunk Type Value | Description | -| ------------- | ---------------- | ------------------------------------ | -| Retired | 0 | Rejected; never reused | -| WAL | 1 | Write-ahead log entries | -| BTree page | 2 | B-tree page storage | -| Page index | 3 | Page index metadata | -| Stream | 4 | Native chunk streams | -| S3 | 5 | S3 object data | -| Iceberg table | 6 | Iceberg immutable file data | -| Reserved | 7-255 | Reserved for future use | - -**Note:** Chunk type is independent of strip type. Any chunk type can use -either mirror or EC strips based on configuration and requirements. - -**Metadata chunks**: Contain only mirror strips for maximum redundancy and -fast access. Used for critical metadata. - -**Repo chunks**: Used for user data storage. Can use either mirror or EC strips -based on configuration and access patterns. - -**WAL chunks**: Used for write-ahead log entries. Typically use mirror strips -for durability and fast recovery, but EC is possible for space optimization. - -**BTree page chunks**: Used for B-tree page storage in the crowdb-tree engine. -Can use mirror or EC strips based on performance vs space trade-offs. - -**Page index chunks**: Used for page index metadata. Typically use mirror strips -for fast lookup, but EC is possible for large indexes. - -## 6. Topology Management - -chunkdb maintains a **TopologyCache** with the cluster hierarchy: - -``` -Site (data center) - └── Rack (physical rack) - └── Node (server) - └── Disk-Group (logical disk group) - └── Disk (physical disk) -``` - -**TopologySnapshot**: Point-in-time immutable snapshot containing: -- `sites`: Map -- `racks`: Map -- `nodes`: Map -- `disk_groups`: Map -- healthy physical-disk locations, allocatable capacity observations, and a - monotonic publication generation. - -Located DiskIO registrations overlay hardware intent with current placement -availability. A configured disk group is eligible only while a matching live -process reports it opened. Retained expired registrations exclude stopped -physical owners; expiry is evaluated at selection time even without a refresh. -This does not change durable hardware status or relax failure-domain constraints. - -Usage observations include only allocatable disks and carry their sample time. -Missing or stale observations remove capacity ranking information; they never -relax a rack, node, or disk safety constraint. Per-disk-group in-flight bytes -are atomic reservations shared by snapshot generations, so concurrent -allocations account for projected utilization without a placement lock. - -**TopologyRefresh**: Hybrid approach combining periodic refresh with watch/notify: -- **Periodic refresh**: Background task that periodically (configurable, - default 30s) fetches full topology from group-0 via `HardwareClient` for - consistency and missed notification recovery. -- **Watch/notify integration**: Register for group-0 watch on disk-group and - node status keys via crowdb-kv watch/notify client. Notifications coalesce - into a complete refresh; no point mutation is published. -- Fetch all nodes with status and rack assignment -- Fetch all disk-groups with capacity and node assignment -- Fetch all racks with site assignment -- Atomically update the cache - -**Rationale**: Placement decisions need consistent topology. Point-in-time -snapshots ensure a single allocation operation sees a consistent view. -Periodic refresh balances freshness and performance. - -## 7. Placement Strategy - -Placement has two independent parts: safety and ranking. Safety first limits -how many fragments one failure domain may contain. Ranking then compares -projected usable utilization, `(used + in_flight + planned) / capacity`, only -among candidates that meet that safety constraint. Equal scores use stable -topology identifiers, making retries deterministic. - -`deployment.mode` is `production` or `test_single_node` in release builds. Production startup -requires at least three distinct voting nodes for every KV group and protected -placement, with `deployment.max_node_failures = 1`. Test-single-node startup -requires one voting node per group and `deployment.max_node_failures = 0`, with -explicit colocated placement. The mode never changes in response to topology -loss. In test-single-node mode, new strips must be one-copy 1 MiB mirrors; -EC, extra copies, and mirror-to-EC conversion are rejected. Production rejects -new mirror strips with fewer than two copies. The production profile normally -places two mirror copies on distinct nodes, including for journal and tree-page -data. - -Debug builds also accept `test_unsafe_placement` for legacy colocated EC -integration fixtures. Release builds reject it during configuration loading; -it is separate from the single-node deployment profile. - -`placement.mode` selects one placement strategy at process construction. The -allocator depends on the `ChunkPlacementStrategy` interface and does not branch -on the mode while allocating, converting, repairing, or deciding whether a -degraded disk result may be published. Each mode is a separate strategy type: - -- `protected` uses failure-domain-aware mirror and EC selectors. The granular - degraded-placement settings below remain available only within this mode. -- `unsafe_colocated` selects one healthy disk group for the explicit - test-single-node deployment. This deployment uses one mirror copy and has - no data protection; a read or write error reaches the caller. - -The mode is an explicit deployment property, not an automatic fallback. A -protected deployment never changes to `unsafe_colocated` because topology is -small or unavailable. With two healthy nodes remaining, a new EC strip retains -its requested data and parity geometry. Its fragments may span both survivors -with a degraded-placement marker and a durable repair task. A new mirror strip -still places two copies across the survivors. New placement -policies are added as strategy implementations and selected at the composition -root, keeping policy branches out of the allocation hot path. - -For an EC `data_num + code_num` strip, a protected rack, node, or physical -disk contains at most `code_num` fragments. For a mirror strip, losing a -protected domain leaves at least one copy. A single-copy mirror is explicitly -non-redundant and remains allocatable while its assessment reports no domain -protection. - -`placement.failure_domain_priority` selects `rack_first` (the default) or -`node_first`. It orders safe candidates lexicographically; it does not change -the node/disk recovery limits. Rack diversity is a preference: normal placement -uses distinct racks when available and distinct nodes within one rack otherwise. -An unmet rack guarantee is reported in the assessment and does not reject the -allocation or EC publication. `placement.allow_degraded_failure_domains` is -required to relax node protection; EC placement exceeding its node or disk -recovery budget additionally requires `placement.allow_unsafe_ec`. - -### 7.1 Mirror placement - -Mirror placement aims to place replicas on **distinct racks** to survive -rack failures: - -1. Select N nodes (copy count) from online nodes -2. Prefer nodes on different racks -3. If not enough distinct racks, place multiple copies on same rack -4. Rank racks, nodes, then disk-groups by projected allocatable utilization -5. Return a plan containing its policy, topology generation, usage freshness, - and planned rack/node protection - -**Negative hints**: Nodes can be excluded from placement (e.g., during -recovery to avoid re-using failed nodes). - -**Example**: 2-copy mirror on 3-rack cluster → 2 replicas on 2 distinct racks. -If fewer distinct healthy nodes than copies are available, normal placement -returns a typed failure before any DiskDB allocation. A one-rack result retains -node protection and explicitly reports that it cannot survive a whole-rack loss. - -### 7.2 EC placement - -EC placement distributes data+parity blocks **across multiple racks** -with per-node block limits: - -1. Calculate total blocks = data_num + code_num -2. Select nodes such that: - - No node has > code_num blocks (safe mode) - - Each protected rack and physical disk also has at most `code_num` blocks -3. Rank the requested primary domain, secondary domain, and disk-group by - projected allocatable utilization -4. Return a plan with the requested policy and planned protection assessment - -**Safe mode**: Ensures no single node failure exceeds code_num, guaranteeing -recoverability. Requires enough nodes to satisfy constraints. - -Within `protected` mode, `placement.allow_unsafe_ec` permits a degraded EC plan -that exceeds the normal recovery budget when the cluster is too small. It does -not select colocated placement. Insufficient topology otherwise returns a typed -placement error without allocating blocks. - -**Example**: 8+4 EC on a healthy 3-node cluster places 12 fragments with at -most 4 per node, so loss of any one node leaves 8 fragments. If one node is -already unavailable, the same geometry may span the two survivors and is -marked for placement repair when the third node returns. - -### 7.3 Physical validation and degraded-placement repair - -DiskDB remains the authority for physical allocation. After every allocation, -ChunkDB resolves returned `disk_id`s through the same immutable topology and -computes the actual rack, node, and disk maxima. A physical-disk violation is -rolled back and retried against an alternative disk-group; it is never silently -published. The assessment, effective policy, and any EC repair marker are -persisted with the strip and carried through the RPC and storage codecs. - -Every marked EC strip has one deterministic persistent placement task. Task -admission happens after foreground publication and a bounded metadata scan -recreates a missing task after an interrupted admission. The task waits with -backoff while topology cannot improve; it does not become terminal merely -because the cluster is undersized. Once a better destination exists, it reads -one source fragment, writes and fsyncs one replacement fragment, validates the -new physical layout, and atomically replaces only that fragment. The old -fragment remains under the normal layout-validity cleanup fence. The marker is -cleared only after all recorded guarantees are satisfied. - -### 7.4 Active cross-domain rebalance - -ChunkDB consumes live DiskDB usage summaries and runs a deliberately low-rate -planner. It requires utilization skew to remain above the configured threshold -for the full hysteresis window, scans a bounded chunk page, skips strips with -placement repair already pending, and admits exactly one fragment move per -cycle. Missing or stale summaries disable capacity ranking, never physical -safety checks. - -For a candidate move, ChunkDB allocates one exact tentative target on the cold -disk-group, substitutes it into a copy of the strip, and recomputes rack, node, -and physical-disk assessment. It frees the candidate and emits no handoff if -any protection flag would become false or any recorded domain maximum would -increase. - -An accepted candidate is sent to the target DiskDB through -`ExecuteRelocation`. DiskDB durably adopts the exact source/target pair, copies -and fsyncs data, and asks the current ChunkDB range owner to claim the move. -ChunkDB persists a deterministic relocation task before acknowledging it. The -task conditionally publishes only when the exact source and expected chunk -revision still match. Duplicate delivery observes the same task or installed -target. ChunkDB first confirms the target after its successful CAS; DiskDB -verifies and idempotently confirms that exact target before freeing the source -after `Published`. ChunkDB never frees the source from this path. - -The task checkpoint makes the tentative target visible to DiskDB's owner -scanner as `TaskPending`. Current metadata reports `Referenced`; a deleted or -superseded owner reports `Absent`. This joins crash recovery and ordinary -relocation under one ownership decision instead of treating target age as -permission to reclaim data. - -## 8. Allocation Flow - -**AllocateChunk** operation: - -1. Generate 192-bit chunk ID (metadata + UUID + reserved) -2. Fetch topology snapshot from TopologyCache -3. Calculate strip layout based on requested capacity and strip type -4. For each strip: - - Call placement selector to get node/disk-group assignments - - Group placements by DiskDB data group and allocate each group in parallel - - On failure: rollback (free all allocated blocks), return error -5. Persist the complete chunk metadata with state = Active -6. Start grouped DiskDB block commits asynchronously -7. Refresh the chunk cache and return the Active chunk to the caller - -**Parallel allocation**: Strips remain sequential. Within a strip, ChunkDB -sends one request per DiskDB data group and runs those requests concurrently. -DiskDB persists each allocated busy block as Tentative before responding. - -**Allocation routing failure**: Only a typed connection-establishment failure -before RPC submission permits rerouting the same allocation. ChunkDB refreshes -the service registry and makes at most one attempt at a different endpoint. -If discovery fails, the endpoint is unchanged, or no owner is advertised, it -returns the original failure. Any failure during RPC submission or while awaiting -the reply remains ambiguous and is never replayed by the allocator. Discovery -may refresh for the next independent request. Successful allocation adds no -registry read or lock beyond the existing periodic discovery policy. - -**Success boundary**: The Active chunk and every referenced Tentative busy -block are durable before ChunkDB returns success. DiskDB commit changes the -matching busy records to Committed in one ordinary batch write. A -reconciliation scanner can resolve a crash in this interval from the Active -chunk reference and allocation incarnation. - -**Rollback**: If allocation or Active metadata persistence fails, every known -segment from every prior strip is freed through its exact DiskDB group before -returning. A failed rollback is surfaced. Background commit failure is -reported by metrics and logs for later reconciliation; it does not retract an -already durable Active chunk. - -## 9. Chunk Lifecycle - -``` -Init ──> Active ──> Sealed ──> Deleted -``` - -| State | Description | -|---------|--------------------------------------------------------------| -| Init | Reserved transient state; the allocation path publishes Active directly. | -| Active | Chunk is open for writes. Strips can be appended. Returned by AllocateChunk. | -| Sealed | Chunk is read-only. Records final length and seal timestamp. | -| Deleted | Durable cleanup intent; retained segments still need freeing. | - -**State transitions**: -- `Active → Sealed`: Via `SealChunk` RPC. Validates state and records the exact - `seal_bytes` boundary in `acknowledged_cursor`, including zero for an empty - chunk. The RPC has one exact-byte form. KiB `sealed_length` fields summarize - geometry and do not enlarge the valid byte range. Finalization preserves the - exact acknowledged cursor. Data alignment and physical block allocation never - add logical bytes to the seal boundary. -- `Active → Deleted`: Persist cleanup intent, free segments, persist tombstone. -- `Sealed → Deleted`: Persist cleanup intent, free segments, persist tombstone. -- Invalid transitions (e.g., `Sealed → Active`) return errors. - -**Concurrency**: An approved bounded `tokio::Mutex` per chunk serializes the -read-modify-write lifecycle. Locks for different chunks are independent and -tests prove cross-chunk progress. Immutable topology, routing, range, endpoint, -and disk-group reverse-map snapshots remain lock-free on their read paths. - -### 9.1 Public frames and active-chunk liveness - -Every persistent chunk user stores public, self-validating frames. The frame -header names a versioned kind and payload bounds; its footer binds CRC32C and -chunk ID. Locations retain only chunk ID, physical frame start, and logical -length, so integrity state is never duplicated in high-cardinality metadata. -Readers verify a complete frame before returning a payload. A failed -verification excludes the serving mirror or shard and uses the existing -protection-read and repair path. - -An Active chunk has exactly one durable `FinalizeChunk` liveness task. Chunk -creation atomically creates it with a 15-minute deadline. The owner renews it -on the 12-minute cadence by conditionally replacing the one deadline-index -entry and task record in one task-partition transaction; no second expiry is -stored in chunk metadata. The owner self-fences before that authority can -expire. The task scanner reads due task-index entries only, never scans the -chunk table. After claim and the shared request-age/skew safety window, it -requires ordered synchronization proof from every segment in the captured -layout, including ordinary empty chunks and unavailable mirror identities. -Failure retains the layout and retries without scanning or freeing blocks. -It then observes content using at-most-1-MiB windows and the shared bounded frame -parser. Stream/Wal scans include allocated capacity beyond the persisted -checkpoint and commit only complete append groups. Ordinary object publication -already persists its physical cursor; its finalization validates that prefix. -An exact-capacity end needs no extra header request. Drain and scan share one -30-second IO budget; drain has at most its first half. Observation and comparison -reserve time within the remainder rather than restarting a timeout per copy. - -Available mirror candidates run concurrently within one IO deadline. Select -the shortest healthy complete prefix and verify agreement on overlapping -contents. Failed, partial or corrupt observations do not become shorter -candidates; valid conflicting contents fail explicitly. No readable candidate -means retry while retaining the Active layout, including a Stream/Wal with a -zero checkpoint. A scan must cover the persisted `acknowledged_cursor` before -sealing or deleting. The task writer generation must match the chunk epoch. -Sealing compares the exact captured chunk metadata while holding its lifecycle -guard; a concurrent revision/layout change forces another observation. Empty -chunks first seal through that same fence, then resume tombstone publication -after restart without accessing physical segments that were already retired. -Finalization uses this durable task path exclusively; an expired metadata -checkpoint cannot independently authorize sealing. A restarted owner always -allocates a new Active chunk. - -## 10. Per-Chunk-ID Lifecycle Lock + Chunk Cache - -`LifecycleHandler`'s six mutating RPCs (allocate/append/seal/delete/ -delete-range/update-strip) each perform a read-modify-write cycle on a -chunk record. Without per-chunk serialization, two concurrent -`AppendChunk` RPCs on the same chunk ID both read the chunk, both -append strips, both `put_chunk`. The second overwrite loses the -first's strips. A per-chunk mutex serializes the RMW cycle. A payload -cache avoids the `get_chunk` store round-trip on every mutating RPC -(the latest chunk is known in-process right after the previous -`put_chunk`). - -The lock and the payload have different eviction requirements (lock: -evict only when uncontended; payload: evict freely by -recency/frequency), so they are separate structures. See -[`design-crowdb-chunkdb-range-binding.md`](design-crowdb-chunkdb-range-binding.md) -for the range guard that ensures a chunk reaches exactly one chunkdb -instance. The per-chunk lock assumes that one-owner invariant. - -### 10.1 ChunkLockMap - -`app/crowdb-chunkdb/src/lifecycle.rs`: - -```rust -pub struct ChunkLockMap { - locks: SkipMap<(u64, u64), Arc>, - chunks: Arc>, - metrics: Arc, - hold_warn_threshold: Duration, -} -``` - -- `new(cache_capacity, metrics, hold_warn_threshold) -> Self` — creates - an empty ordered lock-free index and a `Cache::new(cache_capacity)`. -- `acquire(&self, chunk_id, store, policy, hint) -> Result` — for existing chunks (append/seal/delete). Steps: - 1. `get_or_insert` an immutable `ChunkLockEntry`, atomically retain its user - count, and recheck that the exact entry is still current. - 2. Record lock-wait start time. - 3. Acquire the mutex per `policy`: - - `TryLock` → `try_lock_owned()`. On `Err`, increment - `lock_busy_count`, return `LockBusy`. - - `Wait(d)` → `lock_owned()` with `tokio::time::timeout(d, ...)`. - On timeout, increment `lock_timeout_count`, return - `LockTimeout`. - 4. Record lock-wait time into the histogram. - 5. Serve payload: if `hint == Cache` and `self.chunks.get(&chunk_id)` - returns `Some(chunk)` → cache hit (increment `cache_hit_count`), - return guard with that chunk. Otherwise → cache miss (increment - `cache_miss_count`), `store.get_chunk(chunk_id)`. On - `StoreError::ChunkNotFound` → return `ChunkNotFound`. On success, - if `hint == Cache`, `self.chunks.insert(chunk_id, chunk.clone())`. - 6. Return `ChunkGuard` with the exact retained lock entry, mutex guard, - chunk, hint, hold start, and metrics. -- `acquire_for_create(&self, chunk_id, policy, hint) -> Result` — for `allocate_chunk` with caller-supplied ID. Same - lock acquisition as `acquire` but does NOT fetch from store (chunk - does not exist yet). Returns a guard with `chunk: None`. Caller must - `refresh()` after creating the chunk. -- `populate_cache(&self, chunk_id, chunk)` — for `allocate_chunk` with - auto-generated ID (skips the lock; UUID collision negligible). -- `reap_idle(&self)` — scans the ordered index and changes an entry's user - state from zero to `REAPING` before removing that exact entry. An owner, - waiter, or acquisition in progress has already retained the entry and - prevents this transition. Increments `reap_idle_count` and - `reap_idle_entries_removed`; the separately bounded payload cache is - untouched. -- `invalidate_chunk(&self, chunk_id) -> bool` — calls - `self.chunks.remove(&chunk_id).is_some()`. Increments - `invalidate_count`. Used by range migration. -- `invalidate_range(&self, bucket_start, bucket_end) -> u32` — iterates - cache entries and removes those whose chunk ID hashes to a bucket in - `[bucket_start, bucket_end]`. Returns the count removed. Increments - `invalidate_count` by the count. O(n) on cache size — acceptable for - rare range migrations. -- `cache_len() -> usize`, `metrics_snapshot() -> LifecycleMetricsSnapshot`. - -### 10.2 Lock policy + cache hint - -```rust -pub enum LockPolicy { - TryLock, - Wait(Duration), -} - -impl Default for LockPolicy { - fn default() -> Self { Self::Wait(Duration::from_secs(10)) } -} - -pub enum CacheHint { - Cache, - NoCache, -} - -impl Default for CacheHint { - fn default() -> Self { Self::Cache } -} -``` - -- `LockPolicy::TryLock` → fail fast with `LockBusy` on contention. -- `LockPolicy::Wait(d)` → park the task up to `d`, then `LockTimeout`. - No `WaitForever` variant — a bounded wait prevents indefinite stalls. -- `CacheHint::Cache` (default) → populate cache on miss, write to cache - on `refresh`. -- `CacheHint::NoCache` → skip cache population; the guard's local copy - is still updated so the current operation sees the chunk. - -All mutating RPCs use `LockPolicy::default()` and `CacheHint::Cache`. -These are internal in v1, not exposed in the RPC API. - -### 10.3 ChunkGuard - -```rust -pub struct ChunkGuard { - lock: ChunkLockHandle, // retains the exact index entry - guard: OwnedMutexGuard<()>, // held for Drop — releases the lock - chunk: Option, - hint: CacheHint, - chunk_id: ChunkId, - hold_start: Instant, - metrics: Arc, - hold_warn_threshold: Duration, -} -``` - -- `chunk(&self) -> Option<&Chunk>` — returns the latest chunk record. - `None` for `acquire_for_create` before `refresh`. -- `refresh(&mut self, chunk: Chunk)` — updates the guard's local copy. - If `hint == Cache`, also writes to the lock map's cache. Caller MUST - have persisted via `put_chunk` first. -- `Drop` — records lock-hold time into the histogram (from `hold_start` - to now). If hold time > `hold_warn_threshold`, emits a `warn!` log - with chunk_id and hold duration. - -### 10.4 Chunk cache - -`quick-cache = "0.7"` (`app/crowdb-chunkdb/Cargo.toml`). Default capacity -10_000 entries (configurable via `lifecycle.cache_capacity`). The -design supports 100_000+. `quick_cache::Cache::new(capacity)` accepts -any `usize`; the only constraint is memory (~1-2 KB per `Chunk` → 10k -entries ≈ 10-20 MB, 100k entries ≈ 100-200 MB). - -Cache operations: -- `Cache::get(&key) -> Option` — cache hit check. O(1). -- `Cache::insert(key, value)` — populate on miss, refresh after - `put_chunk`. O(1) amortized. -- `Cache::remove(&key) -> Option<(Key, Val)>` — `invalidate_chunk`. - O(1). -- `Cache::entry_count()` — gauge for metrics snapshot. - -### 10.5 Sweep task - -The lock map grows unbounded without reaping (one entry per chunk ever -touched). `reap_idle` removes uncontended entries periodically, keeping -the map bounded by concurrent locks, not by chunks-ever-touched. - -`main.rs` spawns a background task (`run_sweep_loop`) that calls -`locks.reap_idle()` every `lifecycle.sweep_chunk_lock_interval_secs` -(default 60s). Uses the same `watch::channel(false)` stop signal -pattern as the topology refresh loop. `reap_idle` performs a lock-free ordered -scan with exact-entry removal and does not block request tasks. - -### 10.6 LifecycleHandler integration - -`LifecycleHandler` gains a `locks: Arc` field. Each -mutating RPC acquires the per-chunk lock before its RMW cycle: - -- `allocate_chunk` (caller-supplied ID): `check_range` → - `acquire_for_create` → existence check (`store.get_chunk`; return - `ChunkAlreadyExists` if taken) → build chunk, allocate strips, - `put_chunk`, start background segment commit → `guard.refresh(chunk)`. -- `allocate_chunk` (auto-generated ID): skip the lock (UUID collision - negligible). After `put_chunk`, `populate_cache(id, chunk)` directly. -- `append_chunk`: `check_range` → `acquire` → state/revision check. A matching - `modify_ts` allocates strips, increments the revision, persists the chunk, - and returns only the new strips plus the revision. A stale revision performs - no allocation and returns the complete current chunk so the client can - refresh and retry. Successful mutation ends with `guard.refresh(chunk)`. -- `seal_chunk`: same as append but no diskdb calls (fast path). -- `delete_chunk`: `check_range` → `acquire` → state check → persist Deleted - with segments as cleanup intent → free segments → clear the segment list - and persist the tombstone → `guard.refresh(chunk)`. -- `delete_chunk_range`: offset and length are independent u32 byte values, - describing an exact half-open object range without KiB rounding. The caller - sends the immutable chunk identity and logical range; an optional nonzero - layout revision adds a strict precondition. ChunkDB qualifies its current - sealed user-data chunk before accepting the notification. Dedicated chunks - accept only their complete sealed object extent. The caller must already establish absence of live and future - references. The server atomically records merged exact dead ranges in a - co-located resource record, guarded by the chunk's KV revision; the unchanged - chunk participates in that conditional batch without changing its layout. - Resource identity includes the writer epoch and sealed extent. Existing - retirement replays without another mutation; a new range with an explicit - stale layout condition fails. An active writer or unacknowledged shared range - returns a retryable conflict. The client performs one range notification, - without a preliminary QueryChunk or whole-chunk deletion. - Records are bounded to 4096 intervals and decoded within 128 KiB. Foreground - QueryChunk does not load or return this resource metadata. Conflict or unknown - write results do not trigger confirmation GETs inside the retirement handler. - A successful response means ChunkDB has accepted cleanup ownership. Access - finishes its notification without waiting for the physical `reclaimed` flag; - physical capacity release is a separate ChunkDB/DiskDB responsibility. - Chunk identities remain tombstones after deletion and cannot be allocated - again. Strip replacement preserves logical offsets and total capacity; - physical relocation does not change the range being retired. An intersecting strip is never permission - to free blocks containing live neighbours. Mixed-strip physical compaction and - independent orphan qualification are not implemented by range retirement. -- `update_chunk_strip`: compatibility wrapper over the one-strip form of - `replace_chunk_strip_range`. -- `replace_chunk_strip_range`: `check_range` → `acquire` → validate Active or - Sealed state, expected revision, exact old range, contiguous capacity, - monotonic sequences, segment ownership, and mirror/EC geometry → commit only - new-only segments → atomically publish the replacement plus an old-only - cleanup intent → `guard.refresh(chunk)`. An identical operation retry at the - successor revision returns the installed chunk. Any other revision or range - is a conflict. -- `allocate_replacement_segment`: inspect the current owning strip and return - one geometry-compatible tentative segment. Mirror replacement excludes - survivor nodes. EC replacement excludes nodes already at the strip's - per-node failure-domain limit; explicitly unsafe layouts retain their - balanced per-node ceiling. Every old and surviving strip disk is excluded, - so no two shards share one disk. -- `discard_replacement_segment`: under the lifecycle guard, free a tentative - segment only when it belongs to the chunk and current metadata does not - reference it. -- `reserve_strip_group`: allocate ordinary hidden strips or one special 8+4 - conversion group, persist its writer/lease/placement fences separately from - `Chunk.strips`, and leave chunk capacity unchanged until confirmation. A - lock-free atomic admission gate checks the instance's proportional share of - the configured cluster-wide reserved-block and reserved-byte limits before - allocation. Startup rebuilds usage from durable reservation records. -- `mutate_strip_reservation`: durably consume before DiskIO, confirm one strip - in sequence, cancel or renew idempotently, or publish a complete special - group after current-topology survivor selection. Publication commits one EC - strip and the redundant-replica cleanup intent atomically. -- `query_chunk` / `list_chunks` — unchanged (no lock, no cache). - -`query_chunk` advertises `layout_validity_ms`. Retired segments remain allocated -until that reader-layout window expires. Startup and the periodic lifecycle -scan resume cleanup intents, recheck that retired identities are absent from -the current layout, ask DiskDB to perform its ownership-qualified free, and -only then clear the intent. Metadata publication therefore remains committed -even if post-commit reclamation is temporarily unavailable. - -Reservation reconciliation scans reservation records in bounded rotating -pages, so an old prefix cannot starve later records. It deterministically -admits a completed special group to the normal conversion task path. An -expired incomplete group is cancelled and reclaimed when its consumed strips -carry persisted planned cursors and the writer lease deadline plus -`lifecycle.layout_validity_ms` has elapsed. This grace bounds late request -admission; ordered completion proof and producer exclusion are also required -before physical reuse because an expired RPC can still have backend IO in flight. -Current reservation records require explicit planned -cursor/sequence vectors; a consumed strip with a zero planned cursor is malformed -and fails before reconciliation changes storage. Placement records require -explicit policy and repair-state fields. DiskIO does not validate allocation -ownership. Terminal reservation -records release their quota only after the durable record has been removed; -ambiguous persistence retains the permit unless a linearizable read proves that -no record exists. - -### 10.7 Error variants + service mapping - -```rust -#[error("chunk lock busy — retry later")] -LockBusy, -#[error("chunk lock acquire timed out")] -LockTimeout, -``` - -`map_error` in `service.rs`: -- `LifecycleError::LockBusy => Status::unavailable(e.to_string())` -- `LifecycleError::LockTimeout => Status::unavailable(e.to_string())` - -Both map to `UNAVAILABLE`. The client's existing retry logic handles -this (same as `NotLeaderHint` transient errors). - -### 10.8 Metrics + HTTP endpoints - -`app/crowdb-chunkdb/src/metrics.rs` — `LifecycleMetrics` with -`AtomicU64` counters + `Mutex` for lock wait/hold -latency. Counters (all `AtomicU64`, `Relaxed` ordering): - -- `lock_wait_time` — `PreciseHistogram` (wait duration in acquire). -- `lock_timeout_count` — incremented on `LockTimeout`. -- `lock_busy_count` — incremented on `LockBusy`. -- `lock_hold_time` — `PreciseHistogram` (hold duration in guard Drop). -- `cache_hit_count` — incremented on cache hit in `acquire`. -- `cache_miss_count` — incremented on cache miss in `acquire`. -- `cache_size` — gauge, read from `Cache::entry_count()` at snapshot. -- `reap_idle_count` — incremented each `reap_idle` run. -- `reap_idle_entries_removed` — entries removed per `reap_idle`. -- `invalidate_count` — incremented on `invalidate_chunk`/ - `invalidate_range`. -- `reservation.blocks.g` and `reservation.bytes.g` — current durable hidden - reservation usage rebuilt at startup and updated on admission/release. -- `reservation.rejections.c` — reservation requests rejected by either atomic - quota limit. - -`snapshot() -> LifecycleMetricsSnapshot` drains counters, reads -histograms, returns a serializable struct (JSON). - -HTTP endpoints (`main.rs` HTTP server, alongside `/ready` + -`/health`): - -- `GET /metrics` → returns `LifecycleMetricsSnapshot` as JSON. -- `POST /invalidate_chunk` → body `{ "chunk_id": { "high": u64, "low": - u64 } }` → calls `ChunkLockMap::invalidate_chunk`. Returns - `{ "invalidated": bool }`. -- `POST /invalidate_range` → body `{ "bucket_start": u16, "bucket_end": - u16 }` → calls `ChunkLockMap::invalidate_range`. Returns - `{ "invalidated_count": u32 }`. - -All internal (no auth, same as `/ready` and `/health`). - -`/ready` requires validated storage routing and at least one owned service -slot. Empty or uninitialized ownership rejects work, including reads; there -is no allow-all compatibility mode. - -### 10.9 Edge cases - -- Lock map entry does not exist → created on first `acquire` through the - ordered index's `get_or_insert` operation. -- Lock holder panics → `tokio::sync::Mutex<()>` auto-releases (no - poisoning for `Mutex<()>`); cache slot may be stale but next - `acquire` re-fetches on miss. -- `reap_idle` races an acquirer → either the acquirer increments the exact - entry's user count first and reaping skips it, or reaping marks the entry - first and the acquirer retries against its replacement. Overlapping callers - cannot split across mutex identities. -- Process crash → all in-memory state lost; KV store is source of - truth. -- Cache evicts a chunk between two operations → next `acquire` is a - miss, re-fetches. Correctness unaffected. -- `CacheHint::NoCache` → skip `insert` on miss and `refresh`; the - guard's local copy is still updated so the current operation sees - the chunk. -- `delete_chunk` keeps the Deleted-state chunk cached via `refresh` → - next `delete_chunk` retry gets a cache hit and returns the same tombstone - successfully without a store round-trip. -- `acquire` returns `ChunkNotFound` (store miss during acquire) → the - chunk does not exist; `append`/`seal`/`delete` return - `ChunkNotFound`. -- `LockBusy` / `LockTimeout` → mapped to RPC `UNAVAILABLE`. - -## 11. EC Encoding/Decoding - -**EC encoding** (strip allocation): -1. Allocate data_num + code_num blocks via diskdb -2. Fill data blocks with actual data (by caller, not chunkdb) -3. Call isa-l encode to generate parity blocks -4. Mark EC state = Complete - -**EC decoding** (recovery, deferred): -1. Identify failed blocks (via disk failure notification) -2. Read surviving data+parity blocks -3. Call isa-l decode to reconstruct lost data -4. Write reconstructed data to new blocks -5. Update strip with new block locations - -**crowdb-common EC module**: Provides safe Rust interface to isa-l: -- `make_gf_table(data_num, code_num)`: Generate Galois Field table -- `encode(data_num, code_num, segment_len, gf_table, data_ptr, code_ptr)`: Encode -- `decode(...)`: Decode with error recovery -- Buffer management: `make_buffer`, `destroy_buffer` - -**Rationale**: isa-l is highly optimized for AVX2/AVX512. Strip-level EC -aligns with redundancy unit design. - -## 12. Crate Layout - -``` -app/crowdb-chunkdb/ # chunkdb server binary -├── Cargo.toml -├── src/ -│ ├── main.rs # CLI entrypoint and server wiring -│ ├── lifecycle.rs # pure lifecycle index -│ ├── lifecycle/ -│ │ ├── handler.rs # lifecycle orchestration -│ │ ├── lock_map.rs # per-chunk serialization and payload cache -│ │ └── state.rs # lifecycle state transitions -│ ├── service.rs # pure service index -│ ├── service/ -│ │ ├── chunkdb_rpc_service.rs -│ │ └── chunkdb_rpc_service/ -│ │ ├── service.rs # state and handler registration -│ │ ├── mutations.rs # mutating chunk handlers -│ │ ├── queries.rs # query and list handlers -│ │ └── wire.rs # FlatBuffer frames and error mapping -│ ├── allocator/ -│ │ └── pool.rs # allocation pool -│ ├── selector/ -│ │ ├── mirror.rs # Mirror placement -│ │ └── ec.rs # EC placement -│ ├── topology/ -│ │ ├── notify.rs # topology notifications -│ │ └── refresh.rs # topology refresh task -│ ├── storage.rs # KV persistence -│ ├── routing.rs # bucket routing -│ ├── range_guard.rs # owned-range enforcement -│ └── runtime_context.rs # isolated lifecycle/task runtimes - -lib/crowdb-chunkdb-client/ # chunkdb client library -├── Cargo.toml -└── src/ - ├── client.rs # ChunkdbClient - └── rpc_transport.rs # crowdb-rpc transport - -lib/crowdb-protocol/ # Protocol definitions -└── src/fbs/ - └── chunkdb.fbs # service and data types -``` - -The lifecycle index preserves the public `LifecycleHandler`, `ChunkLockMap`, -and state-machine API. The handler owns chunk operations; the lock-map module -owns the approved bounded per-chunk mutex and payload cache. RPC registration, -chunk handlers, and FlatBuffer construction are separate private modules -behind `ChunkdbRpcService`. - -## 13. Concurrency Model - -- **Async everywhere**: All public APIs are async (`async fn`). -- **Shared state**: Immutable topology is published through `ArcSwap`; placement - reservations use per-disk-group atomics. Lifecycle's existing bounded - per-chunk guards protect metadata mutation, not allocation ranking. -- **Lock scoping**: Existing lifecycle guards are acquired in `{}` blocks and - dropped before `.await`; placement introduces no hot-path lock. -- **Parallel allocation**: Use `futures::join_all` for parallel strip/block - allocation. -- **Background tasks**: `tokio::spawn` for topology refresh, rollback cleanup. - -**Rationale**: tokio provides efficient async I/O. Lock scoping prevents -deadlocks with `.await`. Parallel allocation minimizes latency. - -## 14. Configuration - -Key configuration parameters: - -| Parameter | Default | Description | -|---------------------------------------------------|------------|---------------------------------------------------------------| -| disk_block_size | 1 MB | Size of disk blocks from diskdb | -| deployment.max_node_failures | 1 | Protected production node-failure budget | -| mirror_copy_count | 2 | Production mirror copies on distinct nodes | -| default_ec_scheme | 6+3 | Default EC scheme (data+parity) | -| topology_refresh_interval | 30 s | Topology cache refresh interval | -| placement.mode | protected | Select `protected` or explicit `unsafe_colocated` strategy | -| placement.allow_unsafe_ec | false | Permit explicit degraded EC placement | -| placement.allow_degraded_failure_domains | false | Permit an explicit unmet rack or node guarantee | -| placement.failure_domain_priority | rack_first | Prefer rack or node protection when ranking safe plans | -| placement_repair.max_concurrency | 2 | Maximum concurrent degraded-EC repair tasks | -| placement_repair.scan_interval_secs | 1 | Marker reconciliation interval | -| placement_rebalance.scan_interval_secs | 300 | Cross-domain usage scan interval | -| placement_rebalance.imbalance_threshold_pct | 20 | Required utilization-point skew | -| placement_rebalance.hysteresis_secs | 900 | Required sustained-skew window | -| placement_rebalance.min_target_free_bytes | 1 GiB | Minimum target disk-group free capacity | -| placement_rebalance.max_moves_per_cycle | 1 | Exact per-cycle relocation bound | -| max_allocation_parallelism | 10 | Max parallel strip allocations | -| lifecycle.cache_capacity | 10_000 | Per-chunk payload cache capacity (§10) | -| lifecycle.sweep_chunk_lock_interval_secs | 60 | Idle lock reap interval (§10) | -| lifecycle.lock_hold_warn_threshold_ms | 1000 | Lock hold warn threshold (§10) | -| lifecycle.layout_validity_ms | 30_000 | Minimum retired-layout and expired-writer reuse grace | -| server.keepalive_interval_secs | 10 | Service-registry heartbeat interval | -| server.rpc_workers | 2 | Inbound RPC workers; static and must be positive | - -chunkdb requires a typed TOML file at startup. Omitted fields use typed -defaults, and explicitly supplied CLI fields override file values according to -the shared [`service configuration design`](../config/design-crowdb-config.md). - -## 15. Full-Stack Deployment and Benchmark - -The shared OCI node lifecycle prepares Group 0, the ordinary metadata group, -DiskDB, DiskIO and ChunkDB through accepted node policy and committed service -intent. The common cluster lifecycle creates runtime resources and observes -readiness; benchmarks attach to the selected cluster without creating groups, -resetting topology or starting and stopping servers. Production placement uses -actual failure domains; a Local cluster on one physical host has zero physical -host failure tolerance. See the [deployment design](../deploy/design-crowdb-deploy.md). - -`crowdb-cli bench chunkdb allocate` measures mirror or EC allocation with -configurable shape, duration, seed, and concurrency. `bench chunkdb mix` uses -a deterministic allocate/query/append/seal/delete distribution. Both query -every surviving chunk through the public client, validate state, owner tags, -and nonoverlapping physical segments, and require the exact referenced byte -total to equal the DiskDB busy-space delta after compaction. Capacity -exhaustion is a successful stop reason; any correctness error invalidates the -sample. - -The allocation regression uses an explicit test-only measurement fixture with -three co-located logical nodes, one rack, three data groups and four logical -4-TiB disks per DiskDB. It measures one EC 8+4 strip per allocation with varied -concurrency and retains exact space-accounting checks. The fixture owns every -server by PID and process-start identity, records accepted bootstrap inputs and -provides normal clients with explicit targets. Each invocation retains metrics, -logs and a checked shutdown receipt. This fixture does not qualify physical host -redundancy or replace real-storage OCI acceptance. - -## 16. Implementation Scope - -**v1 (R85)**: -- Basic chunkdb server and client -- Mirror and EC strip allocation -- Synchronous and background mirror-to-EC conversion through persistent tasks -- Rack/node-aware placement -- Topology cache with group-0 integration and watch/notify for real-time updates -- Basic chunk lifecycle (allocate/seal/delete) -- EC wrapper (crowdb-common module) -- Chunk ID generation (crowdb-common): 128-bit format with getrandom -- Wal, BtreePage, PageIndex, Stream, S3 and IcebergTable purposes -- E2E tests - -**Future work** (separate requirements): -- In-chunk GC operations (reclaim, collapse, merge) -- Shared and dedicated Repo-chunk write policies in the client data path -- Recovery flow (disk failure handling, EC rebuild) -- Metrics and observability -- Console/CLI integration -- Custom RPC for performance - -## 17. References - -- CROWDB diskdb design: `doc/design/diskdb/design-crowdb-diskdb.md` diff --git a/doc/design/chunkds/design-crowdb-chunk-kv-client.md b/doc/design/chunkds/design-crowdb-chunk-kv-client.md deleted file mode 100644 index a2f259cf3..000000000 --- a/doc/design/chunkds/design-crowdb-chunk-kv-client.md +++ /dev/null @@ -1,178 +0,0 @@ - - - -# Chunk KV Routed Client - -`crowdb-chunk-kv-client` is the application-facing router for the chunk-backed -KV service. It validates and caches the immutable range catalog, sends each -single-partition operation directly to the current owner, and composes bounded -multi-partition operations without claiming distributed atomicity or a global -snapshot. - -## Catalog and Routing - -`CatalogSource` supplies a catalog head and all referenced pages. The client -decodes them as one ordered `CatalogMap`, reusing the protocol validation that -requires complete, non-overlapping coverage of the binary keyspace. A valid -generation is published through `ArcSwap`; invalid or regressing refreshes do -not replace a warm map. - -Point lookup binary-searches partition starts. No data server acts as a network -forwarding proxy. During same-owner split grace, the old endpoint may dispatch -an old parent point route directly to the exact child handle already hosted in -that process. This preserves the logical request identity and never reopens the -parent writer. Scan and seek topology are not forwarded. A cold client must -load group 0 before sending a nonempty operation. A warm client may continue -trying the cached owner while refresh is unavailable, but the server's owner -epoch and serving lease remain the final authority. - -Each operation has one total deadline covering discovery, refresh, transport, -and backoff. `NotMyRange`, `RefreshRequired`, lease loss, and connection loss -cause bounded rerouting. Only `TargetNotReady` enters the initialization retry -path: it uses the server retry delay without refreshing the route, inventing -another owner, or changing request identity, and returns the third unsuccessful -response to the caller. Normal serving requests do not execute this retry -bookkeeping. Overload, recovery, and write-stall responses retain the general -retry policy. The route-refresh and total-attempt budgets are independent and -explicit in `ClientConfig`. - -When `NotMyRange` reports an older catalog, or the same generation with no -different owner/epoch hint, the client retries the cached endpoint -within its serving-attempt and operation-deadline bounds. This lets a newly -published owner activate without repeatedly loading an unchanged catalog. -Newer generations and different owner hints retain bounded catalog discovery. -Exhausting discovery refreshes does not itself exhaust serving attempts. - -## Request Identity - -Every client handle generates a random nonzero 128-bit instance ID from the -operating system. An atomic counter allocates nonzero sequences beginning at -one and closes permanently before wraparound. A logical mutation allocates its -`ClientRequestId` once, before routing, and retains it across every retry. - -`execute_with_identity` accepts an application-persisted identity and operation -for explicit recovery after process restart. The server compares the operation -digest against its retained result, so changing an operation under a reused -identity remains a typed `RequestConflict`. Connections, endpoints, and RPC -ports never contribute to identity. - -## Direct Operations - -The client wraps get, put, delete, put-if-absent, compare-exchange, -conditional delete, ceiling, higher, floor, and lower. It returns the complete -`ChunkKvResponse`, including journal position, applied or observed revision, -condition outcome, map revision, retry hint, and typed failure. Gets can carry -a minimum journal position for explicit read-after-write ordering. - -After split, a get routed to a child may carry either a retained parent-stream -position at or below the split cutover or a child-stream position. The server -resolves the former through the inherited overlay and the latter through the -child applied frontier. A client never compares offsets from different stream -names. After balance, new writes return only the target stream position. - -Seek is an owner-local ordered operation. It is transported directly and is -not emulated with point reads. Directional scan likewise consumes native -partition scan pages and validates that every returned key is strictly ordered, -inside both requested and partition bounds, and consistent with the returned -continuation topology. - -## Multi-Get and Batch Mutation - -`multi_get` retains every input index, including duplicates, groups keys by -partition, dispatches groups with bounded parallelism, and restores one result -per input position. A failed group produces per-item typed errors without -discarding successful partitions. Each partition observes its own applied view; -the result is not a global snapshot. - -`batch_mutate` allocates or accepts one stable request identity per operation. -It rejects reads and oversized input before dispatch, preserves relative input -order inside each partition group, and returns per-operation results in original -input order. Successful operations are removed from the retry set. Only -transient or transport-unknown operations retry, using their original identity. -There is no atomic commit or ordering promise across partition groups. - -The protocol request types expose `validate_for_range` preflight methods. -Multi-get validation rejects the whole group if any key is outside the declared -partition. Batch validation additionally rejects empty groups, reads, and bad -request identities. Owners run this preflight before reads or WAL admission. - -## Multi-Partition Scan - -A scan validates its binary half-open bounds and optional continuation, pins a -catalog generation, intersects the interval with that generation, and visits -partitions in ascending or descending byte order. Execution is sequential, so -the number of buffered partition pages is one and therefore stays within the -configured page bound. Global item, caller byte, and client response-byte caps -apply across every partition. - -`scan_bounded` is a forward-only metadata read with positive item and byte -budgets. Its dedicated RPC requires server enforcement; unsupported transports -fail without falling back to ordinary scan. Owners cap the caller budget by -their response limit and check native value lengths before copying external -memory cells or assembling overflow payloads. An oversized first value or a -native traversal deadline with no progress is an explicit failure, not an empty -completed range. Partial native pages retain their last emitted key. - -The strict client returns after a nonempty owner page so the next request gets -a fresh byte budget. It returns a terminal page when the entire owner response -has been consumed, the owner reports no continuation, this is the last relevant -partition in the pinned catalog, and the response matches that catalog generation. -Otherwise it retains the conservative continuation, which may require a final -empty page to establish completion. Dataset metadata reads additionally bound -total request time and observe cancellation while waiting for the client. Ordinary scan retains -its existing first-entry and response-validation behavior. - -The public continuation contains direction, original bounds, last emitted key, -and observed catalog generation. Pagination resumes strictly after that key for -forward scans and strictly before it for reverse scans. When a split or transfer -invalidates the plan, the client refreshes and rebuilds only the unconsumed -interval. This prevents repeated emitted keys and stable-key gaps across -topology replacement. It does not make the scan a cross-partition snapshot; -concurrent changes to already consumed keys need not appear. - -A parent continuation is always refresh-only after split. A balance catalog -entry in `TargetCatchingUp` is also non-readable: the routed client retains the -same logical operation and waits or retries `TargetNotReady` until a later -catalog generation marks the target `Serving`. It never resumes a scan from an -incomplete target prefix. - -An empty scan limit, empty multi-get, or empty batch succeeds without catalog -discovery. Malformed bounds, mismatched continuations, oversized requests or -responses, out-of-order server pages, and request-sequence exhaustion fail -explicitly. - -## Resource Bounds and Transport Boundary - -`ClientConfig` caps attempts, catalog refreshes, owner connections, in-flight -partition groups, batch items, response bytes, buffered scan pages, and the -total operation duration. The composition layer enforces group, retry, item, -page, and byte bounds. `ChunkKvTransport` is injected so protocol behavior is -testable independently of sockets; its production implementation owns the -bounded connection pool keyed by owner instance and endpoint. - -Cancellation before server admission leaves no retained request. Cancellation -after admission may hide a committed result, so mutation callers persist and -resubmit the complete identity and operation when the outcome matters. - -## Consistency and Non-Goals - -Single-partition operations inherit the sequencer, durability, request-result, -and serving-fence semantics of `crowdb-chunk-kv`. Multi-get and batch are -partition-local compositions with partial success. Multi-partition scan is -globally ordered but not globally snapshot-consistent. - -TTL, watches, range delete, atomic multi-key transactions, network forwarding -proxies, and merge-specific continuation behavior are outside this client -contract. - -## Open Issues - -- The production group-0 subscription and periodic-refresh adapter is not yet - wired because the chunk KV server has no production catalog RPC surface. -- The production `crowdb-rpc` transport and bounded owner connection pool await - the server's FlatBuffers handlers. -- Native ordered cursors, internal group handlers, and the real chunk KV server - process are still required for real-process seek, scan, batch, and lifecycle - coverage. -- Restart, response-loss, owner-death, split, balance, control-plane outage, - graceful-drain, and merge-specific scenarios remain real-process follow-up. diff --git a/doc/design/chunkds/design-crowdb-chunk-kv-server.md b/doc/design/chunkds/design-crowdb-chunk-kv-server.md deleted file mode 100644 index dc1152802..000000000 --- a/doc/design/chunkds/design-crowdb-chunk-kv-server.md +++ /dev/null @@ -1,804 +0,0 @@ - - - -# CROWDB - Design: Chunk KV Server - -`crowdb-chunk-kv-server` is the standalone ownership, routing, and network -boundary for chunk-backed metadata partitions. It hosts zero or many R142 -partition handles but never owns or bypasses a raw tree or stream. Group 0 is -the durable catalog and monitor authority; monitor-issued serving grants are -the only authority to admit data requests. - -## Contents - -- [1. Catalog Publication](#1-catalog-publication) -- [2. Domain Monitors](#2-domain-monitors) -- [3. Serving Authority](#3-serving-authority) -- [4. Request Contract](#4-request-contract) -- [5. Ordered Reads](#5-ordered-reads) -- [6. Split Publication](#6-split-publication) -- [7. Child Balance State Machine](#7-child-balance-state-machine) -- [8. Placement Policy](#8-placement-policy) -- [9. Lifecycle and Observability](#9-lifecycle-and-observability) -- [Open Issues](#open-issues) - -## 1. Catalog Publication - -The catalog is one checksummed generation head over ordered immutable pages. -Each entry contains an exact half-open binary-key range, stable partition ID, -owner endpoint, monotonic owner epoch, lifecycle state, stable tree ID, stable -stream name, optional tail-overlay artifact, and optional transition ID. The -overlay separately binds a chunk root-catalog generation, a tree snapshot -sequence and applied sequence, source partition/epoch/stream and stream -manifest generation, retained replay and cutover offsets, cutover sequence, -and target-WAL start sequence. A valid generation starts at the empty byte string, has -exact adjacent bounds, ends unbounded, and covers each binary key once. - -Publishers validate the complete successor, including retained-partition epoch -non-regression, before I/O. They write only new pages, reread and validate every -referenced page, and write the head last. An ambiguous head write succeeds only -when rereading proves the exact intended head. Readers retain their last valid -immutable snapshot when a new head or page is absent, corrupt, incomplete, or -regressing. Unchanged pages may be referenced by later heads through their -original page generation. - -An owner's connection address may change without transferring its partitions. -Under the Group-0 leader fence, a fresh instance registration advertising the -same hosted partition and owner epoch may publish an address-only successor. -The partition, range, owner instance ID, epoch, tree, stream and transition -remain unchanged, and old pages remain immutable. A recovering owner may update -its route, but receives no serving authority until normal recovery and grant -checks succeed. Fixed transition comparisons use owner instance identity rather -than requiring its historical connection address. - -## 2. Domain Monitors - -`EnsureDomainMonitor` carries a persisted descriptor with domain and registry -names, compiled driver and capability versions, failure policy, balance policy, -and all health/lease timing. Identical registration is idempotent. Any field -conflict fails closed, and an unsupported compiled driver returns -`UnsupportedMonitorDomain` without changing the prior descriptor. - -Every group-0 replica prepares tasks from persisted descriptors. A staged tick -may observe, plan, and publish only under the current group-0 leader fence. -Registry, catalog, binding, or transition observation failure ends the tick -before planning; it is never represented as an empty cluster. A supervisor -restarts failed tasks from durable state and a new leader resumes outstanding -transitions. - -An unfinished split reserves both partition identities and participating owners -against automatic dead-owner transfer, as it does against ordinary balancing. -The published parent alone cannot prove that a competing transfer may claim its -WAL: durable split handoff can precede catalog publication. Owner restart resumes -that split; heartbeat expiry never authorizes discarding its recovery state. - -Normal service discovery filters expired registrations. Monitor observation -uses the raw scan, preserving expired entries and heartbeat timestamps so -absence cannot be confused with owner death. The default heartbeat is two -seconds, `Suspect` begins at six seconds, and `Dead` recovery begins at ten. -Automatic dead-owner planning and lease exclusion also require the absence of -a live WAL process lease. Independent renewal starts before partition replay, -whereas discovery publication can wait for replay to finish. A renewing recovery -boot therefore retains its assignments despite stale discovery; it receives no -serving authority until normal recovery and grants complete. Expired processes -remain eligible for failover under the existing heartbeat and exclusion limits. - -When the catalog advances during initial replay, completed recovery remains -usable only if every local recoverable entry is identical in both validated -catalogs, including ownership epoch, range, state, transition, and artifact. -The service installs those handles with the newer head without replaying them -again. Remote assignment changes do not invalidate local recovery. Changed -local assignments still require recovery under the original reconciliation -deadline; failed replay is never accepted as completed recovery. - -WAL replay retains a SHA-256 fingerprint of each canonical complete record for -duplicate-sequence validation, rather than retaining its key and value payloads. -The fingerprint includes identity, epoch, request, recorded result, and operation. -Exact late duplicates remain valid; conflicting duplicates, sequence gaps, and -operation-digest mismatches remain corruption. Replay windows and retry-result -retention keep their existing independent bounds. -After each 16 MiB of estimated applied key/value and entry bytes, native replay drains its transient -memtables into the in-memory tree. This bounds overwritten versions retained -while normal maintenance is not yet running. It does not snapshot, publish a -checkpoint, advance the checkpoint replay offset, or trim WAL; a drain failure -still rejects recovery. - -## 3. Serving Authority - -Each boot claims one fixed Group-0 WAL process lease. It retains its predecessor -identity while claiming; a replaced boot cannot claim again or renew. Independent -background renewal starts before partition recovery. Local expiry or a renewal -gap permanently fences that boot. Lease records retain process epochs and prior -process/grant expiry boundaries after expiration. - -The monitor issues one aggregate grant per instance and process epoch. It contains a monotonic -lease sequence, catalog generation, wall-clock expiry, sorted exact -`(partition_id, owner_epoch)` assignments, and their canonical digest. An -instance heartbeat advertises health but cannot authorize a request. - -Heartbeat publication compares the exact process identity and mutates the same -lease key atomically with the observation. Grant publication compares that key -and atomically preserves its sequence and greatest expiry alongside the grant. -Replacement, renewal and grant publication therefore serialize on one record; -replacement cannot miss a concurrent extension. An observation for another -process epoch cannot receive a grant, and a boot rejects grants for other epochs. -Grant publication attempts each independently ready instance once per pass in -instance-ID order. A failed instance publication does not skip other instances; -the pass reports all publication failures after processing them. Each publication -retains its process-identity, revision, expiry and atomic lease/grant checks. -Invalid shared catalog, readiness or exclusion evidence still rejects the pass -before instance publication. No conditional retry or authority extension is added. -One bounded control-plane task submits a boot's renewal and discovery writes -sequentially so their periodic ticks cannot exhaust each other's conditional -admission budget. Recovery, data IO and load sampling remain outside that task. -Queueing and publication retain the captured process-lease deadline; neither -can extend a boot's authority while metadata is unavailable. - -On receipt, an owner subtracts maximum clock skew and its self-fence margin -from the wall-clock remaining duration, then projects that duration onto its -local monotonic clock. Request admission reads an immutable grant snapshot and -checks its local deadline, catalog generation, partition identity, and epoch -without taking a lock. The default 12-second lease, one-second skew, and -one-second self-fence margin fence the old owner by ten seconds. A replacement -waits until the old wall-clock expiry plus skew unless an explicit source fence -is durably proven. - -Authority revocation retains the highest accepted grant sequence. The same old -grant cannot reinstall after a clear. Each assignment also carries local -continuity across live renewal: expiry, removal or clear permanently invalidates -an admitted context, even if a later grant restores the same partition and epoch. -Grant publication uses the current monotonic time, so delayed installation -cannot hide a local expiry gap. - -Normal point execution captures that context before asynchronous WAL/tree work -and validates it again before returning success. A revoked context returns -`LeaseExpired` without advertising a successful journal position; already durable -data remains available to replay and a retry with the same request identity. -The captured IO deadline stays at its original value across continuous renewal. -Catching-up targets retain their separately validated transfer-artifact authority. -Both paths also require the live process scope before execution and before -returning success. Grant installation cannot revive a locally fenced process. - -Client catalog revisions and owner epochs are routing references, not serving -authority. A server may resolve an older reference through a process-local -lineage, but it always authorizes the resolved current assignment against one -internally consistent catalog and aggregate grant. The grant's catalog -generation and exact `(partition_id, owner_epoch)` assignment remain mandatory; -the selected partition repeats the epoch check at its WAL and tree boundary. -This separates stale-client tolerance from owner fencing without weakening -either fence. - -Catalog reconciliation matches the tree identity as well as partition identity, -epoch, range, and stream. When a Serving descriptor has materialized its tail -overlay, an unactivated Prepared overlay handle must be replaced by recovery -from the independent checkpoint. It cannot satisfy the materialized assignment -or bypass the split commit proof. Ordinary grant activation applies only after -that independent recovery has completed. - -## 4. Request Contract - -Every request carries a random 128-bit client instance ID, nonzero monotonic -client sequence, catalog revision, partition ID, owner epoch, optional minimum -journal position, and optional deadline. The logical request identity survives -transport retry and rerouting. Endpoint, socket, and connection identities do -not participate in deduplication. - -The contacted server validates the deadline and request envelope, resolves the -key or logical range against its current catalog and recorded local lineage, -then authorizes every partition it will access against the current aggregate -grant before R142 admission. A client topology mismatch is observed for -refresh and metrics but is not itself an admission failure when the current -owner can resolve and authorize the request locally. A same-owner split keeps -an old parent dispatcher for point, multi-get, batch, seek, and scan requests; -the dispatcher selects the current writer by key and never proxies to another -node. If the current assignment is remote, unprepared, catching up, or absent -from the local grant, the server returns the corresponding typed failure and -performs no WAL I/O. - -Lineage dispatch preserves the original request identity and minimum journal -position. A selected writer accepts either its own stream position or an -inherited source-stream position covered by its durable overlay; it rejects an -unrelated stream instead of silently discarding the ordering requirement. -Point operations preserve get, put, delete, put-if-absent, compare-exchange, -and conditional-delete conditions and results. Successful and failed -conditions return the R142 journal position. `Overloaded`, `WriteStalled`, -`Recovering`, `TargetNotReady`, `LeaseExpired`, `RequestExpired`, -`RequestConflict`, `NotMyRange`, and `RefreshRequired` remain distinct wire -outcomes. - -A deadline observed before sequencer admission creates no WAL record. Once the -sequencer accepts a mutation, dropping the transport response does not cancel -the partition worker; retrying the same identity retrieves the recorded result. -Reads may carry a mutation's returned journal position for explicit -read-after-write ordering. - -Multi-get and batch mutation validate their complete key group before IO, then -capture authority for every resolved serving assignment. All captured contexts -are checked against one grant snapshot before and after each member. Revoking -any context stops later members from entering the WAL; a replacement grant cannot -resume the old group. Continuous live renewal preserves the group. Time spent -waiting for earlier members counts toward the original request deadline. -Batch mutation preserves each member's stable identity and individual outcome, -so retry can retrieve already durable results without replaying their mutations. - -## 5. Ordered Reads - -Seek provides ceiling, higher, floor, and lower operations within one logical -partition view. A same-owner split seek may cross the local writer boundary. -Scan intervals are validated and clipped once against the logical routed range; -the dispatcher visits retained and child writers in key order, or reverse key -order for a reverse scan. A forward scan includes its lower bound and excludes -its upper bound. Its continuation resumes strictly after the last emitted key -so the split key and page boundaries neither duplicate nor skip a key. Reverse -scans use the corresponding exclusive upper cursor. - -A continuation binds direction, last key, and the request's logical partition, -epoch, and catalog revision. It remains valid across a same-owner split while -that exact local lineage is retained, because the last key determines the next -writer. A mismatched continuation envelope or an owner transfer that removes -the local lineage returns `RefreshRequired` or `NotMyRange`; the server never -guesses a remote resume position. Multi-partition composition belongs to the -routed client. - -## 6. Split Publication - -### 6.1. Publication boundary and durable steps - -The group-0 catalog head publication is the external range and assignment -cutover. A local writer handoff is a separate operation: the same process can -dispatch the old parent route to the retained parent and child before the new -catalog is visible. This compatibility must preserve every acknowledged WAL -record; it does not authorize a different owner to serve an unpublished range. - -The parent retains its existing tree and journal. Only the child receives new -storage. Handoff is parent-local range dispatch, not replacement of the parent -writer with another tree/WAL and not remote ownership transfer. The successful -durable handoff status update commits the obligation to resume this split; -in-memory dispatch changes afterward. Handoff status is distinct from final -`ChildPrepared` readiness and from external catalog publication. A candidate -epoch or a process-local dispatcher cannot substitute for this status. - -- **S1 — Plan:** persist the transition identity, expected parent assignment, - split key, both successor epochs and complete storage identities. Reserve the - affected partition and owner against another topology transition. -- **S2 — Prepare:** persist `ParentPreparing` before creating artifacts. Pin the - exact base, construct the bounded child view and establish the common WAL - frontier `C`. Persist the exact handoff status before changing local dispatch. - Parent journal appends and reads continue while the status is persisted; - no handoff waiting queue or mutation-worker wait for that update is allowed. - Bind the original parent storage, child storage, pinned base and replay - boundaries. Cover the additional parent suffix before actual dispatch in - child recovery; fixing `C` does not by itself end parent right-range writes. -- **S3 — Ready:** persist the exact retained-parent and child artifacts, - overlays, common cursor and readiness proof as `ChildPrepared`. A process-local - handle or heartbeat cannot substitute for this durable proof. -- **S4 — Publish:** validate that the current catalog still contains the planned - parent. Publish one successor head replacing it with two adjacent Serving - entries. This head is the external split commit point. On an ambiguous result, - reread the authoritative head and accept only the exact intended successors; - an unrelated newer generation is not proof that this split committed. - Materialization may remove overlays and transition markers before the separate - `CatalogCommitted` status write. Recovery also accepts these materialized - successors only when partition, range, owner, epoch, tree and stream identities - still match exactly; it preserves the newer head instead of restoring overlays. - A transition marker on another assignment remains an identity conflict. -- **S5 — Reconcile:** persist `CatalogCommitted` after proving publication. - Install the matching catalog and grants; preserve old-route dispatch without - allowing a retired writer to append. Restart must use the published assignment - together with its exact durable split proof. -- **S6 — Materialize and retire:** checkpoint independently recoverable halves, - publish removal of their overlays and transition markers, then release pins. - Reclaim only artifacts with no catalog, replay, retry or pin references. - -Crash recovery must distinguish these boundaries: - -- Before handoff status commit, recover the published parent and retry or abort - preparation; no child mutation may have been acknowledged. -- After status commit but before in-memory dispatch, resume the same split. - Recover the complete original parent journal, including writes made while the - status update was in flight; an empty child WAL is valid. -- After dispatch but before catalog publication, resume the same split from the - original parent journal and the child journal. The old parent route remains - published; do not discard the child or choose authority by the maximum local - epoch. Neither final readiness nor catalog publication is needed to establish - that the committed handoff must be recovered. -- An uncertain handoff update requires exact durable transition reread. Stale - preparation or abort work must not overwrite a committed handoff. - While the process remains live, retain and retry the exact prepared child - base, WAL handle and pins. Do not reopen its WAL, release its pins or rebuild - a different candidate merely because publication returned an error. Writes - arriving on the parent meanwhile are covered by the later dispatch frontier. - Once dispatch is installed, its live writers own finalization; only a proven - pre-handoff abort may discard the pending candidate. -- After publication but before `CatalogCommitted` persistence, prove the exact - successor entries and finish the transition marker; do not roll back the split. -- After publication, recover each assigned half's complete own WAL. The child - inherits only the parent prefix through the final readiness frontier `C2` - recorded in its catalog artifact, then replays its own WAL. This bound applies - even when the child WAL is empty: later retained-parent records belong to the - parent's next epoch and must not extend child inheritance. Require matching - grants before admitting requests. - -The initial handoff status fixes `C`, before the eventual in-memory dispatch -frontier `C2`. While only that initial status exists, recovery derives `C2` from -the first child record's sequence minus one, or inherits the parent tail when -the child WAL is empty. Final readiness records the exact boundary explicitly; -published catalog recovery uses that boundary instead of rediscovering it from -the current parent tail. - -The durable handoff proof is stored in the split transition while its phase is -`ParentPreparing`. Revision CAS preserves an existing proof and reconciles an -unknown write result by exact reread. Initial and final replay frontiers remain -distinct; additional real-process crash windows are recorded under Open Issues. - -### 6.2. Local execution and storage retention - -The split transition phase symbols are `Planned`, `ParentPreparing`, -`ChildPrepared`, `CatalogCommitted`, and `Aborted`. A separately persisted -`handoff_proof` distinguishes preparation from committed local handoff before -final readiness. It binds the pinned tree base, original parent stream and -initial inherited frontier while the transition remains `ParentPreparing`. -Final readiness extends that frontier without replacing its base. The server and -group-0 monitor advance them in this order: - -1. Group 0 persists the complete plan before local work begins. The existing - parent keeps its identity, tree, stream, owner, and lower boundary; its next - epoch and the one new child's identity, range, epoch, tree, and stream are - fixed by the plan. -2. The parent process persists `ParentPreparing`, renews its serving grant as - an active owner, checkpoints the exact parent base, builds the child's - range-bounded base, fixes `C`, and persists the handoff status without stopping - parent writes or reads. It then installs parent-to-child dispatch. Left-range - writes retain the original parent journal; right-range writes move to the - child journal only when dispatch changes. Replay coverage includes parent - writes between the fixed frontier and that dispatch change. -3. Readiness binds the exact common cutover, the retained-parent next epoch, - and the child's complete tail overlay. - Split plans retain the original parent tree/WAL identity in every phase. - The strict readiness record contains one child overlay and rejects unknown - fields; the planned timestamp is required. - It is persisted before catalog publication. -4. One catalog generation atomically replaces the old parent entry with a - smaller entry having the same partition/tree/stream identity and the next - epoch, then inserts the one child entry. A retry accepts an already-published - result only if transition IDs, ranges, owners, epochs, trees, streams, and - the child overlay all match. -5. The parent owner has already activated both local writers before publishing - the catalog. Catalog reconciliation adopts those handles for external - routing; it neither resumes the retained parent nor activates either writer. - An old parent route dispatches point, grouped, seek, and scan operations - through the retained local lineage. The current catalog and grant authorize - the actual writer or writers used by the operation. - -Before child materialization, heartbeat load reports the child as dependent. -After process restart, the retained parent reopens its original tree and WAL -through ordinary range-filtered recovery; it has no split tail overlay. The -child reopens its exact pinned base, replays the range-filtered parent suffix, -and then replays its own WAL. A matching serving grant alone cannot activate -the child overlay. Startup loads the persisted split transition and requires its committed -phase, transition identity, range, owner, epoch and complete storage artifact to -match the current Serving catalog entry. Transfer overlays retain their separate -committed-transfer validation. Missing or conflicting evidence leaves the writer -Prepared; repeated matching grant refresh is idempotent. - -For independent serving assignments, the local maintenance loop collects retired -ordered WAL chunk records under the existing trim horizon and reader pins. It -requires a live serving grant, an exact local assignment, and no catalog transition -or tail overlay. Each bounded pass releases at most one eligible chunk and record; -a deletion schedules another pass after yielding across hosted partitions. Periodic -scans continue at an unchanged generation, while split materialization keeps its -cadence. WAL trim and tree-root retention do not advance as part of this cleanup. - -For split overlays, the maintenance loop performs bounded ownership -materialization and a serving checkpoint. A heartbeat then proves independent recovery, group 0 -publishes a generation that clears both the overlay and the completed split -`transition_id`, and only that catalog state is eligible for another split or -owner balance. Releasing both markers atomically prevents a later transfer from -mistaking the independently recoverable child for an active split participant. - -The child root-catalog generation is pinned under the split transition identity -before the server persists child readiness. The pin key is scoped by child tree -ID and its value is the exact root generation; it survives both tree and server -restart. Root reclamation treats the oldest durable pin as an upper bound. The -owner releases the pin only after installing the newer catalog generation whose -child has no overlay and whose parent and child have no split marker. Pin delete -is idempotent and remains reconciliation work after an ambiguous response or a -crash. An abort before handoff commit may remove unpublished child state and its -pin. After handoff commit, absence of readiness or a child catalog entry is not -permission to abort or delete child storage; recover and complete the same split. - -The parent also retains its existing checkpoint generation while the child -depends on the inherited parent WAL prefix. Publishing that prefix into the -child's live tree does not release this dependency: the pinned recovery base -still needs those journal records after a crash. Parent checkpoint publication -remains suppressed by the existing transition pin until the catalog records an -independent child. Parent reads and journal appends continue throughout this -interval. The same completed catalog releases both generation pins. - -## 7. Child Balance State Machine - -### 7.1. Owner cutover and durable steps - -The first successful catalog head publication naming the target in -`TargetCatchingUp` is the owner cutover. The later `Serving` publication changes -readiness, not ownership. A candidate target epoch in a plan is not effective -tree or serving authority. Source release precedes the owner cutover and creates -a recovery obligation; it is not itself a catalog owner change. - -- **T1 — Plan:** persist the exact source assignment, target identity and higher - epoch, artifacts, lease bounds and transition identity before preparation. -- **T2 — Prepare source:** persist `SourcePreparing`, pin an immutable root, - initialize the independent target WAL and record replay start cursor `P`. - Source remains the published owner and accepts writes under its existing grant. -- **T3 — Prepare target:** persist `TargetPreparing`; open the pinned base and - replay the source suffix into a Prepared view. Persist readiness before - `TargetPrepared`. Preparation must not advance the shared tree's effective - owner epoch or disable source recovery. Target cannot admit data mutations. -- **T4 — Seal source:** after exact target readiness, close source admission, - drain admitted writes and persist release proof with final durable cursor `C`. - With an unreachable source, wait for lease exclusion and recover the complete - durable tail instead of treating `P` as final. No target data admission is - allowed while the catalog still names source. A durable release cannot be - undone merely because the catalog still has the old assignment. -- **T5 — Publish owner:** validate the source assignment and durable release, - then publish the target epoch, `TargetCatchingUp` and complete replay artifacts - in one catalog successor. Resolve ambiguous publication by exact head reread. - Effective target storage authority must follow this published assignment; - a crash before that authority update is completed must allow idempotent repair. -- **T6 — Catch up:** recover the complete source prefix through `C`, followed by - the target WAL. The published release proof permits durable unconditional - target appends; reads and conditional operations wait for the complete view. - Source cannot append, including through a stale client route or old grant. -- **T7 — Serve:** persist final readiness, publish the same target assignment - as Serving, persist `CatalogCommitted` and activate only with its exact grant. - A crash between head publication and phase persistence must finish the marker - from exact catalog evidence rather than strand or reverse the assignment. -- **T8 — Retire:** clear the completed overlay only after independent recovery - is proven. Release pins and reclaim old resources only after all recovery, - retry and forwarding references have cleared. - -Restart first reads the authoritative catalog and then the matching durable -transition. Before release it restores source; after release but before owner -publication it keeps source fenced and completes publication; after owner -publication it restores target. A local tree epoch, heartbeat or cached page -cannot independently choose the owner. Missing or conflicting proof fails closed. - -An abort is permitted only before durable release and owner publication. Persist -the exact abort before discarding the prepared target; stale workers must not -publish or activate that aborted transition. Candidate epochs are not reused, -and cleanup must retain any source-referenced base, WAL or pin. Restoring source -does not require synchronous deletion of unrelated target temporary objects. - -The effective-tree-authority ordering and publication/phase crash windows still -require implementation verification; known gaps are recorded under Open Issues. - -### 7.2. Execution, proofs and recovery - -A balance transition persists source and target owners, increasing target -epoch, source and target artifacts, readiness limits, old-grant deadline, -phase, release proof, initial readiness proof, final catch-up proof, and -failure. The live-source phases and actions are: - -1. `Planned` → `SourcePreparing`: the source remains `Serving`, checkpoints a - pinned base, registers the target WAL's stable `StreamRecord`, and records - the initial source cursor in the target overlay. It acquires no writer lease - and allocates no chunks for the target WAL. - After pinning the base, an ordered mutation-worker command publishes and - captures the WAL sequence, byte offset and Stream generation together. - Independently sampled counters cannot define a replay cutover: concurrent - mutations may be newer than the previously published Stream prefix. -2. `TargetPreparing`: the remote target validates the range, page-root and - stream identities, initializes or recovers the target WAL under its own - process lease, opens the exact immutable tree-manifest generation named - by the source-base proof rather than the latest root, and starts one shared - async initialization coroutine that replays the source suffix into a - `Prepared` overlay. The source remains writable. The live target handle and - coroutine survive into final catch-up; reopening the same pinned generation - is required only after target process or handle loss. Failure here may abort; - source authority is unchanged. - Prepared transfer handles are staged separately from catalog-owned handles. - Refreshing unrelated catalog entries cannot evict their heartbeat readiness. - Publishing the matching target assignment promotes the same handle before - retiring staging; authoritative abort removes only the exact prepared epoch. - After restart, persisted `TargetPrepared` or `AwaitingFence` evidence restores - that readiness without rewinding the transition or granting serving authority. -3. `TargetPrepared` or `AwaitingFence`: the monitor requests release only when - record, byte, estimated catch-up, deadline, capacity, request-rate, - cooldown, and one-transition-per-owner bounds pass. The source enters one - explicit fencing lifecycle that closes new admission while the existing - mutation worker drains only already-admitted requests, then persists - sequence and byte cursor `C`. If the source is unreachable, the transition waits until old grant - expiry plus skew. A live handoff enters `AwaitingFence` only after group 0 - observes a healthy target heartbeat hosting the exact partition and target - epoch. If the source dies before an explicit fence, the unpublished target - overlay is discarded and the lease-excluded target reopens the source - stream directly, so recovery observes its complete durable tail rather than - treating preparation cursor `P` as the final cursor. -4. `TargetCatchingUp`: group 0 publishes the target owner and epoch with catalog - state `TargetCatchingUp`. The source is no longer catalog authority and - returns the target hint without appending. The persisted release proof is - narrow target journal authority: ordinary unconditional mutations append - immediately to the target WAL from `C+1`, while reads and conditional - mutations coroutine-await target initialization within their request - deadlines. Each wait is capped by the smaller of the remaining request - deadline and a server initialization-wait budget. A process-wide lock-free - admission counter bounds total initialization waiters; a full counter or - server-cap timeout returns `TargetNotReady` with a retry delay. Awaiting does - not block an executor thread and uses no separate transfer request queue. - Successful unconditional writes mean the target WAL append is durable; they - do not consume initialization-wait capacity. Every state other than - `TargetCatchingUp` retains its simpler existing request path. -5. `CatchupPublished`: the existing target coroutine incrementally replays only - the sealed source suffix after its preparation cursor through `C`, then - persists the final catch-up proof. Target-WAL records already admitted after - `C` remain durable and are applied after that complete source prefix. The - target cannot answer a read or evaluate a condition from a shorter prefix. -6. `TargetReady`: group 0 replaces the catching-up entry with `Serving` in a - second generation. Only a heartbeat advertising the exact prepared target - allows a matching serving grant. The transition then becomes - `CatalogCommitted`. If materialization advances the catalog before this - separate status write, recovery proves the same target range, owner, epoch, - tree and stream, preserves the newer head and finishes the status write. - -An owner-loss transfer created after lease exclusion may adopt the original -tree and stream under the higher epoch instead of constructing a live overlay. -This path is valid only after old lease expiry plus skew; writer-epoch adoption -then fences any lower-epoch stream handle. Its `TargetReady` proof publishes -directly from the exact source Serving assignment; no catching-up generation -is required. A retained marker from a previous completed split does not block -this publication when the source has no inherited overlay. A conflicting -range, owner, epoch, artifact or reused operation identity still fails closed. - -For a recovered target without an inherited overlay, normal placement cleanup -may remove its completed transition marker before its first root checkpoint. -The already prepared root handle may then claim publication authority from the -exact current `Serving` partition, range, owner, epoch, tree and stream, with -both assignments free of overlays. Marker removal alone cannot invalidate that -published ownership. A different nonempty transition, another assignment, or a -catching-up entry still cannot authorize publication through this exception; -root authority continues to fence lower owner epochs. - -Recovery uses persisted evidence only. Before source release, a target failure -leaves the source serving and target preparation is repeatable. After release, -abort is forbidden: catalog reread decides whether to publish or resume -catch-up. An ambiguous catalog write succeeds only if the exact intended entry -is observed. A restarted target reconstructs from base plus source and target -tails. A restarted source cannot resume writes after an explicit release or -lease-exclusion proof. Heartbeats and loaded pages are readiness observations, -never authority proofs. - -The recovery decision matrix is: - -| Durable evidence | Authoritative action | -|-----------------------------------------------------|---------------------------------------------------------------------------------------------| -| Plan or source base only; no target readiness | Keep source serving; retry preparation or abort unpublished target state | -| Initial target readiness; no source release | Keep source serving; recheck budgets before requesting the fence | -| Explicit release; catalog still names source | Keep source fenced; publish `TargetCatchingUp` or resolve head ambiguity | -| Lease exclusion; catalog still names source | Recover target under the higher epoch; source cannot reactivate | -| Catalog names `TargetCatchingUp`; no final proof | Coroutine-await reads/conditions; append unconditional target WAL; replay through `C` | -| Final catch-up proof; catching-up catalog entry | Publish the exact `Serving` successor and then issue the target grant | -| Serving catalog entry; grant absent or expired | Keep target prepared and reject admission until the exact grant arrives | -| Ambiguous catalog head write | Reread head and pages; accept only the byte-exact intended generation | -| Conflicting artifact, cursor, range, epoch, or proof | Fail closed; never infer authority from local state | - -The root-catalog generation in a target overlay is a recovery pin, not an -observation. It is distinct from the tree snapshot sequence stored inside that -root. `TargetPreparing`, `CatchupPublished`, catalog refresh, and restart must -all open the exact root generation and then compare the recovered tree snapshot -sequence and applied sequence with the proof. Opening the latest root and -merely checking afterward is invalid: a concurrent checkpoint may move latest -forward, while stale root-cache state may leave it behind. - -For both split and balance, exact-open initializes the page store from the named -immutable root and bypasses latest-root cache refresh until that store publishes -its own successor. Successor publication compares the exact bootstrap generation -and checksum with the current root first; if another writer has advanced the -lineage, publication fails instead of branching. The transition pin is persisted -before readiness publication and is removed only after authoritative catalog -state no longer contains the corresponding overlay and transition dependency. - -The balance invariants are: - -- **SERVER-ONE-TRANSITION:** one partition and each participating owner have at - most one active topology transition; -- **SERVER-NO-DUAL-GRANT:** `TargetCatchingUp` is never included in a serving - grant; -- **SERVER-PROOF-BEFORE-PUBLISH:** every catalog phase is justified by the - corresponding durable readiness or release proof; -- **SERVER-LIVE-TARGET-CONTINUITY:** normal catch-up retains the prepared target - handle and replays only the remaining suffix; exact-root reopen is recovery; -- **SERVER-ASYNC-INITIALIZATION:** readiness waits suspend Rust futures and - never block executor threads or create a transfer-owned request queue; only - `TargetCatchingUp` requests enter this bounded path; -- **SERVER-AMBIGUITY-FENCES:** an unknown publication outcome never reopens the - source writer; and -- **SERVER-CLEANUP-AFTER-PINS:** source tree, stream, retry history, and shared - packs remain retained until catalog references, recovery pins, retry floors, - and forwarding grace have all cleared. - -## 8. Placement Policy - -Balancing targets at least `live_owner_count * target_partitions_per_owner`, -defaulting to four partitions per owner. Split chooses the largest eligible -partition using approximate retained pack estimates. A split boundary comes from -a resident index separator or a bounded small key window; exact byte or key -medians are unnecessary. Live-key witnesses establish nonempty children when -the observation is made. Concurrent mutations can change that observation. -The structural query never loads cold pages or flushes the tree. A service -owns at most one background sampling job, and heartbeats use epoch-fenced cached -samples without waiting for that job. Every serving partition is considered; -an unsplittable largest partition does not exclude the remaining candidates. -Retained pack estimates use the opened manifest, including shared packs, rather -than cumulative write counters. The estimate is deliberately coarse and can -lag unsnapshotted mutations; observation does not scan remote metadata or data. -The retained parent and new split child stay local. Range splitting deliberately -uses an approximate boundary and does not guarantee equal child sizes. Split -sizing and minimum partition supply do not require exact placement equality. -A move uses assigned partition counts; byte estimates remain capacity filters. -An indivisible range may prevent further useful count balancing, in which case -placement remains unchanged. - -### 8.1 Count-Based Placement - -Placement currently uses assigned split count only: `w_i = p_i / P`, including -healthy owners with zero assignments. `byte_weight_percent` is reserved and must -be zero; nonzero settings are rejected until root/range statistics are available. -The data contribution is disabled. Physical shared-pack bytes, memtable bytes, -write counters and accumulated split ratios are not placement data weights. -Tolerance defaults to 20% relative to equal owner share; minimum global-loss -improvement defaults to 25%. Missing or epoch-mismatched assignment observations -still defer planning. Capacity/headroom remains a separate safety filter. - -Tree statistics do not yet expose current root/range page, logical-byte or live-KV -aggregates. Existing retained-pack estimates remain coarse inputs to automatic -split sizing/candidate ranking and capacity checks; they can include shared packs -and lag ordinary flushes. Count balancing does not make those estimates exact. -The split execution, boundary sampling, recovery and publication contracts remain -unchanged. Console does not display Weight or weighted balance explanations. - -### 8.2 Tolerance and Move Selection - -Equal owner weight is `1 / N`. The observation's relative deviation is -`D = max_i(abs(N * w_i - 1))`; `imbalance_tolerance_percent` bounds `100 * D`. -Within tolerance, no load-balancing move is planned. Outside tolerance, evaluate -candidate moves against the same global loss -`L = sum_i((N * w_i - 1)^2)` before and after the move. The partition count -contribution moves; the byte contribution is zero. The normalized denominators -stay fixed for each candidate evaluation. Only the two affected owner terms -change, so each candidate's loss delta requires constant arithmetic. - -A candidate must strictly reduce loss and meet -`minimum_weighted_improvement_percent` of the current loss. A count difference -does not authorize a move that worsens the global count score. Choose the greatest loss -reduction among safe eligible candidates; stable partition/target identities -break ties. Use validated bounded integer/fixed-point arithmetic, sufficient -intermediate width, and conservative threshold comparisons; UI rounding does -not decide eligibility. With unchanged observations, a reverse move increases -the same loss and cannot qualify. Observation changes remain subject to -thresholds and cooldown, rather than triggering actions for every small change. - -For example, two owners with weights 20% and 80% can move a partition whose -contribution is 40 percentage points to obtain 60% and 40%. A relative tolerance -of 20% includes that result because equal owner weight is 50%. This is an -illustration of configured thresholds, not a promise that split yields two -40-point children. When no safe indivisible candidate offers enough improvement, -report no useful move and keep the placement. - -Request rate, target headroom, fresh health, exact assignment and independent -recoverability are safety filters. A parent-tail overlay is ineligible. One -partition and each participating owner have at most one active transition. -Default per-partition cooldown is one minute; cooldown supplements scoring and -tolerance. Split pacing includes recent splits and transfers; repeat-placement -pacing includes only transfers, so split does not postpone first placement. -Transfer preparation, estimated catch-up and forwarding retain independent -ten-minute safety windows. A safe candidate outside tolerance that meets the -improvement threshold must make placement progress within forty seconds when -its cooldown has elapsed; splitting alone is not placement progress. - -### 8.3 Decision Observation - -The monitor's bounded decision observation identifies policy coefficients and -thresholds, catalog generation, assignment/owner identities, observation time, -source freshness, counts, estimated bytes, count/byte weight contributions, -owner weights, deviation and loss. It records the selected or best rejected -candidate's predicted weights and improvement, plus move/no-move reason: -within tolerance, insufficient benefit, unavailable observation, cooldown, -active transition, inherited overlay, unhealthy owner or capacity/rate limit. -Policy version identifies the scoring formula. Diagnostics retain at most 256 -owners, 4096 partition contributions and 2 MiB, independently of the planner's -complete metadata input. Omitted details display unavailable; they do not alter -planning. Unchanged diagnostics are renewed within half their freshness window -rather than written on every tick. Do not emit or persist all candidate pairs -on every tick. This diagnostic -observation is not catalog authority and does not advance catalog generation. -The observation remains backend diagnostic metadata; Console currently hides -Weight and balance explanations. Catalog generation/identity mismatch remains -explicit; UI refresh does not bypass stale cursor rejection. - -## 9. Lifecycle and Observability - -Startup ensures the monitor, registers the instance, loads a complete catalog, -opens each assignment's latest tree root and Stream chunk record as `Prepared`, -reads the applied sequence and WAL replay offset from that tree root, replays -through the stream's current durable tail, reports readiness, and serves only -after installing a matching grant. Shutdown -atomically stops new admission and clears authority; already admitted R142 -operations retain handles and finish before bounded checkpoint/drain work. - -`begin_drain` closes sampling admission before background workers stop. Before -draining hosted WAL writers, `drain_writers` waits for the sampling job's partition -payload and captured Tokio `Handle` to finish teardown while the driver remains -alive, including when queued work is discarded or the job unwinds. Closed sampling -cannot relaunch; completed drain clears its cache and permits no further publication. -This barrier owns sampling only. OS I/O remains uncancellable, and the existing -10-second monitor and 15-second console stop grace periods remain unchanged; -forced termination retains normal writer-lease exclusion. - -Catalog refresh recovers every new or changed local assignment first, then -replaces the catalog and hosted-partition snapshots; unchanged exact-epoch -handles remain live and departed assignments are dropped. -An unknown journal or post-journal apply outcome moves the affected writer to -`Recovering`, closing its admission and excluding it from serving grants. -An unchanged catalog cannot repair that terminal handle. The process detects -this state, clears authority, stops RPC admission and exits so its supervisor -can relaunch it. The successor reloads the durable catalog, tree checkpoint, -complete WAL and persisted split/transfer proofs before obtaining a fresh grant. -No uncertainty flag is cleared in place and no recovery artifact is deleted. -Diagnostics identify the partition, owner epoch and durable/applied sequences; -ordinary `Prepared` activation and idle instances do not trigger a restart. -The recovery check includes prepared transfer targets and retained split -dispatchers with both ingress writers, even after a newer catalog replaces their -assignments. Current-range readiness cannot hide terminal uncertainty in a -compatibility route that remains reachable by older client references. -Catalog-head watch notifications trigger reconciliation immediately and check -the matching serving grant afterward. Instance-specific grant notifications -install authority without reopening or rescanning catalog assignments. -The configured periodic refresh remains -the fallback when notifications are unavailable; grant renewal also continues -on the heartbeat path. Publication therefore does not wait for a full polling -interval before a healthy owner observes a new routing generation. - -Configuration exposes identity, group-0 seeds, separate RPC listen and routable -advertise addresses, dedicated 15xxx HTTP/RPC ports, hosted-partition capacity, -refresh/drain intervals, timing policy, and balance policy. Validation rejects -unsafe timing, zero bounds, bad addresses, unspecified advertise addresses, and -empty discovery seeds. Automatic child-owner balance is enabled by default and -may be disabled explicitly with the balance policy's `enabled` setting without -disabling local partition splitting. Lock-free counters distinguish successes, redirects, -local stale-route dispatch, lease and deadline rejections, overload, split -preparation/base/tail/fence/overlay/materialization work, and internal errors. -Balance observation records selection inputs, base and tail cursors, readiness -failure, catalog and lease phase, catch-up, and background work. Health and -heartbeat views derive from the same sorted partition snapshots and catalog -generation. - -## Open Issues - -- **Prepared transfer crash coverage:** preparation preserves source tree - authority; mutable target authority requires the exact published owner - assignment. Fault injection must cover source and target restart around - that publication, including overlapping source checkpoint work. -- **Split handoff crash coverage:** the durable handoff status precedes local - dispatch and preserves the original parent tree and WAL. Recovery composes - the exact child base, range-filtered parent suffix and child WAL. Fault - injection must cover status persistence with continuing parent writes, - dispatch before final readiness and an unconfirmed status response. Status - persistence remains outside the mutation worker. -- **Head publication before phase persistence:** transfer and split publication - and their committed transition markers are separate writes. Recovery must - prove exact catalog results and finish markers after a crash in between; - fault injection must verify admission and eventual recovery at both transfer - publications and the split publication. -- **Abort racing publication:** an abort decision must be serialized with - publication under the monitor fence and transition revision. A previously - loaded readiness proof must not publish after an authoritative abort. Verify - this race before treating cleanup as safe. - -- Real-process fault injection should continue expanding coverage of ambiguous - catalog writes and process death at every balance phase. -- Overlapping remote transfer preparation, local split materialization and - source restart need fault coverage that proves exact catalog and transition - reconciliation without weakening single-writer or replay epoch checks. -- Completed transfers remove their catalog overlay, but physical reclamation of - the departed source tree and stream, plus expiry of the stale-route - forwarding grace period, still need a manifest-fenced background policy. diff --git a/doc/design/chunkds/design-crowdb-chunk-kv.md b/doc/design/chunkds/design-crowdb-chunk-kv.md deleted file mode 100644 index e9090788e..000000000 --- a/doc/design/chunkds/design-crowdb-chunk-kv.md +++ /dev/null @@ -1,333 +0,0 @@ - - - -# Chunk-Backed Range KV - -`crowdb-chunk-kv` is the embeddable range-partitioned state machine for the -shared-storage metadata index. It composes crowdb-tree's runtime-selected -chunk page store with a private chunk-stream WAL. It does not depend on the -Paxos-facing `crowdb-kv::KVEngine`, and partition count is independent of node -count. - -## 1. Identity and Ownership - -A partition has a stable 128-bit ID, an unsigned-bytewise half-open range -`[start, end)`, one ownership epoch, one tree, one distinct durable stream -name, and one partition-local mutation sequence. Unbounded endpoints are -allowed. A split key must be strictly inside the source range and its two -child ranges must be adjacent and exactly cover the parent. - -The lifecycle is `Closed`, `Recovering`, `TransferQuiesced`, `Prepared`, -`Serving`, `TransferFencing`, `SplitPreparing`, `SplitFinalizing`, `Retired`, -or `Faulted`. -Data-path admission reads atomics and reserves bounded request and byte -capacity. Lifecycle control closes mutation admission and waits -asynchronously for the admitted count to reach zero. The manager registry and -split transition record use lifecycle-only locks; point reads and mutation -admission do not acquire them. - -## 2. Tree and Journal Ownership - -Every partition owns a `PartitionTree` and `PartitionJournal`. Production -adapters wrap crowdb-tree and R141 `ChunkStream`; injected implementations -support deterministic tests. The raw stream handle is private, so a server -cannot bypass partition ordering, trim, or epoch checks. - -The tree enforces its logical request range at construction. A retained parent -keeps the original physical root, whose pages may still contain keys and -separators outside the narrowed range until background pruning completes. -Reopening that root validates frame checksums and structure without treating -those physical keys as corruption; point reads, scans and mutations still -enforce the current logical range. Importing a newly rebuilt range snapshot -continues to require every imported key to fit its destination range. -Rust never implements the tree page -path through `crowdb-chunk-client`; the production tree selects R140's native -page-store backend at runtime. A balance target opens the source's pinned tree -manifest directly from shared page chunks and owns a distinct target WAL. It -does not copy keys through Rust or copy page packs before handoff. - -Point reads, ceiling/higher/floor/lower, and bounded forward/reverse scans read -only the applied tree prefix. A forward scan uses an inclusive lower bound and -exclusive upper bound; its distinct continuation cursor resumes strictly after -the last emitted key. Forward operations use the native merged lower bound. -Reverse operations use a native predecessor descent that fixes the L0 -memtable set, root page, and GC floor for the page, merges the highest revision -for each L0/L1 collision, and skips tombstones before returning descending -keys. Both directions enforce count and byte budgets inside C++; Rust does not -compose point reads, materialize the range, or sort results. - -## 3. Mutation Ordering - -`RequestId` contains a 128-bit client instance ID and a 64-bit client sequence. -A canonical SHA-256 digest covers operation kind, key, value, and condition, -but excludes routing and ownership fields. Reusing an ID with another digest -returns `RequestConflict` before I/O. - -One bounded MPSC sequencer assigns monotonically increasing mutation -sequences. It evaluates put, delete, put-if-absent, compare-exchange, and -conditional delete in queue order against the applied tree plus successful -earlier entries staged in the same batch. Both applied and condition-failed -outcomes become checksummed WAL records. Therefore a lost response can be -retried without reevaluating its condition. - -The sequencer appends frames to the chunk stream in mutation order. It changes -the tree only after the append is durable, applying one successful record per -tree call and advancing a failed condition as a no-op. A response is released -only after the applied frontier advances and carries the frame's stream name -and starting logical offset. Ordinary concurrent reads see the prior applied -prefix; a minimum journal position waits asynchronously for explicit -read-after-write ordering. - -## 4. WAL and Recovery - -Each physical frame has magic, version, flags, a bounded body length, a -serialized logical record, CRC32C, and a canonical 128-bit physical chunk ID -trailer. The CRC covers the header and body, not the trailer. The journal -submits header+body+CRC through chunk-bound stream append; the stream selects -the chunk and adds its identity. Recovery validates frame completeness, -checksum, source-chunk identity, partition identity, epoch, operation digest, -result revision, sequence continuity, and request-ID uniqueness. The durable -acknowledged cursor is the recovery upper bound, so identity validation never -promotes residual bytes. Recovery never reevaluates a conditional result. A -timestamp is omitted; any future timestamp is diagnostic-only. - -A checkpoint contains the explicit 64-bit R140 tree identity, tree manifest -identifier, applied mutation sequence, stream name, and replay offset. The -tree identity is allocated and persisted; it is never derived by truncating -the partition's 128-bit identity. Recovery rejects a checkpoint or prepared -child whose tree identity differs from the injected tree. The replay offset is -the oldest WAL record needed by retained retry results, rather than simply the -current tail. -Recovery reads those pre-checkpoint frames to rebuild the retry cache without -reapplying them, then applies only the suffix above the checkpoint sequence. -WAL trim cannot cross the checkpoint's replay offset. Requests below the -declared retained floor return `RequestExpired` and are not executed anew. - -Serving maintenance scans retired ordered WAL chunk records within the configured -record and byte budgets. Collection retains the current tail, records above the -existing trim offset or generation horizon, and chunks protected by live interval -pins. It releases at most one eligible physical chunk and its record per pass, -then recaptures the retention proof before another deletion. A successful deletion -schedules another bounded pass after yielding to other hosted partitions; a pass -without deletion resumes on the periodic tick, even if the generation is unchanged. -Maintenance does not publish a new tree root or advance WAL trim or tree retention. -Split preparation and exact-root transition pins suppress metadata reclamation -under the existing maintenance exclusion. Source checkpoint publication and its -durable generation pin share that exclusion, so GC cannot pass between them. - -Physical WAL trim at a nonzero replay offset requires durable expired-request -floors as well as retained results. That floor state is not yet checkpointed, -so the explicit trim API rejects such reclamation with `MaintenanceDegraded`. -The WAL remains retained until the recovery format carries this state; keeping -only the recent result window would let expired requests execute after restart. - -The frontiers satisfy -`checkpoint_seq <= applied_seq <= journal_durable_seq`. The stream resolves -uncertain cursor and manifest outcomes against durable state before completing -the append. If a journal append nevertheless returns an error, the partition -enters `Recovering`; later writes and reads require recovery rather than -mistaking that failure for a quiesced transfer source. A non-OK tree result -after durable append is `ApplyStateUnknown`, also moves only that partition to -`Recovering`, and prevents later records from applying. The C ABI catches C++ -exceptions before they can cross into Rust. - -## 5. Overlay Split and Writer Handoff - -A `SplitPlan` names one transition, the exact parent ID/range/epoch, an -interior split key, the parent's next epoch, and one new child identity. The -existing parent retains its partition, tree, stream, and owner identities and -shrinks to `[old_start, split_key)`; the new child receives -`[split_key, old_end)`. Repeating the same active plan is idempotent; changing -any identity, range, epoch, or stream fails closed. The new child initially -remains on the parent owner. Placement is a later balance operation. - -Split creates only the child's tree and WAL. The retained parent uses its -existing tree, WAL and mutation path; it is not rebuilt into a second -destination tree or moved to a replacement WAL. Local handoff means that -traffic still enters the parent, which dispatches the child range to the child. -It is distinct from the later catalog publication that exposes both ranges to -external routing. A higher candidate epoch alone proves neither boundary. - -Split preparation follows these ordered steps: - -1. Enter `SplitPreparing` and range-rebuild the child base from one exact - pinned parent tree view. The parent remains the only mutation sequencer and - continues its normal WAL and memtable path while the child tree, WAL, and - live memtable are prepared. -2. Prepare the child storage and establish an ordered parent frontier `C`. - Fixing this frontier does not stop parent journal appends or reads. Persist - a handoff status in the split transition, binding the parent and child - identities, epochs, pinned base and WAL replay boundaries. Successful durable - status update is the local handoff commit point. Resolve an uncertain result - by reading that exact transition; an in-memory `SplitCutover` message is not - durable evidence. Do not await this update in the mutation worker or add a - split-owned request waiting queue. -3. After the handoff status is committed, install in-memory key dispatch to - the child. Until dispatch changes, writes continue to the original parent - WAL, including writes after `C`; afterward, left-range writes continue to - that same WAL and right-range writes enter the child WAL. The handoff proof - and replay rules must cover the parent suffix between fixing `C` and changing - dispatch exactly once. `C` alone cannot be treated as the final inherited - parent frontier if parent writes continued beyond it. The child view must - include those acknowledged writes for reads and conditional mutations. - Seed its bounded request result cache and recover any acknowledged child-WAL - tail without reevaluating conditions. A mutation racing dispatch must have - exactly one writer and preserve its logical request identity. -4. The route change does not seal, publish, checkpoint, or wait for a - memtable. The child immediately reads its base, its live memtable, and the - right-range inherited parent view, including the covered handoff interval. - Conditional mutations consult that same merged view. -5. Complete inherited-view publication in background, preserving the original - parent tree and its normal checkpoint path. The child materializes only its - range; newer child revisions win over older inherited entries. Retain the - shared view and replay dependencies until the complete inherited frontier - and durable storage proofs are recorded in one immutable `SplitArtifact`. - Handoff status is not final readiness and does not replace this artifact. -6. Publish one catalog generation that shrinks the retained parent (same ID, - new epoch) and inserts exactly one new child. The parent owner already has - both writers active, so refresh only publishes this catalog and serving - grant to external routing; it does not gate local handoff, recover the - retained parent, or replay a cutover. - -The child base checkpoint records two independent counters: its logical tree -snapshot sequence and its chunk root-catalog generation. Immediately after the -child base is durable, preparation persists -`root//pin/ = child_root_generation`. Readiness is -not recorded until that operation succeeds. A retry with the same generation is -idempotent; another generation under the same transition identity is corruption. -Preparation may remove the unpublished child's pin after an authoritative abort -only before handoff commit. After handoff commit, recovery continues the same -split and retains its base, WAL and pin even if final readiness is absent. - -A prepared or restarted child recovers in three layers: open the exact base -manifest, replay the retained parent suffix through the final inherited frontier -with range filtering, then replay its own WAL from the recorded child start. -The proof must include parent writes accepted while handoff status was being -persisted; the earlier fixed `C` is not automatically that final frontier. -Records at or below the base sequence restore retained request outcomes without -reapplying tree mutations. Failed -conditions remain no-ops with their original result. Sequence gaps, a wrong -source stream or epoch, a mismatched range, or a child WAL beginning anywhere -other than the proof's exact child start reject recovery. - -The child recognizes two minimum-position namespaces. A retained parent-stream -position covered by the inherited suffix is satisfied by the inherited overlay -frontier. A child-stream position waits for the child's applied frontier. New -child mutations return only child-stream positions; retained-parent mutations -continue returning parent-stream positions. This preserves read-after-write across a -catalog split without treating two streams as one offset space. - -After activation, bounded background passes independently prune the retained -parent tree to its smaller range and materialize the child's shared page -ownership. A checkpoint of the complete child view into its own tree root and -WAL permits the catalog to clear the parent-tail overlay and mark the child -independently recoverable. Parent manifests, stream bytes, retry results, and -page packs remain pinned while the child catalog artifact, a retained -checkpoint, forwarding grace interval, or retry floor references them. -The catalog generation that clears the child overlay also clears the retained -parent and child transition markers atomically. Only after the child owner has -installed that authoritative generation may it idempotently release the -transition's child-root pin. A crash between catalog installation and unpin is -a retention leak resolved by reconciliation, never permission to reclaim early. - -The split invariants are: - -- **SPLIT-PARENT-STABLE:** split preserves the parent partition, tree, stream, - owner, and lower range boundary and creates exactly one new child; -- **SPLIT-ONE-WRITER:** before in-memory dispatch changes the parent sequences - the full range; afterward parent and child sequence disjoint ranges; -- **SPLIT-ROUTE-FRONTIER:** the fixed preparation frontier, durable handoff - commit and in-memory dispatch are distinct boundaries. Recovery includes all - parent writes through the final inherited frontier and the complete child - tail; no acknowledged mutation falls between them or executes twice; -- **SPLIT-HANDOFF-DURABLE:** persist handoff before child write admission; - restart after commit resumes the same split even before catalog publication; -- **SPLIT-OVERLAY-DURABLE:** base plus parent suffix plus child WAL reconstructs - values and request outcomes before activation; -- **SPLIT-NO-FOREGROUND-WAIT:** routing does not wait for memtable seal, - parent-history replay, filtered publication, checkpoint, catalog publication, - or ownership materialization; and -- **SPLIT-PIN-BEFORE-RECLAIM:** physical deletion never precedes the last - catalog, snapshot, forwarding, or retry reference. - -Implementation verification remains required for the durable handoff status and -the exact replay/sequence representation covering continued parent appends -between fixing `C` and installing dispatch. It must not be resolved by pausing -parent writes, queueing requests behind status persistence, or retaining a -second destination tree/WAL for the parent. - -## 6. Remote Balance Storage Contract - -Balancing one independently recoverable child reuses the same base-plus-tail -representation. A live source checkpoint supplies the pinned base manifest, -base sequence, Stream chunk record generation, and replay offset. The target owns -a distinct WAL and opens the shared tree root as `Prepared`. One shared async -initialization coroutine replays the source stream while the source continues -serving. It first reaches a persisted preparation cursor `P`, then follows the -small moving source tail. Awaiting this coroutine suspends only the caller's -Rust future; it does not block an executor thread, poll an atomic frontier, or -place the request in a transfer-owned pending queue. - -After target readiness, the source checks record, byte, estimated catch-up, -and preparation-deadline budgets before changing journal ownership. It stops -assigning new records to the source WAL, drains only records already assigned -there, and records their final durable sequence and byte offset as `C`. One -short `TransferFencing` lifecycle closes public mutation admission while the -existing mutation worker drains the already-admitted set; it is not a second -queue or an independent authority flag. Only after that drain does the source -enter `TransferQuiesced`, which permits the exact handoff checkpoint. The -target artifact is extended from `P` to `C`. The live target keeps its opened -tree, memtable, and replay coroutine and reads only `P+1..C`; reopening the -pinned base and replaying the complete retained suffix is crash recovery, not -the normal handoff path. - -Once the release proof fixes `C`, ordinary unconditional mutations may append -immediately to the target WAL beginning at `C+1`; they do not wait for source -tail replay, a checkpoint, or page materialization. Success means that target -WAL append is durable; their tree application remains ordered after the source -suffix and proceeds asynchronously. Reads and conditional mutations await -the initialization coroutine because they require the complete source prefix. -After the coroutine applies through `C`, the partition enters its normal -serving path: reads execute directly and conditional mutations evaluate -against the complete view. Target-WAL application then continues in order. -No request handler retains a thread while awaiting initialization. Both each -wait and the process-wide waiter count are bounded; capacity exhaustion returns -the same retryable initializing result as a bounded wait timeout. -Only catalog entries in `TargetCatchingUp` execute these checks. A `Serving` -entry takes the ordinary direct read or mutation path without inspecting an -initialization future or touching waiter accounting. - -After catalog and grant activation, only the target WAL can advance. A -dead-source recovery may adopt the original stream directly, but only after -the old lease expiry plus clock skew proves exclusion. - -The balance invariants are: - -- **BALANCE-PREPARE-WHILE-SERVING:** live target preparation never fences the - source writer; -- **BALANCE-RELEASE-BEFORE-ACTIVATE:** target mutation authority requires a - durable source release or lease-exclusion proof; -- **BALANCE-APPEND-WITHOUT-TREE-WAIT:** after release, unconditional target - mutations may enter the target WAL before source-tail application completes; -- **BALANCE-COMPLETE-PREFIX:** a target serves only after replay through `C`; -- **BALANCE-ASYNC-READINESS:** reads and conditional mutations coroutine-await - initialization without an executor-thread block or transfer request queue; -- **BALANCE-INDEPENDENT-SOURCE:** a child retaining a split-parent suffix is - ineligible for another owner handoff. - -## 7. Failure Containment and Observability - -Range, epoch, size, condition, request-conflict, expiry, and admission errors -do not damage partition health. Availability failures on a tree read remain -localized to that read. Corruption, conflicting recovery records, and unknown -post-journal apply state require recovery of the affected partition only. -Checkpoint and GC failures retain the prior manifest and WAL authority. - -Per-partition lock-free counters cover mutation requests and outcomes, ordered -seeks and scans, range and stale-epoch rejection, admission backpressure, -journal failures, unknown apply outcomes, recoveries, checkpoints, and split lifecycle -events. Split counters distinguish preparation and base-checkpoint time, -tail records and bytes, catch-up lag and finalization duration, overlay replay records and -bytes, and materialization duration. Snapshot frontiers expose lifecycle, -stream identity, durable sequence and byte offset, and applied sequence without -combining independent partitions. diff --git a/doc/design/chunkds/design-crowdb-chunk-stream.md b/doc/design/chunkds/design-crowdb-chunk-stream.md deleted file mode 100644 index 786b92104..000000000 --- a/doc/design/chunkds/design-crowdb-chunk-stream.md +++ /dev/null @@ -1,627 +0,0 @@ - - - -# CROWDB - Design: Chunk Stream - -The chunk stream presents finite mirrored chunks as one durable, ordered -logical byte space. Append completion depends on mirrored data persistence; -Stream metadata records lifecycle transitions and bounded chunk record checkpoints. -The stream owns physical framing and addressing; the partition WAL owns -business-record framing, request identity, and replay policy. - -Depends on: [chunk IO](../chunkio/design-crowdb-chunkio.md), -[chunk reader](../chunkio/design-crowdb-chunkio-reader.md), -[DiskIO](../diskio/design-crowdb-diskio.md), -[chunkdb](../chunkdb/design-crowdb-chunkdb.md), and -[group-0 sysdata](../kv/design-crowdb-kv-group0.md). - -## Contents - -1. [Ownership and API](#1-ownership-and-api) -2. [Storage contracts](#2-storage-contracts) -3. [Metadata](#3-metadata) -4. [Append protocol](#4-append-protocol) -5. [Rollover and recovery](#5-rollover-and-recovery) -6. [Reads](#6-reads) -7. [Trim and reclamation](#7-trim-and-reclamation) -8. [Bounds and observation](#8-bounds-and-observation) -9. [Errors and invariants](#9-errors-and-invariants) -10. [Open issues](#10-open-issues) - -## 1. Ownership and API - -`crowdb-chunk-stream` is an asynchronous Rust library. `StreamName` is an -opaque, time-ordered 128-bit identity bound to a logical partition, never to a -process or node. It renders as 32 hexadecimal digits. `ChunkStream` exposes -`create`, `open`, `open_read_only`, `tail`, vectored `append`, exact `read_at`, -seekable bounded sequential reading, `trim_prefix`, and `close`. -An owner-side flush/checkpoint barrier captures and persists a synchronized -prefix on demand without changing ordinary append completion. - -Every nonempty append returns the selected chunk identity and exact logical -range. Logical lengths count caller payload bytes, excluding physical frame -headers and footers. A zero-length ordinary append returns no chunk identity -and performs no IO. `append_chunk_bound` and `append_chunk_bound_batch` keep -chunk selection in the ordered worker; the outer Stream frames carry the -selected chunk identity and CRC. They add no payload trailer. The stream does -not interpret the caller's business-record format or retry identity. - -One append payload is one indivisible validity unit. A successful append makes -the whole payload available; recovery never exposes a proper prefix of that -payload. Physical frame and IO splitting are internal. Batching preserves each -request's append boundary rather than making the entire batch one group. - -Each writable handle owns one bounded Tokio MPSC queue and one worker task. -The task exclusively mutates the active chunk's in-memory cursor and mappings, -logical tail, and lifecycle publication state. Producers and readers use -atomics and immutable `ArcSwap` views. Append and snapshot publication add no -hot-path lock. Batching preserves request order and exact returned ranges. - -`open_read_only` validates one exact writer epoch and captures a finite view. -It does not adopt writer authority, allocate a successor, seal a chunk, or -publish writer state itself. It may request a flush/checkpoint from the current -owner. If the owner is unavailable it uses the consistent prefix persisted in -ChunkDB and records that checkpoint as its finite bound; that view can lag -successful appends. A caller requiring a newer exact cutover must not silently -accept the older fallback. The handle supports source-tail replay during split -and balance with these explicit bounds. - -## 2. Storage Contracts - -Three injected async traits separate Stream state from placement and transport: - -- `StreamRegistry` loads and creates the stable group-0 `StreamRecord`. - It is used by create/open, never by ordinary append or data reads. Typed - binary keys contain the Stream ID; metadata defaults to nonzero group 1. -- `StreamMetadataStore` selects the nonzero group containing ordered - `StreamChunkRecord` values. A shared record store implements conditional - lifecycle transitions, reverse seek, candidates, interval leases and GC. -- `StreamChunkStore` allocates and seals mirrored chunks, writes and synchronizes - the required replicas, exposes current chunk layout for bounded reads and - recovery, and idempotently releases trimmed chunks. It does not require a - separate ChunkDB cursor advance for each append. - -Chunk IO and ChunkDB own `chunkinfo`: strips, physical capacity, replica -placement, growth, replacement, and their durable internal transitions. Each -such transition updates chunk information independently. Stream metadata refers -to stable chunk identities and logical mappings; internal chunk changes are -transparent and never trigger Stream chunk record publication. -Recovery and read-only opening resolve current capacity and layout through the -chunk layer instead of trusting capacity cached in an older Stream checkpoint -or a retained adapter. Opening queries authoritative chunk metadata even when -the adapter has a cached layout; an unavailable query fails the open rather than -falling back to that cache. The checkpoint must still fit the current capacity. - -DiskIO writes carry a boolean sync option. For a synchronized request the server -performs the ordered write, then the required flush, and returns one response -after both succeed. The client does not wait for a write reply and then send a -second fsync RPC. Buffered writes do not request the flush. When transport -limits require several requests, preserve strip-block write ordering and -complete required synchronization before acknowledging the append. - -Every allocation preserves the durable `StreamPurpose`: system journals use -`ChunkType::Wal`, business streams use `ChunkType::Stream`. Both use the -canonical `stream/` owner key followed by the 128-bit stream identity. Reopen -rejects purpose disagreement between StreamRecord, StreamChunkRecord, chunk identity, and -chunk information, including a missing or mismatched owner key. Chunk-KV -journals require Wal streams. - -`ProductionStreamRuntime` shares the process KV client and discovered chunk IO -routes. The Chunk-KV server chooses the metadata store; the StreamRecord -chooses its metadata group. `create_registered` initializes a control-plane -StreamRecord. The `test-util` implementation supplies deterministic mirror-write, -sync, tail-content, ownership, and metadata-publication fault injection. - -## 3. Metadata - -### 3.1 Identity and ordered keys - -Group 0 stores the stable `StreamRecord`: purpose, Stream ID, metadata group, -record version, registration state, consumer kind and ID, and creation time in -Unix milliseconds. Writer leases are separate Group-0 records. Append, -preparation, rollover, close and writer relocation preserve the identity value -and original creation time. Consumer identity describes who owns the Stream; -it does not identify the currently leased process. - -A nonzero metadata group stores `StreamChunkRecord` values under typed keys: - -```text -record namespace | stream_id[16] | kind[1] | order[8 big endian] -Attached: kind = 0, order = logical_start -Candidate: kind = 1, order = reservation_sequence -``` - -Each record has a chunk ID, purpose, state, reservation/attachment sequence, -writer epoch/session, publication generation, trim watermark, closed status and -compressed mappings. Attached records contain explicit half-open -`[logical_start, logical_end)` bounds. A Candidate has no assigned logical -range and accepts no user data. Attached Active and Sealed records sort by -logical start; candidates occupy a distinct suffix range. Reverse scanning the -attached prefix returns the current tail without a separate moving head. - -The tail record carries Stream trim/closed state and the publication counter. -A successor inherits that control state. An empty Stream has an empty Active -record from creation; close seals it and persists closed status. The tail -record is retained even when every byte is trimmed. The Active record's end -is a durable checkpoint, not a limit on recovery or the live in-memory tail. - -There is no persisted Stream-wide physical offset and no duplicated -`physical_end` in a StreamChunkRecord. Chunk-local physical positions derive -from append geometry; the chunk's exact seal length is authoritative at release. - -### 3.2 Compact complete-append mappings - -One mapping describes one complete append group, regardless of frame count. -Store append lengths, from which both logical and chunk-local physical offsets -are derived. With grouped-frame payload capacity 65,489 and overhead 47 bytes, -a group of length `L` advances physical position by -`L + 47 * ceil(L / 65489)`. A physical frame maximum of 64 KiB does not impose -a 64 KiB append maximum. - -Lengths are encoded in blocks of at most 256 entries. Equal lengths use a -constant run; other blocks choose canonical variable-length u32 values or raw -u32 values by encoded size. In-memory samples store each block's logical and -physical base and encoded position. A random seek decodes at most one block. -These blocks are an encoding detail inside one record, not persisted pages or -a Stream-owned index tree. - -The checksummed record validates key/start identity, purpose, exact decoded -logical coverage, group lengths and bounded physical geometry. A new checkpoint -preserves all previously published append boundaries. Rollover validates exact -continuity between the finalized predecessor and successor. Bounds detect -inconsistent coverage; CRC32C also detects same-length corruption. - -### 3.3 Physical framing - -All consumers use one versioned physical frame format and common Rust/C++ -codecs. Each frame is at most 64 KiB including header and footer. Magic identifies -purpose and version; a common header role identifies ordinary, head, or follower -frames. Only Stream uses grouped roles. User-data and tree consumers use ordinary -frames; user-data completeness remains a whole-object publication condition. -Native buffers retain bounded streaming and zero-copy ownership. - -Each nonempty Stream append is exactly one verifiable frame group in one chunk. -Its head declares total payload bytes and frame count, including the head. Even -a single-frame append is a head with count one. Followers identify their head -by its physical offset in the same chunk and carry their ordinal. Frame lengths, -roles, group identity, order, count, total payload length, CRC, and expected chunk -identity must all agree before the group is valid. There is no separate commit -frame or commit RPC. CRC32C covers the common header, group fields, and payload; -the footer chunk identity is validated separately against the expected chunk. -All codecs use the same CRC input and byte layout. -Declared lengths/counts are checked against append limits, remaining chunk -capacity, and checked arithmetic before scanning or reserving memory. A parser -does not allocate buffers proportional to an untrusted declared group size. - -Role-dependent header sizes determine payload capacities. Interior frames fill -their role's payload capacity and only the last may be short. Location arithmetic -derives group geometry from the format version, role capacities, and total -logical length, without one mapping per frame. - -Revision 2 retains the 14-byte fixed prefix: little-endian magic, payload offset, -payload size, and write timestamp. The low magic byte is 2; the high byte keeps -the public purpose. The role byte follows the prefix. Ordinary frames use role -0 and a 15-byte header, leaving 65,501 payload bytes. Stream heads use role 1 -and add total payload length (u64) and frame count (u32); followers use role 2 -and add head physical offset (u64) and ordinal (u32). Both grouped headers are -27 bytes and leave 65,489 payload bytes. Group fields are little-endian. -The 20-byte footer stores the little-endian CRC32C followed by the big-endian -chunk identity. Revision-2 CRC starts at zero and excludes the footer. - -Only the canonical revision-2 frame format is accepted. Unknown revisions, -roles outside their purpose and noncanonical header offsets are rejected. - -`StreamChunkRecord` uses a single current `CSCR` version-1 envelope with -exact body length and CRC32C covering header and body. Decode rejects unknown -versions, trailing bytes, invalid states, excessive counts and oversized bodies -before using mappings. No migration or older-service fallback is supported. - -### 3.4 Atomic publication - -A lifecycle publication conditionally updates the captured tail key revision. -Rollover atomically finalizes the predecessor, deletes its registered Candidate -reservation and inserts the new Active record under its actual logical start. -An empty predecessor and successor may share the same logical-start key; only -the successor remains attached there. Same-chunk checkpoint, trim and close -update only the tail record. Historical chunk records retain their bytes. - -The first attachment reserves Candidate sequence zero, then atomically deletes -that reservation and inserts the empty Active record, guarded by the Candidate's -concrete revision. It checks that no attached tail exists after reservation. -It never uses an absent attached key as its initialization fence: that key can -be deleted by later trimming, so an excluded initializer could otherwise insert -it again. A later initializer can replace an older owner's initial reservation; -its new revision excludes the delayed publication without a mutable Stream head. - -A revision conflict invalidates the captured proof. Re-read the tail and any -Candidate identity before retrying. For an unknown outcome, compare the complete -durable tail with the proposed transition; an exact match is committed. A -superseding writer/session rejects stale publication. Pin and Candidate changes -also fence through the tail revision, preventing stale retention observations -from racing publication or reclamation. - -## 4. Append Protocol - -Admission reserves bounded request slots and queued bytes. Oversized appends are -rejected before enqueue. An idle worker dispatches the first request immediately -and drains only requests already queued. It stops at the request limit, byte -target, or current chunk capacity, without adding a batching delay. One large -request may exceed the batch byte target while remaining within append bounds. - -The normal path for an already attached chunk is: - -1. Select non-overlapping logical ranges and encode one complete frame group - per append. Keep one location per payload rather than one extent per frame. -2. Submit mirror writes concurrently, requesting synchronization in the write - requests. The chunk layer owns strip ordering, repair images, replica - replacement, and durable updates of its own chunk information. -3. Wait for successful synchronized completion from every required replica in - the configured mirror set. Three-copy configuration waits for three replicas; - no fixed replica count is assumed by the Stream API. A smaller set of readable - replicas does not lower the write-completion requirement. The chunk layer - restores required protection after replacement before returning success. -4. Publish the completed in-memory cursor, logical tail, and extent snapshot, - then complete each request with its exact logical range. - -There is no per-batch `advance_chunk_write`, group-0 update, Stream chunk record -write, or metadata CAS. Encoding, buffer ownership, and staging remain bounded; -frame splitting does not justify repeated payload coalescing or per-frame KV -writes. Periodic lease maintenance is separate from the append data path. - -An ambiguous mirror response is not evidence of absence. Resolve it through the -chunk write/recovery contract; do not retry at a guessed offset or use an old -ChunkDB cursor to declare new bytes absent. A failure on one request cannot let -later requests skip a logical hole. Watchdogs observe the original future and -do not cancel, resubmit, or complete it. An interrupted append contributes no -partial logical range. Recovery may retain a complete append whose response was -lost, but never its proper prefix; consumer retry semantics handle that uncertain -completion. Differing valid replica tails follow section 5.2. - -## 5. Rollover and Recovery - -### 5.1 Lifecycle ordering - -One append never straddles chunks. A chunk contains at most the configured -physical framed-byte limit, 256 MiB by default. Chunk IO may allocate physical -strips incrementally; this remains internal to the chunk layer. - -Capacity preparation runs ahead of consumption. Measured write bandwidth -and preparation duration size the initialized spare capacity with a two-interval -margin, bounded between one frame and one physical strip. The physical allocator -rounds this target to strip boundaries without adding another growth multiplier. -As consumption approaches the hard chunk boundary, one initialized Candidate is allocated -and registered while the Active chunk accepts appends. Its initial capacity -follows observed batch consumption, not the configured maximum batch size. -Small cold streams retain their first strip without eagerly initializing spare -chunks. Normal writes use ready capacity; exceptional -allocation/initialization delays apply backpressure instead of exposing unsafe -capacity. - -When an append cannot fit, drain preparation and writes for the current chunk, -physically seal its exact final length, take the prepared Candidate, and commit -the atomic record transition. The successor accepts data only after attachment. -Switching does not allocate or rewrite historical metadata. Seal and attachment -are distinct durability boundaries: a crash after seal but before publication -leaves the one last referenced chunk to reconcile. An unattached Candidate has -no acknowledged user data and can be reclaimed after owner exclusion. - -Different segment IO streams can drain concurrently, with at most eight drains -in flight across the complete layout, including during tail recovery. Strip -boundaries do not serialize independent drain requests or consume the prefix -scan's remaining owner deadline. Every submitted drain must finish, including -after a peer fails; physical sealing requires all drains to succeed and current -writer authority. Recovery retains its original captured deadline and requests -durable mirror repair only after all submitted drains have completed. - -### 5.2 Tail reconstruction - -The latest physical cursor is the end of the contiguous prefix of complete -append groups in the last referenced chunk. Recovery starts -at that chunk's beginning or a validated synchronized checkpoint and parses -forward within current authoritative capacity. It checks frame kind, lengths, -CRC, chunk identity, and address arithmetic. It tracks a tentative frame cursor -inside a group and advances the recovered physical cursor and logical tail only -after validating the entire group. It then adds one compact location. - -Before publishing a fresh attachment or admitting writes into newly allocated -capacity, zero and synchronize each new physical strip on every required mirror. -Initialization uses one submission per replica, without automatic retries. An -initialization pass admits at most two strips at once; their required mirrors -still synchronize together. This bounds zeroing bursts while ordinary writes -use the initialized prefix. Every submitted initialization finishes before the -pass returns, and capacity is exposed only after every strip succeeds. An -uncertain result leaves that capacity unavailable to the writer; the owner cannot -repeat initialization after later data writes. A replacement strip receives its -complete retained prefix and a zero suffix with the same single-attempt contract -before its metadata publication. - -Ordinary append writes exactly its complete group range. It never writes a marker -past that end: a delayed duplicate must not overlap the next acknowledged group. -Unwritten headers are supplied by the initialized suffix. Capacity preparation -reserves a complete 14-byte header below the hard chunk limit, growing before -writing when necessary. This keeps the previous complete prefix bounded inside -already initialized capacity if growth stops before its new strips are ready. -An exact hard-capacity end requires no extra header. Allocation, initialization -and repair traffic are accounted separately from ordinary append bytes. - -The bounded repair shadow retains only the unfinished strip prefix. Validation -failures before the first data IO do not consume that owner's fresh-write -admission. Block alignment must preserve all physical bytes outside the requested -range even when the partial-block cache is absent. - -An unwritten suffix, incomplete final frame, or incomplete group bounds the -candidate prefix at the previous complete append. No later group can skip that -hole. Individually valid frames in an incomplete group do not advance the tail. -Corruption in required retained data is not silently treated as EOF: use mirror -validation/repair or fail recovery. Bytes with a wrong chunk identity cannot -extend the tail. Syntactically valid bytes beyond the last checkpoint remain -candidates; a stale persisted cursor is not an authority to discard them. - -The shared parser keeps one at-most-64-KiB frame and constant group/hash state. -Input windows are at most 1 MiB; output batches contain at most 1,024 complete -locations. Observed bytes and parsed frames are distinct from committed group -progress. An exact-capacity end needs no extra header read. Required grouped -checkpoints are exact append boundaries. A SHA-256 digest covers only the encoded -complete prefix; incomplete groups do not enter it. A failed observation cannot -subsequently be finalized as a healthy shorter prefix. - -After owner exclusion, observe the current available mirrors concurrently -within a bounded IO budget. When retained DiskIO topology covers every segment, -order each strip's replicas by rack and node for consistent observation lanes -across strips. Allocation order alone can alternate racks, making every lane -depend on a slow cross-rack link. These route observations only order existing -segments: incomplete hints preserve original ordering, no replica is added or -omitted, and the original authoritative layout remains unchanged. Continue -checking every completed lane's checksum and complete-prefix agreement under -the same deadline. Use the shortest complete-append prefix among healthy -mirrors that finish observation. A mirror that fails or times out during the -read is unavailable, not a zero-length or partially observed prefix. Corrupt -replicas are not valid shorter candidates; use healthy copies or report an -error. All candidates must agree on retained overlapping contents. Conflicting -valid contents fail explicitly rather than waiting for an external arbiter. -No surviving readable candidate means recovery is unavailable. Repair restores -configured protection before the next successful append. Replica comparison and -checkpoints use complete-group boundaries, never an interior frame boundary. - -Only the last referenced chunk may require content parsing. Earlier chunks -have finalized bounds and chunk record mappings and are not scanned. A last -chunk whose seal and final mappings are already published needs no tail scan. -Recovery after an interrupted seal may still reconstruct that one last chunk. -The ChunkDB finalizer may reclaim and tombstone a physically empty tail before -the Stream publishes its successor. Recovery accepts that exact tombstone only -with its seal timestamp, valid purpose and nonzero writer epoch, zero capacity, -zero sealed/acknowledged/checkpoint boundaries, and no remaining strips or cleanup -intents. It retains the logical position, leaves the tombstone unchanged, and -attaches a fresh chunk under current writer authority. A missing chunk or a -nonempty or unproven tombstone never becomes an empty prefix. -Bound reads, buffered bytes, parsed frames, and total scan bytes by the active -chunk geometry, independently of retained Stream length. - -### 5.3 Writer ownership - -Writer open validates purpose and current ownership before exposing a writable -handle. A lower epoch is rejected. A higher epoch adopts retained history -without copying historical chunk records, resolves the old tail, seals/finalizes it, and -attaches a fresh chunk before new appends. Writers never resume appending to a -previous process's chunk. - -Group-0 owner leases remain authoritative between metadata checkpoints. The -writer uses a conservative local monotonic deadline derived from its grant and -checks authority before accepting work and before acknowledging it. Background -renewal is independent of append batches. Expiry or explicit fencing stops both -new admission and successful acknowledgment, including after a process pause. -Metadata CAS still fences lifecycle publications; DiskIO success alone does -not authorize an acknowledgment and DiskIO does not own Stream leases. - -Explicit transfer stops the old owner's admission, drains its in-flight work, -persists the final synchronized boundary, and records a fenced release proof -in Group 0. Once release and the new grant are authoritative, the new owner can -activate without waiting for the old lease's natural expiry. A verbal/local -promise or an unresolved release response is not the durable release proof. - -If the old owner cannot be contacted, activation waits for its lease's safe -expiry boundary, including the existing clock-skew allowance. Resolve in-flight -old writes using the bounded write-age/drain rules before fixing the old tail. -The new owner recovers that tail and publishes a newly allocated chunk; it -never resumes writing the old owner's chunk. Group-0 unavailability cannot -extend a local grant past its conservative deadline. - -### 5.4 Read-only capture - -Read-only open validates the requested epoch and captures the tail chunk record, -trim point, active identity, and one finite validated tail: - -- When the current owner is reachable, request an ordered flush/checkpoint - barrier. The owner captures its synchronized prefix, persists the ChunkDB - complete-group boundary, then publishes the corresponding chunk record checkpoint - before returning the matching epoch/chunk/physical cursor/logical end. A failed - record publication leaves the prior checkpoint authoritative even if ChunkDB - already holds a newer cursor. Later append does not widen that view. Already - synchronized appends do not require another data fsync to capture the prefix. -- When the owner is unreachable, use the captured persisted ChunkDB boundary - with matching Stream metadata. Reconstruct missing mappings only through that - boundary without writer adoption or publication. The result is an explicitly - older checkpoint view, possibly empty for the active chunk. - -Epoch, chunk identity, chunk record/checkpoint, and end must form one consistent view; -both physical and logical checkpoint bounds end complete groups. A raw newer -ChunkDB cursor is not combined with an unrelated chunk record or epoch. Do not combine -observations across ownership changes. Expose the captured bound -and whether it came from owner flush or checkpoint fallback. A requested exact -upper cutover beyond that bound is unavailable, not a successful shorter replay. -Checkpoint fallback is a read-visibility rule: writer failover still recovers -the content-derived tail and cannot discard acknowledged data beyond it. - -## 6. Reads - -Reads reject ranges below `trim_offset` or beyond their captured logical tail. -For persisted data the reader reverse-seeks the greatest attached key whose -logical start is at or below the requested offset, validates coverage, and -locates its complete append through the record's compressed mapping samples. -Active unpublished data uses append-only in-memory mappings. Expand only the -intersecting physical frames and return the requested payload after validation. -Reads cross append and chunk boundaries while preserving logical order. - -A sequential reader accepts a start and finite byte hint or `ToEnd`, captures -its end once, and returns EOF there. The default retained window is 8 MiB, each -physical request is at most 1 MiB, and at most eight requests run concurrently. -Independent readers may run concurrently. Observation watchdogs do not cancel -or retry underlying IO. - -The lock-free record cache has 32 slots, each admitting at most 1 MiB of -encoded mappings plus seek samples. Oversized records remain readable but are -not cached. A record lookup loads only its target chunk, independently of -historical Stream length. Snapshot publication uses immutable ArcSwap views; -finite readers retain their captured view and interval lease through trim. - -A provenance-aware reader yields physical chunk identity along with payload. -The journal owns business-record validation, sequence continuity, filtering, -and retry-result semantics. Physical Stream frame validation applies regardless -of consumer. Readers and overlays preserve their captured bounds through later -append, rollover, and ownership changes. - -## 7. Trim and Reclamation - -`trim_prefix(g)` rejects regression and `g > tail`. Persist the new trim -watermark on the tail before reclaiming data. Only complete Sealed chunks whose -logical end is at or below the watermark are eligible. Partial trim preserves -the chunk record's original bounds and mappings. The current tail is retained. - -A finite reader leases its logical interval and exact Active chunk identity. -The latter protects unpublished local mappings. Pin creation, renewal and -release conditionally fence the tail revision; collection captures that revision -before observing pins. A changed revision invalidates the entire proof. -Reader pins are limited to 512 identities of at most 1 KiB each. They renew -within a 60-second lifetime and enforce an earlier local monotonic deadline. -Expired pins protect IO for a further 31 seconds before bounded cleanup. -Expiry cannot be renewed into a new live view. WAL consumers publish their -safe replay/retry watermark before trim; split/transfer checkpoint suppression -and finite captures preserve their retained dependencies. - -Collection combines the trim watermark, caller-proven generation horizon, -whole-chunk state and all live interval protections. It scans a bounded prefix -with a logical-start continuation and processes at most one eligible physical -chunk per pass. Mark the record Deleting under the captured tail revision, -release the chunk using its exact derived complete physical length, and delete -the record only after release is confirmed. A failed release or delete leaves -a retryable Deleting record. Repeating trim resumes cleanup. At least one -eligible whole chunk can progress even if it exceeds the physical byte target. - -A generation alone never authorizes deleting retained data. There are no -Stream directory roots, page reachability walks or historical metadata copies. - -## 8. Bounds and Observation - -Defaults bound admission at 1,024 requests and 64 MiB, a batch at 64 requests -and a 1 MiB byte target, one append at 64 MiB, and one chunk at 256 MiB with one compressed record -per chunk. Sequential reading uses an 8 MiB window, -eight concurrent physical requests of at most 1 MiB each, and GC uses a 64 MiB -pass budget. Bounds are configurable and validated; record/decode/cache budgets, -and optional checkpoint budgets must also bound their own work. Replica count -is configured, not implied by a physical-byte metric. - -One encoded chunk record is limited to 24 MiB, and its mapping count is bounded -by the physical chunk geometry. Decode requires canonical 256-entry blocks, -except the final partial block. Collection enumerates at most 128 keys and loads -values individually under its byte target, allowing at most one final record -to cross that target; it never buffers every enumerated value at once. - -Lock-free counters and phase histograms distinguish: - -- append admission, queue, capacity preparation, batching, framing, synchronized - mirror RPC completion, in-memory publication, and end-to-end completion; -- chunk-internal layout updates from Stream lifecycle publications, seal, - ownership maintenance, and optional checkpoints; -- total chunk sealing latency and its segment-drain portion, including failures; -- chunk record build, get, scan and conditional publication latency, mapping - counts/encoded bytes, and cache hits/misses; -- recovery chunks/bytes/frames scanned, replica resolution/repair, cursor and - extent reconstruction, and read-only capture; -- logical/physical bytes, queue high-water marks, retained buffers/cache, read - amplification, reclaimed bytes, orphan backlog, and watchdog observations. - -Periodic perf output and cumulative benchmark output retain count, average, -p50, p99, and maximum. Phase scopes and whether counts describe requests, -batches, or replicas are explicit. Nested percentiles are not added. Ordinary -append samples must not conceal control-plane waits inside a mirror stage. -Publication counts and latency include initial attachment and writer adoption; -opening a read-only handle does not count as a publication. - -Performance acceptance uses release artifacts and a matched three-node native -fixture with `mem-block` KV/WAL and retained MemDisk payload bytes. Record the -replica count, buffer/queue settings, checkpoint policy, binary identities, and -fixture parameters. The fixture removes physical-media latency without skipping -framing, transport, synchronization responses, or read validation. It proves -process/reopen behavior while retained media exists, not power-loss durability; -durability fault tests use storage that survives the tested failure. - -The regression workloads and minimum gates are: - -- 4 KiB append, one producer: at least 250 ops/s and p99 at most 10 ms. -- 4 KiB append, 32 producers: at least 3,000 ops/s and p99 at most 20 ms. -- 1 MiB append, eight producers: at least 125 MiB/s and p99 at most 150 ms. -- 4 MiB rollover, eight producers: at least 200 MiB/s and p99 at most 500 ms. -- 4 KiB random reads, 32 producers: at least 30,000 ops/s and p99 at most 5 ms. -- 320 MiB replay, one producer: at least 300 MiB/s and p99 at most 2 s. -- 576 MiB prefix trim: at least 1,000 MiB/s and p99 at most 100 ms. - -These are regression floors, not a claim that multi-millisecond memory-backed -append latency is the performance objective. Matched before/after distributions -must show the effect of removed KV waits and the combined write/sync RPC. -Separately report standalone Stream append, actual Chunk-KV WAL, and client -mutation latency. - -Sentinels require zero errors/watchdogs, bounded admission, less than 256 MiB -peak workload RSS growth, no per-append ChunkDB cursor or chunk record writes, -history-independent normal append work, and at most one tail chunk scanned on -recovery. With optional checkpoints disabled and close outside measurement, -chunk record publications do not exceed initial attachment plus rollover count. Cache -and lookup accounting follow accessed records and configured budgets. GC must -release at least one complete eligible chunk. - -## 9. Errors and Invariants - -`StreamError` distinguishes invalid requests, backpressure, stale writers, -definitely absent operations, unresolved outcomes, stalled writes, unavailable -reads, corruption, and internal invariant failures. Absence cannot be inferred -solely from a stale checkpoint cursor. - -- **I1 — Durable completion.** A successful append has exact ordered logical - ranges and synchronized data on every required mirror replica. Per-append - ChunkDB cursor and chunk record updates are not completion prerequisites. -- **I2 — Bounded record work.** Historical chunk records retain their bytes; - publication touches only the tail and successor, and open/seek loads only - bounded target records. -- **I3 — Publication and ownership.** Data is not admitted to an unattached - chunk; attached records describe complete append groups; epochs never regress; - Group-0 lease expiry or release excludes stale acknowledgments/publications. -- **I4 — Bounded recovery and views.** Only the last referenced chunk can need - parsing. Recovery preserves acknowledged bytes beyond old checkpoints and - establishes a validated contiguous prefix. Read-only ends are finite; an - owner-unavailable checkpoint fallback explicitly permits an older view and - cannot silently satisfy a newer requested cutover. -- **I5 — Retention.** Logical trim precedes deletion and reclamation respects - whole-chunk liveness, logical intervals and every retained consumer pin. -- **I6 — Consumer separation.** Wal and business streams share the same storage - contract. The consumer owns business-record and retry-result semantics. -- **I7 — Logical addressing.** One append stays in one chunk, framing bytes never - enter logical offsets, compatible multi-frame payloads use compact locations, - recovery accepts the entire append or none of it, and rollover/trim never - renumber retained bytes. -- **I8 — Chunk isolation.** Internal chunk layout and `chunkinfo` changes do not - mutate Stream chunk record generations or copy Stream metadata. -- **I9 — Synchronized write RPC.** A synchronized write response follows both - write and flush; it does not require a separate client fsync round trip. -- **I10 — Resource and observation bounds.** Queues, batches, buffers, record - caches, recovery, and GC are bounded; the hot path adds no lock, and watchdogs - observe without taking over IO completion. - -## 10. Open Issues - -There are no unresolved decisions for the append/recovery contract. Verifiable -bounded frame groups, all-required-mirror completion, Group-0 owner leases, and -explicit live-view bounds define the contract. Byte packing, decoder fixtures, -and rollout are implementation work under those constraints. - -Metadata-group rebinding, per-stream sharding, and sealed-chunk EC conversion -remain separate scale-out topics. diff --git a/doc/design/chunkio/chunkio-small-io-flow-analysis.md b/doc/design/chunkio/chunkio-small-io-flow-analysis.md deleted file mode 100644 index a224e32e1..000000000 --- a/doc/design/chunkio/chunkio-small-io-flow-analysis.md +++ /dev/null @@ -1,372 +0,0 @@ - - - -# CROWDB - Design: Small IO Flow Analysis - -This document records the full-stack small-write and read benchmark flow, -baseline measurements, correctness limits, and ordered optimization work. The -data-path contracts are defined by -[`design-crowdb-chunkio.md`](design-crowdb-chunkio.md), -[`design-crowdb-chunkio-small-object-writer.md`](design-crowdb-chunkio-small-object-writer.md), -and [`design-crowdb-chunkio-reader.md`](design-crowdb-chunkio-reader.md). - -## Contents - -1. [Benchmark boundary](#1-benchmark-boundary) -2. [Small-write flow](#2-small-write-flow) -3. [Read flow](#3-read-flow) -4. [Baseline results](#4-baseline-results) -5. [First divergence](#5-first-divergence) -6. [Bottleneck analysis](#6-bottleneck-analysis) -7. [Ordered improvement plan](#7-ordered-improvement-plan) - -## 1. Benchmark Boundary - -The sentinels use a co-located three-node cluster with three instances each of -KV, DiskDB, ChunkDB, and DiskIO. KV data and WAL use `mem-block`; fsync is -disabled. DiskIO uses `NullDisk`, so requests traverse client routing, -crowdb-rpc, the DiskIO request handler, and io_uring without retaining payload -data. Each small-write matrix case starts a fresh combined deployment. Process -exit releases the in-memory KV and WAL contents before the next case, bounding -RSS independently of allocator behavior and detached task completion. Read -preparation performs real writes before the measurement window and retains the -resulting locations. - -| Parameter | Value | -| -------------------- | ------------------------------------------ | -| Timed duration | 20 seconds per case | -| Client concurrency | 1, 4, 32; up to 128 and 256 for small IO | -| DiskIO connections | 8 in sentinels, 4 in CLI; configurable | -| DiskIO RPC workers | 1 client and 1 server worker by default | -| Small object sizes | 1 KiB and 8 KiB writes; 8 KiB reads | -| Large read object | 16 MiB | -| Large read EC | 8+4, 1 MiB shards | -| Read dataset | 16 objects per selected class | -| Mixed read | 50% small and 50% large by request count | -| Write pipeline limit | 32 | -| Write batch limit | 1 MiB or 1,024 objects | - -A result is valid only when every admitted operation is accounted for, -`errors=0`, `incomplete=0`, and the watchdog count is zero. Throughput after the -first error is diagnostic only because immediate failures can increase the -request counter without doing IO. - -## 2. Small-Write Flow - -```text -worker - -> prepare_small_write - -> enqueue whole object in shared writer pool - -> queue-byte/object threshold may add a pipeline - -> completed pipeline immediately drains the objects currently queued - -> drained objects form the next mirror strip without a batching delay - -> ChunkDB allocation and DiskDB/KV metadata commit - -> DiskIO mirror writes to NullDisk - -> completion returned to every object -``` - -Pipeline scale-out is based only on queued bytes or queued object count. A new -pipeline is added only when every active route is under queue pressure, so a -startup burst on pipeline zero cannot create 31 empty pipelines. Scale-in also -uses state rather than elapsed time: all routes must be empty and idle, and the -global reservation count must be zero. - -The benchmark exposes both logical batches and physical aggregation. For every -normal mirror write it counts the number of represented objects, submitted -buffers, logical bytes, and actual full-strip payload bytes. Repair and -mirror-to-EC traffic retain their separate counters. It also records the peak -pipeline count. Dedicated E2E coverage stops admission and observes scale-in -to the configured minimum from queue and busy state. Neither scale-out nor -scale-in uses elapsed idle time as a decision input. - -The pipeline follows the RPC send-queue scheduling model: completion of the -current batch immediately fetches all currently available work, bounded by the -object and byte caps. The 500 ms batch watchdog only reports a stuck durability -operation. It never delays a batch, triggers a flush, or cancels work. - -## 3. Read Flow - -```text -untimed preparation - -> real small or large write - -> ChunkDB/DiskDB/KV metadata commit - -> retain writer-produced Location - -timed request - -> query ChunkDB layout - -> read requested mirror or EC segment through routed DiskIO RPC - -> on segment error, try mirror fallback or partial EC recovery - -> persist observed failed segments in ChunkDB - -> assemble and return object -``` - -The failure-observation path distinguishes durable segment failures from -transient transport, overload, timeout, alignment, and short-read results. -Only durable failures are persisted in `unavailable_segments`; transient -failures participate in fallback or EC recovery for the current request only. - -## 4. Baseline Results - -Measurements were taken on 2026-09-09 on the Intel Core i9-7960X development -host. DRAM totals are passive host-wide PMU samples and include all co-located -services. - -### 4.1 Small Write - -| Size | Threads | TPS | MiB/s | p50 ms | p99 ms | Objects/batch | Peak pipes | Errors | Valid | -| ----- | ------: | --------: | ----: | -----: | -----: | ------------: | ---------: | -----: | :---: | -| 1 KiB | 1 | 424.42 | 0.4 | 2.309 | 3.532 | 1.00 | 1 | 0 | yes | -| 1 KiB | 4 | 856.48 | 0.8 | 4.627 | 6.095 | 2.00 | 1 | 0 | yes | -| 1 KiB | 32 | 6,275.96 | 6.1 | 5.032 | 9.448 | 8.26 | 2 | 0 | yes | -| 1 KiB | 128 | 19,508.32 | 19.1 | 6.166 | 13.627 | 15.36 | 5 | 0 | yes | -| 1 KiB | 256 | 31,313.08 | 30.6 | 7.549 | 19.388 | 21.21 | 8 | 0 | yes | -| 8 KiB | 1 | 390.94 | 3.1 | 2.492 | 3.909 | 1.00 | 1 | 0 | yes | -| 8 KiB | 4 | 675.29 | 5.3 | 5.323 | 10.882 | 2.03 | 1 | 0 | yes | -| 8 KiB | 32 | 4,464.84 | 34.9 | 6.701 | 18.595 | 8.65 | 2 | 0 | yes | -| 8 KiB | 128 | 13,461.05 | 105.2 | 8.791 | 23.066 | 11.25 | 7 | 0 | yes | -| 8 KiB | 256 | 23,446.56 | 183.2 | 9.811 | 27.503 | 18.97 | 9 | 0 | yes | - -The timer-free single-worker p50 is 2.31 ms for 1 KiB and 2.49 ms for 8 KiB; -the earlier roughly 56 ms rows were invalidated by the io_uring wake defect and -the batching deadline. The highest measured rates in this run are 31,313.08 TPS -for 1 KiB and 23,446.56 TPS for 8 KiB, both at 256 workers. These are observed -maxima, not demonstrated saturation peaks. Aggregation rises naturally with -queue occupancy even though there is no batching delay. All watchdog counters, -errors, and incomplete counts are zero. The retained matrix is -`bench-log/chunkio-small-completion-drain-20260909/results.tsv`. - -A dedicated 1 KiB, one-worker distribution rerun completed 8,489 requests at -424.42 TPS: p50 2.309 ms, p90 2.665 ms, p95 2.918 ms, p99 3.532 ms, and max -26.575 ms. Maximum observed queue delay was only 90 us. Server and client stage -windows place the steady-state cost primarily in two serial durability phases: -the three parallel NullDisk mirror writes average roughly 0.72-0.85 ms, followed -by the fenced `advance_chunk_write` metadata commit averaging roughly -1.1-1.4 ms. The retained distribution run is -`bench-log/chunkio-small-latency-distribution-20260909/results.tsv`. - -### 4.2 Read Before the Correctness Fix - -| Workload | Threads | Success | Errors | Success TPS | MiB/s | p50 ms | p99 ms | Valid | -| -------- | ------: | ------: | --------: | ----------: | ----: | -----: | -------: | :---: | -| Small | 1 | 387 | 0 | 19.35 | 0.2 | 51.643 | 52.110 | yes | -| Small | 4 | 31,988 | 0 | 1,595.21 | 12.5 | 2.479 | 3.242 | yes | -| Small | 8 | 24,599 | 220,333 | 1,229.86 | 9.6 | 2.250 | 3.299 | no | -| Small | 16 | 16,478 | 527,755 | 823.84 | 6.4 | 0.962 | 2.872 | no | -| Small | 32 | 7,479 | 1,397,831 | 373.94 | 2.9 | 1.062 | 1.841 | no | -| Small | 128 | 29,419 | 2,880,180 | 1,470.86 | 11.5 | 2.008 | 4.631 | no | -| Small | 256 | 11,676 | 2,985,510 | 583.74 | 4.6 | 4.633 | 58.562 | no | -| Large | 1 | 24 | 0 | 1.19 | 19.0 | 850.917 | 952.716 | yes | -| Large | 4 | 312 | 20 | 15.43 | 246.9 | 227.787 | 387.001 | no | -| Large | 32 | 629 | 445 | 30.26 | 484.2 | 260.863 | 818.468 | no | -| Mixed | 1 | 88 | 0 | 4.37 | 35.8 | 119.247 | 866.818 | yes | -| Mixed | 4 | 652 | 61 | 31.40 | 228.2 | 22.008 | 358.932 | no | -| Mixed | 32 | 551 | 1,134 | 26.36 | 253.5 | 248.930 | 1,009.263 | no | - -Only error-free rows are baselines. Failed rows do not establish a peak because -the failure cascade truncates real IO and then spins on metadata-marked -segments. - -### 4.3 Final Read Baseline - -NullDisk now completes its generated-content contract at the requested length, -transient read errors no longer update durable layout metadata, cluster reset -waits for fresh service registrations, and flushing keeps frozen memtables -visible until their entries are published in L1. All final 20-second cases -completed with zero errors and zero incomplete requests. - -| Workload | Threads | Success TPS | MiB/s | p50 ms | p99 ms | -| -------- | ------: | ----------: | ----: | -----: | -----: | -| Small | 1 | 783.11 | 6.1 | 1.268 | 2.270 | -| Small | 4 | 3,418.85 | 26.7 | 1.164 | 1.725 | -| Small | 8 | 8,691.62 | 67.9 | 0.874 | 1.661 | -| Small | 16 | 21,254.08 | 166.0 | 0.730 | 1.183 | -| Small | 32 | 36,815.96 | 287.6 | 0.843 | 1.558 | -| Small | 128 | 57,098.87 | 446.1 | 2.139 | 3.799 | -| Small | 256 | 56,879.50 | 444.4 | 4.359 | 6.947 | -| Large | 1 | 13.79 | 220.7 | 71.241 | 88.609 | -| Large | 4 | 70.20 | 1,123.2 | 55.410 | 86.171 | -| Large | 32 | 173.76 | 2,780.2 | 181.237 | 275.468 | -| 50/50 mix | 1 | 24.30 | 194.9 | 40.443 | 94.923 | -| 50/50 mix | 4 | 135.18 | 1,082.4 | 28.849 | 82.793 | -| 50/50 mix | 32 | 352.08 | 2,818.1 | 36.473 | 236.431 | - -Small read reaches its useful saturation region around 128 workers: increasing -to 256 does not improve TPS and roughly doubles p50. The main retained matrix -is `bench-log/chunkio-read-final-20260909/results.tsv`; the two preparation -cases rerun after the readiness fix are in -`bench-log/chunkio-read-readiness-fix-20260909/results.tsv`. - -### 4.4 Read After the io_uring Wake Fix - -The common DiskIO io_uring poller previously accepted an idle-ring submission, -woke from `epoll_wait`, and then waited up to 50 ms for a completion before it -published the new SQE. It also reused the external completion `eventfd` as its -private submission wakeup. A private wake `eventfd` now separates those roles, -and the poller publishes submissions immediately after wakeup. - -| Threads | Success | TPS | p50 ms | p99 ms | Errors | Valid | -| ------: | --------: | --------: | -----: | -----: | -----: | :---: | -| 1 | 15,735 | 786.71 | 1.269 | 2.011 | 0 | yes | -| 4 | 68,215 | 3,410.15 | 1.168 | 1.712 | 0 | yes | -| 32 | 744,514 | 37,198.96 | 0.841 | 1.462 | 0 | yes | -| 256 | 1,116,927 | 55,789.01 | 4.441 | 7.224 | 0 | yes | - -At one thread this changes p50 from 51.7 ms to 1.27 ms and throughput from -19.36 to 786.71 successful reads/s. Stage histograms show ChunkDB query at -about 0.8 ms and DiskIO read at about 0.4 ms for one thread. At 256 threads, -ChunkDB query remains about 0.4 ms while DiskIO read grows to about 4.7 ms and -dominates the request. The retained stage result is -`bench-log/chunkio-read-stage-final-20260909/results.tsv`. - -## 5. Resolved First Divergences - -The earliest confirmed read divergence is a short DiskIO read from NullDisk: - -```text -disk I/O error: PartialWrite -``` - -The return-code name is shared with writes, but the server emits it when a read -completion returns fewer bytes than requested. The eight- and sixteen-worker -small-read runs both eventually show this exact error. Earlier occurrences on -other mirror replicas had already been persisted as unavailable, so the next -short read exhausts all three replicas. - -The failure then amplifies: - -1. DiskIO returns a transient short-read or transport failure. -2. `StripReader` adds the segment to `failed_segments` without an error class. -3. `ChunkReader` persists every observed segment in `unavailable_segments`. -4. Mirror reads eventually have no eligible replica; EC recovery eventually - has more missing shards than parity can reconstruct. -5. Subsequent requests fail before payload IO and rapidly inflate the error - counter. - -The 32-worker client also records `rpc.send.queue.full.c=19`. This confirms RPC -backpressure is reached, but the observed short read is the earliest directly -reported trigger. Queue-full events and short reads are both transient and -must not be converted into durable media-failure metadata. - -The fix makes NullDisk synthesize the complete requested buffer after any -nonnegative backing completion. DiskIO RPC/short-read failures are typed as -transient, while confirmed missing disks, missing zones, and media IO errors -remain durable. Mirror and EC readers persist only the durable class. A real -ChunkDB E2E verifies that an injected transient mirror failure leaves -`unavailable_segments` empty and that the original client can still read the -object. - -The next divergence was the 50-ms low-concurrency latency in the common -io_uring poller described in section 4.4. A C++ regression submits consecutive -operations after the ring becomes idle and verifies that neither waits for the -poll timeout. - -At 256 workers, the next first divergence was a transient metadata `NotFound`. -Repeated probes showed the same key becoming visible tens of microseconds -later without a rewrite. `Crowdbtree::flush()` had removed the draining frozen -memtables before publishing their entries in L1, creating an L0/L1 visibility -gap for concurrent readers. The draining tables now remain in `frozen_` until -L1 publication completes. A stable sentinel in the concurrent freeze/drain -test proves that an immutable key never disappears during this transition. - -Cluster reset had a separate readiness race: DiskDB could start before DiskIO, -and stale service heartbeats could satisfy registration checks. Reset now -starts DiskIO first and requires post-restart heartbeats for DiskIO, DiskDB, -and ChunkDB. Combined deployment also waits until every ChunkDB `/ready` -refreshes and exposes the published range assignment. - -## 6. Bottleneck Analysis - -### 6.1 Small Writes - -A post-instrumentation 8-KiB/32-worker run completed 62,567 objects in 32,424 -batches. Three mirror copies produced 97,272 normal DiskIO writes representing -187,701 object buffers, or 1.93 buffers per physical request on average. The -requests carried 1.43 GiB of replicated new logical bytes but 94.99 GiB of -full-strip payload. Peak pipelines reached 32. This confirms that the next -small-write optimization should reduce full-strip rewrite amplification and -improve batch fill together, rather than optimizing object admission alone. -The retained result is -`bench-log/chunkio-small-write-20260909-095047/results.tsv`. - -Two queue-accounting problems explained the excessive scale-out. A carried -object that did not fit the current batch was subtracted from queue pressure -twice, underflowing the unsigned counter. Synchronized batch boundaries also -looked idle and caused scale-in/scale-out thrashing while admitted objects still -held reservations. The corrected policy has real E2E coverage for carried -objects, byte-pressure scale-out, object-pressure scale-out, and final -state-driven scale-in. - -With a 16-object pressure threshold, 8-KiB/256 reached 23,257.11 successful TPS, -used at most 9 pipelines, and performed 51,288 physical mirror RPCs for -1,397,154 represented objects: 27.24 objects per physical request. Compared -with the earlier 8-KiB/32 run, throughput is 7.46 times higher and aggregation -is 14.1 times higher while the peak pipeline count falls from 32 to 9. The -retained result is -`bench-log/chunkio-small-tune-balanced-q16-256-20260909/results.tsv`. - -### 6.2 Reads - -The added `chunkio.chunk.query.e2e` and `chunkio.diskio.read.e2e` histograms -separate metadata from payload IO. KV engine get is about 4-5 microseconds and -KV RPC get is about 36-82 microseconds, so KV get is not the observed limit. -At 256 threads the DiskIO stage, including RPC, queueing, io_uring, and response -delivery, accounts for nearly all of the 4.44-ms median. - -Connections and both RPC worker counts are now independently configurable: - -- benchmark client: `--diskio-connections` and `--diskio-rpc-workers`; -- DiskIO service: `--rpc-workers`; -- combined deployment: `--diskio-rpc-workers`; -- regression scripts: `CHUNKIO_*_DISKIO_CONNECTIONS`, - `CHUNKIO_*_DISKIO_RPC_WORKERS`, and `CHUNKIO_*_SERVER_RPC_WORKERS`. - -An A/B small-write run at 8 KiB and 256 threads measured 22,802.91 successful -TPS with 8 connections and 1+1 workers, versus 23,993.26 TPS with 16 -connections and 4+4 workers. The 5.2% single-run increase is modest and is not -yet sufficient to change the defaults. The valid results are retained in -`bench-log/chunkio-small-diskio-default-ab-retry-20260909/results.tsv` and -`bench-log/chunkio-small-diskio-c16-cw4-sw4-final-20260909/results.tsv`. - -The initial read A/B attempts exposed the ownership-readiness and memtable -visibility bugs above, so they are not retained as performance results. The -final default-configuration matrix is fully valid; a repeated worker-count A/B -remains future measurement work. - -### 6.3 Foreground EC Sentinel - -The steady-state sentinel runs the foreground-EC path for 20 seconds per case, -reports reservation startup latency and parity bytes, and requires nonzero -foreground parity bytes with zero errors, incomplete objects, and watchdogs. - -Latest 20-second EC results (errors/incomplete/watchdogs all zero): - -- 1 KiB, 32 threads: 47,104.93 objects/s, p50 493 µs, p99 7,236 µs, - parity 234,881,024 bytes -- 1 KiB, 128 threads: 128,382.64 objects/s, p50 598 µs, p99 11,172 µs, - parity 184,549,376 bytes -- 1 KiB, 256 threads: 207,246.10 objects/s, p50 686 µs, p99 15,063 µs, - parity 209,715,200 bytes -- 8 KiB, 128 threads: 32,430.72 objects/s, p50 1,141 µs, p99 29,215 µs, - parity 306,184,192 bytes - -Source: `bench-log/chunkio-small-write-20260910-103318`. The 8 KiB 256-thread -case is excluded (7 batch watchdog expirations at extreme concurrency). - -Historical A/B baseline (60s, mirror vs EC): EC/mirror ratios were 100.23% -(32t) and 92.01% (128t). The mirror-only mode was removed from the sentinel -after the small-write path stabilized; EC is now the default and only mode. - -## 7. Ordered Improvement Plan - -1. Make pipeline retirement independent of a slow replacement-allocation RPC. - A sustained 32-pipeline run can scale in partway and then block the manager - while one retiring pipeline waits on DiskDB; pending conversion groups must - remain recoverable by ChunkDB tasks. -2. Repeat the 20-second DiskIO connection and worker matrix with at least three - samples per cell now that the metadata visibility and readiness bugs are - fixed. -3. Add DiskIO server queue depth, per-connection in-flight, and response-queue - latency metrics. The current aggregate DiskIO stage cannot distinguish the - single io_uring pipeline from RPC response delivery at 256 threads. -4. Keep the default at 8 connections and 1+1 workers until the valid repeated - matrix demonstrates a stable latency or throughput improvement. diff --git a/doc/design/chunkio/chunkio-write-flow-analysis.md b/doc/design/chunkio/chunkio-write-flow-analysis.md deleted file mode 100644 index a5c636767..000000000 --- a/doc/design/chunkio/chunkio-write-flow-analysis.md +++ /dev/null @@ -1,204 +0,0 @@ - - - -# Chunk IO Large-Write Flow Analysis - -Large-object write flow from the benchmark workload through chunk -preparation, fetch, EC encode, DiskIO RPC, and chunk seal. The benchmark -sentinel is `tools/benchmark/bench-chunkio-write-regression.sh`. The write pipeline -architecture is in -[`design-crowdb-chunkio.md`](design-crowdb-chunkio.md); this doc traces the -measured hot path and records benchmark results. - -## 1. Flow - -```text -prepare_large_writes (before timer) - -> ChunkIoClient::prepare_large_writes - N sessions, each allocates first chunk (2 strips) - distributed round-robin across workers - -> timer starts - -per object: - -> pop PreparedLargeWrite from queue - -> push replacement session to back of queue - (concurrent chunk allocation, off the data path) - -> write_stream / write_buffers - -> [Fetch] AsyncRead → 1 MiB Bytes blocks - stream: socket read into owned BytesMut, freeze, send - direct: caller-owned Bytes slices, no copy - -> [ChunkWriter::push] - block-granularity drive loop: - push data block → DiskWriter::write (1 RPC per block) - strip full → spawn parity task, auto-rotate to next strip - chunk full → seal, pull next Chunk from prefetch - -> [Parity Task] EC encode → parity blocks → DiskWriter::write - encode_parity_from_shards (isa-l, pre-split shards) - data + parity handles grouped per strip - -> [Completion] join in-flight parity handles - seal_chunk RPC - return ProtoLocation - -> record latency + per-stage timings - -> next object until deadline or object_count - -timer stops after all workers drain -DRAM BW measured from after prefetch to all writes complete -``` - -## 2. Measured Stages - -The benchmark result (`LargeWriteBenchmarkResult`) aggregates per-object -timings across all concurrent workers. Each stage time is total microseconds -divided by object count — milliseconds per object, averaged across writers. -Stages overlap and must not be summed into latency. - -| Stage | What it measures | -| -------------------- | ----------------------------------------------------------- | -| Source/read copy | Time in the fetch stage reading from `AsyncRead` into blocks. Zero in direct-buffer mode. | -| Fetch assembly copy | Buffer assembly copies inside the writer. Zero in both modes when the pipeline holds pre-split shards. | -| EC encode | `encode_parity_from_shards` — isa-l parity computation per strip. | -| Write-completion wait| Time joining in-flight DiskIO write handles + `seal_chunk` RPC. Dominated by loopback RPC round-trips. | - -Additional result fields: - -- **logical_mib_per_sec** — application bytes written (object size × objects) / - elapsed. The user-visible throughput. -- **physical_mib_per_sec** — EC-expanded bytes (logical × 1.5 for 8+4) / - elapsed. What the DiskIO layer actually transports. -- **objects_per_sec** — objects completed / elapsed. -- **latency_p50_us / p99_us** — per-object wall-clock latency from admission - to seal completion. -- **dram_read/write/total_mib_s** — host uncore IMC counter bandwidth over - the workload window (after prefetch). Multiplex-corrected. `None` when the - PMU is unavailable. -- **preparation_stalls** — count and microseconds where a worker's - replacement session fell behind and the writer waited for chunk allocation. - -## 3. Workload Configuration - -The regression sentinel uses fixed parameters across all cases: - -| Parameter | Value | Notes | -| ---------------------- | -------- | -------------------------------------------------- | -| Object size | 16 MiB | Two 8-MiB data strips per object. | -| EC scheme | 8+4 | 8 data + 4 parity = 12 blocks per strip, 1 MiB each. | -| Block size | 1 MiB | One DiskIO RPC per block. | -| Chunk size | 1 GiB | Multiple objects per chunk; rotation is off the measured path at 16 MiB objects. | -| Prefetch chunks | 10 | Write sessions prepared before the timer starts. | -| Prefetch strips/chunk | 2 | Strips pre-appended ahead of the write cursor. | -| Duration | 20 s | Admission deadline; in-flight objects drain after. | -| Storage | NullDisk | Real io_uring submission, no stable-media durability. | -| Deployment | 3-node | Co-located KV + DiskDB + ChunkDB + DiskIO, loopback. | - -Cases vary only in concurrency (1, 4, 32) and input mode (stream vs direct -buffers). Stream mode traverses the production `AsyncRead` fetch path; direct -mode passes caller-owned `Bytes` blocks with zero fetch copies. - -## 4. Benchmark Results - -### 4.1 Intel Core i9-7960X — 2026-09-08 - -Intel Core i9-7960X (16c/32t, Skylake-X), 4 DDR4 channels at 2667 MT/s -(~85 GB/s peak), Linux 6.11, `perf_event_paranoid=-1`. - -| Mode | Writers | Logical MiB/s | Physical MiB/s | p50 / p99 ms | DRAM read avg | DRAM write avg | DRAM total avg | -| -------------- | ------: | ------------: | -------------: | ----------------: | ------------: | -------------: | -------------: | -| Stream | 1 | 162.5 | 243.8 | 99.191 / 111.225 | 2,496.6 | 1,393.0 | 3,889.6 | -| Direct buffers | 1 | 229.1 | 343.7 | 69.191 / 85.641 | 1,996.1 | 1,228.7 | 3,224.8 | -| Stream | 4 | 1,965.6 | 2,948.5 | 31.917 / 51.806 | 12,699.7 | 10,943.0 | 23,642.7 | -| Direct buffers | 4 | 2,672.1 | 4,008.1 | 22.811 / 40.000 | 11,799.6 | 12,051.5 | 23,851.1 | -| Stream | 32 | 3,249.3 | 4,873.9 | 151.716 / 267.992 | 19,562.5 | 15,603.7 | 35,166.2 | -| Direct buffers | 32 | 3,547.9 | 5,321.9 | 138.654 / 261.388 | 14,816.5 | 12,766.6 | 27,583.1 | - -Per-object client-stage time (milliseconds per object, aggregated across -concurrent writers; stages overlap, do not sum): - -| Flow step | Stream 1 | Direct 1 | Stream 4 | Direct 4 | Stream 32 | Direct 32 | -| --------------------- | -------: | -------: | -------: | -------: | --------: | --------: | -| Source/read copy | 20.247 | 0 | 3.737 | 0 | 8.704 | 0 | -| Fetch assembly copy | 0 | 0 | 0 | 0 | 0 | 0 | -| EC encode | 20.175 | 13.164 | 15.213 | 12.357 | 22.045 | 20.222 | -| Write-completion wait | 53.990 | 54.238 | 11.741 | 10.118 | 122.786 | 120.972 | - -### 4.2 AMD Ryzen 9 5950X — _pending_ - -| Mode | Writers | Logical MiB/s | Physical MiB/s | p50 / p99 ms | DRAM read avg | DRAM write avg | DRAM total avg | -| -------------- | ------: | ------------: | -------------: | -----------: | ------------: | -------------: | -------------: | -| Stream | 1 | | | | | | | -| Direct buffers | 1 | | | | | | | -| Stream | 4 | | | | | | | -| Direct buffers | 4 | | | | | | | -| Stream | 32 | | | | | | | -| Direct buffers | 32 | | | | | | | - -| Flow step | Stream 1 | Direct 1 | Stream 4 | Direct 4 | Stream 32 | Direct 32 | -| --------------------- | -------: | -------: | -------: | -------: | --------: | --------: | -| Source/read copy | | | | | | | -| Fetch assembly copy | | | | | | | -| EC encode | | | | | | | -| Write-completion wait | | | | | | | - -## 5. Analysis - -### 5.1 Throughput Scaling - -Single-writer throughput is low (162–229 MiB/s logical) because each 16 MiB -object requires 24 sequential-ish DiskIO RPCs (8 data + 4 parity per strip × -2 strips) over loopback. The completion-wait stage dominates at 54 ms/object -— this is the aggregate time joining 24 RPC handles plus the `seal_chunk` -RPC, all serialized through one tokio worker. - -At 4 writers, throughput jumps 12× (1,966–2,672 MiB/s) because the 24 RPCs -per object now overlap across objects. The loopback transport and DiskIO -io_uring pipelines parallelize naturally. Completion wait drops to 10–12 -ms/object as the per-object join overlaps with other objects' writes. - -At 32 writers, throughput rises only 1.6× over 4 writers (3,249–3,548 MiB/s) -while p50 latency balloons to 138–152 ms. The loopback RPC/DiskIO queue is -saturated: more concurrency increases queueing delay rather than bandwidth. -Completion wait rises to 121–123 ms/object. - -### 5.2 Stream vs Direct - -Direct-buffer mode is 1.41× faster than stream at 1 writer (229 vs 163 MiB/s) -because it eliminates the source-read copy (20.2 ms/object). At 4+ writers -the gap narrows to 1.36× (2,672 vs 1,966) as the RPC bottleneck dominates -over the fetch copy. At 32 writers the gap is 1.09× — the pipeline is -RPC-bound, not fetch-bound. - -The fetch assembly copy is zero in both modes, confirming the shard-based EC -design: the pipeline holds data as pre-split 1 MiB `Bytes` blocks and -`encode_parity_from_shards` consumes them directly without re-splitting. - -### 5.3 EC Encode - -EC encode time is stable at 12–22 ms/object across all concurrency levels. -It does not scale with concurrency because isa-l parity computation is -CPU-bound and each parity task runs on one tokio worker. At 32 writers, -EC encode (20–22 ms) is no longer the bottleneck — completion wait (121–123 -ms) dominates by 6×. - -### 5.4 DRAM Bandwidth - -Host DRAM bandwidth scales with concurrency: 3.2–3.9 GB/s at 1 writer, -23.6–23.9 GB/s at 4, and 27.6–35.2 GB/s at 32. The read/write split is -roughly balanced at 4+ writers (EC reads source data, writes parity + -RPC transport copies), which matches the expected traffic pattern. - -At 32 writers, DRAM total (28–35 GB/s) is well below the 85 GB/s peak, -confirming the bottleneck is RPC queueing, not memory bandwidth. The -stream_32t case shows higher DRAM usage (35.2 vs 27.6 GB/s) because the -source-read copy adds memory traffic without adding useful throughput. - -### 5.5 Bottleneck - -The bottleneck is the loopback RPC/DiskIO transport queue. At 32 writers, -completion wait (121–123 ms/object) is 6× the EC encode time and 14× the -source-read time. The RPC path transports 24 MiB per object through -loopback sockets with kernel copies on both sides — this is the dominant -cost, not EC, not memory, not chunk allocation. - -Further work should profile and reduce RPC/kernel transport cost (zero-copy -send, larger batch sizes, connection pooling) and EC/cache contention. -Increasing prefetch or adding wrapper layers will not address the measured -bottleneck. diff --git a/doc/design/chunkio/design-crowdb-chunkio-reader.md b/doc/design/chunkio/design-crowdb-chunkio-reader.md deleted file mode 100644 index ea09f3806..000000000 --- a/doc/design/chunkio/design-crowdb-chunkio-reader.md +++ /dev/null @@ -1,164 +0,0 @@ - - - -# CROWDB - Design: Chunk Object Reader - -The chunk object reader reconstructs full, ranged, and streamed objects from -writer-produced locations across current mirror and erasure-coded layouts. - -Depends on: [chunk IO](design-crowdb-chunkio.md), -[chunkdb](../chunkdb/design-crowdb-chunkdb.md), and -[mirror-to-EC conversion](../chunkdb/design-crowdb-chunkdb-mirror-to-ec.md). - -## Contents - -1. [API](#1-api) -2. [Location mapping](#2-location-mapping) -3. [Strip reads](#3-strip-reads) -4. [Read consistency](#4-read-consistency) -5. [Memory bounds](#5-memory-bounds) -6. [Errors](#6-errors) -7. [Durable repair handoff](#7-durable-repair-handoff) -8. [Invariants](#8-invariants) - -## 1. API - -`ChunkIoClient` owns one `ChunkReader` sharing its metadata client and -lock-free DiskIO route snapshot. It exposes: - -- `read_object`: return the complete logical object as `Bytes`. -- `read_range`: return one half-open logical interval. -- `read_range_partial`: return ordered successful intervals and exact failed - intervals without substituting bytes. -- `read_stream`: return a pull-based `ChunkReadStream` whose item size is - bounded by policy. - -`ChunkReadPolicy` configures stream-window bytes, a 256-MiB process-wide -EC-recovery budget, layout safety margin, bounded layout retries, and the -ad-hoc full-fragment threshold. An optional preferred node supplies a mirror -read locality hint; ChunkKV uses its configured node identity for WAL replay. - -## 2. Location mapping - -Locations are sorted by `logical_offset` and must cover one gap-free logical -interval. Zero-length entries are ignored. Physical length must cover logical -length, and all additions use checked arithmetic. - -For each intersecting location, the reader queries current chunk metadata. It -validates that stored strip intervals are ordered and non-overlapping, binary -searches the first overlap, and walks explicit -`[chunk_offset, chunk_offset + capacity)` intervals. It never derives geometry -from strip vector index or assumes uniform width. This keeps locations stable -when several mirror strips become one equal-capacity EC strip. - -Independent locations are fetched concurrently and assembled by logical -offset. The output length must equal the requested logical length. - -## 3. Strip reads - -Mirror reads try usable replicas in metadata order by default. With a preferred -node and complete retained route hints for a strip, they first try that node, -then other nodes in its rack, preserving metadata order within each tier. -Unknown preferred nodes or incomplete strip hints preserve metadata order. -This ordering changes only the read snapshot: it neither publishes placement -nor adds or removes replicas. Layout validity deadlines, exact segment -incarnations, checksum verification, and corruption reporting remain unchanged. -Each request uses the exact segment-relative offset and length; DiskIO supports -arbitrary byte ranges. - -EC reads derive the shard width from stored segment geometry. The fast path -reads only data-shard intersections needed by the request. If a required data -read fails, the reader reads the same byte interval from surviving data and -parity shards and invokes ISA-L decode. It starts only enough surviving reads -to obtain `data_num` shards and issues another read only after a candidate -fails. A 16-KiB request therefore reconstructs 16 KiB rather than rebuilding -the whole shard. Missing shards beyond `code_num` are unrecoverable. - -An EC strip with incomplete parity still permits direct reads from healthy -data shards. A failed data read in that state is unrecoverable because parity -that has not reached the durable `Parity` state is never a decode source. - -Zero-filled data shards beyond a partial strip's durable length participate in -decode without issuing DiskIO. Returned data is always clipped to exact -location length, so stored KiB rounding and parity padding never escape. - -## 4. Read consistency - -Each metadata query advertises a maximum layout validity duration. The reader -records the local query start and subtracts a safety margin. If all dependent -DiskIO does not finish before that deadline, it discards every byte from the -attempt, re-queries, and retries within policy. ChunkDB retains replaced -segments through the same window. - -The chunk's durable `acknowledged_cursor` provides the exact written byte -boundary for both sealed chunks and active shared mirror chunks. Derive each -strip's valid prefix from that cursor and its chunk offset; later bytes are -`NotYetAvailable`. Legacy chunks without a byte cursor retain their coarse -`sealed_length` contract. EC data shard `i` has -`min(shard_bytes, max(0, valid_strip_bytes - i × shard_bytes))` valid bytes; -every code shard has `min(shard_bytes, valid_strip_bytes)` valid bytes. Recovery -reads only those prefixes and treats absent tails as mathematical zeros, -independently of stale bytes on disk. Repair writes only the valid prefix. - -Only an unparseable returned frame or a write-frame checksum mismatch proves -physical corruption. The reader reports its exact serving segment to ChunkDB; -ChunkDB first fences the DiskDB BusyBlock to `Corrupt`, then adds the segment -to `unavailable_segments` and admits `RepairStrip`. Network, timeout, and -unknown I/O failures may be decoded around but never create these markers. -This metadata update can advance -the revision of an active shared chunk. Its owning writer refreshes the chunk, -verifies state and writer epoch, recognizes an ambiguously committed cursor, -and retries the cursor advance against the new revision. - -## 5. Memory bounds - -Partial EC recovery reserves decode shard buffers plus output from a -client-wide Tokio semaphore before issuing recovery I/O. A request larger than -one reservation is split into same-offset slices. The limit covers recovery -scratch, not the caller-owned result or ordinary direct-read buffers. No lock -is introduced. - -`read_object` necessarily owns the complete returned object. Callers use -`ChunkReadStream` for large objects; each pull reads at most -`stream_window_bytes` and retains no prefetched second window. - -## 6. Errors - -`ReadError` separates malformed locations/ranges, deleted chunks, -not-yet-durable data, expired layouts, metadata failures, DiskIO failures, EC -decode failures, data loss, and exact failed logical ranges. Strict reads fail -at the first missing range. Partial reads preserve healthy ranges before and -after failures. A stream emits its contiguous successful prefix, one ranged -error, and then terminates; it never silently truncates or zero-fills. Empty -objects and empty ranges perform no RPC or DiskIO. - -## 7. Durable repair handoff - -Verified corruption is reported through a versioned chunk-routed RPC carrying -the expected revision, strip sequence, exact segment incarnation, and -operation ID. A small read still decodes only its requested slice; repair -admission proceeds independently and may begin before that read returns. -Matching client failures accumulate for one second. At a complete fragment or -the lesser of 1 MiB and half a fragment, one bounded full-fragment request is -sent to ChunkDB and same-key readers share its result future. Saturation falls -back to slice recovery and the durable repair task. - -ChunkDB coalesces full-fragment requests across clients under 32 concurrent -jobs and a 512-MiB shared decode/result budget by default. The existing -`RepairStrip` task provides the only allocation and publication authority; -rebuilt bytes can reach waiters after target fsync while fenced publication -continues. Task key/value, scheduling, publication, and crash behavior are -defined by -[Mirror-to-EC Conversion and Chunk Tasks](../chunkdb/design-crowdb-chunkdb-mirror-to-ec.md). - -## 8. Invariants - -- I1: every returned byte is covered by a validated location and strip. -- I2: bytes from an expired layout are never returned. -- I3: active-strip reads never exceed the durable acknowledged cursor. -- I4: EC recovery never exceeds its shared scratch-memory budget. -- I5: stream items never exceed the configured window. -- I6: unrecoverable redundancy loss is explicit and never zero-filled. -- I7: a verified-corrupt segment is durably marked before its recovery is - returned; an ordinary I/O error is never persisted as corruption. -- I8: incomplete parity is never used to reconstruct a data shard. diff --git a/doc/design/chunkio/design-crowdb-chunkio-small-object-writer.md b/doc/design/chunkio/design-crowdb-chunkio-small-object-writer.md deleted file mode 100644 index c01fe2e96..000000000 --- a/doc/design/chunkio/design-crowdb-chunkio-small-object-writer.md +++ /dev/null @@ -1,343 +0,0 @@ - - - -# CROWDB - Design: Small-Object Shared-Chunk Writer - -The small-object write path batches independent objects into shared mirrored -chunks while preserving per-object completion, bounded memory, and a durable -prefix after process loss. - -Depends on: [chunk IO data path](design-crowdb-chunkio.md), -[chunkdb lifecycle](../chunkdb/design-crowdb-chunkdb.md), and -[diskio](../diskio/design-crowdb-diskio.md). - -Satisfies: bounded small-object admission, lock-free routing, shared-chunk -batching, fenced durable cursors, in-place mirror repair, elastic pipelines, -and orphan sealing. - -## Table of Contents - -- [1. Scope and Concepts](#1-scope-and-concepts) -- [2. Admission and Object Handles](#2-admission-and-object-handles) -- [3. Routing and Pipeline Ownership](#3-routing-and-pipeline-ownership) -- [4. Batching and Physical Layout](#4-batching-and-physical-layout) -- [5. Durable Cursor and Completion](#5-durable-cursor-and-completion) -- [6. Chunk Lifecycle and Recovery](#6-chunk-lifecycle-and-recovery) -- [7. Elasticity, Failure, and Shutdown](#7-elasticity-failure-and-shutdown) -- [8. Policy and Metrics](#8-policy-and-metrics) -- [9. Correctness Invariants](#9-correctness-invariants) - -## 1. Scope and Concepts - -One client-owned pool multiplexes small objects across a bounded set of -pipelines. Each pipeline exclusively owns one active Repo chunk containing -mirror strips and may own one empty prepared replacement. A pipeline batches -whole objects and writes the physical range to every mirror. Ordinary completion -returns independent locations while cursor progress runs asynchronously; callers -can opt into completion after the readable cursor is durably confirmed. - -The shared path can incrementally form 8+4 EC groups while writing mirrors. -It retains one open-strip image and four parity accumulators, but never retains -eight completed data images. Incomplete or failed groups remain mirrored and -can be completed by later background conversion. - -## 2. Admission and Object Handles - -`prepare_small_write_for_key(object_size, key)` validates the policy and uses -the complete tenant/bucket/object identity as a preferred pipeline hint. The -single-frame interface charges bytes as immutable input fragments arrive. -`prepare_shared_object_write_for_key` atomically reserves the complete declared -payload on an available pipeline before receiving it. It probes the other -routes when the preferred route cannot reserve enough capacity. Neither path -uses a global or per-tenant semaphore. - -The returned single-use writer retains caller-owned byte fragments. Successful -input is never partially accepted. Overflow or underflow is terminal, drops -retained fragments, releases the reservation, and returns an exact size error. -An empty object completes locally without starting the pool. Abort before -submission also completes locally and releases all resources. - -Finishing an exact non-empty object transfers its fragments, byte charge, and -completion channel to the pool. Cancellation after that transfer does not -cancel or reshape a physical batch: the worker completes the durable operation, -and an undeliverable result leaves a reclaimable range in the shared chunk. - -## 3. Routing and Pipeline Ownership - -Admission reads an immutable `ArcSwap` route snapshot. The hash selects the -first candidate; a full or closed route causes probing of other candidates -before waiting. Moving a retained byte charge reserves capacity on the target -atomically before releasing the previous route's charge. Failed transfers keep -the object intact. The submission path adds no mutex or read-write lock. - -When all routes are unavailable, the caller waits for a capacity notification -with a control-interval fallback that also observes newly published routes. -Socket admission leaves the body unread until retained-byte capacity is -reserved, so TCP applies backpressure. Queue capacity and byte reservations -bound pending work independently. - -For retirement, the manager publishes a snapshot without the route before it -signals the worker. The worker closes its receiver, establishing the acceptance -boundary, and drains every accepted object. A sender racing an old snapshot is -therefore either accepted and drained or rejected intact for rerouting. - -## 4. Batching and Physical Layout - -A pipeline waits for the first object only while it has no work. After the -previous batch completes, it immediately drains whole objects that are already -queued until reaching the byte limit, object-count limit, or remaining chunk space. It -never delays an admitted object to wait for a batching timer. Concurrent work -naturally accumulates behind the in-flight batch and is aggregated on the next -completion-driven drain. An object larger than the normal batch target is -written alone as long as it is within the object limit. - -Native small uploads allocate one exact-sized receive owner containing payload -and all frame regions. The receive task can fill it across multiple socket -reads, prepares frame checksums in place, and submits the complete owner once. -The worker sets each frame's Chunk ID after placement. Generic immutable input -fragments are framed with separate header/footer views without copying payload. - -Objects are packed consecutively and may span mirror strips within one chunk. -The worker slices shared `Bytes` views at strip boundaries and sends only each -new range to DiskIO through scatter/gather writes. It does not rotate merely -because a frame or object reaches the strip boundary. Every returned location -covers one object's consecutive physical frames. - -The current strip retains immutable prefix views for repair. Ordinary mirror -success creates no continuous strip shadow. Repair materializes the complete -prefix only after a failure; EC conversion materializes the closed strip image -for parity computation. Closing a strip releases its retained views. A view -crossing the boundary can retain its object's owner until the final open-strip -prefix is released, so memory headroom includes a maximum-sized owner per pipe. - -### Transport descriptor bound - -A strip aggregate may retain more views than one RPC frame permits. The DiskIO -semantic client validates the entire segment range, then submits consecutive -bounded scatter/gather frames at increasing offsets. Within one strip's disk -block, each frame must finish successfully before the next is submitted, even -when the boundary is aligned or the logical byte ranges are disjoint. Different -connections can reorder arrivals; a partial-block tail's zero padding can then -overwrite a later frame and corrupt stored chunk headers or payload. - -Independent strips' disk blocks may overlap. Different disks holding one strip's -EC shards or mirror replicas may also overlap; they do not share the padded -physical block. Callers preserve ordering between separate writes to the same -strip block. An error stops subsequent frame submissions before repair. Views -keep their original owners without coalescing payload. Requested fsync follows -the complete range once, and cursor publication follows successful completion -of all frames. A descriptor bound cannot be treated as a disk failure or trigger -repair. - - -A caller can attach a `SmallWriteIntent` to durable completion. Once the batch -has assigned exact locations, every attached callback completes before any of -the batch's DiskIO. Failure aborts the batch and its pipeline; cancellation of -the caller does not detach ownership registration from the physical operation. -The callback has no default storage policy. Legacy tree FileIO uses it for a -catalog-sharded block ledger; native streamed files rely on chunk allocation -ownership and Chunk-KV WAL. Range reclamation defers active shared-chunk ranges -that extend beyond the acknowledged cursor, preserving uncertain writes. - -Chunk allocation reserves a bounded group of hidden strips. A reserved strip -becomes `Consumed` immediately before its first DiskIO and is confirmed into -the readable layout only after its mirror write succeeds. Confirmation runs on the ordered metadata chain. Allocation/refill has its -own background task and does not delay that chain; reserve failures fall back to a -bounded batch of already attached mirror strips. Seal cancels never-consumed -reservations and removes attached strips beyond the written length. Objects and batches may straddle strips, but never chunks. Mirror-only groups -contain 32 strips by default; at 16 remaining strips a new group is allocated -asynchronously after the full preceding reservation, including hidden strips. -The client verifies the returned append offset before accepting a refill; an -older backend that ignores this field fails safely rather than overlapping -hidden strips. Upgrade ChunkDB before enabling this client prefetch behavior. -Resource allocation releases the existing chunk lifecycle guard; publication -reacquires it and revalidates ownership and sequence. No extra persistent -high-water record is required. - -Lease renewal refreshes the authoritative readable layout while retaining the -writer's exact staged, unconsumed strip as a local write view. This also applies -to an empty prepared replacement. Renewal does not add that strip to published -capacity or advance the readable cursor. Consume still precedes the mirror -write, and Confirm still follows its durability. A changed chunk identity, writer epoch, or staged strip -fails the writer instead of selecting a later reservation and leaving a gap. - -When automatic conversion is enabled and at least eight strips remain, one -special reservation allocates eight mirror sets with the configured copy -count and four parity segments from a joint placement plan. Each completed -strip updates the four incremental parity accumulators and releases its input -image. After strip eight, -the client writes and fsyncs only the parity segments. ChunkDB then reselects -one healthy survivor per mirror set against current topology and atomically -publishes the 8+4 EC strip. If optimal publication is unavailable, mirrors stay -authoritative and the ordinary durable conversion task is admitted. - -The configured pool memory budget subtracts one maximum-sized retained owner (at least 1 MiB) per maximum -pipeline and one foreground conversion group, then divides the remaining -buffer capacity between possible routes. Conversion is a cold-path bounded -operation and may use its own permit; ordinary object admission never does. -Published EC capacity is divided by its data width when deriving the next -reservation, so consecutive groups keep the same per-shard geometry. - -## 5. Durable Cursor and Completion - -Every shared chunk carries a nonzero writer epoch, acknowledged physical byte -cursor, optional closed-strip sequence, writer lease deadline, and metadata -revision. Cursor advancement executes under chunkdb's existing per-chunk -lifecycle guard and requires: - -- an Active chunk and matching nonzero writer epoch; -- the caller's expected metadata revision; -- a strictly increasing cursor within chunk capacity; and -- a valid, nondecreasing closed-strip marker. - -The operation updates the cursor and marker, renews the lease from server time, -increments the revision, persists the record, and refreshes the cache. Stale -epochs or revisions conflict; backward or out-of-range cursors are invalid. - -The response barrier is: - -1. Frame the owner in place and write immutable slices concurrently to every - mirror, splitting views at strip boundaries when necessary. -2. Repair each failed replica from the retained prefix and fence the new segment into - chunk metadata or, while it remains hidden, into its durable reservation. -3. Publish all object-specific locations together. -4. Coalesce cursor progress in the background metadata chain. Strip close and - batched append execute there in revision order. - -The physical mirror-strip flow is shared with chunk streams: it writes mirrors -in parallel, excludes failed disks, writes a prefix-complete replacement image, -and publishes the fenced strip swap. Each single-owner caller retains its own -current-strip prefix and controls its publication barrier. Journal streams also -fsync the final mirror set before advancing their durable cursor and resolve an -uncertain replacement result against chunk metadata before retrying it. - -No location is visible before its complete physical range exists on every -configured mirror. Cursor persistence is an asynchronous availability and -orphan-recovery checkpoint; readers can transiently report `NotYetAvailable` -until it catches up. - -Callers publishing immediately readable immutable authorities use -`SharedObjectWriter::finish_durable`. A batch containing a durable-completion -request waits for the existing metadata chain, then confirms any remaining -cursor suffix before delivering locations. Metadata failure fails that batch -instead of exposing an unreadable reference. The pipeline does not start its -next batch before this barrier completes. Arrivals queue during the wait and -form the next batch; multiple pipelines provide concurrency. Shared-object -writers require this barrier by default. The single-frame `on_finish` retains its -asynchronous cursor behavior; no additional lock or reader-side retry is needed. - -## 6. Chunk Lifecycle and Recovery - -A pipeline allocates its first chunk and its initial strip batch before -publication. The worker owns the write cursor while one background metadata -chain owns revision-ordered cursor commits and batched strip appends. When -remaining chunk capacity falls below the larger of one prefetch group and the -object limit, it starts at most one asynchronous replacement allocation so ordinary rotation does not wait for allocation. - -Idle workers renew the current chunk and any ready replacement at half the -writer lease interval. A matching live chunk writer lease also protects hidden -reservation groups, avoiding a separate renewal write for every group. - -Closing a strip releases its prefix views immediately. Retirement seals a non-empty -current chunk at its acknowledged cursor and deletes an empty current or -replacement chunk. A write or metadata failure -fails the affected batch and every already accepted queued object, removes the -route, and retires its chunks. - -Chunkdb periodically scans Active shared chunks. When a persisted writer lease -has expired, it acquires the normal lifecycle guard, rechecks the record, seals -at the persisted cursor, cancels and frees never-consumed reservations, closes -only acknowledged strips, persists, and refreshes the cache. Recovery retains -consumed reservations until the writer lease deadline plus the configured -reuse grace has elapsed, so a timed-out write cannot reach a recycled extent. -Consumed reservations without planned cursors remain allocated fail-safe. The -old writer can no longer renew or advance the sealed chunk. DiskIO performs raw -I/O and does not validate allocation ownership. - -## 7. Elasticity, Failure, and Shutdown - -One manager owns pipeline membership. It publishes a scale-out candidate only -after the candidate's first chunk is ready; initialization failure leaves the -old snapshot intact. When all routes reach their queue or byte-capacity high-water marks, the -manager can add one pipeline, up to 32 by default. -Scale-out depends only on queued work, not worker utilization, request age, or -elapsed idle time. When the whole pool has no queued or active work, an extra -route can be unpublished and drained while preserving the configured minimum. - -Unexpected worker termination removes the failed route and creates replacements -until the minimum is restored. Explicit shutdown closes admission, unpublishes -all routes, drains accepted work, joins all workers, and seals or deletes their -owned chunks. Dropping the pool closes admission but cannot await cleanup. - -A mirror write failure records the submitted segment's disk in one client-wide, -TTL-based lock-free negative list. Repair allocation excludes all live failed -disks and the nodes holding surviving replicas. The worker writes the complete -shadow to a tentative replacement, then publishes a one-strip fenced metadata -swap. Allocation and replacement-write failures may select a new target within -the bounded attempt count. An ambiguous metadata result retries the identical -operation ID and replacement, so it cannot allocate or install a duplicate. -A definite conflict discards the tentative block when it is not referenced. -Exhaustion fails the batch once, records an unavailable replica for an already -acknowledged prefix, and retires the pipeline for background recovery. - -## 8. Policy and Metrics - -Defaults accept objects up to 8 MiB and target batches up to 1 MiB, budget -1.25 GiB pool-wide, use one to 32 pipelines, and allow 1,024 queued objects per -pipeline. Queue high-water marks are 4 MiB or 128 objects. Shared chunks have -a 256 MiB client-side capacity, prefetch 32 strips per ordinary group, and use -two mirrors. Foreground conversion uses the configured data-width group and -its joint parity placement; single-node deployments disable conversion and -select one mirror explicitly. The access service routes by the logical data -capacity of a strip multiplied by its threshold ratio, not socket read size. - -Configuration validates nonzero bounds, reachable queue high-water marks, -ordered pipeline limits, and a budget covering retained owners, one conversion -group, and request buffers. The retained `scale_in_delay` and `cooldown` fields -are configuration-compatible but do not participate in scale decisions. - -Foreground parity writes use a dedicated connection per DiskIO endpoint and -64 KiB byte-range requests, followed by a parity-only fsync barrier. Ordinary -mirror traffic retains its configured connection pool and request sizing. - -Atomic metrics cover submitted, completed, and failed objects; reserved bytes; -batch sizes and fill; queue delay; active and draining pipelines; scale changes; -strip-tail waste; and batch-watchdog expirations. The watchdog reports a batch -that remains in flight beyond its configured interval, then continues awaiting -the same durability future. It neither delays queue draining nor cancels an -operation whose physical or metadata outcome may already be committed. The -default 500 ms observation cadence matches the RPC pending-request reaper -scan interval; underlying RPC clients retain their own 5- or 10-second request -timeouts. Repair metrics expose attempts, repaired and exhausted -replicas, failed-disk exclusions, active pipelines, pipeline replacements, repairs -that avoided rotation, complete-shadow bytes, and repair latency totals and -maxima. Snapshots compute aggregates without locking submission. - -## 9. Correctness Invariants - -- **SW-I1 — Incremental bounded input.** Received object frames are charged to - one route at a time; shared-object admission atomically reserves its declared - payload, and rerouting preserves bounded accounting. -- **SW-I2 — Single acceptance.** A submission racing retirement is accepted by - exactly one draining receiver or returned intact for rerouting. -- **SW-I3 — Exclusive ownership.** Exactly one live pipeline epoch can advance - a shared chunk. -- **SW-I4 — Whole placement.** An object remains in one chunk; views can cross strips without - introducing padding or changing its consecutive frame layout. -- **SW-I5 — Physical completion.** Object locations are published only after - every mirror write succeeds. Native FileIO additionally waits for readable - cursor publication; generic callers can select physical-only completion. -- **SW-I6 — Monotonic prefix.** The acknowledged cursor moves strictly forward, - and recovery seals only at that persisted prefix. -- **SW-I7 — Terminal completion.** Every accepted object completes or fails - exactly once; caller cancellation may discard delivery but not batch work. -- **SW-I8 — Repair publication.** A replacement segment is written from the - complete retained prefix and fenced into metadata before any affected location is - published. -- **SW-I9 — Queue-only elasticity.** Pipeline membership changes are driven by - queued bytes, queued objects, and empty/non-busy state, never elapsed time. -- **SW-I10 — Hidden reservation.** A reserved or consumed strip is never - readable through `Chunk.strips`; only fenced confirmation publishes it. -- **SW-I11 — EC publication.** Mirror replicas remain authoritative until four - parity segments are durable and one current-topology survivor from every - mirror set is atomically published. diff --git a/doc/design/chunkio/design-crowdb-chunkio.md b/doc/design/chunkio/design-crowdb-chunkio.md deleted file mode 100644 index fd2835645..000000000 --- a/doc/design/chunkio/design-crowdb-chunkio.md +++ /dev/null @@ -1,625 +0,0 @@ - - - -# CROWDB - Design: Chunk IO Data Path (Overview) - -The chunk IO data path is the client-side layer that writes and reads -large-object data as mirror or EC strips across diskio servers, using chunkdb -for chunk lifecycle management (allocate, append, seal, delete). It -lives in the `crowdb-chunk-client` crate and is consumed by object store -layers and application upload handlers. The chunkdb server design -(chunk lifecycle, placement, EC integration) is in -[`doc/design/chunkdb/design-crowdb-chunkdb.md`](../chunkdb/design-crowdb-chunkdb.md); -the diskio block IO engine is in -[`doc/design/diskio/design-crowdb-diskio.md`](../diskio/design-crowdb-diskio.md). -This doc does not repeat their architecture — it covers the data path -that sits between them: the write pipeline, its backpressure and memory -model, and the design choices that make a 1 TB upload cost the same -~15 MB of RAM as a 50 MB one. The shared mirrored path for small objects is -specified in the -[small-object writer design](design-crowdb-chunkio-small-object-writer.md). - -`ChunkWriter` selects a mirror or EC strip writer from each persisted strip, -so consecutive strips in one chunk can have different layouts and capacities. -The mirror writer sends each write to every mirror segment and synchronizes -them before strip completion; any failed copy returns a write error. The EC -writer derives its shard scheme from that strip's metadata and retains its -separate parity and repair flow. `ChunkReader` likewise dispatches each strip -to mirror or EC recovery based on stored geometry. - -`MirrorStripFlow` routes both ordinary mirror writes and replacement images -through synchronized `DiskWriter` operations. The production routing and metrics -adapters preserve that completion contract: matching DiskIO peers perform ordered -write and flush before one response. Oversized immutable view lists complete in -byte order and only their final request synchronizes the preceding range. -Missing synchronization proof fails directly. A replacement is synchronized -before ChunkDB publishes its new segment mapping; successful surviving replicas -are retained. Stream and WAL replacement images initialize their complete strip -with a retained prefix and zero suffix in one attempt, without automatic retries. -An uncertain initialization cannot authorize publication or immediate reuse. - -`DiskWriter::drain_segment` preserves the segment ordering base and one absolute -deadline through metrics and routing adapters. Its empty synchronized request -waits for earlier backend writes and disk flush; a missing completion proof fails -directly. The caller must first exclude old producers and late admissible requests. -Recovery and allocation reuse cannot substitute an independent disk fsync for -this ordered barrier. - -EC shard capacity is the persisted segment's `unit_count × unit_kb × 1024`, -independent of the allocation-unit size. ISA-L consumes only each data shard's -actual byte prefix. A partial first shard produces equally short code shards; -once a full data shard exists, each code shard spans the full shard width. -No logical padding is written. Device alignment belongs to DiskIO. -Writers seal through one exact-byte RPC, including zero for an empty chunk. -Rounded KiB fields summarize chunk/strip geometry and do not define valid bytes. - -## Table of Contents - -- [1. Non-Goals](#1-non-goals) -- [2. Key Design Decisions](#2-key-design-decisions) -- [3. Write Flow](#3-write-flow) -- [4. Backpressure and Memory Budget](#4-backpressure-and-memory-budget) -- [5. EC Integration](#5-ec-integration) -- [6. Chunk Rotation and Location](#6-chunk-rotation-and-location) -- [7. Completion and Error Handling](#7-completion-and-error-handling) -- [8. Interaction with Neighbors](#8-interaction-with-neighbors) -- [9. Tunables and Defaults](#9-tunables-and-defaults) -- [10. Application API and Routing](#10-application-api-and-routing) -- [11. Performance Workload](#11-performance-workload) - -## 1. Non-Goals - -- **No small-object writer details.** Shared-chunk packing is a separate - component specified by the - [small-object writer design](design-crowdb-chunkio-small-object-writer.md). -- **Reader is a separate component.** Location resolution, mirror/EC fetch, - partial decode, range reads, and bounded streaming are specified in - [Chunk Object Reader](design-crowdb-chunkio-reader.md). -- **No GC of leaked partial chunks.** Best-effort cleanup on abort - leaves Active chunks for a future reaper; this doc does not specify - the reaper. -- **No object protocol server.** Applications call the client library - directly. An S3-compatible or other object protocol server can use the same - API later without owning chunk placement or DiskIO routing. - -## 2. Key Design Decisions - -- **Block-granularity pipeline, not strip-granularity.** The first disk - write starts after 1 MiB (one data block), not after the full 8 MiB - strip. Three strips stay in flight simultaneously (N parity, N+1 - data, N+2 fetch) without unbounded memory. Strip-granularity would - double time-to-first-byte and halve steady-state throughput. -- **Push-based "always store" contract.** `ChunkIoWriter::on_data` - never rejects a buffer; it awaits until internal capacity is free. - This puts the retry-loop in one place (the writer) instead of every - caller. A `FeedStatus` (`Continue` / `Pause`) answers "would the next - push block?" so a dedicated upload task can ignore it and block, - while a shared handler task can pre-check via the non-async - `require_data` hint and return 503 instead of stalling. -- **Bounded preparation, not eager allocation.** A 1 TB object does - not allocate all 250K strips at once. Consumption-driven channels stay only - `prefetch_strips_per_chunk` strip results (default 1) and - `chunk_preparation_depth` chunks (default 1) ahead, - keeping allocation rate and KV metadata pressure bounded regardless - of object size while keeping the cursor fed. -- **Warm admission, then continuous replacement.** Services that know their - write policy prepare a configurable queue of write sessions before admitting - load. The first chunk of every queued session is allocated concurrently. - Consuming a session starts its replacement while the active object writes, - so ChunkDB convergence and ordinary allocation are outside the data path. -- **Shard-based EC, no re-split copy.** The pipeline already holds data - as separate 1 MB `Bytes` blocks. Re-splitting a contiguous buffer - just to feed `crowdb_common::ec::encode` would copy 4 MB per strip for - no benefit. `encode_parity_from_shards` takes pre-split shards - directly and reuses the existing isa-l FFI path — no new C++ code. -- **Repair one failed shard, not the strip.** A durable data or parity write - failure keeps every successful shard and retries only the failed segment - through ChunkDB placement and fenced publication. Exhaustion aborts the - unpublished object; reduced redundancy is never reported as success. -- **Memory budget per pool, not per object.** A `WriterPool` tracks a - total `memory_budget` and an atomic `in_use` counter; `try_acquire` - rejects with `MemoryBudgetExhausted` when full, enabling backpressure - up the call stack. Per-writer footprint is constant (~15 MB peak for - 4+1 EC, 1 MB blocks, defaults), so `max_concurrent = budget / - per-writer-footprint` — a 1 TiB and a 50 MiB upload cost the same RAM. -- **Real-process E2E is the primary write-flow coverage.** Large- and - small-object E2E tests run the client against KV, DiskDB, DiskIO, and - ChunkDB processes, query committed chunk metadata, and read payloads back - from their allocated segments. Large-write coverage also verifies stored EC - parity. The `ChunkAllocator` and `DiskWriter` seams support focused tests for - fault injection, backpressure, cancellation, accounting, and exact boundary - conditions that a real cluster cannot trigger deterministically; they are - auxiliary coverage rather than substitutes for the end-to-end flow. -- **Drive loop in `ChunkWriter`, not the object layer.** The - strip-level drive loop (push block → write to disk → auto-rotate - strips when full → spawn parity) lives inside `ChunkWriter::push`. - The object layer calls `push` + `is_full` + `seal` — it does not - track strip boundaries, block indices, or parity handoff. This - eliminates the `StripPlacement` bridge type and keeps the object - layer thin. -- **Own flatbuffer types directly.** `ChunkWriter` owns `Arc`, - `EcStripWriter` holds `Arc` + strip index, and `seal()` - returns `ProtoLocation` directly. No parallel wrapper structs - (`StripPlacement`, `Location`) — the flatbuffer types are the canonical - representation throughout the write path. - -## 3. Write Flow - -The writer runs three concurrent stages: a fetch stage, a -`ChunkWriter` (which owns the strip-level drive loop + internal strip -prefetch), and a bounded pool of background parity tasks. -`LargeObjectWriter` exposes two driving modes over the same pipeline — -stream mode (`write_stream`, pulling from an `AsyncRead`) and push mode -(the `ChunkIoWriter` trait, §2). In stream mode a fetch stage pulls -from `AsyncRead`; in push mode the caller's bytes go directly to the -block channel. The `ChunkWriter` drive loop is identical in both modes. - -``` - ┌────────────────────┐ - │ ChunkPrefetch │ background, bounded: - │ 1 chunk ahead │ pre-allocates next Chunk - │ (1 strip each) │ for a 16 MiB 8+4 write - └─────────────┬──────────┘ - │ pre-allocated Chunk - ▼ - AsyncRead ──► [Fetch] ──► block_buf ──► [ChunkWriter::push] ─────► disk (data) - read ≤1MB (1 MB per │ write 1 data block → 1 disk - directly into block) │ (immediately, no EC wait) - send to write │ - max_cached_buffer │ when all 4 blocks of strip N written: - = 4 MB (default) ├──── spawn parity ──────────────────── - (backpressure if full) │ ▼ - │ [Parity Task N] (background) - │ EC encode → 4 parity blocks - │ write parity → disks 9..12 - │ (no join — handles collected) - ▼ - auto-rotate to strip N+1, block 0 - (no wait for parity) - │ - │ internal strip prefetch: - │ append_chunk ahead of cursor - │ (bounded: prefetch_strips_per_chunk=1) -``` - -The flow, step by step: - -- **Canonical framing.** Ordinary `LargeAsyncObjectWriter::on_data` input is - copied into its logical frame tail, then each payload is copied once into - its final encoded batch. Header and CRC footer preparation uses stack - regions; full frames have no temporary encoded payload allocation. A partial - final frame is emitted at EOF. The payload-copy counters include these - explicit copies and repeated encoding before a rotation accepts the batch; - they exclude framing bytes, allocator growth, source reception, EC, and - transport staging. Owner-backed `on_framed_data` retains its final frame - allocation and contributes no payload-copy bytes to those counters. -- **Fetch.** Reads from the `AsyncRead` stream directly into the owned - `BytesMut` block. A socket read may return less than 1 MiB; the fetch stage - accumulates in that same allocation, freezes it without copying, then sends it to - `ChunkWriter::push` immediately — it does not wait for the full 4 MB - strip. The fetch stage sends sequential `Bytes` blocks and does not - track block indices or strip boundaries; `ChunkWriter` owns indexing. - On EOF with a partial last block, the partial block is sent. -- **ChunkWriter::push (drive loop).** The central coordinator. It - owns `Arc` and shares it with `EcStripWriter` by ref count. - On each `push`, if the current strip is full, it finishes the strip - (submits bounded data/parity writes and collects completion handles, - no join), checks `is_full()` (chunk-level), and either returns - `Pause` (chunk full — caller rotates chunks) or opens the next strip - (from `chunk.strips` if pre-appended by the internal strip prefetch, - or via inline `append_chunk` RPC). It then pushes the block to the - new strip's data segment via `DiskWriter::write` — one disk per - block. Independent data blocks are submitted without serial completion - waits, and strip N+1 overlaps strip N's parity completion. -- **Parity.** One background task per strip. It receives the strip's - data blocks, EC-encodes via `encode_parity_from_shards` (§5) into - `code_num` parity blocks and writes them to the remaining segments via - `DiskWriter::write`. - Data and parity handles are grouped into one completion per strip. - `ChunkWriter` retains at most `parity_depth` strip completions and joins all - remaining completions at `seal` - time (not at strip finish) — this decouples parity durability from - strip rotation, allowing strip N+1's data writes to start before - strip N's parity completes. -- **Strip prefetch (internal to ChunkWriter).** A background task - appends strips to the current chunk via `append_chunk` ahead of the - write cursor, bounded by `prefetch_strips_per_chunk` (default 1). Larger - known-size objects batch two strips per append when more than four remain. A normal - `append_chunk` response contains only new strips plus `modify_ts`; the - writer merges them locally. A stale revision response supplies the complete - current chunk and is retried once. The result replaces `self.chunk` (Arc-swap — old Arc in - any in-flight `EcStripWriter` stays alive). For known-size objects, - the prefetch stops after enough strips are allocated; for - unknown-size objects, it stays `prefetch_strips_per_chunk` ahead. -- **Chunk prefetch (ChunkPrefetch, object layer).** Pre-allocates the - next `Chunk` with `prefetch_strips_per_chunk` strips ahead of rotation, up to - `chunk_preparation_depth` ahead. On chunk rotation, the object layer - pulls the next `Chunk` from the prefetch receiver (fast path — - pre-allocated) or calls `on_demand` (slow path — prefetch fell - behind). The `Chunk` is passed to `ChunkWriter::open`, which wraps it - in `Arc` and starts the internal strip prefetch. - -### 3.1 Partial Last Strip - -Partial strips occur only at EOF, never mid-chunk. When EOF arrives -before the current strip is full, the writer persists only the filled -data. EC strips also release empty blocks and hand the partial set to -parity (§5). At chunk seal, ChunkDB records the durable `sealed_length` -of every written strip, whether mirror or EC, so a reader can cross -strip boundaries and read the partial final strip. - -## 4. Backpressure and Memory Budget - -Two independent limits bound the pipeline; neither depends on object -size. - -- **`max_cached_buffer`** (default 4 MB = one strip) bounds un-written - data in the fetch channel — blocks sent to the main write task but - not yet written to disk. When disk write is slower than network - receive, the cache fills; once full, the fetch stage blocks and - throttles the stream to the disk write speed. -- **`parity_depth`** (default 2) bounds in-flight parity tasks. When - the parity pool is full, the main write task blocks at hand-off — - backpressure on the write path, decoupled from the fetch cache. - -If prealloc falls behind, the main write task awaits on the bounded -strip channel; the fetch stage keeps filling `max_cached_buffer`, then -blocks when full. No data is lost. - -### 4.1 Per-Writer Footprint - -For 4+1 EC, 1 MB blocks, defaults: - -- `max_cached_buffer` — 4 MB un-written data in fetch cache. -- 1 block being written — 1 MB. -- Up to `parity_depth` (2) parity tasks, each holding the strip's data - blocks (4 × 1 MB = 4 MB, shared via `Bytes` ref count — not copied, - but resident until EC compute completes) + 1 parity block (1 MB). - -Peak: 4 + 1 + 2 × (4 + 1) = **15 MB**. The conservative 15 MB assumes -both in-flight parity tasks hold data refs simultaneously; realistic -steady-state peak is ~11 MB, since EC compute is fast relative to disk -write + fsync and parity tasks stagger. - -### 4.2 WriterPool - -`WriterPool` tracks a `memory_budget` and an atomic `in_use` counter. -`try_acquire` returns `MemoryBudgetExhausted` when the budget is full; -release decrements `in_use` on `Drop`. `max_concurrent = -memory_budget / per-writer-footprint`. Many concurrent large-object -uploads are thus bounded by available RAM, not by object size — the -pool rejects new writes when the budget is exhausted, propagating -backpressure up the call stack. - -## 5. EC Integration - -The pipeline holds data as separate 1 MB `Bytes` blocks; the existing -`crowdb_common::ec::encode` re-splits a contiguous buffer into shards, -which would force a 4 MB copy per strip just to re-split. Instead, -`encode_parity_from_shards(scheme, data_shards)` takes pre-split data -shards directly and reuses the existing isa-l FFI path — no new C++ -code. - -`data_shards.len()` is `data_num` for a full strip, or `< data_num` -for a partial strip. isa-l supports partial EC: missing shards are -treated as zero for the encoding matrix, so no padding is written to -disk — the reader reads only `sealed_length` bytes. The function -returns `code_num` parity shards. - -Edge cases: - -- Full strip → standard EC. -- Partial strip → parity from present shards; reader reads only - `sealed_length` bytes. -- Single-block object (< 1 MB) → 1 partial data shard, 1 parity shard. - -## 6. Chunk Rotation and Location - -Very large objects (`> max_chunk_size`) span multiple chunks. Chunk -size is always a multiple of strip data capacity — the writer only -appends whole strips — so rotation happens at strip boundaries, never -mid-strip. When `ChunkWriter::push` finishes a strip and `is_full()` -returns true, it returns `FeedStatus::Pause` without pushing the -buffer. The object layer then calls `seal()` (joins all in-flight -parity handles, `seal_chunk` RPC, returns `ProtoLocation`), records -the location, pulls the next `Chunk` from `ChunkPrefetch`, and opens a -new `ChunkWriter`. The buffer is re-pushed to the new chunk. The -`ProtoLocation` array accumulates one entry per rotated chunk, ordered -by `logical_offset`. - -Object writers use the common revision-2 ordinary frame codec: 15-byte header, -20-byte footer and at most 65,501 payload bytes per 64 KiB frame. The role is -ordinary; Stream head/follower markers are never added to object payloads. -Unknown-length uploads keep bounded tail buffers and publish an object only -when the complete upload finishes. Native socket buffers reserve the same -header/footer geometry and retain payload ownership through scatter/gather IO. -Readers retain explicit V1 support, including both historical CRC conventions; -revision 2 has canonical header/payload CRC coverage with separate chunk-ID -validation, shared with the C++ codec. - -### 6.1 Location - -`ProtoLocation` (from `crowdb-protocol::chunkdb::rpc::Location`) records -which chunk holds a contiguous byte range of an object, the byte range -within that chunk, and the object-level logical offset/length so a -multi-chunk object reads back as one contiguous stream. An object -spanning N chunks has N locations ordered by logical offset, contiguous -and non-overlapping. The within-chunk offset is always 0 for the -large-object writer (dedicated chunks filled from the start); it -exists for future shared-chunk packing and range reads. Serialization -is flatbuffers via the flatbuffer `Message` trait (`encode_to_vec` / `decode`). - -Edge cases: - -- Empty object (size 0) → `Vec` is empty; no chunk allocated. -- `logical_length` counts object payload bytes. `length` counts physical - bytes including ordinary frame headers and footers. Readers infer the - supported frame geometry from this pair and verify each actual frame kind - and revision before exposing payload views. -- `max_chunk_size` not a multiple of strip data capacity → rotation - still happens at strip boundaries; the actual chunk size is the - multiple-of-strip value at or just above the threshold. - -## 7. Completion and Error Handling - -`write_stream` must seal the final chunk and return the `ProtoLocation` -array on success. On error or abort it must not leak partial chunks and -must return already-sealed `ProtoLocation`s for caller cleanup. - -- **Completion** — `ChunkWriter::seal()`: finish the current strip if - it has data (spawn partial parity, no join), join all in-flight - parity handles, `seal_chunk` RPC, return `ProtoLocation`. If the - chunk is empty (0 bytes written), `delete_chunk` instead of - `seal_chunk`. -- **Error / abort** (`on_error`) — stop strip and chunk prefetch, retain and - drain every submitted write completion, then `delete_chunk` on the partial - chunk. Draining before deletion prevents a late write from hitting reused - storage. The public prepared-write API returns an error rather than a - partially successful object. -- **Single-segment replacement** — a failed data or parity completion retains - its `Bytes`, inserts the failed disk into the client-wide lock-free negative - list, and queries current chunk metadata. ChunkDB allocates one tentative - placement excluding every existing strip disk and nodes already at the - strip's failure-domain limit. The client writes that segment and publishes a - geometry-identical strip through exact revision-and-range replacement. - Successful shards are untouched. -- **Fencing and retry** — publication uses a deterministic operation ID. - Ambiguous results retry the identical request. A definite revision conflict - discards the tentative segment, re-queries, and retries only while the exact - failed identity remains. Repeated disk failures extend their exclusion TTL. - Attempts are bounded by `large_write_repair_attempts`, three by default. -- **Repair exhaustion** — `ChunkWriter::seal` returns the failure before - `seal_chunk`; its owner drains remaining completions and deletes the active - chunk. It never seals a strip with missing parity or publishes degraded - large-object locations. -- **Dropped writer** — dropping does not perform async metadata cleanup. - Applications call `on_error` when abandoning a started push-mode write; - an Active partial chunk left by process loss remains for future lifecycle - GC. - -Edge cases: - -- `on_data` after `on_finish` / `on_error` → `IoError::Finished`. -- `on_finish` twice → `IoError::Finished`. -- `on_error` with no sealed chunks → `Ok(vec![])`. -- EC encode failure → pipeline aborts immediately. EC encode is a CPU/ISA-L - failure rather than a disk placement failure and cannot use segment - replacement. -- `delete_chunk` fails during cleanup → log + continue (best-effort; - the partial chunk stays Active and is reaped by a future GC task). - -## 8. Interaction with Neighbors - -- **chunkdb** — the writer calls allocate / append / seal / delete / - fenced strip-range replacement / query via `ChunkAllocator`. chunkdb handles - placement and lifecycle; the writer is unaware of internal placement - logic — it receives `Segment` placements and writes to them. -- **diskio** — the writer writes data and parity blocks through the single - durable-completion contract `DiskWriter::write`. - `RoutedDiskWriter` is a narrow foreground-policy adapter over the semantic - `crowdb-diskio-client`; it does not own discovery, endpoints, RPC transport, - or result decoding. Conversion and repair use that client's separately - bounded priority lane. Direct wire access exists only in protocol fixtures. -- **crowdb-common EC** — `encode_parity_from_shards` is the shard-based + - partial-encode entry point used by parity tasks. -- **crowdb-protocol** — `ChunkId`, `*ChunkRequest` types, `Segment`, - `Chunk`, `Location` (`ProtoLocation`) message. - -## 9. Tunables and Defaults - -Chunk capacity is a write-policy limit, while each persisted strip records its -own size and protection layout. A single-node test chunk may contain many -1 MiB one-copy mirror strips. The S3 and Iceberg access policies can select -different small and large chunk capacities in their configuration. Tree page -storage separately uses `storage.tree_chunk_capacity_bytes` from the chunk KV -server configuration and splits a single-copy tree chunk into 1 MiB strips. -Chunk KV stream storage uses `storage.stream_chunk_capacity_bytes` for the -`Stream` chunk type, also independently of its 1 MiB strip geometry. - -| Knob | Default | Role | -| --- | --- | --- | -| `max_chunk_size` | 1 GB | Chunk rotation threshold. | -| `prefetch_strips_per_chunk` | 1 result | Strip-preparation results buffered ahead of the write cursor. | -| `parity_depth` | 2 strips | Completed-strip write groups allowed in flight. | -| `chunk_preparation_depth` | 1 chunk | Chunk preparation ahead of rotation. | -| fetch granularity | 1 MB | One data block per fetch call. | -| `max_cached_buffer` | 4 MB | Un-written data budget in fetch channel (one strip). | -| `memory_budget` | (pool) | `WriterPool` total; `max_concurrent = budget / per-writer-footprint`. | - -## 10. Application API and Routing - -`ChunkIoClient` is the application boundary. `connect` takes management seeds -and discovers ChunkDB endpoints, DiskIO registrations, hardware disks, and disk -group ownership. `prepare_large_write` starts bounded chunk preparation when -the object size and `LargeWritePolicy` become known. The single-use -`PreparedLargeWrite::write_stream` consumes an `AsyncRead`. -`PreparedLargeWrite::write_buffers` transfers caller-owned `Bytes` blocks -directly into the same writer, bypassing fetch and its input copy. Both return -`LargeWriteResult` with locations, logical and EC-expanded physical bytes, -chunk and strip counts, elapsed time, and preparation stalls. - -`prepare_large_writes` starts several single-use sessions concurrently and -waits until each owns its first chunk. A server calls it before opening its load -gate, keeps the returned queue, and starts one replacement session whenever it -consumes one. Queue depth is an application admission setting; the benchmark -defaults to ten through `--prefetch-chunks`. Unused sessions are explicitly -aborted so their Active chunks are deleted. - -The write and read hot paths call the semantic DiskIO client with a checked -segment-relative address, caller-owned `Bytes`, lane, durability policy, and -deadline. That client reads immutable disk-ID route generations through RCU, -owns fixed normal and priority connection groups, and selects healthy members -without a caller lock. Refresh joins authoritative service and hardware -records, constructs and prewarms a complete replacement off-path, and reuses -unchanged endpoint groups. Missing or duplicate ownership rejects the whole -candidate; no chunk caller chooses an arbitrary endpoint. - -`ChunkIoClientConfig::diskio_connections_per_endpoint` configures the normal -foreground group. Conversion and repair use reserved priority resources, so -their queue or connection pressure cannot consume the normal group. For the -native crowdb-tree page store, `ChunkIoClient` passes through an opaque retained -route set produced by the DiskIO client; neither application code nor the CLI -assembles routes or constructs RPC servers and connections. - -Topology refresh follows server ownership. `refresh_chunkdb_routes` refreshes -ChunkDB endpoints and range bindings; `refresh_diskio_routes` asks the DiskIO -client to publish its next complete generation. Metrics wrappers only observe -the two narrow seams and do not own scheduling, discovery, or connection -lifetime. - -## 11. Performance Workload - -Chunk IO performance workloads live in `crowdb-chunk-client`; CLI commands -only map arguments, start the standard process metrics collector, and format -results. - -- `run_large_write_benchmark` retains the deterministic bounded-source writer. - Before its timer starts, it prepares write sessions and distributes them - across workers. Each worker starts a replacement allocation when consuming a - session. -- `run_small_write_benchmark` uses `prepare_small_write`, awaits each real - Location, drains the shared pool, and reports batch fill, queue delay, - active/draining pipeline gauges, queue-driven scale-out/scale-in, throughput, - latency, and exact object accounting. -- `run_read_benchmark` prepares a bounded reusable dataset through the real - small- and/or large-write APIs before timing. Small, large, and deterministic - mixed request modes call `read_object`; mixed ratio is by request count. - Successful reads validate logical length. Under NullDisk they deliberately - do not compare payload contents. - -The CLI commands are `bench chunkio write`, `write-small`, `read-small`, -`read-large`, and `read-mix`. Results separate preparation from timed load and -include aggregate throughput, latency, errors, stop reason, and exact admitted -versus completed accounting. - -The CLI workload admits objects until a shared duration deadline (20 seconds by -default). A worker checks the deadline before admitting its next object; an -object already admitted is allowed to finish, including all DiskIO completions -and chunk seal, before the worker exits. Reported elapsed time includes this -drain tail. The object count remains a safety cap and a deterministic test mode, -not the normal regression stopping condition. - -The regression fixture starts three co-located logical nodes in one rack: -three KV servers, three DiskDB, three ChunkDB, and three DiskIO processes -backed by `NullDisk`. KV state and WAL use `mem-block` with fsync disabled, so -benchmark metadata also avoids real media. Read preparation still writes real -ChunkDB/DiskDB/KV metadata and writer-produced Locations; only payload storage -is synthetic. An 8+4 strip in this intentionally compact local topology -requires the local-test-only unsafe EC placement option; disk ownership and -routing remain strict. Unsafe placement still balances blocks across the -available nodes within a rack; it relaxes the failure-domain limit without -concentrating the strip on the first node. The retained logs -contain `bw_read_mib` and `bw_write_mib` when host PMU counters are available. -Their sum is observed host memory traffic during the workload, not physical DIMM peak bandwidth and not -an application-byte estimate. Loopback TCP, EC expansion, RPC framing, kernel -copies, EC calculation, synchronous writes, and metadata work all keep end-to-end logical -throughput below that hardware envelope. - -### 11.1 Intel Core i9-7960X Baseline - -Reference run 2026-09-08: Intel Core i9-7960X (16c/32t, Skylake-X), 4 DDR4 -channels at 2667 MT/s (~85 GB/s peak), Linux 6.11, `perf_event_paranoid=-1`. - -The sentinel uses 16 MiB objects, EC 8+4, 1 MiB blocks, ten prefetched chunks, -and two strips prefetched per active chunk. Each worker creates one deterministic -random 1 MiB source block and reuses it. Stream cases traverse the production -`AsyncRead`; direct cases pass reference-counted `Bytes` blocks to the same -writer without a fetch copy. Both traverse EC, routed RPC, the DiskIO handler, -and real io_uring submission. RPC sends every 1 MiB data or parity payload through -the loopback socket. The storage target is the only substitution: NullDisk -replaces production BlockDisk and does not request stable-media durability. - -| Mode | Writers | Logical MiB/s | Physical MiB/s | p50 / p99 ms | DRAM read avg | DRAM write avg | DRAM total avg | -| -------------- | ------: | ------------: | -------------: | ----------------: | ------------: | -------------: | -------------: | -| Stream | 1 | 162.5 | 243.8 | 99.191 / 111.225 | 2,496.6 | 1,393.0 | 3,889.6 | -| Direct buffers | 1 | 229.1 | 343.7 | 69.191 / 85.641 | 1,996.1 | 1,228.7 | 3,224.8 | -| Stream | 4 | 1,965.6 | 2,948.5 | 31.917 / 51.806 | 12,699.7 | 10,943.0 | 23,642.7 | -| Direct buffers | 4 | 2,672.1 | 4,008.1 | 22.811 / 40.000 | 11,799.6 | 12,051.5 | 23,851.1 | -| Stream | 32 | 3,249.3 | 4,873.9 | 151.716 / 267.992 | 19,562.5 | 15,603.7 | 35,166.2 | -| Direct buffers | 32 | 3,547.9 | 5,321.9 | 138.654 / 261.388 | 14,816.5 | 12,766.6 | 27,583.1 | - -Per-object client-stage time is measured inside the production writer. Times -overlap and therefore must not be added to predict object latency. - -| Flow step | Stream 1 | Direct 1 | Stream 4 | Direct 4 | Stream 32 | Direct 32 | -| --------------------- | -------: | -------: | -------: | -------: | --------: | --------: | -| Source/read copy | 20.247 | 0 | 3.737 | 0 | 8.704 | 0 | -| Fetch assembly copy | 0 | 0 | 0 | 0 | 0 | 0 | -| EC encode | 20.175 | 13.164 | 15.213 | 12.357 | 22.045 | 20.222 | -| Write-completion wait | 53.990 | 54.238 | 11.741 | 10.118 | 122.786 | 120.972 | - -All values are milliseconds per object, aggregated across concurrent writers; -stages overlap and must not be summed into latency. Every object has two data -strips, 16 data blocks, eight parity blocks, and therefore 24 full 1 MiB DiskIO -RPCs. This is exactly 16 MiB logical and 24 MiB physical traffic: EC 8+4 gives -the expected 1.5 physical/logical ratio. The run recorded zero preparation -stalls and zero `append_chunk` calls: initial chunk allocation returned both -strips requested by an explicit `prefetch_strips_per_chunk=2` override. - -Stream mode has one application payload copy per block: socket/source into its -final owned block. Direct-buffer mode has zero application payload copies; -`Bytes` clones, channels, EC input, routing, and RPC buffer wrapping transfer -ownership only. Both modes still copy or DMA bytes across the loopback transport -and DiskIO boundary. EC reads 16 MiB and creates 8 MiB parity per object, while -the RPC path transports 24 MiB, so physical memory traffic must be multiple -times frontend bandwidth. The multiplex-corrected host read/write counters are reported above, -but is not a complete byte ledger and can be below RPC bandwidth because of its -sampling scope and interval. - -### 11.2 AMD Ryzen 9 5950X Baseline - -Reference run: _pending._ - -| Mode | Writers | Logical MiB/s | Physical MiB/s | p50 / p99 ms | DRAM read avg | DRAM write avg | DRAM total avg | -| -------------- | ------: | ------------: | -------------: | -----------: | ------------: | -------------: | -------------: | -| Stream | 1 | | | | | | | -| Direct buffers | 1 | | | | | | | -| Stream | 4 | | | | | | | -| Direct buffers | 4 | | | | | | | -| Stream | 32 | | | | | | | -| Direct buffers | 32 | | | | | | | - -| Flow step | Stream 1 | Direct 1 | Stream 4 | Direct 4 | Stream 32 | Direct 32 | -| --------------------- | -------: | -------: | -------: | -------: | --------: | --------: | -| Source/read copy | | | | | | | -| Fetch assembly copy | | | | | | | -| EC encode | | | | | | | -| Write-completion wait | | | | | | | - -### 11.3 Bottleneck Conclusion - -Independent blocks are already parallel: each of the eight data writes is -spawned as soon as its block enters `EcStripWriter`, then four parity writes are -spawned together after EC completion. `ChunkWriter` groups the 12 handles per -strip, permits `parity_depth` strip groups in flight, and joins them before -seal. There is no serial per-block completion wait. - -Chunk preparation is also off the measured hot path. The remaining single-write -gap is input ownership plus cache pressure: direct buffers reach 229.1 MiB/s, -1.41 times the stream result, and spend no time in source reads. Four direct -writers peak at 2,672.1 logical MiB/s. At 32 writers throughput rises to 3,547.9 -MiB/s but completion wait rises to 121 ms/object. The loopback RPC/DiskIO queue -is saturated well before 32 writers; more concurrency increases latency rather -than bandwidth. Further work should profile and reduce RPC/kernel transport cost -and EC/cache contention. Increasing prefetch or adding wrapper layers will not -address the measured bottleneck. diff --git a/doc/design/config/design-crowdb-config.md b/doc/design/config/design-crowdb-config.md deleted file mode 100644 index 1f58d9296..000000000 --- a/doc/design/config/design-crowdb-config.md +++ /dev/null @@ -1,195 +0,0 @@ - - - -# CROWDB - Design: Service Configuration - -This document defines the startup configuration contract shared by CROWDB -server processes. Component designs define the meaning of their domain -settings; this document defines how values are owned, loaded, merged, -validated, and activated. - -## Table of Contents - -- [1. Scope](#1-scope) -- [2. Ownership](#2-ownership) -- [3. Resolution and Activation](#3-resolution-and-activation) -- [4. TOML Schema](#4-toml-schema) -- [5. Reload and Restart](#5-reload-and-restart) -- [6. Deployment](#6-deployment) -- [7. Failure Handling](#7-failure-handling) -- [8. Invariants](#8-invariants) -- [9. Non-Goals](#9-non-goals) - -## 1. Scope - -The `crowdb-kv-server`, `crowdb-diskdb`, `crowdb-chunkdb`, `crowdb-diskio`, -and `crowdb-access-server` processes accept typed TOML startup -configuration. A service may require a file or make it optional, but a supplied -file follows the same resolution and failure rules in every process. - -This contract covers process configuration. Durable cluster topology, -membership, allocation metadata, and application data keep their existing -authorities. - -## 2. Ownership - -- A reusable library owns config types for behavior implemented by that - library. The KV schema is therefore owned with the KV library. -- A server binary owns startup inputs and the final merge before constructing - listeners and services. -- `crowdb-common` owns shared Rust TOML loading, validation, watching, and - structural diff mechanics; it does not own service-specific fields. -- `DioConfig` owns the equivalent typed C++ schema and TOML decoding for - diskio. - -Client-side concurrency fields remain named for the dependency they drive, -such as `kv_rpc_workers`, `diskdb_rpc_workers`, and `peer_pool_size`. -`server.rpc_workers` always means workers accepting that process's inbound RPC -traffic. - -## 3. Resolution and Activation - -Every process resolves a complete candidate in this order: - -```text -typed compiled defaults -> TOML values -> explicitly present CLI values - -> complete validation -> runtime construction -``` - -A CLI parser default is not an explicit value and cannot overwrite a file. -Rust CLI fields participating in the merge use optional values. Diskio loads -its single `--config` path before replaying the other explicitly supplied CLI -options, so argument order does not change precedence. - -The final candidate is validated before listener construction. Runtime -components receive values from this candidate; wiring layers do not maintain -independent defaults. - -## 4. TOML Schema - -Common section names describe process roles: - -| Section | Ownership | -|---------------|-------------------------------------------------------------| -| `[server]` | Listener identity, server-side RPC resources, and heartbeat | -| `[engine]` | Local execution and I/O engine tuning | -| `[group0]` | Local group-0 discovery and registration inputs | -| `[metrics]` | Metrics output and cadence | -| Domain tables | Service-owned storage, placement, lifecycle, or policy | - -The service schemas retain their different domain sections: - -| Service | File policy | `server.rpc_workers` default | Principal domain sections | -| ------- | ----------- | ---------------------------- | ------------------------------------------------ | -| KV | Optional | 2 | Paxos, WAL, engine, metrics | -| diskdb | Required | 2 | Storage, heartbeat, persistence, scanner, sync | -| chunkdb | Required | 2 | Storage, placement, lifecycle, topology, clients | -| diskio | Optional | 4 | Engine, group-0 discovery, metrics, disk entries | -| Access | Optional | N/A | HTTP listeners, shared read, S3, Iceberg GC | - -The S3 and Iceberg access processes accept the same `AccessConfig` schema. -Each process activates only its own protocol section plus `[common]`, `[read]`, -and `[small_write]`. The file is selected with `--config`; a supplied file is -validated before connecting to dependencies or binding listeners. Read slots, -the per-read window, the global retained-byte budget, the separate EC recovery -budget, and small-write memory limits are static startup settings. Credentials -and bearer tokens remain in the environment. For access settings that predate the -file, an explicitly populated TOML value takes precedence over the legacy -environment variable; an omitted TOML value keeps the legacy behavior. - -The small-object routing bound is `threshold_ratio × disk_block_bytes × EC data -shards`; mirrored strips use one data shard. `[small_write]` supplies the default -EC layout for both access writers, and S3's EC override changes its small and -large writers together. The default is 8+4 with 1 MiB data blocks and ratio -0.9. The single-node container profile uses 2+1. - -All schemas tolerate unknown keys so a newer file can be staged before a -binary upgrade. A known key with the wrong type or invalid value rejects the -candidate. Omitted fields take their typed defaults. - -Diskio represents disks as ordered `[[disk]]` entries. A disk ID is a string -because it accepts either a single hexadecimal low word or `high:low`. Each -entry supplies a path and one zone capacity. An empty path selects a dummy -disk. Fault latency bounds and error rate are optional `[server]` fields. - -## 5. Reload and Restart - -`server.rpc_workers` is static in all services because changing it requires -reconstructing the RPC listener. A file watcher may detect and report the -change, but the running listener retains its startup value until restart. - -Other fields follow their component's reload contract. A watcher never makes -a startup-only field dynamic merely because it can observe a changed file. - -## 6. Deployment - -Tracked templates are executable examples guarded by loader tests. Local -deployment writes node-specific files where addresses, roots, or group-0 seeds -vary, and restart reuses the same file path. Explicit deployment overrides are -represented as explicit CLI options and therefore retain the standard -precedence. - -The single-node container uses a named, validated `DeploymentProfile` and -rendered service configurations. `crowdb-monitor` owns service supervision, -dependency order, probes, restart budgets, bounded logs and shutdown. A durable -bootstrap manifest binds the profile/configuration digest and generated -credential identity to the mounted data volume. Restart replays completed steps; -conflicting identity fails closed. Child recovery revalidates storage, catalog -and Web authority before readiness returns. - -Release programs and UI are compiled incrementally on a Linux amd64 host with -the locked repository toolchain. Docker receives a staged runtime directory -containing only those artifacts, required shared libraries and the deployment -profile. It performs no source compilation and contains no Pixi or compiler. -Runtime linkage and source/version metadata are checked while packaging into -the digest-pinned Ubuntu image. The publication job reuses the verified staged -artifacts rather than compiling a second set. - -Group 0 owns hardware/topology and service registration. It does not store -container mounts, process PIDs or restart policy. Docker Web reads live Group 0 -topology and fresh monitor process status independently; missing authority does -not produce an empty or cached topology. Automatic single-node startup disables hardware/process mutation at both UI and -backend. Manual node startup runs only monitor, SSH and Web until an operator -prepares nodes and initializes Group 0; then shared authority enables management. -Both startup policies use the same image and services. Discovery UUID, SSH keys, -host keys, binding and operation recovery records reside on the persistent root. -`CROWDB_KV_MANAGEMENT_ADVERTISE_ADDR` specifies the reachable management address -for service registration independently of the local listening address. See the -[deployment architecture](../deploy/design-crowdb-deploy.md) for ownership and -recovery ordering. - -Public examples and the container profile use Console 9090, S3 9091 and -Iceberg 9092. Standalone Web defaults to 9090; explicit service configuration -and managed deployment port allocations take precedence. Container publication, -volume and endpoint usage are defined by the container deployment files. - -## 7. Failure Handling - -Malformed TOML, an unreadable named file, a wrong known-field type, or failed -validation terminates startup before bind. The loader does not silently fall -back to defaults after an operator explicitly names an invalid file. - -Diskio file loading is transactional: decoding and validation use a temporary -`DioConfig`, and the caller's config changes only after the complete candidate -is valid. - -## 8. Invariants - -- **I1 — deterministic precedence:** The same defaults, file, and explicit CLI - inputs resolve to the same candidate independent of CLI argument order. -- **I2 — complete activation:** No runtime component observes a partially - decoded or partially validated candidate. -- **I3 — worker viability:** Every server starts with at least one inbound RPC - worker. -- **I4 — one runtime authority:** Listener construction reads - `server.rpc_workers` from the final candidate and has no shadow default. -- **I5 — static-change honesty:** A detected static change is not reported as - active until a restart has constructed runtime state from it. -- **I6 — restart stability:** Deployment restart uses the same serialized - configuration path as initial startup. - -## 9. Non-Goals - -Configuration is local to each server. Group 0 does not publish configuration -revisions, resolve scoped overrides, track applied revisions, or coordinate -dynamic activation. The local file remains the startup authority. diff --git a/doc/design/console/design-crowdb-console-ui.md b/doc/design/console/design-crowdb-console-ui.md deleted file mode 100644 index 243492726..000000000 --- a/doc/design/console/design-crowdb-console-ui.md +++ /dev/null @@ -1,1865 +0,0 @@ - - - -# CROWDB - Design: Console UI Specification - -Depends on: [Console architecture](design-crowdb-console.md), -[KV architecture](../kv/design-crowdb-kv.md). -Satisfies: [Console architecture](design-crowdb-console.md). - -This is the authoritative UI design and acceptance specification. Each numbered -contract defines observable behavior, not a claim that its implementation has -passed. Implementation status, runtime evidence, defects, and execution steps -belong in the working plan. There is no separate UI spec to reconcile with this -file. Backend architecture and wire formats remain in the linked component docs. - -The scope covers all seven domains and their shared interaction rules. Backend -prerequisites and inspection APIs are part of acceptance; a rendering fixture -does not establish that a native service operation succeeds. - -## Contents - -- [1. Product contract](#1-product-contract) -- [2. Reference environment](#2-reference-environment) -- [3. Navigation and layout](#3-navigation-and-layout) -- [4. Visual language and service identity](#4-visual-language-and-service-identity) -- [5. Cluster](#5-cluster) -- [6. Properties and cross-links](#6-properties-and-cross-links) -- [7. Modes and embedding](#7-modes-and-embedding) -- [8. Bounded observation](#8-bounded-observation) -- [9. Creation and mutation dialogs](#9-creation-and-mutation-dialogs) -- [10. Accessibility](#10-accessibility) -- [11. Verification contract](#11-verification-contract) -- [12. KV](#12-kv) -- [13. Service lifecycle](#13-service-lifecycle) -- [14. DiskGroup and disk management](#14-diskgroup-and-disk-management) -- [15. Capacity](#15-capacity) -- [16. Loading, failures, and recovery](#16-loading-failures-and-recovery) -- [17. Fixed-cluster operator flow](#17-fixed-cluster-operator-flow) -- [18. Acceptance scenarios and scope](#18-acceptance-scenarios-and-scope) -- [19. Chunk](#19-chunk) -- [20. Chunk-KV](#20-chunk-kv) -- [21. Iceberg](#21-iceberg) -- [22. S3](#22-s3) -- [23. Actions and properties](#23-actions-and-properties) -- [24. E2E organization and timing](#24-e2e-organization-and-timing) -- [25. Dataset](#25-dataset) - -## 1. Product contract - -- **UI-01:** One Console serves one configured cluster. Opening the UI enters - that cluster directly; no Connect dialog, endpoint input, or access-key form - precedes ordinary browsing. -- **UI-02:** The seven top-level tab labels are Cluster, PaxosKV, Chunk, ChunkKV, - Iceberg, Dataset and S3. KV and Chunk-KV remain their internal domain identities. - Each domain has an independent selected resource and query window. Cluster - shares topology/capacity selection and tree expansion without a view switch; - topology visibility follows CLU-05. -- **UI-03:** Assume an administrator/root session until UI authentication is - introduced. Available backend operations remain authoritative; capability - restrictions are not inferred from a missing login screen. -- **UI-04:** Display real resources and structured values. Example graphs, - inferred ownership, and placeholder counts must not look like observed data. -- **UI-05:** Every remote list, expansion, preview, and layout has a work/size - bound. A large cluster must not require a complete scan before the UI responds. -- **UI-06:** Visible UI behavior is governed by this specification. A proposed - change to behavior or layout is first discussed and added here, including its - ASCII layout when applicable; implementation follows the agreed spec and may - not silently introduce a conflicting control or tree level. Focused tests or - E2E coverage accompany the implementation. -- **UI-07:** Iceberg and Dataset work areas and Properties show resource data, - controls and concise current state. Do not keep tutorials, pagination limits, - upload/publication internals, immutability explanations or GC explanations as - permanent prose around tables and editors. General guidance belongs in the - existing Help documentation. Keep field labels, identities, measured values, - empty states, upload progress, validation errors and contextual conflict or - destructive-action information. Draft feedback uses one short status: - `Saving…` or `Saved locally`; a failed save must not display success. - -## 2. Reference environment - -The main acceptance environment has one Rack and three Nodes. Each Node has one -KV, DiskDB, ChunkDB, DiskIO, Chunk-KV, and Access Server instance. Group 0 has -three replicas; an ordinary data Group 1 has three replicas. Storage has valid -DiskGroup bindings, live owners, and writable disks. Allocated and free blocks -provide observable capacity and bitmap fixtures. - -- This is normal multi-node deployment. Single-node/unsafe colocated settings - cannot be silently enabled to make acceptance pass. -- Rack diversity is preferred when multiple racks exist. A single rack with - enough distinct nodes is valid; properties report actual node/rack protection. -- Three local processes on one host verify logical placement and UI behavior; - they do not demonstrate physical-host fault tolerance. -- API provisioning may prepare this environment. The setup method does not - replace the dedicated tests for creation and lifecycle dialogs. -- Empty, partial, unavailable, and very large datasets are additional fixtures, - not reasons to replace the populated reference environment with mocks. - -## 3. Navigation and layout - -- **NAV-01:** The header keeps the seven domains in a stable order. Domain - changes preserve that domain's bounded query state and selected identity. -- **NAV-02:** The left panel selects hierarchy/scope; the center presents the - selected resource or a paged child list; the right panel presents selected - properties. Resource operations follow the context-menu contract in §23. -- **NAV-03:** Cluster row/card selection updates properties without replacing the center - scope. Names enter that resource; closing properties retains the work area. - Cluster breadcrumbs identify the current physical scope; they are navigation, not duplicate - Back links for inline details. - This graph inspection distinction does not apply to ordinary resource tree/list - rows: activating their label, icon or row selects the same resource. Expansion - controls are separate. At DiskGroup scope, both Disk icon and Disk ID enter - the same Disk workspace, including from a DiskDB ownership reference. -- **NAV-04:** Cross-links identify the exact destination resource. Returning - restores the originating selection and page. A stale/deleted destination has - an explicit state rather than silently selecting a different item. The header - and browser Back/Forward share one session history with at most 32 retained - visits. History stores identities, bounded - query cursors and scroll coordinates, never object bodies or metadata payloads. - New navigation clears Forward. Returning refetches the saved bounded window; - scroll restoration waits for that window to render. Resource types and parent - identities participate in selection equality, so equal numeric IDs in - different resource kinds cannot share a selected highlight. Returning to an Iceberg reference verifies the - saved metadata generation before inspecting it again. -- Shared physical/logical trees retain expansion and local search per domain. - Return restores both expanded and collapsed branches. KV captures bounded - byte cursors and focused Key identity, then refetches the window; it does not - keep Value payloads in history. Capacity retains at most 32 Disk query states - containing zone window, selected Zone and bitmap block window. Parent - navigation can unmount a detail without losing its return query. -- **NAV-05:** Trees expand lazily. Each large child collection has continuation - or Load more and an explicit end/partial state. Routine loaded-item counts are - not shown in tree footers; they are internal pagination state unless a domain - contract explicitly makes a count part of the selected resource. Collapsed - branches do not recursively load their descendants. -- **NAV-06:** The center remains usable with the right panel open. Tables and - diagrams scroll within their work area; action bars and pagination remain - reachable. Resizing does not reset selection or trigger unbounded refetches. - Capacity observations retain the last successful inventory, usage and scan - independently after a failed refresh. Unknown values and refresh errors appear - with the affected capacity data; raw observation metadata is not a global strip. Runtime usage is keyed by group/disk - scope; an unobserved scope remains unknown instead of borrowing other totals. -- **NAV-07:** Every domain uses the shared Tree appearance and draggable - panel dividers. Cluster/KV left default width is 240 px; data workbenches use - 280 px. Properties default is 320 px. - Cluster navigation resizes between 200 and 360 px, properties between 280 - and 480 px, preserving 480 px for the center when space permits. Below - 1100 px in Cluster, properties overlays the work area as a dismissible drawer; Escape - closes it and restores focus. Navigation remains visible without a hide/show control. Other domain dividers retain their 220–600 px bounds. - Switching domains introduces no panel replacement/slide animation. -- **NAV-08:** Parent cards in center topology diagrams have separate named controls for - opening the resource and collapsing or expanding children. Card bodies inspect - without navigating. Children retain their own - expansion state. Sidebar focus reveals hidden ancestors. Right-click opens - the entity menu without toggling expansion. Fit All fits visible cards. - Hiding a topology panel interrupts viewport motion before zero-size geometry - is observed. Rapid domain switches retain finite coordinates; centering runs - only in a visible, measured canvas. - The default diagram initially expands Datacenter/Rack and stops at visible Node cards; - service descendants appear when a Node is explicitly expanded. - -The header shows the CrowDB brand, domain navigation, then Back/Forward and -Refresh in the right action area. Access instance `access-1` displays as `AS-1` -while API operations retain the original identifier. - -Shell layout (NAV-01/02/04/07): - -```text -+------------------------------------------------------------------------------------------------------+ -| CrowDB Health | Cluster PaxosKV Chunk ChunkKV Iceberg Dataset S3 | [Back] [Forward] [Refresh] | -+------------------------+------------------------------------------------------+----------------------+ -| Resource tree | Current scope | Properties | -| v Parent | Query / filter [Refresh] | Selected identity | -| Child | Primary content or paged child list | Structured values | -| | [Previous] [Next] | | -+------------------------+------------------------------------------------------+----------------------+ - draggable divider draggable divider -``` - -All wireframes show layout contracts, not runtime evidence. Names, IDs, counts -and statuses are illustrative. `>`/`v` denote collapsed/expanded branches, `*` -marks selection, and `[Auto]` denotes the accessible Auto icon. Menu entries -remain subject to the resource capabilities and lifecycle rules in the text. - -## 4. Visual language and service identity - -- **VIS-01:** Use muted colors with readable text and visible boundaries. - Buttons, including Run Recalc, must have adequate text/background contrast. - Color alone cannot encode state, role, or selection. -- **VIS-02:** Service instances use one compact `TYPE-ID` convention everywhere: - `PKV-1`, `DDB-1`, `CDB-1`, `DIO-1`, `CKV-1`, and `AS-1`. Trees, topology cards, - selected titles, and instance menus use the same label. -- The suffix identifies the service instance in its own type. It must not be - fabricated from a Node ID when those identities differ. Exact backend IDs - remain available in properties and are used for API operations. -- Full names such as ChunkDB and Access Server belong in type properties, - deployment choices, or help text. A tree label must not mix a verbose name, - a separator, and another backend identifier. -- **VIS-03:** All integer IDs and offsets preserve their exact values; large - identifiers do not pass through lossy JavaScript number conversion. -- **VIS-04:** Selected resources use a consistent outline/highlight. Health, - lifecycle, readiness, and data protection are separate labelled values. -- Capacity bitmap: muted blue means used, muted green means free. Unknown, - unavailable, and reserved states have distinct labels/patterns; they are never - rendered as free. - -## 5. Cluster - -- **CLU-01:** The hierarchy is Datacenter → Rack → Node → service instances. The center - topology uses physical identities and shares selection with the left tree. - The left tree separates each Node into Services and Storage. Physical groups - and disks belong under Storage regardless of their DiskDB owner; service - instances remain under Services. Logical Paxos groups are managed in KV. - A DiskDB instance also expands to references to the DiskGroups it currently - owns. These use authoritative instance ownership, not physical Node membership. - They open the same resource as Storage references and update on ownership changes; - unknown ownership never assigns a group to an arbitrary instance. - Candidate nodes forms another Datacenter branch, alongside Racks, consistent - with disk candidates under Storage. Only unjoined candidates for this cluster - appear there; confirmed members appear only in their Rack. Raw discovery logs - are not rendered in the tree. Legacy Capacity links resolve to Cluster. - DiskGroup ownership references use the compact `DG-` label directly; - do not prefix them with `Managed storage`. The candidate branch is labelled - `Candidate nodes`; while its authoritative list is loading it may expand to - show the loading state, but once the list is known to be empty it has no - expand arrow and renders no `No candidate nodes` placeholder. Candidate - rows appear only when there are eligible unjoined nodes. Storage candidate - rows may show `Candidates (N)` when the authoritative count is known, - including zero, but never `Candidates (?)`. - Capacity properties use the same hardware-inventory and runtime-usage scope - at Storage/DiskGroup, Node, Rack and Datacenter levels. A complete known - aggregate shows capacity, used and free values; an incomplete observation - remains explicitly unknown and is never rendered as zero. - Add Rack and Add Store belong to the respective Datacenter context menu; - uninitialized KV exposes Initialize Cluster there. -- **CLU-02:** Rack actions include Add Node. A Node context menu independently - manages each service type and each existing instance, using the same lifecycle - pattern as KV/DiskDB. It must not route other service types through KV APIs. -- **CLU-03:** Standard OCI Add Node opens the existing candidate flow with its - Rack preselected. Verify and review authenticates SSH and reads actual disk - inventory without admitting the node or initializing storage. Join and start - submits the reviewed choices and accepted preflight identity. Existing accepted - or pending-review Rack choices take precedence over a new default. -- Defaults select Paxos-KV, DiskDB, DiskIO, ChunkDB, Chunk-KV and Access Server. - Each service and its listeners can be disabled. Access has independent Iceberg, - S3, Dataset and Health ports; DiskDB's related RPC listeners must not overlap. Node - admission owns the fixed KV bootstrap listeners. Canonical service type is - `paxos-kv`; a legacy `kv` wire value is rejected. -- With no supplied input, standard setup chooses a 16-GiB preallocated file and - 1-GiB host reserve. Sparse allocation is explicit. Supplied media retain exact - observed identities and capacities; each raw device requires initialization - acknowledgement. Diskless setup creates no disk. Invalid supplied choices - never fall back to a managed file. -- The target monitor durably accepts storage and service intent before reporting - acceptance. Progress continues after the dialog, Web connection or browser - closes; restart resumes the same intent. Node progress polls every two seconds - and retains last confirmed rows on observation failure. A completed setup - receipt is separate from current service or I/O health. -- During creation, visible topology and service health refresh independently - every second; a slow capacity request must not stop service observations. - Opened Storage branches refresh without collapsing or reopening them. - Returning to the browser immediately observes current state without F5. - Polls preserve selection, expansion and the topology viewport. -- Storage waits for Group 0 and an ordinary metadata binding. Dependent services - wait for real prerequisites. A failed disk/service retains successful siblings; - explicit retry uses their original operations and fixed placement. Later - operator Stop/Restart/Delete commands take precedence over setup replay. - Consecutive failed startup/readiness attempts have a durable five-attempt - budget per service operation; successful readiness clears that budget. -- Legacy standalone Add Node retains its one-dialog registration and saved - six-service plan. Its session reconciler saves revision-checked progress before - deployment and resumes waiting steps while that Console session is active. - Unavailable process observations and failed steps continue read-only observation - every two seconds, as defined in §13. A fresh, identity-matched running - observation reconciles interrupted progress to deployed. Explicit Retry is - required to initiate another failed Start, Restart, or Deploy mutation. It never - duplicates a registered Node or successful service. This session-owned plan is - distinct from the standard target monitor's accepted node setup. -- Internal service health probes the RPC endpoint independently for each - service. Startup records lacking RPC endpoints use process observation. - Before its first store exists, PKV reserves but does not bind its RPC port; - process liveness covers that startup phase. Once a store exists, a failed - RPC probe remains Failed even when the process is alive. - Access health probes its dedicated listener; missing retained health - configuration is Unknown and requires reconciliation before restart. - Current service observations take precedence over persisted node projections. - DiskDB instances are associated by complete advertised RPC addresses; equal - ports in different node namespaces do not imply the same instance. -- Cluster selections never display or request Chunk ownership. -- **CLU-04:** Cluster readiness is reported by capability: KV quorum, storage, - Chunk-KV, and Access protocols. A PID or registry entry alone is not Ready. - Missing prerequisites link to the domain where they can be resolved. Remote - Access readiness uses the committed dedicated health listener on the owning - node; missing configuration remains Unknown and failed probes are Down. - -- **CLU-05:** The cluster overview and Datacenter/Rack/Node scopes show topology. - Datacenter/Rack/Node capacity appears below it. Storage, DiskGroup, Disk and - Zone scopes hide topology and release its space for storage content/capacity, - including when entered through a DiskDB ownership reference. Returning to a - higher scope restores topology. Service selections and an unselected cluster - do not show capacity. Node details and - configuration editing belong in the properties panel. Storage selection exposes - its disk operations without a second node summary. There are no Topology/Capacity - tabs, diagram toggle, or node-card toggle. -- **CLU-06:** Auto join starts enabled for a new standard workspace and retains - an explicit target. Three authenticated eligible nodes are ordered by UUID; - the smallest UUID initiates Store 0 / Group 0 with those fixed three voters. - Fewer nodes wait. Later admission is nonvoting and never expands/replaces - Group-0 voters. Explicit single mode auto-starts the same initialization - lifecycle with one local voter. Turning Auto join off stops new triggers; - accepted initialization continues and conflicting intents require recovery. - Eligible unjoined candidates default to Auto join, including later admissions - into a bound cluster. Each candidate can retain a durable explicit override. - Enabled candidates join the default Rack; later node editing can change its - Rack. Disabling one candidate stops its new admission triggers without changing - another candidate's policy or cancelling accepted work. Initialization - progress and failure recovery appear in Datacenter properties; completed - initialization is not a persistent banner. -- **CLU-07:** Each candidate is one ordinary selectable tree row labelled - `node-`. An Auto icon, accessible label and tooltip express the effective - policy; unloaded policy is Unknown, never an unchecked/disabled substitute. - Full identity, addresses, discovery/admission state, errors and policy appear in - properties. Enable/Disable Auto join is available in its context menu and - properties, backed by the same state. Manual admission, cancellation and - discarding a prepared review appear only when applicable, in that node's menu. - There are no row checkboxes, inline action stacks, raw discovery logs or - general-purpose Update node / Clean up Group 0 controls in the candidate tree. - -Cluster: Datacenter/Rack/Node scope (CLU-01/05): - -```text -+----------------------------+--------------------------------------------------------------+------------------------------+ -| v datacenter * | datacenter | Properties | -| v R-1 | Topology | Datacenter | -| > N-1 | [datacenter] | Name datacenter | -| > N-2 | : | Health observed | -| > N-3 | [R-1] | ... | -| > Candidate nodes | / : \ | | -| | [N-1] [N-2] [N-3] | Initialization progress | -| | ---------------------------------------------------- | (only when relevant) | -| | Capacity [Refresh] | | -| | Capacity / Allocated / Free / Unknown coverage | | -| | Resource Capacity Usage | | -| | R-1 ... ... | | -+----------------------------+--------------------------------------------------------------+------------------------------+ -Nodes / services / opened Storage: live updates; returning to browser -> refresh -``` - -Cluster: candidate selection and effective Auto policy (CLU-06/07): - -```text -+----------------------------+--------------------------------------------------------------+------------------------------+ -| v datacenter | Cluster overview | Properties | -| > R-1 | Topology remains the primary overview. | node-a1b2 | -| v Candidate nodes | | Discovery ID | -| node-a1b2 [Auto] * | Candidate details are in Properties. | Addresses | -| node-c3d4 | No inline candidate cards or operation log. | State Eligible | -| | | Auto join Enabled [v] | -| | | Target Rack Default Rack | -| | | Admission Waiting | -| | | Error None | -+----------------------------+--------------------------------------------------------------+------------------------------+ -``` - -Candidate menu and DiskDB ownership references: - -```text -Right click node-a1b2 Ownership reference Physical reference -+---------------------------+ N-1 N-2 -| Disable Auto join | Services Storage -| Join node... | DDB-1 DG-7 -| Cancel admission | DG-7 Disk-a -| Discard access review | Disk-a -+---------------------------+ \ / - same DG / Disk identity -Joined candidate -> default Rack -> normal Node tree item -Explicitly disabled candidate -> no new automatic admission -``` - -Candidate menu entries depend on admission state. A DiskDB ownership reference -can point to disks on another physical Node. Selecting its DG hides topology -just like selecting that DG below Storage. - -## 6. Properties and cross-links - -- **PROP-01:** Right-side properties describe the exact selected Rack, Node, service, - Store, Group, Replica, DiskGroup, Disk, Zone, or allocation block. -- Show exact ID, type, parent/owner, source, observation time, and relevant - lifecycle/health fields. Missing information is Unknown or Unavailable. -- Properties provide copyable full identifiers and exact byte/unit values. - Abbreviating diagram labels must not destroy the original value. -- **PROP-02:** Resource links navigate by exact identity: service → Node, - Replica → Node, Disk → DiskGroup, Zone → Disk, and DiskGroup → owner/binding. Failed optional - lookups do not erase a successfully loaded primary resource. -- **PROP-03:** Raw JSON may be a diagnostic affordance. It is not the default - inspector for supported structured resources. - -## 7. Modes and embedding - -- **MODE-01:** Standalone mode exposes administrator topology, service, storage, - logical KV, and data operations supported by the backend. -- **MODE-02:** Standard manual OCI nodes expose authenticated admission and - confirmed Group-0 topology/service/storage operations. Explicit single-profile - preview mode uses the same seven-domain UI for observation and supported logical/ - data operations, but rejects physical topology/deployment/storage mutations. - Backend enforcement accompanies hidden/disabled controls in either mode. -- **MODE-03:** Readonly embedding disallows mutations. Every fetch uses the - configured API prefix. Styles remain scoped to the Console; embedding does - not change resource identities or connect to another cluster. -- Embedding keeps `apiPrefix`, `basePath`, `readonly`, `modules`, `initialDomain`, - and `onEvent`. Domain selection has no Swagger or per-node connection mode. - -## 8. Bounded observation - -- **DATA-01:** API pagination is authoritative. Continuations are opaque and - scoped to the query/source/generation. The browser does not invent offsets - or fetch all pages to provide a count, badge, filter, or layout. -- Defaults: Capacity zone pages contain 32 zones. Every other collection declares its page size at its boundary and - exposes bounded continuation instead of an unbounded initial expansion. -- **DATA-02:** Prev/Next replace a window. Type/scope/query changes clear cursor - history. End-of-list is explicit; no exact total is shown unless supplied by - an authoritative bounded API. Selection remains stable across page changes. -- **DATA-03:** Label local filtering as applying to the loaded window. It must - not launch a hidden cluster scan or imply that an empty local result means - no matching resource exists globally. -- **DATA-04:** Cancel or ignore obsolete responses after scope/selection changes. - A late response cannot overwrite the current selection. Polling cannot stack - overlapping requests for the same resource or grow retry work indefinitely. -- Automatic Access deployment checks live DiskIO ownership before provisioning - credentials or catalog state, including when an existing Chunk-KV process is - available. A transient ownership gap keeps the plan waiting and resumes - without another Submit when the prerequisite is confirmed. -- Active views refresh at a bounded cadence; hidden views suspend observation - polling. Accepted deployment work is independent of observation polling. -- **DATA-05:** Preserve partial successes, identify failed sources, and show - observed/scanned/matched counts separately when they differ. Ownership or - generation changes invalidate incompatible continuations and layouts. - -## 9. Creation and mutation dialogs - -- **FORM-01:** Every Create/Deploy dialog opens with usable, nonconflicting - defaults for IDs, listener ports, and available parent/dependency references. - With prerequisites present, default OK succeeds without manual editing. -- Defaults are fetched/revalidated at submission, across all service types on - the relevant host. Include internal listener ranges, such as DiskDB's ports. - A user edit is not overwritten by a late default response. -- **FORM-02:** Missing prerequisites disable the unsupported operation and name - the prerequisite. Do not prefill a plausible value that inevitably fails. - Real device paths are explicit; never infer a physical device to overwrite. -- **FORM-03:** Submission prevents duplicates. Field validation and backend - errors stay in the dialog or center editor with entered values intact. Toasts are supplementary, - not the only evidence of success or failure. -- **FORM-04:** Show the current operation and elapsed/waiting state for a slow - request. A timeout means outcome unknown until reconciled, not automatically - failed. Do not retry a potentially completed creation blindly. -- Successful creation selects or reveals the resource after authoritative - refresh. Partial creation shows exactly what exists and what remains. -- Cluster reset fences new plan/deployment writes, waits at most 30 seconds - for accepted work, then stops registered children before deleting workspaces. - If accepted work remains pending, reset returns a conflict and removes nothing. - Cleanup operations use the reset's exclusive context; another browser cannot - start a deployment during cleanup. -- **FORM-05:** Delete names the exact target and effects. Deleting a service, - removing configuration, and deleting stored data are distinct operations. - Reset lists its affected scope and stops queued deployment work before teardown. - -Shared dialogs and operation feedback (FORM-01–05, ACT-05): - -```text -Parent menu: Create Dataset... Resource menu: Drop table... -+----------------------------------+ +---------------------------------------+ -| Create Dataset [X] | | Drop table [X] | -| Namespace: dataset-ns | | Target: analytics.events | -| Name [.........................] | | Remove this Catalog reference. | -| | | Physical reclamation follows GC. | -| | | | -| Validation / request error | | Exact scope and blockers appear here. | -| [Cancel] [Create] | | [Cancel] [Drop table] | -+----------------------------------+ +---------------------------------------+ - -Dataset: center edits -> [Submit Snapshot] -> upload / prepare / conditional head -Operation feedback: exact target / operation ID / waiting / progress / error -Unknown outcome: reconcile the same operation before retrying -``` - -Forms open only after a menu action. Progress follows the actual operation; -simple creation need not acquire publication stages. Closing a dialog does not -cancel accepted work. - -## 10. Accessibility - -- All controls have semantic roles and unambiguous labels. Keyboard operation, - focus trapping, Escape/Close behavior, and focus restoration work in dialogs. -- Focus/selection/error indicators remain visible on the dark palette. Text - contrast targets WCAG AA; color-coded states include text or a legend. -- Large diagrams retain accessible summaries and selected-object properties. - Errors and asynchronous progress are announced without stealing focus. - -## 11. Verification contract - -- **TEST-01:** Prepare and verify backend prerequisites through APIs before - visual acceptance. Browser E2E uses real services, persisted metadata and file - bytes. Response mocks, interception and HAR replay are forbidden. Pure unit - tests remain separate and cannot satisfy end-to-end acceptance. -- Every acceptance result records: contract/scenario ID, fixture, actions, - visible result, relevant bounded API result, pass/fail, and concrete defect. -- Use role/label/test-id selectors. Prefer a short scripted flow to repeated - full-page snapshots. Capture one useful screenshot per completed scenario or - failure; do not repeatedly inspect unchanged trees. -- Test mutation results from durable API state and resulting UI, not a toast. - Poll readiness/leadership with a deadline; no arbitrary sleeps or hidden retry. -- Reuse the populated cluster for read-only cases. Isolate destructive cases - and avoid resetting the whole environment for each tab. -- Scope evidence honestly: rendering passed, API passed, and end-to-end passed - are distinct. Backend failures remain failures even if the UI handles them. - -## 12. KV - -- **KV-01:** Left tree is Datacenter → Store → Group → Replica. Store creation - belongs to the Datacenter context menu. The center is the bounded data browser; - exact Store/Group IDs, health, leader and replica membership remain in properties. -- Before initialization, explain the dependency on deployed KV servers and offer - Initialize Group 0. The three-node fixture selects all three eligible nodes. -- **KV-02:** Store/group/replica creation, membership operations, and deletion - use the selected scope and valid defaults. Group 0 is visibly identified as - the system group; ordinary groups are not confused with it. -- **KV-03:** Selecting a Group directly opens its data window; there is no - Overview/Data toggle. Membership and node placement remain in properties. - Unknown leader is not rendered as zero. The right inspector does not poll or - render generic internal metrics. -- **KV-04:** Read operations target an explicit Store/Group. Query and Scan - show encoding and exact key/value interpretation. Scan is bounded and paged; - empty values, missing keys, and request failures are distinguishable. -- A selected replica or group may expose supported diagnostics; a generic - physical Cluster diagram does not replace the logical KV workbench. -- **KV-05:** A data window contains at most 20 entries. Previous/Next replace - rows; each Group has an independent cursor when browsing a Store. Clicking - a row shows the full Key and Value. Printable UTF-8 runs remain text; only - undisplayable characters and malformed bytes become uppercase hexadecimal - runs without `0x`, distinguished by warm gold text, a dark background and border. - Display conversion cannot change bytes sent in a mutation or continuation. -- **KV-06:** The KV data browser is read-only: Query performs an exact Get and - Scan reads a bounded window. There are no Put/Delete/demo injection controls. - Topology and membership operations remain in resource context menus and use - management APIs. A uniform menu style does not add data mutation capabilities. - -PaxosKV: Group data inspection (KV-01–06): - -```text -+----------------------------+--------------------------------------------------------------+------------------------------+ -| v datacenter | PaxosKV | Properties | -| v S-0 | Store [S-0 v] Group [G-1 v] | Group G-1 / selected Key | -| > G-0 | Prefix [........................] [Scan] | Store S-0 | -| v G-1 * | Exact key [.....................] [Query] | Leader LR-2 | -| LR-1 Follower | ---------------------------------------------------- | Members LR-1,2,3 | -| LR-2 Leader | Key Value | Key | -| LR-3 Follower | key-a | Value | -| | key-b | Encoding UTF-8 / hex | -| | ... | | -| | [First] [Previous] [Next] | | -+----------------------------+--------------------------------------------------------------+------------------------------+ -``` - -Create Store belongs to Datacenter; Group/Replica management belongs to its -resource menu. Data browsing is read-only; Query/Scan remain center query tools. - -## 13. Service lifecycle - -- **SVC-01:** Each instance has Start when stopped, Stop when running, Restart, - and the supported remove operation. Menus target exact type and instance ID. -- **SVC-02:** Preserve desired configuration independently of PID. A restart - retains endpoints, workspace, and data; refreshed properties show the new PID - and separately observed readiness. Process identity is verified before signal. - Lifecycle acceptance first checks the submitted Start/Restart response, then - observes the new PID and process. An in-flight startup is not a completed - operation and must not trigger a duplicate cleanup restart. -- In the legacy session-owned deployment flow, page unload stops its runner. - Standard accepted node setup runs on the target monitor independently of the - browser, as defined in CLU-03. Cancellation of an old page's - observation or progress request cannot publish a service failure. The next - page reloads the persisted plan and reconciles interrupted deployments; - restoring a cached history page also reloads before resuming its runner. -- **SVC-03:** Startup observes dependencies: Group 0 before dependent metadata - services; valid disks/ownership for storage; journal/data prerequisites before - Chunk-KV bootstrap; Chunk-KV catalog before Access startup. -- Plans waiting for a registered DiskIO restart or authoritative storage readiness - observe dependencies every two seconds, including when Capacity data has not - been loaded. A live PID does not establish disk-group ownership. The ten-second - missing-disk backoff does not apply to an outstanding storage-readiness probe or - a deployment rejected because its storage prerequisite changed. -- Access provisioning durably records catalog initialization and activation - request identities before issuing either mutation. Interrupted deployment - reconciles committed state and resumes the same request. An existing catalog - retains its identity and policy; restart does not reset operator changes. - Missing or conflicting recorded identity fails explicitly. -- Service failures name the failing step and permit a safe retry. Existing - instances are not silently duplicated or treated as healthy merely by presence. -- A registered service whose current process observation is unavailable remains - waiting while the session observes it every two seconds. Missing, expired, - rejected, or failed observations do not prove that the service stopped. The - plan retains its registration and successful siblings without sending another - Start, Restart, or Deploy request. A fresh, identity-matched running observation - reconciles the step to deployed. Endpoint health remains independently reported - and never supplies a missing PID or proves process absence. -- A fresh, identity-matched observation that confirms a registered service is - stopped marks the step failed when no accepted startup is pending. Failed steps - continue read-only observation every two seconds, including when no other - deployment is queued. A fresh, identity-matched running process automatically - reconciles that registered service's progress to deployed. This also reconciles - an interrupted deployment whose owned process is later confirmed running. -- A plan whose selected steps are all deployed continues read-only process - observation every two seconds. A short-lived running process cannot freeze - progress after it exits. Confirmed stopped diagnostics retain the service name - and configured listener overrides; successful siblings and overrides persist. -- Read-only recovery never sends Start, Restart, or Deploy again. Missing or - unverified process evidence does not clear a recorded failure. Explicit Retry - is required to initiate another failed lifecycle mutation; an accepted startup - retains its existing operation and launch budget. Error observation does not - use the missing-Capacity backoff. -- An unavailable service-inventory read preserves successful steps, recorded - failures, and fixed-slot warnings. Unattempted queued steps keep waiting for - current inventory instead of becoming failed lifecycle operations. -- Operational type descriptions remain available even though instance labels - use the compact convention in §4. - -Session-owned Node service plan with unavailable process observation -(relevant progress rows and existing controls): - -```text -+--------------------------------------------------------------------+ -| Node 1 services | -| | -| crowdb-access-server deployed | -| crowdb-chunk-kv deployed | -| crowdb-chunk-db deployed | -| crowdb-disk-db deployed | -| crowdb-disk-io deployed | -| crowdb-paxos-kv queued · waiting | -| Waiting: current process observation is unavailable | -| | -| [Cancel] [Done] | -+--------------------------------------------------------------------+ -``` - -Existing dialog guidance remains unchanged. Done closes progress; observation -continues while this Console page is active. Confirmed stopped with no pending -startup becomes failed and continues observation. A later fresh running process -reconciles progress to deployed automatically without a lifecycle request. The -existing Retry failed services action initiates a needed lifecycle mutation. - -A previously completed plan after a confirmed stopped process: - -```text -+--------------------------------------------------------------------+ -| Node 1 services | -| | -| crowdb-access-server disabled | -| crowdb-chunk-kv disabled | -| crowdb-chunk-db disabled | -| crowdb-disk-db failed | -| crowdb-disk-db (rpc_port=7100): Registered service is stopped | -| crowdb-disk-io disabled | -| crowdb-paxos-kv deployed | -| | -| [Cancel] [Retry failed services] | -+--------------------------------------------------------------------+ -``` - -## 14. DiskGroup and disk management - -- **DISK-01:** The physical tree ends at Disk: Rack → Node → Storage → DiskGroup - → Disk. Zones and allocation bitmaps remain in the center workspace. - DiskGroup creation establishes its authoritative registration, data binding, - and owner, or reports which prerequisite/step prevents completion. -- The dialog does not spin indefinitely when Group 0, a data group, or DiskDB - is unavailable. It reports the actual failure/unknown outcome and reconciles - whether a resource exists before offering retry. DiskGroup creation returns - within eight seconds. An unknown outcome keeps the ID fenced until accepted - work finishes; reopening the same ID/name reconciles registration, binding - and owner instead of allocating another group. Binding can exist before a - DiskDB service is registered; this partial state names the missing owner - prerequisite explicitly. -- **DISK-02:** Node storage separates registered disks, operation progress and - candidates. Create File Disk seals a generated path, capacity, existing group, - reserve and allocation mode. Accepted preparation/admission continues on the - monitor. Pending/failed outcomes do not count as usable capacity. -- Join Candidates selects at most 16 eligible supplied media, each with an - existing compatible bound DiskGroup and media class. File/device backend and - HDD/SSD class are separate. Unverified identity, changed observations, missing - mappings and insufficient capacity disable admission and explain why. - General initialization consent and per-raw-device acknowledgement are explicit. -- Bulk Join submits at most four independent operations concurrently. Each row - retains its operation and exact reviewed medium across response loss/retry; - successful siblings are not resubmitted. Closing stops unsent submissions, - while accepted work continues. Durable history pages contain at most 16 rows; - the Node workspace retains at most 64 observations and reports partial coverage. - Nonzero usable bytes fail without wiping or publishing disk membership. -- Legacy metadata-only disk editors retain geometry and uniqueness checks; they - are separate from verified physical provisioning. -- Standard-node group menus open Create File Disk, while disk lifecycle links - open Node storage. Direct legacy Up/Down/Delete controls are unavailable in - that mode. Maintenance diagnostics remain available. A disk row with no observed - provisioning receipt reports Unknown media rather than inferring a class from - compatibility geometry. -- **DISK-03:** Distinguish physical location, DiskDB owner instance, and metadata - Store/Group binding. Neither Node ID nor service ID implies ownership. -- Owner/status/binding operations are available only when supported and expose - the resulting authoritative state. Container restrictions apply to all entry - points, including context menus and direct API requests. - -- **DISK-05:** Offline applies exclusion to the current owner and every configured - DiskIO participant, then waits for I/O quiescence. Remove requires authoritative - Offline and the exact revision/owner evidence. A lost reply is reconciled with - Group 0 before showing success. Removed media appear as retained candidates. - Rejoin restores their original identity and allocation metadata Offline, then - explicit Online validates the retained format and current descriptor. No - recovery migration, evacuation or physical file deletion is implied. Missing - or replaced registered media remain quarantined; discovery creates no substitute. -- A retained receipt preserves its parent even after Remove. DiskGroup deletion - requires no retained media, owner or metadata binding; empty membership alone - does not permit a cascade. Confirmation explains the blocking prerequisites. - -Storage scope: inventory without topology (DISK-01/02, CLU-05): - -```text -+----------------------------+--------------------------------------------------------------+------------------------------+ -| v datacenter | Storage - N-1 [Refresh] | Properties | -| v R-1 | Capacity / Allocated / Free / Unknown coverage | Storage / selected DG | -| v N-1 | ---------------------------------------------------- | Physical Node N-1 | -| > Services | DiskGroup Owner Binding Status | DiskDB owner DDB-1 | -| v Storage * | DG-7 DDB-1 S-0 / G-1 Online | Binding S-0 / G-1 | -| > DG-7 | ... | Observed | -| > Disk candidates | [Previous] [Next] | | -| | | | -| | No topology canvas or reserved canvas space. | | -+----------------------------+--------------------------------------------------------------+------------------------------+ -``` - -DiskGroup: Disk icon and ID enter the same Disk (NAV-03, DISK-03): - -```text -+----------------------------+--------------------------------------------------------------+------------------------------+ -| v N-1 | DG-7 [Refresh] | Properties | -| v Storage | Capacity / Allocated / Free | DiskGroup DG-7 | -| v DG-7 * | ---------------------------------------------------- | Owner DDB-1 | -| Disk-a | Disks | Binding S-0 / G-1 | -| Disk-b | +------------+ +------------+ | Physical Node N-1 | -| | : Disk icon : : Disk icon : | Status Online | -| | : usage : : usage : | | -| | +------------+ +------------+ | | -| | Disk-a Disk-b | | -| | Online Offline | | -| | | | -| | Click icon or ID -> that Disk workspace | | -+----------------------------+--------------------------------------------------------------+------------------------------+ -``` - -The same view opens through DDB-1 or Storage. Supported storage lifecycle -operations remain in resource menus with the mode restrictions in §14. - -## 15. Capacity - -- **CAP-01:** Datacenter/Rack/Node/Storage/DiskGroup/Disk selections show relevant - capacity, used/free allocation and disks. Observation time belongs with the - affected detail, not in a global strip. Missing statistics are not - zero and must not produce an all-free disk map. -- **CAP-ALLOCATION:** Used/busy capacity denotes allocated physical blocks, including - system chunks, reserved Chunk capacity and blocks awaiting compaction. It is not - user payload size or host filesystem usage. Capacity tooltips and Zone bitmap - accounting explain this distinction; the displayed bitmap window reports its own - used/free count separately from the whole Zone. -- **CAP-02:** Disk detail retains its summary and paged zones. Default 32 zones, - Prev/Next, current visible range, and optional cheap indexed zone lookup. - Avoid full-zone scans for filtering or drawing a large disk. -- **CAP-03:** Selecting a Zone renders its details below the disk view. Keep the - parent disk and zone list in place; no redundant Back to parent Disk link. - If a dedicated zone view is used, provide an actual disk breadcrumb instead. -- **CAP-04:** Zone bitmap uses muted blue for used and muted green for free, - with a legend, used/free counts, allocation unit, and zone geometry. Bit index - maps to the correct block offset according to the protocol's decode order. - Bitmap counts and usage identify their scope, observation and denominator. - Partial windows or aggregation cannot imply that the whole Zone is full. - Comparable allocation counts reconcile with usage; differing scopes or times - are explicit. Allocation, reserved/preallocated space, logical data bytes and - backing-file usage are distinct measurements. Initial system allocations are - attributable to their actual sources, not described as user data by inference. -- Hover/selection provides exact block index and offset. Large bitmaps use - bounded drawing/tiling or labelled aggregation; an aggregate is not one block. -- **CAP-05:** Refresh bitmap, scan/recalc, compact/rebuild, and status operations - identify their scope and show progress/results. RPC or owner lookup failure - remains visible at the affected zone and never becomes an empty/free bitmap. - -- **CAP-06:** Opening Cluster does not issue DiskDB runtime usage or scan - requests. Capacity usage is scoped to the selected DiskGroup/Disk when that - scope is known. Selecting Datacenter, Rack, Node or Storage requests the runtime summary - needed for that scope automatically. DG and Disk requests retain their narrower - scope. Inventory capacity remains available independently; Busy/Free are known - only when exact node/DG/disk identities and geometry cover the selected inventory. - Moving from Disk to an upper scope never reuses the Disk report as a full summary. - Cached candidate/setup summaries are bounded to 64 nodes and never initiate - node-by-node discovery. - Disk inventory belongs to one shared source. Expanding a Node's Storage branch loads its - DiskGroups; expanding/selecting a DiskGroup loads its disks. Selecting a - linked Disk loads the required ancestor inventory directly. Four workers - serve opened branches, merging duplicate requests for the same branch. - Refresh revisits requested branches and never traverses unopened disks. - Failed branch refresh retains its known inventory and reports failure. -- **CAP-07:** The physical tree reads Node health from the service projection; - it does not ping every Node or fetch a second per-node KV catalog. Capacity - Storage branches and DiskGroups begin collapsed; Datacenter/Racks/Nodes and - Services remain visible. - Default-service prerequisite inspection is explicit for its selected Node. - -Disk and inline Zone bitmap (CAP-02–05): - -```text -+----------------------------+--------------------------------------------------------------+------------------------------+ -| v N-1 | Disk-a [Refresh] | Properties | -| v Storage | Capacity / Allocated / Free | Selected allocation block | -| v DG-7 | Zones: current bounded window | Disk Disk-a | -| Disk-a * | Zone Allocated Free Usage | Zone zone-1 | -| Disk-b | zone-0 ... ... ... | Index 17 | -| | zone-1 * ... ... ... | State Free | -| Tree stops at Disk. | [Previous zones] [Next] | Unit | -| | ---------------------------------------------------- | Zone offset | -| | zone-1: block window, not the entire Zone | Disk offset | -| | BBBBGGGGGG?? B=used/blue G=free/green ?=unknown | Observed | -| | BBBBGGGGGG?? Selected block: 17 | | -| | Window counts / Zone totals / unit / geometry | | -| | [Previous blocks] [Next] [Go to block] | | -| | | | -+----------------------------+--------------------------------------------------------------+------------------------------+ -``` - -Letters represent the bitmap colors. Rendering uses actual protocol bit order -and allocation geometry. Unknown/reserved data never becomes free. Free blocks -in another window are not visible in this one. - -## 16. Loading, failures, and recovery - -- **ERR-01:** Distinguish initial loading, refreshing existing data, empty, - partial, unavailable, and failed. Keep usable previous data visibly stale - while refreshing; do not flash an empty page on every poll. -- **ERR-02:** Display a concise cause, failed dependency/operation, and retry - action. Technical endpoint/HTTP details may be expanded for diagnosis. -- **ERR-03:** A long-running or timed-out mutation preserves its target and - inputs. Reconcile its result before retrying. Closing a modal does not prove - server cancellation; UI text must not imply otherwise. -- **ERR-04:** Failure of detail/placement/one child branch does not erase the - parent resource or other successful results. Unauthorized/unsupported actions - are explicit and do not fall back to another protocol or cluster. -- Polling and automatic dependency checks have bounded cadence and fan-out. - Identical failures do not flood notifications; explicit retry remains usable. - -Collection and refresh states (ERR-01–04): - -```text -Initial loading Confirmed empty collection -+--------------------------+ +--------------------------------------------+ -| Loading namespaces... | | No namespaces. | -| No invented zero totals. | | Right-click Catalog to create a namespace. | -+--------------------------+ +--------------------------------------------+ - -Refresh failed Mutation outcome unknown -+--------------------------+ +--------------------------------------------+ -| Last observed rows | | Target: exact resource | -| Stale - refresh failed | | Operation: accepted ID | -| Cause / dependency | | Outcome unknown - reconcile before retry | -| [Retry] | | Inputs / last confirmed progress retained | -+--------------------------+ +--------------------------------------------+ -``` - -## 17. Fixed-cluster operator flow - -1. Open Cluster directly. Inspect/create Rack and Nodes; add each Node through - one dialog with six default services and explicit waiting dependencies. -2. In KV, initialize three-replica Group 0 and create ordinary data groups. -3. In Cluster, select Storage to register DiskGroups/disks with owners and bindings; - DiskIO becomes ready. Enter Disk/Zone to inspect capacity and its bitmap. -4. Select a Datacenter/Rack/Node in Cluster to inspect topology and service readiness. - Resolve logical group dependencies in PaxosKV and physical storage in Cluster. - -This is a dependency guide, not a blocking wizard. A working capability remains -usable while another is unavailable. No domain asks the operator to reconnect. -Data-browser flows are covered by their domain sections below. - -## 18. Acceptance scenarios and scope - -Each scenario uses the contracts above; failures are recorded in the working -plan, never edited out of this specification to make a run pass. - -- **A01 / entry:** open the fixed cluster; all seven tabs appear; no connection - or access-key form; restoring a domain does not change cluster identity. -- **A02 / create:** with one Rack, create three Nodes using valid defaults; - verify one instance of every requested type per Node, conflict-free ports, - accepted operation progress, and safe retry after one injected service failure. - Closing a standard setup dialog does not stop accepted monitor work. -- **A03 / lifecycle:** independently stop/start/restart an auxiliary instance - from its Node menu; verify exact target, preserved data/configuration, updated - PID, and readiness. All instance labels follow §4. -- **A04 / KV:** verify Group 0 and ordinary Group 1 with three replicas; select - a group, inspect membership, and perform bounded exact/scan data operations. - Add members to the same group sequentially, refreshing membership after - each result; independent group preparation may run concurrently. -- **A05 / Capacity:** create/register DiskGroup and disk, inspect owner/binding; - select a disk with more than 32 zones, page, select Zone, and verify inline - bitmap, decode offsets, muted colors, and error handling for unavailable owner. -- **A10 / limits:** large topology, group/replica lists, disk/zone lists, and KV - scans remain bounded. Observe request counts/bytes and rendered item counts; - no hidden fetch-all, invented totals, or selection replacement on page changes. -- **A11 / failures:** inject unavailable owner/group, malformed API response, stale cursor, - and slow/unknown mutation outcome; verify §16 and retry without duplication. -- **A12 / modes:** Single preview rejects physical topology/disk mutations through UI and API; - supported admin data operations remain usable. Readonly embedding forbids - mutations and respects API/style isolation. - -- **A06 / Chunk:** default real list, 10-entry replacement windows, exact ID, - multiple Mirror/EC Strips, selected block properties and Disk/Node round trips. -- **A07 / Chunk-KV:** registered servers including empty owners, bounded graph, - collapse, exact split identity, selected Tree/Journal and stale continuations. -- **A08 / Iceberg:** Catalog to Parquet footer through a real committed table; - nested schema, snapshots, paged manifests/files and zero data-page reads. -- **A09 / S3:** create a bucket using its prefilled name with one click; verify - bucket/object replacement pages, HEAD, bounded preview, single and batch upload - using filenames as object keys, duplicate-key replacement, download/multipart, - locations and return from Chunk to the source object. -- **A13 / resource interaction:** parent context menu creates a supported child; - selecting it reads details; its own context menu exposes supported lifecycle - operations. Create Dataset, Iceberg Table and S3 Bucket with their prefilled - default names without typing (add data for Dataset; upload Parquet for Iceberg), including ordinary HTTP container origins. Verify tree/list/property refresh, keyboard - access, readonly and unknown capability/outcome states across Iceberg, Dataset - and S3. -- **A14 / Dataset:** create/import, parent-linked Snapshots, paged Samples/Fields, - explicit verified preview, Split revisions and supported lifecycle operations. - No browser Session activity or default raw-JSON properties replace this content. - -Out of scope: UI login/role design and new data-plane protocols. -Unsupported features are explicit rather than represented as working controls. - -## 19. Chunk - -- **CHK-01:** Left tree is Datacenter → Rack → Node → CDB and KV Server; - KV branches expand Store → Group without Replicas. Chunk ownership has two - independent maps over slots 0–1023: Serving assigns a ChunkDB instance; - Storage assigns the metadata Store/Group. Chunk-KV ordered splits and payload - disk placement are separate concepts. The left tree has no Serving ownership / - Storage ownership wrapper or repeated footer. Ownership meaning belongs in - selected-resource properties and the scoped center inspection. Details are - requested only when a real ChunkDB or replica target - exists. Unprovisioned selections do not send ownership probes. -- Selecting a CDB or Group displays its full 32×32 bitmap; parent selections - display one combined bitmap per layer. Each cell is exactly one slot, ordered - left-to-right then top-to-bottom. Owner colors remain stable across scopes; - outside-scope slots are gray, unknown membership uses a pattern. Each map - carries its own generation. Invalid or unavailable maps never become empty - or gray observations; retained failed-refresh data is marked stale. -- Storage groups are deduplicated across replicas. Node/Rack scopes include a - group when it has a replica in that scope. Unknown replica/service membership - remains unknown. Legend hover highlights an owner's slots; clicking an owner - navigates to its bitmap. Legend windows contain at most 12 owners. Selecting - a cell shows exact slot, owner, generation, source and node associations; - arrow keys navigate the grid. Service and storage boundaries need not match. -- Bitmap observations read a validated complete map and return exactly 1024 - lossless owner identities per layer. No owner-page stitching is needed. -- **CHK-02:** Entering Chunk requests an All-type page automatically. The center - lists at most 10 chunks without an inner vertical scrollbar. Previous/Next - replace the page. Type is a protocol-defined selector; exact Chunk ID lookup - sits above the list in the center. Arbitrary ID-prefix/global filtering is not - required. Source failures identify partial coverage rather than an empty list. - Datacenter and narrower lists expose Type, Status, Strips, Size and Sealed. - Status uses the observed lifecycle state; Size reports Chunk capacity and Sealed - reports sealed length, both with explicit units matching the detailed metadata. - Sealed length is not a boolean, nor is Chunk capacity physical used space. - Unknown fields are explicit; valid zero lengths/counts remain visible. - List and detail refer to the same Chunk identity. - Aggregation deduplicates authoritative Chunk identities rather than counting - replicas as separate chunks. Window counts are not global totals. System and - user-data types remain distinct; a large initial count alone proves no leak. -- **CHK-03:** Selected Chunk summary and layout appear below the list. Each Strip - has a compact two-line identity: `Strip Mirror` or `Strip EC k+m`, - followed by logical interval such as `[8M, 16M)`. Stable protocol identities - remain distinct from presentation indexes. Capacity, written bytes and physical - usage are separate; missing values are unknown. -- **CHK-04:** Muted selectable block cards form each Strip. Mirror cards use - `Mirror 1`, `Mirror 2`, etc.; EC cards distinguish data and parity. Small cards - show identity/role, while properties carry full Node, DiskGroup, Disk, Zone, - offset and length. No redundant horizontal Strip bar consumes a third row. -- **CHK-05:** Selecting a block updates right properties. Disk and Node links - retain the originating chunk, page/cursor, Strip window, block and scroll. - Repair status reflects actual protection deficit, not lack of rack diversity - alone. Missing placement retains known Disk IDs rather than invented targets. - -Chunk: scope, ownership inspection and list fields (CHK-01/02): - -```text -+----------------------------+--------------------------------------------------------------+------------------------------+ -| v datacenter * | Chunk ownership - datacenter | Properties | -| v R-1 | Serving slots [32x32 map] [Refresh slots] | Selected resource / slot | -| v N-1 | Storage slots [32x32 map] [Refresh slots] | Full identity | -| CDB-1 | ---------------------------------------------------- | Owner / binding | -| v PKV-1 | Chunks Type [All v] [Refresh chunks] | Generation | -| v S-0 | Exact Chunk ID [...................] [Lookup] | Observation / coverage | -| G-1 | ID Type Status Strips Size(KiB) Sealed(KiB) | ... | -| | | | -| No ownership footers. | ... WAL Active ... ... ... | | -| | ... S3 Sealed ... ... ... | | -| | Coverage / scanned count [Previous] [Next] | | -+----------------------------+--------------------------------------------------------------+------------------------------+ -``` - -Chunk: selected Strip and block placement (CHK-03–05): - -```text -+----------------------------+--------------------------------------------------------------+------------------------------+ -| v datacenter | Chunk list stays above its selected detail. | Properties | -| v R-1 | ---------------------------------------------------- | Mirror 1 | -| v N-1 | Chunk / type / state / capacity | Node N-1 | -| CDB-1 | Strip 4 Mirror | DiskGroup DG-7 | -| | [0 MiB, 16 MiB) | Disk Disk-a [link] | -| | [Mirror 1 *] [Mirror 2] [Mirror 3] | Zone zone-1 | -| | Strip 5 EC 4+2 | Offset | -| | [16 MiB, 32 MiB) | Length | -| | [Data 1] [Data 2] [Data 3] [Data 4] [P1] [P2] | Protection | -| | | | -| | Select block -> Properties | | -| | Disk link -> Cluster/Disk; Back restores this page | | -+----------------------------+--------------------------------------------------------------+------------------------------+ -``` - -## 20. Chunk-KV - -- **CKV-01:** Left shared Tree is Datacenter → Rack → Node → CKV Server → Split. - No extra sidebar title duplicates the root. Registered servers stay visible - even with no assigned splits. Compact split labels retain full ID and bounds - in hover/properties; five rendered splits are a window, not a global count. -- **CKV-02:** The center is a node-link diagram, not a second indented sidebar: - Chunk-KV root → CKV Server → Split → actual Tree ID. Its visible window is - at most eight servers and five splits per server. Server cards collapse their - children. Graph paging is separate from authoritative catalog pagination. - Do not add a loaded-ID filter or an ambiguous All loaded servers control. -- **CKV-03:** Selecting a Split opens its scoped Tree/Journal information below - the graph. Properties retain complete identities, bounds, epoch, owner/source, - checkpoint and selected journal extent. Root/child KV Pages must come from a - real bounded page-inspection API; journal fences are not KV Pages. Byte keys - default to hex with an optional validated text interpretation. -- Returning from Chunk inspection restores the selected Split, Tree/Journal tab, - catalog generation/window, graph server/split windows, collapsed branches and - selected journal extent. History stores identities and cursors, not runtime - payloads. Return revalidates the catalog and stream generations; stale state - stays explicit until Refresh catalog/runtime starts a new observation. -- **CKV-04:** Tree and journal observations have bounded replacement pages. - Continuation pins catalog/stream generation. Stale generation requires refresh - from the first page; parent/child recovery dependencies retain separate stream - offsets. Placement and split lineage are separate fields. -- **CKV-05:** Page inspection follows actual inner-child indices from the root, - with at most 32 child steps, one 1 MiB base frame and 20 entries per window. - It may load cold structural pages, but never flushes, checkpoints, scans live - KV records or follows overflow values. Physical base records, inline deltas - and the count of external delta pages remain explicit; pending writes are not - presented as part of this base-page view. Keys and inline values have bounded - 256-byte previews with exact lengths and truncation markers. Version fences - protect paths; page fingerprints additionally protect entry continuations. - Changed observations require an explicit root refresh. History retains path, - fences, offset, selected entry and byte format, then revalidates on return. - -- **CKV-06:** Split properties, Overview and graph cards do not display Weight. - Full Range bounds remain visible and wrap independently. -- **CKV-07:** Weighted balance explanations and data/count contribution values - remain hidden until tree statistics can describe current range data reliably. - No tree statistics scan is triggered by opening or refreshing the console. -- **CKV-08:** Count-based placement remains a backend policy. Existing bounded - observations retain catalog/epoch freshness fences, but are not presented as - data weights. Browser assertions verify that Weight and weighted explanations - are absent while tree/page inspection and range navigation remain functional. - -ChunkKV: distribution graph and selected Split (CKV-01–08): - -```text -+----------------------------+--------------------------------------------------------------+------------------------------+ -| v datacenter | ChunkKV range distribution [Refresh catalog] | Properties | -| v R-1 | Catalog generation / page / loaded scope | Split | -| v N-1 | [ChunkKV] | Range [start,end) | -| v CKV-1 | / \ | Owner CKV-1 | -| split-a * | [CKV-1] [CKV-2] | Epoch | -| v N-2 | : : | Tree ID | -| CKV-2 | [split-a] [split-b] | Checkpoint | -| | : : | Selected extent / page | -| | [Tree ID] [Tree ID] | No Weight field | -| | Graph window controls [Fit All] | | -| | ---------------------------------------------------- | | -| | Selected split-a: [Tree] [Journal] | | -| | Bounded page / journal extent content | | -+----------------------------+--------------------------------------------------------------+------------------------------+ -``` - -Graph windows and catalog pages have separate continuation. Selecting a Tree -uses real bounded base-page inspection; the graph does not scan records for weights. - -## 21. Iceberg - -- **ICE-01:** The configured cluster Catalog loads automatically. Left shared - Tree is Catalog → Namespace → Table → Snapshot. Manifest and File are center - inspection data, not tree levels. Namespace and Table types are labelled - explicitly. CrowDB supports nested Namespace - components as represented by the [Iceberg REST Catalog specification](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/open-api/rest-catalog-open-api.yaml). No Metadata leaf or Snapshots wrapper - duplicates content; Manifest List is a center inspection level. Current Snapshot is marked - `Current`; immutable IDs remain exact. -- **ICE-02:** Catalog/Namespace centers list their children. Table center shows - a compact overview followed directly by a Snapshot table: time, operation, - records and files. Overview/Schema/Files controls belong only to selected - Table, disappear for other resources, and have no independent Snapshots tab. - Catalog root uses one Namespaces heading; Namespace uses its actual name. - No Catalog breadcrumb/type/title stack duplicates the left tree. Namespace - and Table lists page in the center with visible continuation controls. -- **ICE-03:** Schema is an expanded tree table with field ID/name/type/required - and hierarchy for structs, lists and maps. Element/key/value identities stay - visible. Large schemas use bounded row windows; collapse is optional, not a - prerequisite to seeing ordinary fields. Create Table and Snapshot Drafts show - this Schema as a separate read-only preview. Field names, types, required flags - and nested definitions come from the first uploaded Parquet for a new Table, - or the current Table Schema for an existing Table. These workflows have no - Add/Remove Field or field-definition inputs and do not publish Schema edits. -- **ICE-04:** Selecting a Snapshot keeps the left tree at the Snapshot level and - shows one scoped summary plus bounded center inspection levels: - `Manifest List → Manifest File list → Data/Delete File list`. These are center - tables/sections, never tree nodes. Expanding one Manifest File requests its - Data/Delete File subtable. Manifest List, Manifest File and Data/Delete File - pagination are independent and lazy; no full-snapshot fetch to calculate a - total. The right properties retain Manifest List path and known size. - File-name columns use a bounded width, left alignment and ellipsis for long - names. The full path/name remains available in the selected object's - Properties and in the filename tooltip. Tree levels are contiguous: Catalog, - Namespace, Table and Snapshot have no empty spacer row or duplicate text-only - leaf between them. - Center drill-down keeps the left tree selected on the Snapshot. Back returns - to the previous center scope with its bounded page, cursor and selection; - browser Forward/Back uses the same bounded center history contract. -- **ICE-05:** Manifest/file inspectors render decoded fields. Parquet shows - footer information and sized Row Group/column blocks. Selecting a column - shows complete metadata in properties. Inspection reads footer/metadata only, - not data pages. Unsupported file types show an explicit capability state. -- **ICE-06:** Catalog context menu creates Namespace; Namespace context menu - creates Table and, when supported, child Namespace. A Namespace's own menu - creates one literal child name component; a dot in the entered name does not - create another level. Full identity is the ordered namespace component array. - Its menu also edits supported properties or drops that empty Namespace. A Table's menu - exposes supported Edit as Draft, Rename, Commit metadata and Drop table operations. A new - Namespace form is prefilled with an editable unique name such as - `namespace-`. A new Table form is prefilled with an editable unique name such as `table-`, - so creation needs no manual name entry; a new Table requires a Parquet upload. Drop - removes the catalog reference; physical reclamation follows service GC. - Snapshot and Parquet menus expose table-managed maintenance as defined in ICE-10. - Manifest and metadata files have no arbitrary physical Delete action. Demo operations are not primary - browsing controls. Properties describe only the selected identity. -- **ICE-07:** Reference windows contain at most 100 entries; Parquet windows - contain 20 Row Groups and 12 columns. Paging replaces the visible window. - Column widths represent compressed sizes and file spans retain exact byte - identities. History restores the selected reference, footer/layout view and - column selection, then revalidates the table head, snapshot and file identity. - Cached branches do not bypass revalidation. A failed or stale inspection - preserves selection context but removes data and continuation actions. - -- **ICE-08 — Parquet import:** Namespace Create Table and Table/Snapshot - Edit as Draft open the same center editor. A new Table starts with a name and - Upload Parquet action, without a fabricated default Schema. The first uploaded - file supplies the Schema; a separate read-only tree table previews it once. - A file list below it contains file name, record count, exact byte size, Status and - removal actions. It has no Schema columns or repeated type cells. - One Status column replaces Change and Upload: Uploaded for additions, Kept for - existing files, Removed for pending removal, and Reselect to resume for unfinished - uploads. Remove and Undo removal remain row actions. - File selection populates Properties; nested Schema details remain in the preview. - Further files are checked against the inferred Schema; an incompatible file - reports its name and the mismatched field before upload. Existing Tables keep - their current Schema and durable field IDs, including when editing history. - No silent union, coercion or Parquet rewriting occurs. Create Table requires - at least one uploaded file; a zero-record Parquet is valid. Removing every added - file from a new Table resets its inferred Schema, ready for a different upload. - Commit is disabled until there is a file change and, for a new Table, an upload. - Existing Tables may remove every data file while retaining their Schema. - The writer supports unpartitioned format-v2 Tables and standard three-level List/Map - encodings; source Snapshots may retain an older compatible Schema. Incompatible - historical Schemas require an Iceberg client; metadata-only evolution must not - block subsequent file edits of the current Snapshot. - Authoring permits 1000 added files, 256 top-level fields, 20 files per selection - and 100 MiB per file. Existing file windows load one Manifest page at a time, - with explicit continuation; Schema previews use bounded row windows. No total-file count - or full Snapshot read is required to open the editor. -- **ICE-09 — Draft and Commit:** choosing a file reads its bounded footer and - checksum, then immediately uploads it through authorized FileIO. The proxy rejects - foreign prefixes and arbitrary credential endpoints. Small native uploads retain - their returned immutable location; larger files use 8 MiB multipart parts. - Save Draft and autosave persist metadata and upload references only, under the API - origin, Namespace, Table and requested source scope. File/Blob bytes and credentials - are never persisted. Completed uploads survive reload without reselecting files; - interrupted uploads require the same bytes. Invalid references are reported, and - expired uncommitted uploads can be replaced under a new identity. Local revisions - reject stale saves/discards. A local Draft is not a server-side editable Table. - Stage-create supplies canonical UUID/Schema/storage material without publication. - Saved Drafts retain their inferred Schema and upload references. Legacy Drafts - with manual Schema edits must be discarded explicitly before starting this flow; - those edits must never be published silently. Frozen pending submissions retain - their exact request and recovery semantics. - Commit freezes the input and publishes generated Manifests and metadata atomically; - file additions/removals produce one immutable data Snapshot. Property-only updates - do not invent data Snapshots. Remove and replace change - the resulting file set; they do not modify the original Snapshot or file bytes. - The chosen source Snapshot and expected current main are separate fixed identities. - Editing history publishes that source's edited file set as a new child of expected - main, with a visible explanation that newer files are excluded. Table UUID, Schema, - default spec, main and the advertised metadata-location guard protect publication. - Concurrent changes conflict; the editor never silently rebases. Manifest rewriting - preserves original data/file sequence values and rejects unpreservable layouts. - Pending requests retain exact body/key and immutable output paths. A lost response - is reconciled by Table UUID plus Snapshot/Manifest List identity before retrying; - no second Snapshot is created. Unknown outcomes cannot be discarded as failed. - Property editing remains a separate name/value/removal form with metadata-only - publication. Advanced updates retain exact integer tokens and UUID conditions. -- **ICE-10 — Removal and expiry:** a live current Parquet file offers - `Remove from table…`, which opens the same Draft editor with that file marked - Removed. Undo restores it before Commit. Old Snapshots remain readable while - retained. Unaffected Manifests, including delete Manifests, are preserved; the - inspector's rewrite capability prevents losing metrics/encryption/other attributes. - Snapshot `Expire snapshot…` is a separate lifecycle command. Current and Branch/Tag - references prevent expiry; the advertised guarded commit checks metadata location - and UUID under head CAS, including refs created or moved after confirmation. - Ordinary Iceberg metadata updates retain standard semantics. Existing pending - maintenance requests can still resume their frozen body/key after reload. - Removing a file reference or expiring a Snapshot does not physically delete the - file. A live Table's published files are not reclaimed solely because no retained - Snapshot refers to them; physical cleanup follows the Table/file lifecycle. - -Iceberg: editing and publication (ICE-08/09): - -```text -Namespace -> Create Table Table / Snapshot -> Edit as Draft -+--------------------------+------------------------------------------------+----------------------+ -| v Catalog | Create Table / events · New Snapshot | Properties | -| v analytics | Table name [table-abc123] (new Table only) | selected file | -| v events | [Upload Parquet] | path, size, records | -| snapshot-a | Schema | | -| | Field ID Type Required | | -| | id 1 long No | | -| | message 2 string No | | -| | Files | | -| snapshot-b Current | File Records Size Status | | -| | a.parquet 100 4 KiB Kept Remove | | -| | b.parquet 100 4 KiB Removed Undo | | -| | c.parquet 100 4 KiB Uploaded Remove | | -| | [Previous files] [Next files] | | -| | [Close] [Discard Draft] [Save Draft] [Commit] | | -+--------------------------+------------------------------------------------+----------------------+ -New Table: upload first Parquet -> infer Schema -> read-only preview -> Commit -Existing Table: retain Schema -> check uploads / remove files -> Commit -Before first upload: "Upload Parquet to create a Table."; Commit disabled -Save Draft -> close/reload -> reuse uploaded files and inferred Schema -Commit -> frozen source + add/remove -> new Manifest List -> conditional main - existing Snapshot and its referenced bytes remain immutable -``` - -Iceberg: Snapshot file inspection (ICE-04): - -```text -Left tree Center panel -v Catalog Snapshot snapshot-a - v analytics [Back] Manifest List - v events ml-001 2 manifests [Open] - snapshot-a * └─> Manifest Files - mf-001 added [Open] - mf-002 delete [Open] - └─> Data/Delete Files - part-001.parquet [Properties] - delete-001.pos [Properties] - -Open / drill -> next bounded center level -Back -> previous level; left tree remains on snapshot-a -``` - -Iceberg: lifecycle (ICE-10): - -```text -Current S2: [a.parquet, b.parquet, c.parquet] - b.parquet -> Remove from table -> Draft [b: Removed] -> Commit S3 - S3: [a.parquet, c.parquet] S2 still reads all three files - -S2 -> Expire snapshot -> current / Branch / Tag check on fixed metadata - changed base -> conflict unchanged base -> remove S2 - neither reference removal nor Snapshot expiry physically deletes files -``` - -Iceberg: Catalog and Namespace collections (ICE-01/02/06): - -```text -+----------------------------+--------------------------------------------------------------+------------------------------+ -| v Catalog | analytics | Properties | -| v analytics * | [Refresh] | Namespace analytics | -| > events | ---------------------------------------------------- | Full name | -| > users | Name Type Summary | Properties | -| > sandbox | events Table ... | ... | -| | users Table ... | | -| | ... | No duplicate Catalog | -| | [Previous] [Next] | path/type/title stack. | -| | | | -| | Catalog selected -> title: Namespaces | | -| | Namespace selected -> title: actual name | | -+----------------------------+--------------------------------------------------------------+------------------------------+ -``` - -Iceberg context-menu targets: - -```text -Catalog Namespace Table -+----------------------+ +------------------------+ +------------------------+ -| Create namespace... | | Create table... | | Rename... | -+----------------------+ | Create namespace... (*)| | Commit metadata... | - | Edit properties... | | Drop table... | - | Drop namespace... | +------------------------+ - +------------------------+ -(*) Nested Namespace creation requires backend support. -``` - -Iceberg: Table, Snapshot and File inspection (ICE-02–05): - -```text -+----------------------------+--------------------------------------------------------------+------------------------------+ -| v Catalog | events [Refresh] | Properties | -| v analytics | [Overview] [Schema] [Files] | Selected Table / reference | -| v events * | Compact table summary | Full identifier | -| v snapshot-42 | Snapshot Time Operation Records Files | Schema / Snapshot identity | -| v manifest-a | 42 ... append ... ... | Manifest List path | -| file.parquet | ... | File format / size | -| | [Previous] [Next] | Selected column metadata | -| | ---------------------------------------------------- | | -| | Select Snapshot -> paged Manifests | | -| | Expand Manifest -> paged Files | | -| | Select Parquet -> footer, Row Groups, columns | | -+----------------------------+--------------------------------------------------------------+------------------------------+ -``` - -Schema uses a structured field tree table. File inspection decodes metadata -and footer structure; it does not imply reading data pages or SQL querying. - -## 22. S3 - -- **S3-01:** Left Tree is S3 → Bucket. A single namespace introduces no wrapper; - the logical S3 root is not named Datacenter. Root center lists buckets with - 20-row server windows. It directly presents the Prefix query and list without - repeated path/type/heading or ordinary selection/refresh success messages. -- **S3-02:** Selecting a Bucket displays an API page of at most 20 objects. - Bucket and Object lists share `Prefix [input] [List]`. Enter/List applies a - server-side name/key prefix and resets cursor history; Previous/Next replace - rows using the applied prefix, independent of unsubmitted input. Bucket and - per-Bucket Object query windows remain separate. Tree selection and history - restore the relevant scope; loaded-page substring filtering is not Prefix. -- **S3-03:** Selecting an Object reads HEAD fields into properties: key, size, - ETag, last modified, content type, supplied version and user metadata. Missing - fields show `—`. The center presents the selected object's content inspection. - Tree selection and Back/Forward restore the Bucket list and its cursor/prefix. - Preview is a separate bounded operation, not an automatic payload GET. -- **S3-04:** S3 root context menu creates Bucket; the create form is prefilled - with an editable DNS-safe unique name such as `bucket-`, so the - default Create action completes without manual typing. A Bucket's menu uploads - one or more child Objects; batch upload uses each selected file's filename as - its object key, with duplicate keys replacing the existing object. A Bucket's - menu also opens multipart uploads or deletes that empty Bucket. An Object's own - menu exposes Preview, Download, Replace and Delete. Object rows in the center - use the same menu contract without adding every Object to the left tree. - Prefix is a key filter, not a physical folder; Bucket deletion does not silently - cascade. Demo controls are not primary browsing actions. Exact bucket/key is visible. - A successful Object deletion immediately removes that exact Bucket/key from - the current object list before its server refresh completes; other rows remain. - Failed or unknown deletion retains the row. A later refresh failure preserves - the confirmed removal and reports the refresh error. - Multipart state distinguishes uploaded parts, completion, abort, failure and - unknown outcome. Downloads stream; previews read at most the advertised cap. -- **S3-05:** Object details include Storage locations below HEAD, 20 extents per - page: stable index, logical `[start, end)`, Chunk ID, chunk `[offset, end)`, and - physical length. Properties show exact decimal bytes and full identity. - Chunk links preserve object and extent cursors, selected extent and scroll. -- Locations are an admin metadata query, never inferred from HEAD or obtained - by reading object payloads. Empty/missing/corrupt/oversized references have - distinct states. Continuation binds object and metadata generation; overwrite - reports stale and requires first-page refresh. Exact 64-bit offsets survive - JSON/JavaScript without rounding. Bounded decoding and response limits are - mandatory; no recursive placement fetch for every extent. -- **S3-06:** `GET /api/access/s3-inspect/locations` accepts exact bucket/key, - a 1–100 limit (default 20), and an opaque cursor. The Console targets only - the configured Access origin and supplies its server-held management token; - browser credentials and arbitrary endpoint parameters are rejected or ignored. - Access authorizes management privilege before metadata reads and resolves the - configured tenant's bucket/object itself. There is no object-payload client in - the inspection path. -- **S3-07:** Inspection decodes at most 4 MiB of stored references and returns - at most 1 MiB. Access has a five-second metadata deadline; the Console proxy - has a six-second request deadline. Reference-limit errors are 413, corrupt - references 422, missing objects 404, and changed cursor generations 409. - Invalid, expired or oversized cursors and limits are rejected. Cursor expiry - is 15 minutes and its authenticated scope includes bucket identity, exact key, - revision-sensitive generation and next extent index. Identical overwrites also - invalidate old cursors. -- **S3-08:** Location failures preserve HEAD. Empty objects show `No storage - extents`; null Chunk identities show `Location unavailable` without a link. - Previous/Next retain at most 32 cursor positions and replace the page; at most - four bounded location pages are cached. A stale result keeps the previous - page labelled stale and disables continuation until first-page refresh. - Selection is captured before following a Chunk link; return refetches metadata - against the saved generation and restores the exact selected extent. - -S3: Bucket object list (S3-01/02/04): - -```text -+----------------------------+--------------------------------------------------------------+------------------------------+ -| v S3 | photos | Properties | -| photos * | Prefix [...................] [List] | Bucket photos | -| exports | ---------------------------------------------------- | Name | -| logs | Key Size Modified | ... | -| | cat.jpg ... ... | | -| | dog.jpg ... ... | Selecting an Object reads | -| | ... | its HEAD and properties. | -| | [Previous] [Next] | | -| | | | -| | Objects stay in this bounded list. | | -| | Right-click Object -> its own resource menu. | | -| | Deleted Object disappears on success, then list refreshes. | | -+----------------------------+--------------------------------------------------------------+------------------------------+ -``` - -S3 context-menu targets: - -```text -S3 root Bucket Object list row / detail -+-------------------+ +-------------------------+ +------------------------+ -| Create bucket... | | Upload files... | | Preview | -+-------------------+ | Multipart uploads | | Download | - | Delete empty bucket... | | Replace... | - +-------------------------+ | Delete object... | - +------------------------+ -``` - -S3: Object inspection and multipart view (S3-03–08): - -```text -+----------------------------+--------------------------------------------------------------+------------------------------+ -| v S3 | cat.jpg [Refresh] | Properties | -| photos | Content inspection | Object | -| exports | Preview is read only after an explicit request. | Bucket photos | -| | ---------------------------------------------------- | Key cat.jpg | -| | Storage locations | Size | -| | Extent Object interval Chunk ID (link) | ETag | -| | ... [start,end) ... | Content type | -| | [Previous] [Next] | Modified | -| | | User metadata | -| | Bucket menu -> Multipart uploads view | Selected extent identity | -| | Upload key / ID / state; select -> paged parts | | -| | Upload menu -> Abort (when supported) | | -+----------------------------+--------------------------------------------------------------+------------------------------+ -``` - -Preview and Download are explicit reads from the resource menu. Selecting an -Object does not download its payload. Multipart inspection is a separate center -view; its state is not piled beneath the ordinary Object list. - -## 23. Actions and properties - -- **ACT-01:** The left tree owns resource navigation and context menus. Create - child belongs to the parent menu; Delete belongs to the resource's own menu. - Selection is Read; child collections are automatic paged lists in the center. - There is no generic Update requirement. Additional operations use precise - resource-specific names and exist only where that node's capabilities allow. - A lifecycle operation such as Reclaim is not relabelled Delete. -- **ACT-02:** All domains share menu, dialog, button and property styles. The - center has one scope heading, query/filter controls, Refresh and content, with - pagination below its list. It has no duplicate CRUD toolbar, collapsed Actions - strip, inline creation forms or repeated path/type/title stack. Query controls - and bounded Preview/Download are distinct from resource mutations. - Ordinary buttons have visible borders; destructive menu items use muted red. - Readonly, deployment mode, backend capability and lifecycle restrictions apply - to every entry point. Unknown capability does not enable writes. -- **ACT-03:** Right properties show complete information for the clicked item, - including graph blocks/columns/extents. Optional diagnostics never replace - primary details. Metrics panels, entries and polling are excluded from the UI. Future metrics - publishing targets customer-managed time-series systems; it is outside this - UI contract. -- **ACT-04:** Tree labels, icons and ordinary resource rows activate the same - identity. Expansion chevrons only expand. List-only resources use the same - context menu on their center row; a focused detail retains access to that exact - resource menu. Right-click targets the clicked resource, independent of prior - selection. Menu key / Shift+F10 opens the same menu; Escape restores focus. - Matrix cells and diagram blocks retain their documented inspection semantics. -- **ACT-05:** Simple mutations use shared dialogs; Dataset and Iceberg authoring - opens center editors with one Submit action and local recovery. The target - stays explicit and cannot change under an open form. - Switching resources closes obsolete forms without claiming to cancel accepted - work. Successful operations refresh affected lists, tree references and properties; - failures retain input and unknown outcomes require reconciliation. -- **ACT-06:** Empty collections explain how to create a supported child from - the parent menu. Empty, loading, unavailable and denied states are distinct. - Browser Session activity is absent from domain workspaces. Operation feedback - stays with its flow; resource history such as Snapshots and publication receipts - remains available. Default properties are labelled structured fields, not JSON. - -The common layout and menu targets are: - -```text -+----------------------------+--------------------------------------------------------------+------------------------------+ -| Resource tree | Current scope | Properties | -| v Parent | Filter [.............] [Refresh] | Selected identity | -| Child | ------------------------------------- | Full path / ID | -| | Child name Type / summary | Structured fields | -| | ... | ... | -| | [Previous] [Next] | | -+----------------------------+--------------------------------------------------------------+------------------------------+ -``` - -Domain-specific scope and menu mappings: - -```text -Iceberg Center on selection Context menu -Catalog Namespaces (paged) Create namespace - Namespace Tables/children (paged) Create table/namespace; Properties; Drop - Table Overview/Schema/Files Edit as Draft; Rename; Metadata; Drop - Snapshot Paged metadata/files Edit as Draft; Expire when unreferenced - Manifest -> File Decoded metadata Current Parquet: Remove from table - -Dataset Center on selection Context menu -Datasets Datasets (paged) Create dataset - Dataset Snapshots (paged) Import / New Snapshot; Delete empty - Snapshot Samples x Fields Create child; Edit Split revision; - Stable/Retain/Release/Reclaim - Splits Revision/member windows Create Split; Edit revision - Revision Fixed Split collection Edit as a new revision - Split Members (paged) Edit / Remove in a new revision - Sample/Field/Cell in center Selected scope/value Edit in child Snapshot; Preview - -S3 Center on selection Context menu -S3 Buckets (paged) Create bucket - Bucket Objects (paged) Upload; Multipart; Delete empty - Object in center Content inspection Preview; Download; Replace; Delete -``` - -These mappings do not imply every listed action is available for every state. -Immutable references and backend-owned internal structures do not acquire generic -CRUD merely to match another domain. Dataset lifecycle support is defined in §25. - -## 24. E2E organization and timing - -- **TEST-02:** Page-function specs retain each behavior once: shell `0x`, - physical lifecycle `1x`, logical KV `2x`, KV data `3x`, inspector/canvas `4x`, - Capacity/Chunk/Chunk-KV `5x`, Iceberg `6x`, S3 `7x`, Dataset `8x`, cross-function `9x`. - Dedicated creation/reconfiguration specs retain their UI mutations; a smoke - chain does not repeat every dialog or failure permutation. -- **TEST-03:** Use separate unit, backend integration and native browser E2E - layers. Browser fixtures provision actual services, metadata and files; - outages use actual service lifecycle. Deployment and container capability - enforcement require real API acceptance. Share setup per spec and keep - destructive cases last. Reset only for cases that require empty authority. - S3, Iceberg and Dataset basic create/add/list/read/delete flows share one - owned cluster and one startup/teardown under the managed resource owner. - Distinct resource names and protocol scopes isolate their effects. API reads - verify results; they do not replace the UI mutations under test. Dedicated - suites retain pagination, large transfers, concurrency and recovery coverage. -- **TEST-04:** Keep the step timer and slow-test reporter. Measure setup, mutation - response, lifecycle readiness, DOM refresh and teardown separately. Slow steps - at 2 s and very slow steps at 5 s remain logged even when a step fails. Tests - at 10/30 s retain slow/very-slow reports. Compare per-test baselines; investigate - more than 2× rather than hiding time with retries or larger deadlines. -- **TEST-05:** One worker owns the mutable runtime; installed system browser, - no automatic browser installation. Auxiliary-service dependency convergence - may wait up to 10 s for Chunk-KV range activation and matching catalog/grant - authority. After 3 s it records the latest readiness response; it also records - the final response and elapsed time. This prerequisite does not change UI - response or process-observation assertion budgets. - Default assertion deadline is 3 s and - election 10 s, polling every 100 ms. No fixed sleeps. Native large multipart - and SF=1 loader acceptance run separately from ordinary UI iteration; normal - smoke data remain small while preserving multi-Strip and pagination cases. - - -## 25. Dataset - -Depends on: [Dataset control lifecycle](../access-server/dataset/design-dataset-control.md). - -- **DS-01 — Scope:** Dataset is immediately before S3. The fixed `dataset-ns` - namespace is a property, not a single-child navigation level. Catalog discovery - comes from the configured authority, including SDK-created identities. Empty, - missing configuration, unavailable service and denied access remain distinct. - Credentials stay on the Console server; no browser endpoint or bearer editor. - The root center automatically lists Datasets. Root context menu creates a - Dataset in the center table editor, with an editable unique name such as - `dataset-`. Adding data and submitting Create Dataset publishes its - first Snapshot in the same flow. There is no name-only creation dialog or - second creation step. A Dataset menu offers New empty Snapshot; a Snapshot - menu offers Edit Snapshot to open a child Draft. All use §23; there is - no Session activity region. The left panel begins with the Dataset title and - shared Help link above the tree. Properties render named fields rather than JSON. -- **DS-02 — Relationships:** left navigation follows declared parent links as a - forest. Roots and each parent's children page independently; no ancestry scan - or Snapshot-ID sorting reconstructs the tree. Latest and Stable are reference - badges. A Snapshot's Splits group is a peer of its children. Samples and Fields - remain in the center. Direct Snapshot navigation preserves the exact identity. - No separate Selected ancestry panel duplicates the tree. Split hierarchy is - Snapshot → Splits → immutable collection Revision → named Split. Current marks - the active collection revision; history pages lazily under Splits. Same-name - Splits in different revisions have distinct identities. Lazy nested branches - inherit depth, indentation and accessibility levels from the shared Tree. - Dataset and Snapshot inventory pagination is reachable in the center. The fixed - Datasets root retains its first bounded page plus the selected resource; the - catalog has no Previous/Next tree-page toolbar. Snapshot branch continuation - stays lazy and independent. Tree appearance and selection use shared conventions. - Empty child collections do not render diagnostic/footer text such as `No Split - revisions` or `No children` in the tree. If an empty state is useful, it is - shown in the selected resource's center panel; the tree keeps only real nodes, - continuation controls, and actionable errors. -- **DS-03 — Combined inspection:** one Samples matrix uses Schema definitions as - columns, defaulting to twenty rows and twelve fields. Required/optional is the - actual field definition; no inferred type, MIME or nested schema. Header, Sample - row and Sample/Field cell selections have separate properties and history. - Clicking a row highlights it; Properties pages through all of that row's Fields - independently of the center columns. - Missing, tombstoned, inline, payload-backed and inherited values are distinct. - Known lengths and declared checksums are metadata, not verified payloads. -- **DS-04 — Bounds:** each catalog/row/column query carries its immutable scope and - explicit continuation. A stale continuation requires first-page refresh. Loaded - row filtering is labelled local. Opaque binary Sample IDs retain canonical padded - base64; valid UTF-8 is only a display/helper encoding. Decimal u64 values remain - strings. Navigation retains bounded identities/cursors and scroll, not payloads. - Hidden work stops; late responses cannot replace another scope. -- **DS-05 — Preview:** only the explicit Preview action reads a selected field. - Complete verified values of at most 64 KiB render as UTF-8 or hex. Unknown length, - missing/tombstoned values, oversized data, integrity failures and cancelled reads - never appear as a successful truncated value. Browsing starts no Read Plan. -- **DS-06 — Table editor:** Create Dataset and Edit Snapshot use one center table. - One row is a Sample; one column is a Field; Sample ID is a separate editable - row key. New input starts with optional `content`. The sole top toolbar action - is Upload files. With one Field it opens the picker directly; with multiple - Fields it first asks for an explicit target in a temporary chooser. There is - no persistent Upload into selector. `+ Sample` at the table bottom generates - an editable ID; Upload files adds rows with the file's exact filename as the - default Sample ID into the selected Field, without a type or `string` prefix. - Sample ID and `+ Sample` remain visible while scrolling across Fields. - Duplicate IDs are rejected. Existing values are replaced explicitly through their - cells, preserving other pending Field changes. Empty cells show Upload file or - a text input directly. Choosing Upload opens the native picker immediately; text - is edited inline, including newly added columns, without a Text/File dialog or - Apply step. Inherited inline values load on focus, not for every visible cell. - Clear writes a tombstone and Remove changes membership. `+ Field` at the - right edge of the table header adds a column and remains available after files are added; a Field is not itself a - standalone file. A file or inline text is a value in a Sample × Field cell, - so every cell in a new file-valued Field keeps the direct Upload file or text - input. A field-level bulk upload, if later needed, must be a separate explicit operation and - must not replace the cell editor. Every upload or text edit started from a - cell is bound to that exact Sample × Field cell and can never fall back to the - last row or another implicit target. A bulk upload creates one explicit - file-to-row mapping and never chooses a row implicitly. Required parent Fields - cannot be removed or downgraded. A selected row exposes - its Fields in Properties independently of the center column window. Editing a - cell preserves the Draft's Sample row order; changing a value must not move - that Sample to the end of the table or change the target of the next edit. - Edit Snapshot freezes that specific parent and submits with Create Snapshot. - New empty Snapshot opens blank input with no parent, even when Latest exists; - it inherits no Schema, Samples or Splits. Original content remains immutable. - Advanced controls expose - independent roots, parent selection, frozen Split source filtering and explicit - inheritance of parent Splits. Snapshot editing has no Split-definition form. - Creation is bounded to 1000 Sample changes, 64 Fields, 64 KiB per text value, - 4 MiB of editable metadata, 20 files per batch and 100 MiB per file. Larger - committed Snapshots remain browsable with independent row/column windows. - Sample/cell context menus create a child Draft with that identity preselected. -- **DS-07 — Browser Draft and Commit:** each Draft has an independent resource ID. - IndexedDB stores metadata, small text, fixed Dataset/parent/source identities, - upload references and expected head conditions under the API origin/Dataset scope. - Dataset creation Drafts use a stable browser scope independent of the editable - name and appear beneath Datasets as Dataset Draft entries. Uploading the first - file, or submitting inline data, creates the server Dataset automatically and - pins its returned identity; its name then becomes read-only. Subsequent failures - retain the same editor and identity. A conflicting name or unknown creation - response never causes an existing Dataset to be adopted. The user can correct - an unallocated name; an allocated Dataset is not silently deleted on discard. - The tree lists other local Snapshot Drafts beneath Dataset and local Split Drafts - beneath their Snapshot's Splits. Save Draft and autosave retain editable input; - closing does not publish. Browser storage never contains File/Blob bytes. Storage - failure is visible; a transaction compares local revisions to reject stale-tab - saves and discards. A second Draft never silently resumes or replaces the first. - The identity fence prevents a same-name replacement Dataset receiving old input. - Choosing a file starts incremental MD5 and immutable 4 MiB part uploads with two - concurrent requests. Cells show filename, size and state; Properties includes - checksum and unclaimed expiry. Complete references are checked on restore and - Commit. An interrupted upload requires reselecting bytes; the size/digest key - resumes matching whole parts. Expired/aborted uploads receive a new identity. - No complete file must be reselected just because the editor reloads. - Create Dataset / Create Snapshot freezes input, prepares the complete immutable Snapshot, then conditionally - selects Latest. Split Commit updates only currentSplit. Unknown replies retain - the exact ID, input and original conditions for explicit retry; head conflicts - retain the returned observation for review. Neither silently rebases. Definitive - preparation rejection offers Edit as new Draft with a fresh resource ID; the - original input is never changed under a reserved identity. Unknown outcomes must - first be resolved. Successful Commit removes the local Draft and selects its - immutable result. - Discard removes only local input. Uploaded data and unreferenced results follow - existing service expiry/GC; local Drafts do not retain parents or files on the server. -- **DS-08 — Split Draft, View and Commit:** a committed Snapshot's Create Split - menu opens the complete set in the center, preserving other names. Filter rules - support Sample IDs, prefixes, ranges, keywords, inline Field matches, AND/OR/NOT; - they do not search uploaded file content. Weighted partitions, overlap and coverage - remain explicit. Save Draft is browser-local. Commit prepares and conditionally - publishes a new immutable collection revision without changing Dataset Latest. - Split counts are internal completeness metadata, not required navigation or - overview statistics. Browsing neither scans all pages to count nor labels loaded - rows or whole-Snapshot counts as a Split total. - Each rule has View. Draft View uses `split-preview`, a read-only bounded scan of - the fixed Snapshot, not a filter on the loaded page. A page scans up to 20 Samples - and can have zero matches plus a continuation. Cursors bind Snapshot and the full - rule; changing a rule clears the preview. View writes no draft, evaluation progress - or revision. The common matrix and Properties display matching rows. Committed - Split View reads the selected revision's frozen member pages. New Snapshots and - later revisions do not change old membership. Inherited membership is intersected - with child membership; new Samples remain unassigned until explicit reevaluation. - Historical revision editing retains the selected base and publishes under the - observed current head condition. Weighted members rename together; removing a - partition removes its complete group. -- **DS-09 — Lifecycle:** stable, retain, release and reclaim name the exact - Snapshot. Backend protection wins over an observed eligibility hint. Reclaim - distinguishes unknown, pending logical cleanup and completed retirement; physical - cleanup may remain deferred. A Dataset's own menu offers Delete when - `dataset-delete` is advertised. The confirmation pins its immutable Dataset ID. - Nonempty Datasets (including pending creation or cleanup) are rejected by the - backend. An accepted mark hides the Dataset from tree, list and properties; - cleanup follows the configured grace period, twenty minutes by default. - The name remains reserved until cleanup completes. Retries retain the same ID - and deadline; replacement identities are never silently targeted. No Split-history - purge action is implied. - Published Sample/Field changes produce a new Snapshot rather than modifying - the immutable selected one in place. -- **DS-10 — Modes:** `modules.dataset=false` hides the domain and prevents Dataset - observation. API prefixes apply to every request. Readonly disables editing and - the Console API rejects direct writes; exact matrix and Split preview POSTs remain observations. - Managed mode permits supported data operations without enabling topology changes. - -Dataset: catalog and Snapshot inventory (DS-01/02/06): - -```text -+----------------------------+--------------------------------------------------------------+------------------------------+ -| v Datasets | images [Refresh] | Properties | -| v images * | Complete Snapshots | Dataset images | -| v snapshot-a | Snapshot Parent Samples References | Dataset ID | -| > Splits | snapshot-a - ... Stable | Namespace dataset-ns | -| > snapshot-b | snapshot-b snapshot-a ... Latest | Latest snapshot-b | -| > audio | ... | Stable snapshot-a | -| | [Previous] [Next] | ... | -| | | | -| | Select Datasets root -> paged Dataset list | | -| | Select Snapshot -> Samples x Fields | | -| | No Session activity region. | | -+----------------------------+--------------------------------------------------------------+------------------------------+ -``` - -Dataset context-menu targets and immutable editing: - -```text -Datasets root Dataset images Snapshot -+--------------------+ +---------------------------+ +-------------------------+ -| Create dataset... | | New empty Snapshot... | | Edit Snapshot... | -+--------------------+ | Delete Dataset... | | Edit Split revision... | - +---------------------------+ | | - | Set stable | -Delete: empty only; hide immediately. | Retain / Release | -Cleanup: configured grace (default 20 minutes). | Reclaim... | - +-------------------------+ -Sample/Field edits -> reviewed candidate -> new published Snapshot -Published Snapshot data are never edited in place. -``` - -Dataset: table editing and browser Drafts (DS-06/07): - -```text -Dataset Help | Create Dataset | Properties -v Datasets | Name [dataset-xxxx] | Selected Sample / Field - Dataset Draft xxxx | [Upload files] | - v images | +-------------+-------------------+-----------+| - snapshot-a | | Sample ID | content | + Field || - Snapshot Draft b | +-------------+-------------------+-----------+| - | | cat.jpg | cat.jpg · 32 KiB | Remove || - | | note-001 | Upload or [text] | Remove || - | | + Sample || - | +-------------+-------------------+-----------+| - | [Previous rows] [Next rows] < columns > | - | > Advanced | - | [Save Draft] [Create Dataset] [Discard Draft] | - | Saved locally | - -Edit Snapshot: fixed Parent shown; submit = Create Snapshot. -New empty Snapshot: blank content column; no Parent; submit = Create Snapshot. -Upload files with multiple Fields -> choose target Field -> native file picker. -``` - -```text -Choose file -> checksum + bounded parts -> upload reference in local Draft - | interrupted: reselect matching bytes to resume -Save Draft -> browser metadata only -> close/reload -> continue editing -Create Dataset / Create Snapshot -> freeze input -> prepare result -> conditional head update - unknown: retry exact input / ID / conditions -``` - -Dataset: Split preview (DS-08): - -```text -v images | Splits · Draft · local - v snapshot-a | name [cats] rule [Field label = cat] - v Splits | [View cats] overlap / coverage - v revision Current | +-------------+-------------------+-----------+ - train | | Sample ID | content | label | - Split Draft c | | | | | - | +-------------+-------------------+-----------+ - | | cat.jpg | payload · 32 KiB | inline | - | +-------------+-------------------+-----------+ - | [Next Samples] - | [Discard Draft] [Save Draft] [Commit] - | Saved locally -``` - -Dataset: Samples matrix and selected cell (DS-03–05/08): - -```text -+----------------------------+--------------------------------------------------------------+------------------------------+ -| v Datasets | snapshot-b [Refresh] | Properties | -| v images | Parent / Latest / Stable / inspection state | Selected Sample/Field cell | -| v snapshot-a | ---------------------------------------------------- | Sample ID | -| > Splits | Sample image label ... | Field image | -| v snapshot-b * | sample-001 cat | Required | -| v Splits | sample-002 * dog | Storage Payload | -| v rev Current | ... | Length | -| validation | [Previous rows] [Next rows] | Checksum | -| | [Previous fields] [Next fields] | [Preview field] | -| | | Verified complete value | -| | Select Split -> pinned revision member window | or explicit unavailable | -| | Select column / row / cell -> matching properties | | -+----------------------------+--------------------------------------------------------------+------------------------------+ -``` - -Preview is explicit and bounded to a complete verified value of at most 64 KiB. -Snapshot/revision/member history remains business data, independently of the -removed browser Session activity. - -Local batch mapping uses version 1 JSON aligned with Sample/Field edit models, -with the same editor limits. `schema: null` inherits the parent Schema. IDs and -inline bytes use canonical base64. File values in the submitted mapping are completed -upload descriptors; pending file references cannot Commit. Duplicate IDs, invalid -Fields and incomplete references fail without changing published content. - -```json -{ - "version": 1, - "schema": {"version": 1, "fields": [{"name": "text", "required": true}]}, - "samples": [ - {"kind": "upsert", "id": "c2FtcGxlLTE=", "fields": [ - {"name": "text", "value": {"kind": "inline", "data": "aGVsbG8="}} - ]} - ] -} -``` - -A submitted payload value is `{"kind":"upload","id":"","length":"123","md5":""}`. Pending file tokens are local editor state only. An optional -field tombstone is `{"kind":"tombstone"}`. Removal is -`{"kind":"remove","id":"c2FtcGxlLTE="}`. Authority composition combines these -bounded edits with the frozen source; the browser never constructs a partial -Manifest and labels it complete membership. diff --git a/doc/design/console/design-crowdb-console.md b/doc/design/console/design-crowdb-console.md deleted file mode 100644 index 67a027abc..000000000 --- a/doc/design/console/design-crowdb-console.md +++ /dev/null @@ -1,542 +0,0 @@ - - - -# CROWDB - Design: Console (Overview) - -The `crowdb-console` component provides a Web UI and a CLI that share one -Rust core for managing `crowdb-kv-server` clusters. This overview covers -the backend, data model, CLI, and hosting concerns; the frontend SPA -design is detailed in the sub-design `design-crowdb-console-ui.md`. - -## Table of Contents - -- [1. Goals and Non-Goals](#1-goals-and-non-goals) - - [Goals](#goals) - - [Non-Goals](#non-goals) -- [2. High-Level Architecture](#2-high-level-architecture) - - [2.1 Call Path](#21-call-path) - - [2.2 Reuse Boundary](#22-reuse-boundary) -- [3. Data Model](#3-data-model) - - [3.1 Physical (deployment) view](#31-physical-deployment-view) - - [3.2 Logical (usage) view](#32-logical-usage-view) - - [3.3 Source of truth and freshness](#33-source-of-truth-and-freshness) - - [3.4 Design decisions](#34-design-decisions) -- [4. Console Configuration and Authority](#4-console-configuration-and-authority) - - [4.1 Separated local configuration](#41-separated-local-configuration) - - [4.2 Runtime observation](#42-runtime-observation) - - [4.3 Group 0 authority](#43-group-0-authority) - - [4.4 Local runtime namespace](#44-local-runtime-namespace) -- [5. Node Access Model](#5-node-access-model) - - [5.1 Two transports per node](#51-two-transports-per-node) - - [5.2 SSH defaults (russh)](#52-ssh-defaults-russh) - - [5.3 Process lifecycle (deploy / start / stop)](#53-process-lifecycle-deploy--start--stop) -- [6. Web UI Backend (Axum)](#6-web-ui-backend-axum) - - [6.1 Design Rules](#61-design-rules) - - [6.2 In-process test API](#62-in-process-test-api) - - [6.3 Orchestration semantics](#63-orchestration-semantics) - - [6.4 Resolution rules](#64-resolution-rules) - - [6.5 Frontend contract](#65-frontend-contract) -- [7. CLI Design](#7-cli-design) -- [8. Error Model and Operation Logging](#8-error-model-and-operation-logging) -- [9. Observability](#9-observability) -- [10. Hardware mutations](#10-hardware-mutations) -- [11. Cluster teardown and verification](#11-cluster-teardown-and-verification) - -## 1. Goals and Non-Goals - -### Goals -- Single workspace project `crowdb-console` delivering a Web UI and a CLI that share one Rust core. -- Operate against any number of `crowdb-kv-server` instances via their public surfaces (HTTP management API + crowdb-rpc KV / health). -- Model a **Rack → Node → Server Instance → Store → Group → Replica** hierarchy, including a "simulated hardware" mode that runs entirely on `127.0.0.1`. - -### Non-Goals -- Bypassing `crowdb-kv-server` to talk to Paxos / WAL / storage internals. -- Multi-tenancy and a general audit-log service. -- Making a console-local file authoritative for cluster topology. - -Ordinary deployment runs the shared OCI node image. The host lifecycle owns -containers, applied resources and private ownership receipts; each node monitor -owns its service processes and retained data. Local evaluation may place several -logical nodes on one host, with zero physical host failure tolerance. Production -placement uses the confirmed physical failure domains. Test-only measurement -fixtures may own native processes but do not define a second public deployment -lifecycle. - -## 2. High-Level Architecture - -`crowdb-console` is **one project** split across the `lib/` and `app/` -workspace roots: a shared core lib plus two binaries. The console is a -general cluster-management surface (not limited to CROWDB), so crate -names use the `crowdb-*` prefix without `kv`. - -``` -lib/crowdb-console-shared/ (lib) data models, HTTP+crowdb-rpc clients, registry, aggregator, error model, SSH session pool, workload generator -app/crowdb-web/ (bin) Axum backend, static asset server, proxy routes - src/ Rust source - ui/ React + Vite frontend source (TS, shadcn/ui, React Flow) - tests/ integration tests -app/crowdb-cli/ (bin) clap-based CLI; depends on shared -``` - -Targets: -- `crowdb-console-shared` → reusable lib for both frontends. -- `crowdb-web` → bin, serves UI + API on `:9920`. -- `crowdb-cli` → bin, the user-facing CLI. - -### 2.1 Call Path - -The web frontend (SPA backed by Axum) and the CLI follow the same -path through the shared `ops` module: - -- **Web**: `user → crowdb-web (Axum) → shared (ops module) → group-0 sysdata + crowdb-kv-server` -- **CLI**: `user → crowdb-cli → shared (ops module) → group-0 sysdata + crowdb-kv-server mgmt` - -Both frontends build an `OpContext` with Group 0 discovery seeds. Attached CLI -commands select `--cluster` or `--management-endpoint`, verify the confirmed -cluster UUID through the management origin, then verify the same UUID through -native Group 0 before hardware/KV operations and native workloads. Production -Web uses a versioned process configuration. Both read confirmed hardware and logical records through Group 0 -and resolve node management endpoints from live registration. Ordinary OCI nodes -use `crowdb-monitor` for process ownership on both Docker and containerd. A -configured auxiliary Web process may retain a local launch registry; it is not -required for node formation or endpoint-only attachment. - -The CLI talks directly to group-0 system metadata via -`CrowdbSysmdClient` and to individual `crowdb-kv-server` management -APIs via `ServerClient` — no `crowdb-web` intermediary. The `ops` -module in `shared` holds the operation logic that both frontends -call. - -``` - ┌──────────────┐ ┌──────────────┐ - user ───► │ crowdb-web │ or │ crowdb-cli │ (frontend) - └──────┬───────┘ └──────┬───────┘ - │ parse input, │ - │ render output │ - └──────────┬────────────┘ - ▼ - ┌───────────────┐ - │ shared │ (business logic: - │ (lib crate) │ ops module, - └──────┬────────┘ leader discovery, - │ registry controls, - ┌──────────────┼──────────────┐ SSH session pool) - ▼ ▼ ▼ - HTTP crowdb-rpc SSH - │ │ │ - ▼ ▼ ▼ - ┌────────────────────────────────────┐ - │ crowdb-kv-server │ (one per node) - └────────────────────────────────────┘ -``` - -### 2.2 Reuse Boundary - -- All "what to do" lives in `shared` (e.g. `ops::kv_logical::add_group`, - `ops::kv_server::deploy`, `ops::kv_data::put`, `ops::hardware::add_rack`). -- `web` (Axum) and `cli` (clap) only parse input and render output. -- The web SPA **does not** reimplement business logic; it calls - `shared` via the Axum backend, never `crowdb-kv-server` directly. -- Both frontends build an `OpContext` and call `shared`'s `ops` - module directly — the CLI from its verified application target, - the web backend via `AppState::op_context()` (sharing the - cached `CrowdbKvClient` and Group 0 management seeds). -- Shared hardware and logical operations provide the same authority and - conditional publication rules to both frontends. Node count and runtime do not - select UI permissions; required authority and resource ownership still apply. - -## 3. Data Model - -The console exposes **two hierarchy views** of the same cluster. Both -views describe the same underlying entities; they differ only in the -direction from which the cluster is observed. - -### 3.1 Physical (deployment) view - -> "What hardware exists, and what is running on each piece of it." - -Rooted at **Rack → Node → Server → PxStore → PxGroup → {LocalReplica, -RemoteReplica…}**. Every entity below `Node` is described from that -node's vantage point. A `PxGroup` has exactly one local replica plus -N−1 remote-replica proxies. This mirrors the `crowdb-kv-server` internal -data structure, which is why this view is also the "debugging view": -the API surfaces the remote-list explicitly so an operator can spot -bugs where a node failed to register all of its peers. - -Identity is the parent chain -`(rack_id, node_id, store_id, group_id, replica_id)`. - -### 3.2 Logical (usage) view - -> "What stores and groups exist in the cluster, regardless of where -> they live." - -Rooted at **Cluster → Store → Group → Replica…** with a unified replica -list (no local/remote split; each replica carries a `node_id`). This is -the view that KV traffic, leader resolution, and routine cluster -operations use. Shared operations translate logical IDs into confirmed -membership and live endpoints for both frontends. - -Identity is `(store_id[, group_id[, replica_id]])`. - -### 3.3 Source of truth and freshness - -- **Group 0:** rack and node identity, nonsecret SSH connection settings and - credential references, disk hierarchy, bindings, KV stores, groups, - replicas, and service registration. -- **Local process inputs:** a versioned Web process configuration, bare-metal - launch registry, and per-console secret store. Neither topology nor inline - SSH secrets are accepted in the launch registry. -- **Live state:** management health, process identity, and current endpoints. - Docker reads monitor-owned process state; bare-metal process identity is - checked by `LaunchRuntime`. A stopped process is not a live registration. -- **Unavailable authority:** missing or ambiguous registration and Group 0 - outages are reported as unavailable. The monitor and launch registry do not - supply fallback topology. - -### 3.4 Design decisions - -- **No `server_id` namespace.** The server's mgmt/crowdb-rpc URLs live inside - `Node.server` and are never exposed in console-facing JSON URLs. Since - the console enforces one server per node, node identity *is* server - identity. -- **Local/remote split is visible only in the physical view.** The - logical view collapses replicas into a unified list so cluster-level - operations can ignore placement. The physical view keeps the split - for debugging missing peer registrations. -- `StoreView` / `GroupView` / `ReplicaView` reuse `crowdb_kv::cluster::info` - where possible; the console-side wrapper adds the `node_id` - projection that the per-server protocol does not encode. - -## 4. Console Configuration and Authority - -### 4.1 Separated local configuration - -`WebProcessConfig` contains the listener, Group 0 management seeds, UI and log -paths, deployment mode, and (for Docker) the monitor status path. Configured -managed Web requires this versioned input. Docker rejects a launch registry. - -Standalone Web starts without `--config` in the persistent console `default` -namespace. Its private, atomically replaced `config.json` retains pre-bootstrap -hardware, process launch inputs, and a post-bootstrap topology cache. Web restart -recovers KV processes first, discovers Group 0, reloads its confirmed records, -and then recovers auxiliary services from their recorded launch inputs. The -cache never overrides initialized Group 0 topology. A malformed file or failed -initialized authority recovery reports an error without resetting operator data. - -`LaunchRegistry` contains bare-metal process policy: service, node, host, -binary, service config, workspace and auto-start setting. Runtime PID and -start-time identity are retained separately by `LaunchRuntime`. SSH credential -reference IDs are read from Group 0 and resolved against each console's local -secret store. Group 0 never contains private keys, passwords, PIDs, images or -container IDs. - -The `ConsoleConfig` struct is an operation input for bootstrap and local -development; standalone Web persists it locally until bootstrap and retains -launch inputs for restart. A sealed `BootstrapIntent` -retains pre-Group-0 identity across interruption and is deleted only after all -committed records are verified. - -### 4.2 Runtime observation - -The production `/api/preview` snapshot reads Group 0 hardware and logical -records and validates live service registrations. Docker overlays monitor -process status, while bare-metal Web uses its local launch runtime. Missing -monitor status makes Docker runtime observation unavailable. The in-process -Web test router keeps a monitor cache for fixture orchestration; that cache is -not production topology authority. - -Manual node Web refreshes per-node KV topology through the admitted management -endpoints before physical reads. Remote process health comes from this live -observation; remote PIDs never enter Web's local process table. The service -list separately queries admitted monitors over their authenticated control -channels for actual live children, with `pid_node_id` identifying the process -namespace. This observation verifies local cluster/node binding and remains -available during Group-0 failure. The Web host neither probes these PIDs via -its own `/proc` nor stores or signals them as local processes. A failed probe -marks the node down, while local launch observations still require a tracked PID. - -### 4.3 Group 0 authority - -System group (store 0, group 0) replicates hardware and KV-cluster metadata. -Bootstrap initializes selected KV members, wires peers and conditionally -publishes the rack, node, store, group and replica records. A retry compares -sealed identity and already committed content, writes only missing records, -and rejects conflicting content. Nonmember KV processes receive Group 0 seeds -and must register exactly one live identity before logical operations use them. - -The metadata namespaces are `/hw/rack`, `/hw/node`, `/hw/dg`, `/hw/disk`, -`/hw/owner`, `/hw/bind`, `/kv/store`, `/kv/group`, `/kv/replica`, and `/srv`. -Logical mutations confirm all node-side steps before publishing membership; -conditional writes and confirmed reads reconcile a lost response. A local -launch or monitor record never substitutes for a missing Group 0 result. - -### 4.4 Local runtime namespace - -The default local root is `.crowdb-runtime/`, divided into `ephemeral/`, -`persistent/`, `artifacts/`, and `ports/`. A mini-cluster or retained console -deployment is one persistent namespace. Its manifest stores the namespace -identity and the mapping from each `(service kind, logical instance)` to a -port. Service paths live below -`services///{data,config,log,artifacts}`. - -`start_new` creates an identity and its assignments. `restart` reuses the -same paths and assignments and never asks for replacement ports. `stop` -terminates recorded processes while preserving the manifest and data. -Deletion is the only operation that releases persistent claims and removes -the namespace. If a recorded port is held by another owner, restart reports -the conflict; it does not silently select a different endpoint. - -Disposable deployment and E2E fixtures use ephemeral namespaces with the -same layout. A saved CLI cluster inside an ephemeral fixture inherits that -fixture's process and port ownership; its launcher may exit without releasing -ports still used by the fixture. Process records remain outside disposable -data so cleanup can terminate services even after data removal. Cleanup releases -these nested claims along with the fixture. All local CROWDB data, generated -configuration, logs, benchmark output, and coordination state stay below this -root; no default path uses the system temporary directory. Ordinary clean -operations preserve persistent -namespaces. - -## 5. Node Access Model - -### 5.1 Two transports per node -| Purpose | Transport | -| --- | --- | -| Deploy / start / stop `crowdb-kv-server` process; copy binary | SSH | -| Runtime mgmt API (add store/group, list, health) | HTTP | -| Runtime KV ops, paxos health | crowdb-rpc | - -### 5.2 SSH defaults (russh) -- Crate: **`russh`** (pure Rust, async). No shell-out fallback. -- Default auth: `~/.ssh/*` keys (agent + standard key paths). -- Alternative auth: explicit key path; explicit `user/password`. -- Default host: `127.0.0.1` with the current OS user. -- Pre-flight: every operation calls `ssh::probe(node)` which performs a real handshake before any side-effecting work. Failure surfaces as `NodeUnreachable { node_id, reason }`. - -**SSH credential boundary:** Group 0 stores only the SSH user, port and -credential reference associated with a node. Each bare-metal console resolves -the reference in its own local secret store. Bootstrap intent rejects inline -private keys and passwords; the launch registry accepts references only. - -Node admission installs each node's public Ed25519 key and proves access in -both directions with strict host-key checking. Preparation reuses one -key-authenticated candidate session and one session per existing peer; the -two directional proofs run concurrently. Ordered preparation commands reuse -their authenticated connection. A failed stage preserves durable progress for -retry, while initialization passwords remain only in the submitted request. - -### 5.3 Process lifecycle (deploy / start / stop) - -DiskDB deployment accepts an optional positive `keepalive_interval_secs` for -its normal heartbeat policy. Omitting it retains the server default; the -committed remote service configuration preserves it across restart. This -setting does not change ownership or deployment protection rules. - -Chunk-KV deployment optionally accepts `stream_chunk_capacity_bytes` (1–256 MiB) -and positive `stream_extent_page_entries`. Omitted values retain server defaults. -These storage parameters are restricted to Chunk-KV and become part of its -committed service configuration, including subsequent container-owned restarts. - -`LaunchRuntime` uses the validated launch registry for local or SSH process -start, restart, stop and readiness checks. It records PID plus process start -time as local runtime identity and refuses to signal an unrelated process. -Auto-start policy is reconciled on Web startup and reload. A successful process -launch is not a substitute for a Group 0 service registration. Docker delegates -child recovery and status to `crowdb-monitor`. - -## 6. Web UI Backend (Axum) - -### 6.1 Design Rules - -The ordinary OCI entrypoint starts a node Web with an editable UI in both -one-member and three-member clusters. Automatic preparation and runtime choice -do not restrict management writes. Before bootstrap, candidate/rack preparation -uses retained local admission state; afterward, hardware and logical mutations -use confirmed Group 0 authority and authenticated monitor execution. Configured -managed Web retains its versioned process configuration and requires confirmed -records; unknown managed routes never enter an in-memory topology fallback. - -A mutation returns success only after the required node-side steps and Group 0 -publication are confirmed. The Console assumes a root operator until UI login -is introduced. Explicit `readonly = true` rejects non-read requests with 403; -`/api/mode` exposes that setting and the SPA disables writes accordingly. The -ordinary node default is writable. Credentials remain in the established private -store, and invalid authority, ownership or placement rejects the operation for -that cause. The SPA calls the Axum backend rather than KV management endpoints. - -### 6.2 In-process test API - -The in-process Web router and `--test-mode` retain fixture orchestration for -browser and integration tests. Their recursive physical views and monitor cache -help exercise the UI. Standalone Web uses these resource APIs with persistent -bootstrap/recovery inputs; `--test-mode` uses isolated ephemeral state. Neither -provides fallback topology for configured managed requests. - -### 6.3 Orchestration semantics - -For each multi-node operation in the logical tree, the backend obeys -these rules: - -- **Plan first, act second.** Resolve every required node + replica id - from Group 0 membership and live service registrations before issuing - mutation RPCs. Missing or ambiguous registrations fail the operation. -- **Built on physical primitives.** The orchestrator only calls the - per-node physical mutators; it never invents a side channel. -- **All-or-nothing where feasible.** On partial failure, attempt to - undo successful sub-steps and surface the resulting state in the - error body. -- **Confirmed membership publication.** Complete peer wiring before publishing - group or replica membership. A new group and its initial replica records - commit in one conditional batch. Create records conditionally; a concurrent - conflicting record is preserved. A lost write response is resolved only by - a linearizable read that confirms the intended record. -- **Deletion preserves authority on node failure.** Confirm deletion on every - hosting node before removing membership. Store hosts include nodes from - replica records as well as the store record. Remove descendants before - parents; an already absent node-side object permits retry. -- **Idempotent retries.** A repeat of the same logical request must - converge to the same state. -- **Read after write.** A mutation returns only after the required node-side - and Group 0 confirmation steps complete. - -### 6.4 Resolution rules - -- Unknown ids → `404`. -- Unreachable node for a physical-tree call → `502 Bad Gateway`. -- Partial logical-tree failure → roll back and report `409 Conflict` - with structured per-node outcomes (not `207 Multi-Status`). -- All handlers are thin wrappers around `shared` entry points; the - CLI calls the same entry points. - -### 6.5 Frontend contract - -The frontend SPA design lives in `design-crowdb-console-ui.md`. The -backend-facing contract here: - -- Bundle output is `app/crowdb-web/ui/dist/`; `crowdb-web` serves - it via SPA fallback. -- The SPA polls the management API on a short interval. An unavailable - authority clears stale logical rows and is shown explicitly. -- No `/api/cluster/snapshot` aggregate endpoint. - -## 7. CLI Design - -- `cluster start/stop` own OCI runtime lifecycle through a named local ownership - receipt. `start` accepts one or three nodes and delegates preparation to the - normal node monitor. Runtime inspection, logs and image operations use the - runtime's tools; cleanup belongs to the independent host/Pixi helpers. -- `cluster status/check` observe confirmed application readiness over management - endpoints, without a runtime socket. A finite check returns nonzero when any - required dependency is unmet. Runtime facts unavailable remotely remain unknown. -- Hardware, logical KV, S3 and benchmark commands require `--cluster` or - `--management-endpoint`. Every selected origin must report the same confirmed - UUID; saved bindings pin that UUID. Native consumers additionally verify - transport reachability and the committed Group-0 UUID before data access. - There is no system-address, localhost or launch-registry fallback. -- Hardware and logical KV operations call Group 0 and node management APIs - directly after target verification. Signed S3 clients obtain access credentials - and the advertised public URI from the selected management origin. -- Endpoint-only attachment grants no runtime ownership. It cannot start or stop - containers, create a runtime receipt, change the image or prepare infrastructure. -- The retained workload families are KV, DiskDB, ChunkDB, ChunkIO and S3. - `bench kv prepare` writes its declared dataset into an existing group. - Workloads never initialize groups, deploy services, reset cluster topology or - stop a cluster. Workload-owned data cleanup and ordinary client transport - workers remain allowed. Standalone RPC tests own their server/client lifetime. -- `kv store/group/replica/data` retain normal topology and data operations. - Process deployment and reset commands are absent from the command tree. -- Output identifies the copyable command and final result. HTTP exchanges show - response metadata and format structured bodies; binary bodies are identified - by type and byte count. Status/check support structured JSON output. -- Ordinary commands log only to the console. A benchmark writes the latest run - under the runtime root's `artifacts/cli/bench-` directory; the next run - replaces only that family's previous benchmark log. KV workloads also retain - a structured `report.json` with operation counts, errors and measured latency - alongside the human-readable console report. Measurement tools preserve each - completed invocation before another run replaces the latest directory. - -## 8. Error Model and Operation Logging - -- `shared::Error` enum covers `NodeUnreachable`, `UpstreamRpc`, - `Validation`, `NotFound`, `Conflict`. HTTP maps to 4xx/5xx; CLI maps - to exit codes (0 ok, 1 user error, 2 cluster/network error). -- **Console transcript** — every invocation prints the copyable command and - result. Direct S3 requests additionally show request/response headers and a - body marker. Text is visible, XML/JSON is formatted, and binary content is - summarized by byte count without duplicating it in the trace. -- **Retained diagnostics** — only benchmark commands write files. The latest - run for each benchmark family lives beneath the runtime root's - `artifacts/cli/bench-/` - and replaces the previous run. Benchmark tracing is also mirrored to the - console. Routine outbound HTTP calls are debug-level so default output keeps - lifecycle events, warnings, and errors without per-request noise. - -## 9. Observability - -- `tracing` everywhere; `--vv` switches CLI to debug. -- Web backend exposes `/healthz`. **`/metrics` is deferred**. The Rust - Prometheus story has multiple competing crates; we will pick one when - broader observability work for `crowdb-kv-server` begins. -- All console-issued operations attach a correlation id propagated as - `x-crowdb-kv-corr-id` to `crowdb-kv-server` request headers. -- Regression service metrics have a content contract: KV emits `rust`, - `cpp-rpc`, `cpp-tree`, and `misc`; DiskDB and ChunkDB emit `rust`, - `cpp-rpc`, and `misc`; DiskIO emits `cpp-rpc` and `misc`. A tracing or RPC - log may legitimately remain empty when its configured level observed no - events; metric validation therefore checks metric sections and counters - independently of auxiliary log size. - -## 10. Hardware mutations - -CLI and node Web use the shared Group 0 hardware operations. Rack, -node, disk-group and disk changes update parent and child records in one -conditional batch where membership changes. Matching retries are confirmed; -conflicting concurrent writes preserve the existing record. One-member and -three-member OCI nodes expose the same shared management surface after -confirmed Group 0 publication. - -The physical Cluster workspace shares one tree and selection across Topology and -Capacity. Cluster/Rack node cards and capacity tables reuse bounded observations; -Node storage owns discovery, accepted file/supplied operations and retained -candidates. Group 0 defines effective disk membership. Candidate Join validates -reviewed media and retains one immutable operation per disk; accepted work runs on -the owning monitor after the browser closes. Offline/Remove/Rejoin retains bytes -and allocation identity, with exclusion and validation gates before participation. -Physical discovery is never permission to reinitialize or replace an existing disk. - -Concurrent Access deployment reserves one cluster-scoped Console catalog -Initialize/Activate operation pair in Group 0. Nodes replay that pair; conflicting -local journals remain visible and are not rewritten. Access credential bootstrap -also reserves one encrypted user candidate before conditionally publishing its -access-key record. Response loss/concurrent nodes cannot replace an existing -credential; duplicate or disabled users require explicit review. Enumeration is -bounded and rejects truncated/timed-out results instead of authorizing issuance -from incomplete state. Secrets remain in the established private credential path. - -## 11. Cluster teardown and verification - -`cluster start`, `cluster stop`, `cluster status` and `cluster check` form the -common CLI lifecycle. Start and stop require matching private runtime ownership; -stop retains accepted inputs and data. Endpoint-only attachment supports current -observation and workloads without granting runtime stop or cleanup authority. -Independent OCI cleanup validates the recorded daemon, exact container and -resource identities before removing owned runtime resources. It preserves data -by default; explicit owned-data deletion is separate from stopping services. - -Application cleanup is a distinct authenticated operation. It validates the -retained bootstrap, retires further setup/recovery, drains owned services and -removes the matching system store through its local deletion endpoint. Failed -members remain pending for explicit retry; an unavailable node is never proof -that its membership or data should be deleted. No cleanup guesses orphan -ownership or reconstructs topology from host-local process records. - -Manual node discovery and bootstrap follow the -[deployment architecture](../deploy/design-crowdb-deploy.md). The Cluster sidebar -retains a separate candidate list and explicit draft, fixed bootstrap, publication, -active, stale authority and cleanup states. Each UI reads its local monitor's -observations and the shared Group 0 registry. Discovery filtering selects another -cluster for inspection without merging its membership. Authenticated node updates -retain UUID/numeric allocations and physical failure domains. Loss of Group 0 -blocks management; existing data groups use their own Paxos authority and confirmed -endpoint hints for data operations. diff --git a/doc/design/deploy/design-crowdb-deploy.md b/doc/design/deploy/design-crowdb-deploy.md deleted file mode 100644 index 18038b3c5..000000000 --- a/doc/design/deploy/design-crowdb-deploy.md +++ /dev/null @@ -1,850 +0,0 @@ - - - -# CROWDB - Design: Multi-node Deployment - -Node containers run discovery, SSH, monitor and Web before cluster creation. -Group 0 owns confirmed topology and management operations. Linux amd64 images -are built by Pixi-managed standalone BuildKit and run on system Docker or -containerd/nerdctl. The [OCI image design](design-crowdb-oci-image.md) defines -construction, runtime resources and publication. - -The [console architecture](../console/design-crowdb-console.md) defines the -shared operation layer; the [Group-0 architecture](../kv/design-crowdb-kv-group0.md) -defines system metadata; the -[reconfiguration design](../kv/design-crowdb-kv-reconfiguration.md) governs -Paxos voting membership. This design connects those responsibilities without -making node discovery a membership protocol. - -## Table of Contents - -- [1. Goals and boundaries](#1-goals-and-boundaries) -- [2. Components and identities](#2-components-and-identities) -- [3. Invariants](#3-invariants) -- [4. Node discovery](#4-node-discovery) -- [5. Node and rack configuration](#5-node-and-rack-configuration) -- [6. Bootstrap and recovery](#6-bootstrap-and-recovery) -- [7. Shared UI and management operations](#7-shared-ui-and-management-operations) -- [8. Authorization and partitions](#8-authorization-and-partitions) -- [9. Image and runtime deployment](#9-image-and-runtime-deployment) -- [10. Resource and lifecycle management](#10-resource-and-lifecycle-management) -- [11. Configuration and observability](#11-configuration-and-observability) -- [12. Correctness and validation](#12-correctness-and-validation) -- [13. Supported scope](#13-supported-scope) - -## 1. Goals and boundaries - -- Each node has a monitor and a Web UI. Operators can use any member's UI; - there is no permanently designated UI host. -- Monitors discover nodes on a management LAN without a hand-written peer list. - Discovery identifies nodes, not the individual servers hosted on them. -- Before Group 0 is initialized, UI operations are restricted to node/rack - preparation and Group-0 bootstrap. Other server deployment requires a ready - Group 0 and confirmed initial topology. -- Group 0 is the shared information platform and authority for cluster - membership, racks and deployment operations. -- Production deployment uses Linux light containers: OCI images package - binaries/dependencies, while network, disks and performance resources are - explicitly assigned from the host. -- Single-host tests simulate multiple nodes using isolated containers on a - dedicated Docker network and independent simulated disks. - -The deployment layer is not a general scheduler. Kubernetes and Nomad are not -the default deployment substrate. OCI image compatibility allows other runtimes -to consume an image; it does not establish complete Kubernetes deployment -support. Cross-subnet discovery has a seed fallback and does not require a -multicast relay. Independent initialized clusters are never automatically merged. - -## 2. Components and identities - -```text -operator browser - -> any node's Web UI / shared console operations - -> Group 0: committed topology and operation authority - -> target node monitor - -> OCI node container and managed server processes - -node monitor <-> management-network discovery <-> peer node monitors -``` - -- **Node:** a deployment and hardware-management identity. A production node - normally corresponds to a physical Linux host; a test node may be a container. - A node exists before any KV server is started. -- **Monitor:** the node management endpoint, discovery participant and local - process/container supervisor. Reuse `crowdb-monitor`; do not introduce a - second agent with overlapping responsibilities. -- **UI:** a presentation/API entry point into shared operations. It owns neither - confirmed topology nor a private cluster. Losing the initiating UI does not - delete a submitted operation. -- **Server:** a managed service/process deployment on a node. Server deployment - is distinct from node admission and Group-0 voting membership. -- **Cluster identity:** the durable identity associated with one initialized - Group 0. A node cannot infer or replace it from nearby multicast announcements. -- **Rack:** an editable logical grouping of nodes. A node-supplied rack value - is an initial hint; confirmed placement belongs to Group 0. -- **Physical failure domain:** actual shared host/rack infrastructure used by - placement policy. Virtual rack labels do not manufacture hardware independence. - -Persist node identity, cluster binding and accepted bootstrap information in -the node's host-mounted state root. Persist Group-0 replica state through its -existing WAL/storage contract. These identities are not image contents and are -not derived from IP addresses. Existing protocol identifier types remain the -canonical types. Each node generates and persists a discovery UUID before -admission. Candidates are identified by UUID; bootstrap fixes the UUID-to-numeric -node-ID mapping in its accepted configuration. Subsequent admission allocates a -numeric node ID through Group 0 and durably records the mapping. Allocation and -mapping publication are conditional and idempotent under concurrent admission; -IDs cannot collide or be silently reassigned. Endpoint and rack changes do not -change either identity. Duplicate live UUID claims are identity conflicts. -Confirmed admission retains the Web listener proved over authenticated SSH. -Cluster observation uses that listener for each UUID, including containers with -shared host addresses and different port offsets. Retained default-port records -keep their existing serialized identity; each monitor republishes its own current -listener through the existing confirmed-node update operation. - -## 3. Invariants - -- **I1 — Durable identity:** recreation, endpoint changes and rack reassignment - preserve node identity. Conflicting live endpoints claiming one identity - cause a visible conflict, not silent overwrite. -- **I2 — Candidate discovery:** discovery records are observations. They grant - no admission, voting membership, rack authority or execution permission. -- **I3 — Group-0 authority:** confirmed topology and management mutations use - Group 0. A local monitor/cache never substitutes for unavailable authority. -- **I4 — Bootstrap binding:** every selected monitor durably accepts one - compatible cluster/bootstrap manifest before starting its initial replica. - Conflicting manifests are rejected. No timeout permits replacement bootstrap. -- **I5 — Authorized execution:** a connected Group-0 replica is not proof of - authority. Management mutations require quorum-backed authorization, and - stale operations cannot execute after revocation or authority transition. -- **I6 — Explicit failure domains:** virtual rack assignment and physical - failure-domain identity are distinct placement inputs. -- **I7 — Runtime portability:** a supported-architecture OCI image carries - code; runtime inputs carry identity, network, devices and persistent state. -- **I8 — Bootstrap gate:** only Group-0 prerequisite processes and Group-0 - initialization run before the shared information platform is ready. Other - deployment requires committed initial topology. -- **I9 — Identity-bound address recovery:** a changed address is usable only - after the peer proves possession of the SSH host private key pinned to its - accepted cluster UUID, discovery UUID and numeric node ID. Discovery cannot - change this binding or the accepted voting membership. - -## 4. Node discovery - -### 4.1 mDNS and DNS-SD - -Monitors announce and browse `_crowdb-node._tcp.local.` on the configured -management interfaces. Candidate expiry removes observations after five seconds; -it never removes a confirmed member. - -- mDNS transports local-link DNS queries/responses using UDP port 5353 and - multicast groups `224.0.0.251` / `ff02::fb`. -- DNS-SD uses PTR records to enumerate instances, SRV to identify monitor host - and port, A/AAAA to resolve addresses, and TXT for bounded metadata. -- TXT carries only discovery essentials: identity, protocol compatibility and - optional cluster binding. Full node/hardware/rack information is obtained - through the monitor management handshake. - -These mechanisms come from [RFC 6762](https://www.rfc-editor.org/rfc/rfc6762.html) -and [RFC 6763](https://www.rfc-editor.org/rfc/rfc6763.html). The standards define -discovery and caching; CROWDB defines admission and authorization separately. -Announcements, refresh, cache expiry and clean departure follow these mechanisms. - -Each monitor's discovered-peer cache feeds its local UI backend. Browser code -does not send mDNS traffic. Peers query/announce directly within the discovery -domain; there is no requirement for a separate gossip membership layer. - -### 4.2 Network scope and reachability - -Discovery runs only on configured management interfaces. Host-network nodes -use the management NIC/VLAN; Docker test nodes use their dedicated bridge -interface. The advertised monitor endpoint must be directly reachable by peers -in that scope. A wildcard bind address is not an advertised address. - -UI addresses exposed to browsers and monitor addresses used by peers are -separate endpoint purposes. In bridge tests, host UI ports are published while -peer monitor connections use container-network addresses. Production host -networking requires an explicit local port plan. - -mDNS does not normally cross routed subnet/VLAN boundaries. When unavailable, -configured peer/seed monitor endpoints provide the same handshake path. The -seed list is a discovery input, not an authoritative membership list. Monitor -connectivity verifies endpoint usability; discovery expiry alone does not -establish node failure or remove a confirmed member. - -### 4.3 Cluster separation and trust - -Classify a discovered peer as unbound, bound to this cluster, bound to another -cluster, incompatible or identity-conflicting. Only compatible candidates can -be selected for bootstrap/admission. A foreign cluster is displayed separately; -its topology is not imported automatically. - -Announcements are untrusted hints. The direct handshake and admission policy -must validate the management peer before privileged actions. Never advertise -credentials in TXT records. Admission requires authenticated management peers and -either an explicit submission or the workspace's enabled Auto join policy. The Cluster -tab shows a separate Candidate Nodes list below its left-side cluster view. -Moving a candidate into a cluster opens SSH access and rack selection. Before -bootstrap this updates only the visibly uncommitted draft; afterward admission -requires a committed Group-0 operation. - -All admitted nodes support mutual SSH access using Ed25519 keys. Each node -creates and persists its own `id_ed25519` private key; only `id_ed25519.pub` -is exchanged into the other nodes' `authorized_keys`. Private keys are never -copied between nodes or published in Group 0. Joining uses username/password -once when working key access has not already been provisioned. Verify SSH -host identity and its association with the candidate monitor before exchanging -keys. Install existing members' public keys on the candidate and its public -key on every existing member, then verify bidirectional key access before -confirming admission. Failed setup remains visibly pending and resumes the -same admission operation without duplicating keys. Pending nodes cannot receive -general deployments. Cancellation removes authorization entries installed by -that operation without removing unrelated keys. - -The multi-node Docker test profile provides an SSH login account with default -username/password `crowdb` / `crowdb`, prefilled by its UI. Production credentials -are supplied explicitly. Initialization passwords are discarded after key -setup and are not persisted in Group 0. Node keys, `authorized_keys` and SSH -host keys survive container recreation in persistent storage. Runtime service -accounts need not acquire an interactive shell merely to provide this login. -Every management entry point uses the locally provisioned SSH identity; Group 0 -stores only nonsecret connection metadata and credential references. Mutual -SSH access grants each node the configured login user's access to its peers. - -## 5. Node and rack configuration - -Before bootstrap, an operator can inspect nodes, create/name virtual racks and -assign nodes. Node-provided rack hints can be overridden. The UI labels these -values as an uncommitted bootstrap draft, not shared authoritative metadata. -Drafts in different UIs can differ; consensus-backed UI consistency begins only -after Group 0 and topology publication are ready. - -The accepted bootstrap manifest captures the chosen initial node/rack mapping. -After bootstrap, all rack edits and node admissions are committed through -Group 0. Rediscovery cannot overwrite operator-confirmed rack placement. - -Connection and rack edits authenticate the currently discovered UUID over SSH. -`NodeRecord` keeps its discovery UUID, numeric ID, admission operation and physical -host identity. `node_update` persists fixed source/target inputs in Group 0 before -changing hardware, refreshes peer RPC endpoints without changing voting flags, -then publishes the updated mapping. A retry from another UI resumes those same -inputs. Moving a node with disk groups is rejected until those groups are removed; -this prevents losing its hardware children during relocation. - -Physical host/failure-domain information remains independently inspectable. -Several test containers may occupy different virtual racks while sharing one -physical failure domain. Placement policy must not count those labels as -independent physical redundancy. - -## 6. Bootstrap and recovery - -### 6.1 State transitions - -The node's management lifecycle is: - -- **Unbound:** identity exists, discovery is active, no cluster is accepted. -- **Prepared:** a cluster identity and initial manifest are durably reserved; - the monitor accepts retries of that manifest and rejects conflicting ones. -- **Group-0 starting:** selected monitors provision/start their initial replicas - using the same fixed voting-member manifest. -- **Topology publishing:** Group 0 has usable quorum/leadership; initial hardware - and cluster metadata are being published idempotently. -- **Active:** shared topology is confirmed and general deployment is enabled. -- **Authority unavailable:** binding persists, diagnostics/discovery continue, - and new cluster-management mutations are disabled until authority recovers. - -Unavailability is not a transition back to Unbound. Clearing cluster binding -requires an explicit recovery/decommission procedure, not an automatic timeout. - -### 6.2 Initialization flow - -1. Any UI collects compatible candidates and the operator's rack assignments. -2. The operator selects initial Group-0 voting members, or Auto join selects - the explicitly selected one, three or five authenticated eligible candidates in - discovery-UUID order. Candidate - count is not voting-member count; application nodes need not all vote. -3. Submit the bootstrap identity, immutable initial manifest and initial data - group choice to the selected member with the smallest discovery UUID. Its - monitor durably accepts the request before the UI receives acceptance. -4. Start the minimum Group-0 services with the fixed manifest. Use existing - Paxos election and WAL durability, not discovery-cache membership. -5. Wait for quorum-backed readiness and publish initial metadata idempotently. -6. Persist/refresh cluster contact information for member UIs. Enable general - deployment only after initial topology is confirmed. - -The accepting monitor executes preparation, initialization, binding and optional -initial data-group creation independently of the browser and Web request. -It exposes the operation identity, fixed members, current phase and failure. -Closing a page or losing its connection does not cancel accepted work. Monitor -restart resumes interrupted work; a recorded failure requires explicit retry. -Retry preserves the accepted data-group choice when that option is omitted and -rejects changed inputs. There is no automatic initiator takeover. Cleanup rejects -an actively executing initialization. - -Monitor peer commands use the node's durable SSH key and the strict host-key -pins established during mutual admission. Console's separate TOFU store does -not substitute for those peer pins. Initialization completion is observed from -the monitor and quorum-backed publication, not inferred from HTTP acceptance. - -Bootstrap exclusion is enforced by each target KV server before creating -store 0 and covers the complete store-0/group-0 initialization sequence. The -server atomically and durably accepts the first bootstrap operation identity -and its fixed initial configuration. A conflicting operation is rejected before -creating or reusing store 0. Retries of the accepted operation resume unfinished -steps and verify existing content; an existing store 0 without group 0 is not -permission for another operation to take over. Restart retains the accepted -operation. Failure and timeout do not automatically release its ownership. -Monitor preparation and KV-server acceptance refer to the same operation and -configuration; neither UI-local state nor a generic store/group creation path -may bypass the exclusion. - -Every selected member must durably accept the same operation and configuration -before any initial replica becomes eligible for election or data publication. -Overlapping requests cannot both obtain all required acceptances. They may each -occupy part of their selected membership, in which case both remain incomplete -and the UI exposes the conflict for explicit cleanup and retry. This mechanism -guarantees per-node exclusion, not a globally unique cluster on the LAN. -Disjoint selected memberships may create independent clusters; UIs display -them separately and do not automatically merge their logs or topology. - -### 6.3 Recovery and subsequent admission - -Node workspaces persist an enabled-by-default Auto join policy. The Web backend -evaluates it without a browser connection through authenticated admission and -accepted deployment inputs. Local policy supplies a test rack and credentials; -Production requires configured rack assignments and access. Missing Production -credentials remain Needs access, without a Local fallback. Multiple bound cluster -UUIDs require explicit target selection rather than an arbitrary advertisement. - -The accepted initial voter count is one, three or five. An incomplete selected -set remains pending; an observed overfull fresh set reports conflict before acceptance. -Eligible UUID ordering selects the coordinator within that set, while monitor and -KV-server accepted-manifest fencing prevent conflicting execution. Fixed voters -never expand from later advertisements. Turning Auto join off stops new scheduling -while already accepted monitor work continues. - -After Group-0 publication, one committed default preparation intent owns the -ordinary metadata group, replica mapping and shared access credentials. The -replica mapping covers every fixed initial voting member; each receives its own -initial storage binding. Authorized workers resume that intent across -Web/coordinator loss. Each monitor interleaves -disk preparation with dependent service startup; DiskIO may start before Online -because its ownership proof enables Online. Explicit later service stops or group -retirement supersede historical setup and prevent automatic recreation. - -Background service recovery observes readiness once per pass, retaining the -same live child and durable launch attempt until readiness succeeds. Explicit -service commands retain their bounded initial startup wait. Local read-only -health probes run concurrently with their individual deadlines, and publication -rechecks actual owned process liveness and PID before reporting health. Slow -startup or several unavailable listeners must remain visible as unhealthy; -publishing a fresh timestamp cannot substitute for those observations. - -The versioned deployment policy is durable and independent of UI permissions. -Local and Production inputs cannot replace one another through an environment -change. Initial voter count, storage, listeners and scope are sealed before node -identity or setup effects. Retained values outrank new image defaults; corrupt, -conflicting or obsolete populated roots fail without implicit reset. - -Restart resumes accepted identity and replica WAL state. Lost multicast does -not erase stored Group-0 contacts. An interrupted bootstrap reuses its operation -identity/manifest; partial metadata publication resumes without admitting -unselected nodes or repeating destructive provisioning. - -A newly discovered node enters the candidate view. A committed admission -operation binds it to the cluster and assigns its rack. Adding a deployment -node does not automatically alter Group-0 quorum. Voting-member changes follow -the existing reconfiguration contract. - -The UI supports Group-0 member management and explicit group/store deletion -for bootstrap recovery. To consolidate independently initialized clusters, the -operator selects the retained cluster, cleans the other cluster's group 0, -store 0 and binding, then admits its cleaned nodes to the retained cluster. -Voting additions use reconfiguration and catch-up, not merging independent -Group-0 histories. Deleting a group or store alone does not silently release -bootstrap ownership or cluster binding. Cleanup must fence delayed commands -and prevent old replicas from restarting before allowing a new bootstrap; -accepted operations retain durable terminal state so stale retries cannot -recreate deleted resources. Unreachable nodes remain pending cleanup and are -not eligible for reuse. Existing committed clusters require explicit destructive -cleanup rather than cancellation of an uncommitted draft. - -The public fixed `PreparedBootstrap` manifest is a recovery record distributed to -selected monitors over authenticated SSH. It is not a global cluster registry or -an authority before consensus. KV-server durable acceptance arbitrates resource -ownership at each participant. A private Unix control socket handles local -monitor commands; the unauthenticated HTTP handshake only exposes observations. -SSH connection/authentication has a ten-second bound; bidirectional proof has a -twenty-second bound. Passwords are discarded, sessions are reused during setup, -and only public keys cross nodes. - -Each Web process sampler owns one read-only SSH connection per bound remote -node. Process observations retain their one-second cadence and three-second -deadline; connection reuse does not extend observation freshness. The sampler -checks the returned node and complete bootstrap identity on every response. -Connections are discarded after a command error, cancelled read, binding or -node configuration change, and are reauthenticated at least once per minute. -They use the provisioned node key and existing strict host-key verification, -without password fallback or sharing a session with management mutations. - -### 6.4 Authenticated member address recovery - -Node identity survives changes to management and data addresses. Each accepted -member has a durable binding between cluster UUID, discovery UUID, numeric node -ID and its Ed25519 SSH host public key. Admission retains the canonical public host key in its Group-0 node record; -normal address/rack updates cannot replace that pin. The SSH login key authenticates a caller; -the SSH host key authenticates the peer. Both private keys remain on their owning -node's persistent volume. Address recovery reuses SSH and does not introduce a -CA, certificate service or custom signature protocol. - -Initial trust is established by the existing bootstrap/admission procedure on -the deployment's trusted management network. Local test credentials are a -convenience, not production identity verification. An mDNS announcement alone -cannot establish trust. Before bootstrap replicas start, each selected member -durably accepts the same member-to-host-key bindings alongside its fixed -manifest. Subsequent admissions and revocations use committed Group-0 authority; -participants retain the applicable bindings locally for recovery without quorum. -A pending non-voter seals the initial manifest's authenticated pins and its own -admission pin before its first KV start. This does not authorize recovery: only -confirmation distributes `ConfirmedMemberTrust`, whose registry and pins each -monitor verifies through current linearizable authority. The retained snapshot -adds contact hints while preserving the original voting/store configuration. -The independently running contact worker refreshes it when authority returns; -its cached confirmed pins permit SSH identity proofs during quorum loss. -Current authority retirement durably fences the local admission before owned -KV/storage services stop. Missing or corrupt later-member snapshots cannot -resume the member or answer a recovery identity request. Discovery cannot promote a pending admission or revive a removed member. Missing -or corrupt trust records fail closed; a new address never triggers first-use -trust or automatic key replacement for an existing identity. - -Recovery proceeds independently of Group-0 availability: - -1. mDNS/DNS-SD, a configured seed or retained accepted contact supplies a - candidate management address. Retained contacts remain probe hints when - multicast or Group 0 is unavailable. Advertised identities and addresses - never authorize a route without fresh proof. Fresh discovery hints receive - the bounded probe budget before retained addresses, so a full old-address - cache cannot suppress a member's changed IP. -2. The monitor selects an already accepted member binding and opens SSH to that - address with the expected host key, rather than trusting the IP's previous - occupant or accepting a new key. SSH proves possession of the private key. -3. Through that authenticated connection, the monitor reads the peer's durable - discovery identity and cluster/bootstrap binding. All must match the accepted - member, and the verified reply supplies that member's current endpoints. - The SSH identity command reads private retained state and verifies the owning - monitor's independent liveness endpoint. It does not enter the serialized - lifecycle executor or wait for service readiness: otherwise slow service - startup could suppress peer proofs, fence routes and prevent its own quorum - recovery. Offline roots and changed or retired identities still fail closed. -4. The local transport uses the verified endpoints for the same replica IDs in - the existing groups. It does not change votes, membership, WAL, rack placement - or bootstrap identity. Endpoint replacement invalidates old connections and - applies to subsequent reconnects; mappings use node/replica identity, not a - global old-IP-to-new-IP substitution. - Restored replicas fence old routes before their election drivers start. - Consensus requests identify their intended cluster and replica; a reachable - process at a reused address cannot contribute another replica's response. -5. Once quorum returns, normal authorized operations publish updated addresses - in Group 0. Local endpoint evidence never authorizes a management mutation. - Committed replica-to-node ownership is retained locally for subsequent - recovery. Internal service listeners and management contacts change through - conditional new service operations; later stop/delete commands win. Public - client URIs are independent of these internal addresses. - -The contact worker checks retained retirement and accepted identity before -probing peers. After authenticating recovered routes, address publication -checks current authority through those routes before optional confirmation -refreshes retained contacts. Publication and confirmation share a three-second -authority budget; SSH proofs retain their ten-second bound. Unreachable old -authority contacts must not consume the budget before recovered addresses can -be published, nor add another full wait that expires otherwise valid routes at -their fifteen-second freshness fence. Current authority still fences removed -members and conditional service operations before publication. - -This permits all members to restart with changed IPs at once. Neither stable DNS -names nor fixed IPs nor a complete peer-address list is required. Multicast must -reach the selected management scope; across discovery boundaries, seed discovery -remains necessary. Docker bridge DNS is optional and host networking uses the -same monitor discovery and SSH verification path. - -A changed key, foreign cluster/bootstrap, unknown member or conflicting live -claims is rejected and reported. An unverified advertisement cannot overwrite a -verified endpoint. Duplicate authenticated live claimants for one identity stop -automatic selection; a copied data volume is not a new node. Recovery has bounded -handshake timeouts and retains the last accepted identity on failure. Key rotation, -lost-identity replacement and removal remain explicit authority-controlled work; -they are never inferred from address changes or discovery expiry. - -### 6.5 Voting replica admission - -Node admission grants hardware/service management access; it does not add a voter. -Adding a replica uses `JoinGroupRequest` to import a snapshot from a confirmed -leader. Existing peers first register the new replica without voting rights. -The target wires existing peers and catches up through WAL until its contiguous -applied frontier reaches the leader's observed frontier. Only then is the new -replica promoted and published with store membership in Group 0. Once promotion -begins, an ambiguous response retains the caught-up replica for retry rather than -deleting a replica another peer may already count in its quorum. - -System replica join uses identified system ownership before creating store 0. -Generic join cannot bypass an accepted system bootstrap. Independent clusters -remain separate: operators explicitly clean one cluster, wait for every node's -cleanup, admit the released nodes to the retained cluster and add voting replicas. - -### 6.6 Accepted node setup and storage - -Node access review reads authenticated identity and inventory without installing -peer authorization or granting admission. The browser keeps an initial password -only in request/session memory. Final admission retains the original review -operation and sends the selected storage/services to the target monitor. - -The monitor persists an immutable `NodeSetupRequest` and sealed placement before -starting work. Browser disconnect, another Console session and monitor restart -reuse that intent. Provisional disk discovery grants no membership: only ready -Group 0 can register disks. Local default storage is one 16-GiB sparse file with -a 1-GiB reserve; Production automatic storage requires preallocation. Supplied -or diskless choices never silently fall back. -Different media classes use compatible independent placement. A chosen missing -ordinary metadata group remains waiting instead of creating an unrelated group. - -Each supplied medium has a stable operation, opened identity, canonical physical -resource and physical size. Whole-zone logical size excludes any unusable tail. -The monitor verifies every usable byte while retaining its exact descriptor and -exclusive raw-device claim through Init publication. DiskIO acquires and validates -its own descriptor before Online. Nonzero contents fail without a wipe; interrupted -scans start from zero. Already-Up replay checks its retained receipt and does not -mistake existing user bytes for a new blank medium. - -Global Group-0 physical reservations are sticky across interruption and Remove, -keyed by physical host and canonical resource. File/loop aliases share the same -resource. Unknown legacy raw lineage, unknown host or physical membership without -a backing receipt requires explicit migration. Numeric DiskGroup IDs also have -retained global placement reservations. Retry never chooses a different group or -medium after acceptance. Per-disk failures preserve successful siblings. - -Offline requires current revision/owner exclusion and I/O quiescence. Remove -retains backing identity, geometry and allocation metadata. Rejoin restores the -same disk Offline and requires validated Online. Discovery and restart never -replace missing/reassociated media or auto-online a restored path. Recovery and -evacuation remain separate operations. -Retained receipts also prevent parent deletion after Remove. Group deletion -requires an unassigned scope with no retained media, rather than silently -forgetting ownership, bindings or recovery inputs. -Public HTTP owner/binding routes assign or replace; they do not expose explicit -unassignment. A scope blocked by these records remains retained. Forgetting media -and unassigning their parents require a separate explicit operator flow; deletion -must not emulate that flow with an implicit cascade. - -Explicit service Stop/Restart/Delete supersedes old setup intent. Consecutive -process launches have a durable five-attempt budget per operation; readiness -success resets it. A bounded readiness observation that expires while the -same operation's process is alive retains that process and its attempt count. -Further observations continue startup without stopping or launching it again, -including on the fifth launch. Recovery verifies the committed local dependency -intents and their live probes before launching a dependent process. Consumers -that need the complete DiskIO topology also wait for its current authoritative -owner view, including other nodes. Waiting for these dependencies consumes no -process-launch attempt. A new explicit operation can supersede it. -Explicit cluster cleanup retires and drains accepted -setup/provisioning before deleting system metadata. Cleanup dispatches to at most -four retained nodes concurrently and saves each result; unreachable nodes stay -pending under the original confirmed cleanup operation. -Short-lived monitor control and probe commands do not create service-sized -worker pools. Service worker settings apply to the long-running node modes. - -## 7. Shared UI and management operations - -The UI presents explicit lifecycle states: - -- **Unbound draft:** local temporary topology and a separate candidate list; - SSH preparation and rack edits do not constitute committed membership. -- **Bootstrap in progress:** per-node store-0/group-0 progress and the fixed - submitted configuration. Later draft edits cannot alter that operation. - Failure offers resume of the same operation or explicit cleanup. -- **Topology publishing:** quorum exists but initial publication is incomplete; - show initialization progress and keep general service deployment disabled. -- **Active:** load confirmed Group-0 topology and replace the temporary view. - Subsequent changes use Group 0; unused drafts are never silently imported. -- **Authority unavailable:** retain the last confirmed topology with an explicit - stale/unavailable indication and diagnostics. Disable authority-dependent - operations and never return a bound node to replacement-cluster creation. -- **Multiple clusters discovered:** show distinct clusters and let the operator - select one to manage; neither independent topology nor drafts auto-merge. - -Admission with partial SSH setup is shown as joining/recovery required, with -operation progress distinct from confirmed membership. Retry installs only -missing authorization and resumes verification. Cancellation removes only -public-key authorization installed by that operation, preserving pre-existing -or concurrently required trust; failure to finish cleanup remains visible. - -All UIs invoke the same shared console operations and read confirmed state -from Group 0. Consistent reads support action confirmation; watch/refresh keeps -presentation current. A temporarily stale screen does not create a second -authority. The UI displays stale/unavailable state explicitly. - -Management operations have durable identity, target, requested action and -execution outcome. Group 0 commits the management intent/authorization; the -target monitor performs the side effect and records its result. This is not -an atomic transaction between a KV write and a runtime action. - -Explicit Restart requests each receive a new operation identity, even when -the requested service configuration is unchanged. Replaying that same identity -resumes its recorded execution; matching configuration cannot turn a later -Restart into a no-op. Repeated desired Start/Stop/Delete states may resume -the existing operation. - -Retries therefore use operation identity and local execution recovery. Crash -after runtime success but before result publication must inspect runtime state -and complete/reconcile the same operation, rather than blindly repeat it. -Image digest and deployment-instance identity distinguish the intended instance -from a reused process ID or container name. - -Changing UI hosts does not require moving cluster state. Local secrets and -runtime observations are not replicated as public topology. Credentials needed -to authorize another UI must be provisioned through the chosen trust policy. - -## 8. Authorization and partitions - -For a three-voter Group 0 partitioned two-to-one, only the two-voter side can -commit management mutations. The minority cannot use its local replica or -monitor cache as replacement authority and cannot bootstrap a new Group 0. - -The monitor checks cluster binding and the complete current service intent -through linearizable Group-0 reads before executing a management command. Each -read has a three-second bound. It checks the intent again after stopping an old -process and before starting its replacement. A superseded operation or unavailable -authority rejects execution. Recovery reads current desired intents through the -same authority path and does not execute a retained command solely from disk. - -An exact completed KV Stop/Delete can return its retained receipt while that -child remains stopped. The receipt carries the bootstrap identity under which -the operation was authorized; it must match the current accepted local binding -and complete intent, with no live replacement, cancellation or retirement. -This reports a prior effect without executing another one. It must not require -the Group-0 leader that the operation already stopped: background recovery may -finish the intent before its identical SSH request reaches the executor. -Legacy receipts without that bootstrap proof still require current authority. -Incomplete, superseded and new operations retain every authority fence above. - -Admission commands similarly match the current cluster publication and UUID, -numeric ID, operation ID and cancellation state. Local durable cancellation and -retirement markers reject delayed commands after cleanup or restart. Long-running -data ownership uses its existing expiry and fencing contract rather than deriving -an extended lease from these management checks. - -Monitors and UIs remain available for diagnostics during authority loss. -Group-0 unavailability does not by itself stop existing data services. Each -data group continues only under its own Paxos authority and read/write rules. -Operations requiring current Group-0 authority fail, including management -mutations, keep-alive updates, ownership acquisition/renewal and ownership -balancing. Failure to obtain ownership content is an authority-unavailable -result, never an empty ownership set; it cannot trigger release or reassignment. -Existing ownership remains subject to its own expiry and fencing contract; -actions requiring renewed ownership stop when that authority expires. Cached -ownership cannot extend its validity indefinitely. -Group-0 replica processes must be allowed to recover quorum even while general -deployment is gated; stopping all replicas on authority loss would prevent -recovery. - -## 9. Image and runtime deployment - -### 9.1 Common OCI image - -Publish OCI images to Docker Hub or another OCI-compatible registry. The same -image for a supported CPU architecture contains monitor/UI/server binaries and -their dependencies. Node-local state and cluster identity are external mounts. -Single-node and multi-node operation use the same image, monitor and UI. -Fresh Local startup prepares an explicitly one- or three-voter cluster through the -shared authenticated node lifecycle. The default is one voter with file-backed -test storage and zero physical host failure tolerance. Production selects explicit -credentials, resources and protected placement. Both expose editable management; -automatic preparation and node count do not determine authorization. Aggregate -status distinguishes Group-0 authority, historical setup and current full-cluster -readiness. Host helpers create runtime resources and observe this lifecycle; -they do not provision application topology through a second bootstrap path. - -Tags identify releases for humans; production plans pin immutable image digests. -Architecture-specific builds may share a multi-platform image index only when -their dependencies have been validated. - -System Docker and Pixi-supplied containerd/nerdctl manage the node container. -The monitor supervises local managed processes; external runtime lifecycle -recovers the whole node. Retained service instances restart in their release -profile's dependency order, with instance names used only to order peers of the -same service kind. Shutdown uses the reverse dependency order. -Standalone BuildKit explicitly exports and validates -OCI media types and blob digests without Docker or containerd dependencies. -The [OCI image design](design-crowdb-oci-image.md) defines shared construction, -resource verification and publication without changing cluster semantics. - -### 9.2 Production profile - -- Linux host, system Docker or rootful containerd runtime and host networking. -- Each node runs its monitor and Web UI inside a CROWDB node container. - The node container is the management boundary and controls managed services - using explicit mounted resources and local process supervision. Application service containers - do not inherit that runtime access. An external host/runtime startup mechanism - starts and recovers the node container itself; it cannot recover itself after - its own termination. Node identity, SSH identity and management state use - host-persistent mounts. -- Explicit management/data-plane network selection; discovery uses management. -- Stable disk/device identity and explicitly permitted block-device access. -- Host-persistent config, node state, server state, logs and crash artifacts. -- Explicit CPU quotas, memory limits and permitted block devices; the launcher - validates inputs before creating the container and verifies applied resources. - -Host networking removes a container bridge/NAT layer but does not itself -guarantee high performance. Disk passthrough does not itself bypass filesystem -cache: the storage engine's raw-block/direct-I/O path defines that behavior. - -### 9.3 Single-host test profile - -- One CROWDB container per simulated node on a dedicated user-defined Docker - bridge, each running its own monitor, Web UI and SSH endpoint. Starting three - node containers enables mutual discovery without pre-entering peer addresses; - opening any published UI exposes the candidate list and cluster preparation. - Docker availability does not require an implicit runtime socket mount: the - monitor manages local processes for this profile, while future external runtime control - requires separately configured runtime access. -- Unique persistent roots and identities; independent simulated disk files or - loop devices, never overlapping writes to one backing disk. -- Identical internal listener ports are allowed because network namespaces - differ; browser UI access uses distinct published host ports. -- Actual multicast discovery is exercised within the bridge. Separate networks - isolate independent discovery tests. - -Docker bridge behavior is described in the -[Docker reference](https://docs.docker.com/engine/network/drivers/bridge/). -The test profile measures functional correctness, not raw-disk production -performance. Linux host-network acceptance separately validates explicit resource -inputs, credentials and persistent replacement using the same image. macOS-hosted Linux containers require a VM/network -setup that is tested explicitly; image portability does not make the VM's LAN -multicast behavior equivalent to Linux host networking. - -## 10. Resource and lifecycle management - -Containers are packaging, resource/isolation and upgrade units. Deployment -plans explicitly assign host resources; monitors validate and execute, rather -than autonomously choose global placement. - -- **CPU/memory:** the production launcher validates CPU quotas and memory limits - against host capacity, then verifies the selected runtime applied them. Memory is at least - 512 MiB. Hardware observations respect container limits. CPU affinity, NUMA - and IRQ tuning remain host policy. -- **Disk:** stable device references, exclusive intended ownership and supported - engine access modes. Important data never lives only in the writable layer. -- **Network:** host ports and management/data interfaces are explicit. - RDMA device access and TCP fallback remain the responsibility of their - transport contracts, not the discovery mechanism. -- **Lifecycle:** monitor-managed services support explicit deployment, start, - stop and restart using shared operation records. Container replacement or - image upgrade is performed externally with a pinned digest and the same - persistent root; storage/protocol compatibility remains required. -- **Monitoring:** container and process health are separate observations. - Existing monitor supervision can recover eligible process exits; it must not - silently override a deliberate stop, expired authorization or upgrade drain. -- **Runtime access:** the host selects system Docker or rootful containerd. Node containers - supervise local processes and receive no Docker or containerd socket. Explicit - permitted block devices are passed through by the host launcher. - -No automatic global placement/config reconciliation is required. Monitor -detects drift against confirmed deployment inputs and reports it; local -liveness recovery follows explicit policy. Repair/upgrade progress and failure -remain visible to every authorized UI through shared operation records. - -## 11. Configuration and observability - -Local startup inputs are limited to bootstrap prerequisites and process/runtime -settings: persistent root, management interfaces, listen/advertise addresses, -optional discovery seeds, trust inputs and runtime endpoint. Group 0 owns -confirmed cluster topology and management intent. This does not move every -process tuning parameter into Group 0; respect the existing -[configuration architecture](../config/design-crowdb-config.md). - -Expose: - -- Discovery peer state, endpoint reachability, last observation/expiry, - incompatible identities and multicast-interface status. -- Bootstrap manifest identity, preparation/start/publication progress and the - reason general deployment is gated. -- Group-0 authority/renewal status, rejected stale operations and recovery needs. -- Desired deployment inputs, actual image digest/resources, process/container - health, drift and operation outcomes. - -`cluster status` reads the node's aggregate HTTP observation without a local -runtime socket. Its collector has an six-second monotonic deadline after the -bounded local monitor observation. Current linearizable Group-0 publication, -accepted ordinary metadata replicas and ordinary-group authority, physical disk -placement/bindings, live DiskDB owner leases and allocatable capacity are -independent dependencies. Automatic file storage uses its accepted initial group -reservation. Supplied storage resolves every accepted disk ID from a fresh, -identity-checked member setup observation through its unique Group-0 disk and -group membership. All resulting media-class groups must satisfy metadata -binding, current owner and space checks; a missing, duplicated or conflicting -medium stays unknown. Observation never changes sealed placement or creates the -automatic reservation for a supplied-media node. Capacity summaries must be no older than the service -registry's fifteen-second liveness window. Per-member service evidence matches -the current committed operation; required S3, Iceberg and Dataset listeners each -supply their own read-only protocol proof. Historical setup completion cannot -substitute for these observations. Missing or stale evidence remains unknown. -A bounded observation may wait for a slightly faster peer's timestamp within -its existing monotonic deadline and that evidence's freshness window, capped -at three seconds. It then evaluates the unmodified timestamps against the -actual local clock. It never treats future evidence as current, extends a -request deadline or refreshes old evidence to compensate for clock offsets. -Runtime/image facts remain unknown when runtime access was not supplied. - -Access dependency evidence comes from one backend sampler, independently of -status reads and business request outcomes. ChunkDB evidence uses one routed -point read in each of the system and user-data maintenance domains. An -authoritative absent key is healthy evidence; routing, storage, ownership and -protocol failures remain unhealthy. These observations never create probe -objects or enumerate the chunk catalog. Hardware discovery and bounded header -reads on online disks share the same overall probe deadline. Phase latency -histograms retain failed and cancelled observations and are readable without -issuing new probes. - -The aggregate route bypasses deployment recovery middleware and never reserves -preparation, restores configuration or repairs topology. Valid unhealthy status -returns success; `cluster check` returns one while readiness is unmet and zero -only for current ready evidence. Invalid or unobtainable reports return two. -Checks support bounded waiting, and JSON stdout contains only the report while -diagnostics use stderr. Explicit origins must agree with the selected cluster -UUID and name; attaching never grants runtime cleanup ownership. - -Keep local logs/crash artifacts persistent and collect asynchronously. Do not -replicate every log line into Group 0 or expose credentials in discovery/UI -diagnostics. - -## 12. Correctness and validation - -- Restart and endpoint changes preserve identity; duplicated roots cause a - visible conflict (I1). -- Three monitors discover each other on a real test bridge and LAN; expiry - affects candidates, not confirmed topology (I2). -- Multiple UIs can edit local drafts, but only the accepted manifest becomes - shared topology. General deployment stays gated through incomplete publication - (I3, I8). -- Concurrent overlapping bootstraps, partial preparation, initiator crashes - and interrupted publication recover without conflicting replicas or identity - replacement (I4). -- Partition tests distinguish reachable replicas from quorum authority and - deliver delayed/retried commands to verify stale-action rejection (I5). -- Multiple virtual racks on one host preserve one physical failure-domain - identity (I6). -- Docker bridge and Linux Docker host-network profiles use the same image - digest and exercise restart, persistent disk/state and resource boundaries - (I7). -- Existing Group-0 membership changes remain governed by reconfiguration; - deployment-node discovery never changes a voting set. - -Before adoption, measure discovery latency/traffic and stale-view bounds at the -intended LAN size. Broad ecosystem use of mDNS is not proof of unlimited -single-domain scalability. - -## 13. Supported scope - -The discovery identity and monitor placement decisions are settled: durable -UUIDs identify candidates, Group 0 owns admitted numeric IDs, and each node's -monitor/UI run in its node container. No human decisions remain open in this design. Additional runtimes, -architectures and Kubernetes orchestration remain outside this supported scope. diff --git a/doc/design/deploy/design-crowdb-oci-image.md b/doc/design/deploy/design-crowdb-oci-image.md deleted file mode 100644 index a216a22d8..000000000 --- a/doc/design/deploy/design-crowdb-oci-image.md +++ /dev/null @@ -1,374 +0,0 @@ - - - -# CROWDB - Design: OCI Image Build and Runtime - -Depends on: [deployment architecture](design-crowdb-deploy.md). -Satisfies: Docker-free, Pixi-managed Linux image construction and shared images -for Docker, containerd and future Kubernetes deployment. - -Linux amd64 construction uses Pixi-managed standalone BuildKit. Docker and -containerd consume the shared image; explicit probes and resource verification -cover both runtimes. Operational commands and tested versions are in the -[shared image guide](../../../container/oci-image/README.md). -macOS implementation and Kubernetes deployment automation are deferred. - -## Table of Contents - -- [1. Scope and invariants](#1-scope-and-invariants) -- [2. Image and base system](#2-image-and-base-system) -- [3. Pixi-managed construction](#3-pixi-managed-construction) -- [4. Artifacts and publication](#4-artifacts-and-publication) -- [5. Docker deployment](#5-docker-deployment) -- [6. Lightweight Linux deployment](#6-lightweight-linux-deployment) -- [7. macOS and Kubernetes extension points](#7-macos-and-kubernetes-extension-points) -- [8. Validation and supported boundaries](#8-validation-and-supported-boundaries) -- [9. External references](#9-external-references) - -## 1. Scope and invariants - -- **I1 — Docker-free build**: a Linux amd64 host with Pixi and documented kernel - prerequisites can compile, stage, construct and validate an OCI image without - Docker CLI, Docker daemon, Docker socket or containerd daemon. The public - shortcut selects this archive-only behavior explicitly; its default also - loads the validated artifact into local Docker. -- **I2 — One image**: for a given architecture and release, Docker and containerd - consume the same published image digest. Local and Production formation use - accepted startup policies rather than separate image builds. -- **I3 — External state**: UUIDs, SSH keys, cluster bindings, server data and logs - remain outside image layers. Replacing a runtime retains persistent state. -- **I4 — Enforced resources**: production startup fails if requested CPU - policy, memory limits or host networking cannot be applied and verified. -- **I5 — Verified publication**: publication copies the verified image without - rebuilding its root filesystem or converting its manifest media types. -- **I6 — Bounded lifecycle**: build processes have isolated sockets, owned state - and explicit cleanup. Cleanup never terminates unrelated runtime services or - deletes node data or shared image caches. - -Initial build/output platform is Linux amd64. Running these images on another -architecture requires an explicitly supported emulator or a separately verified -architecture build; an OCI index alone does not provide architecture support. - -## 2. Image and base system - -Ubuntu 24.04 remains the runtime base, pinned by digest. OCI specifies image -layout, configuration and layers; it does not require a particular distribution. -Ubuntu supplies the existing SSH, certificate, shell and system-library runtime. -Keep the current runtime dependency checks and UID/GID 10001 data ownership. - -Tini runs as PID 1 under both runtimes and reaps adopted orphan processes, -including SSH session children. It forwards signals to the monitor child, which -retains ordered service shutdown and the exit status of its own children. -The monitor does not perform an indiscriminate wait that could consume an owned -Tokio child status. This uses Tini's standard reaping/signal behavior as documented -in its [upstream implementation](https://github.com/krallin/tini). - -The image contains the same monitor, UI, Rust/C++ servers, shared libraries, -templates and entrypoint for Local and Production deployment. Fresh Local startup -defaults to one initial voter, automatic file-backed test storage and all three -access protocols. Three-container Local startup explicitly selects three voters -on one dedicated discovery scope; missing peers remain waiting. Local supplies -clearly scoped test admission credentials and reports zero physical host failure -tolerance. Production requires configured credentials, explicit resources and -valid protected placement. It never falls back to Local credentials or geometry. - -The monitor seals the versioned deployment policy before identity, admission or -storage effects. Retained accepted values supply omitted inputs on restart; -conflicting overrides or populated obsolete roots fail without conversion or -reset. Both policies expose the same editable UI and authorized management -operations. Initialization occurs through the shared durable node lifecycle, -independently of host helpers or browser connections. Image construction creates -no node identity or initialized storage. - -BuildKit's Dockerfile frontend remains usable, including existing bind mounts in -`RUN` instructions. The recipe name does not imply a Docker daemon dependency. -The final image uses OCI manifest, configuration and layer media types and -standard image annotations. Digest verification checks the actual exported -objects, not only `org.opencontainers` labels. - -Docker-specific `HEALTHCHECK` is not a portable OCI configuration field. Runtime -health checks invoke `crowdb-monitor liveness` and `crowdb-monitor readiness` -explicitly. Container restart policy and application readiness are distinct: -unready Group 0 must not trigger destructive reinitialization or restart loops. -On retained startup, the monitor validates sealed member trust and starts the -owned KV process without waiting for quorum health. Management discovery and -authenticated member identity remain available while the replica recovers its -peers' current addresses. Independent HTTP health observations determine KV -readiness; a live unready process does not consume another restart attempt. -Explicit admission/start actions retain their bounded readiness wait. -Background protocol observations obtain the local node's current address and -service intents from Group 0 within one shared authority deadline. Cluster, -node and discovery identities must match a confirmed, non-cancelled record. -The retained UI topology is not the address authority for these local probes; -status readers only consume the resulting observations. - -## 3. Pixi-managed construction - -Standalone BuildKit uses its OCI worker with `runc`; containerd is unnecessary -for construction. Pixi manages locked versions of `buildctl`, `buildkitd`, -`runc`, and the image inspection/copy tool, Skopeo. Rootless build -also needs RootlessKit and any required snapshotter helpers. Existing Rust, -CMake, Node and packaging tools continue to use the repository's locked Pixi -environment. - -A separate Linux-only Pixi workspace isolates the tools from macOS and ordinary -application development. Conda-forge supplies runc and Skopeo. Checked-in Pixi -source-package recipes supply BuildKit, RootlessKit, containerd, nerdctl, Syft -and Cosign using pinned upstream release checksums. Packages build locally; -an external project channel is unnecessary. No implicit apt/Homebrew or -unverified download bootstraps tools. - -The public build operation has three stages: - -- **Compile/stage**: on Linux, compile release artifacts with the current FFI - feature contract and assemble a source-free runtime directory. This stage - retains Cargo/CMake/npm caches and does not inspect any container daemon. -- **Construct**: start a task-owned BuildKit instance, wait for its private Unix - socket, run the Dockerfile frontend and explicitly export OCI output. Keep - cache/state separate from node data. Propagate errors and clean up the owned - daemon on success, failure and interruption. -- **Validate**: inspect layout, media types, blob hashes, platform, revision and - version; reject incomplete output. Record the image manifest/index digest - separately from archive checksum. Successful output is atomically published - to the requested local artifact path. - -Rootless construction is preferred, subject to user namespaces and snapshotter -support. Before compilation, preflight exercises the pinned RootlessKit with the -same user namespace and mount setup used by the builder and reports complete -startup diagnostics. Dependency installation on Linux amd64 and the public OCI -build entry point set up a per-executable AppArmor profile through sudo when -the namespace probe identifies the host's AppArmor restriction. Setup validates -the profile before installation, keeps checkout paths distinct, leaves global -policy unchanged and verifies namespace startup after loading it. Existing -working hosts require no privilege prompt. Noninteractive runs fail promptly -when sudo authentication is unavailable. No automatic privileged builder -fallback is allowed: a documented administrator-enabled build mode can be -selected explicitly if rootless support is unavailable. Pixi supplies userspace -tools, not kernel capabilities or privilege delegation. - -The common build component lives under `container/oci-image/`. The existing -single-node component delegates staging and image construction to it while -retaining its startup conveniences and acceptance tests. Docker import is a -separate step after Docker-free construction and validation. The public shortcut -performs that import by default and accepts an archive-only option that avoids -all Docker access. The same split supports a -future Linux-VM build backend without putting host-platform branches in the -image recipe. - -## 4. Artifacts and publication - -The canonical local artifact is an OCI image layout/archive with a build receipt: -source revision, project version, platform, base digest, tool versions, runtime -content checksum, OCI root digest and archive checksum. Runtime content and -final image are independently identifiable. Timestamps/package repositories may -affect reproducibility; this design does not promise byte-identical rebuilds. - -Reuse the existing release workflow and release-branch/version policy. It gains -separate construct, runtime verification and publish jobs: - -- Construct through Pixi without Docker access or registry publication secrets. - Upload the OCI artifact and receipt; do not rebuild from staged files later. -- Verify the same artifact on system Docker and on containerd/nerdctl. At least - one CI environment has no Docker installation; otherwise accidental dependency - on Docker cannot be excluded. Each job emits the verified OCI digest. -- Publish only after both runtime gates pass, with the existing protected - publication environment and current-release-head check. Copy the exact OCI - graph using digest-preserving transfer; failure to preserve it is an error. - Verify the registry digest before signing or updating moving tags. - -The canonical Docker Hub repository is `crowdb/crowdb-node`. The Iceberg, S3 and -Dataset repositories are usage names for the same image digest and version tags. -Repository names change neither the accepted deployment policy nor UI permissions. -All three access protocols use the ordinary node lifecycle. Existing populated -state retains its accepted policy through image replacement. - -Keep signing, SBOM and provenance. Generate provenance from the source/tool/base -receipt and attach attestations to the verified digest without rebuilding the -image. Distinguish the runnable image digest from any attestation-bearing index -digest, and verify both where an index is published. Failures leave the verified -artifact available for retry; credentials never enter layers, logs or receipts. - -## 5. Docker deployment - -Use the host's Docker installation. Pull the OCI image from the registry by -digest, then start it with the existing entrypoint/environment/mount contract. -Production uses `--network host`, explicit CPU policy and `--memory`, a persistent -data directory, physical host identity, management interface and credential -mounts. Bridge-mode single-host simulation remains available for tests. -Fresh Local one-node usage needs only the published client/UI ports and persistent -data mount. Three-node Local usage selects three voters without peer addresses or -fixed IPs. Both use automatic authenticated preparation and remain editable. -Production resource/device arguments are separate explicit host inputs. - -The standard monitor accepts private `PrepareDiskFile` requests only for its -confirmed node/cluster binding with available Group-0 authority. It records the -request before allocating a generated file asynchronously; control connections -and service heartbeat processing do not wait for preallocation. One allocation -runs at a time. `DiskFilePreparation` reports accepted, creating, prepared or -failed, including the retained backing identity. Preparation alone grants no -disk membership or usable capacity. - -File preparation checks available filesystem space including the requested host -reserve. Preallocation errors remain failures; sparse allocation must be explicit. -Retry preserves the original intent and verifies the recorded inode. Completed -files are never recreated or resized. An existing file without a durable ownership -receipt is rejected, including after an interruption between creation and receipt -publication. Cleanup is blocked while allocation runs; shutdown waits for the -blocking allocator to finish its durable record. New storage still requires -Group-0 registration and runtime availability confirmation. - -Monitor restart preserves a managed file's filesystem/device identity, inode and -logical size. A preallocated file must remain fully allocated, but its allocated -block count is an observation rather than identity: filesystem extent metadata -can grow as services write. Additional allocated metadata does not indicate a -replacement medium. - -The persistent data mount contains node identity, bootstrap/service journals and -managed disk files. Supplied files and whole block devices use the same container -disks directory in Docker and containerd, with launcher-verified exact mappings. -Replacement retains the original sources and identities; invalid inputs never -trigger managed-file fallback. Local defaults to one 16-GiB sparse managed file -and a 1-GiB host reserve. Production automatic storage uses preallocation and -rejects sparse files. File preallocation failure never authorizes sparse fallback. - -New managed files use 1-MiB allocation units and whole 256-MiB zones. For -capacities not divisible by 256 MiB, the zone size is halved until it divides -the capacity, down to the 64-MiB bitmap-word minimum. The default 16-GiB file -therefore has 64 zones, leaving non-active zones available for background -reclamation while four zones accept allocations. The accepted preparation -persists this geometry: retries and restarts never repartition existing files. -Files smaller than five minimum-size zones cannot provide a spare zone for this -background rotation; their capacity remains valid, but reclamation may wait for -allocation to rotate the full active set out. - - -Supplied provisioning seals the canonical physical host/resource and physical -capacity separately from whole-zone usable geometry. A full-file loop device -shares its backing file's canonical resource; partitions, loop slices and ambiguous -stacked/virtual lineage are rejected. The monitor reads every usable byte before -Init membership, and DiskIO verifies its own opened descriptor before Online. -Reservations survive interruption and removal. Unusable physical tails and -existing retained data are not rewritten by admission or recovery. Detailed -operator commands and candidate/rejoin flows live in the operational guide. - -Local OCI archive loading is version-dependent and must be tested separately. -The baseline production contract is registry pull by digest. If a supported -Docker version needs a Docker archive for offline loading, an explicit import -helper may convert the already built artifact for local use; it must not rebuild -or create a second published release. Converted local image IDs are not assumed -to equal the canonical OCI manifest digest. - -## 6. Lightweight Linux deployment - -The selected alternative is **containerd + nerdctl + runc**. Containerd supplies -image storage and container lifecycle, nerdctl supplies the operator CLI, and -runc executes the container. This excludes the Docker daemon but still requires -a running containerd service. Direct runc bundle management is outside scope. - -Initial production support is rootful Linux with cgroup v2 CPU/memory -controllers. `--network host` shares the host network namespace; no bridge, -NAT, CNI setup or published-port mapping is needed for this profile. Each host -normally runs one node because fixed service ports conflict on a shared host -network. Multi-node tests on one host keep their separate bridge profile. - -Pixi supplies pinned runtime binaries through a separate runtime environment; -an administrator installs/starts the host containerd service with stable binary -paths and explicit configuration. Updating a Pixi lock or cleaning a development -environment must not silently change a running production daemon. System -containerd can also be used if its version/configuration passes preflight. -Use a dedicated CROWDB namespace and never alter a Kubernetes-owned service's -configuration or containers automatically. - -A shared launcher validates digest, platform, data ownership, credentials, -physical host identity, management interface and allowed devices before choosing -Docker or nerdctl. It verifies applied cgroup `cpu.max` and `memory.max`, actual -host network namespace and device access after startup. Requested limits cannot -silently degrade into advisory values. The explicit sharing policy instead -verifies unlimited `cpu.max` and the requested `cpu.weight`: 1024 shares -represent one CPU of relative weight, not a ceiling or exclusive core. -Container E2E and CLI clusters default to this weight per container; fixture -scheduling bounds concurrent clients and hard memory totals without summing -node weights into required physical cores. Node services inherit one Tokio -worker per process in these profiles, so an unconstrained host CPU count does -not multiply idle worker pools beyond PID/memory limits. Idle CPU can be borrowed. -It records runtime/container identity -for explicit stop, restart and cleanup. Host limits are upper bounds, not -guaranteed CPU reservations. Block-device access stays explicit; no blanket -privileged container is needed. - -The common cluster CLI supports Docker and containerd host lifecycle through -these shared validators. A private receipt is sealed before runtime effects; -it records the stable daemon identity, Unix endpoint, context or namespace, -snapshotter, owner token, immutable image, resources, media lineage and input -file hashes. Resume accepts the same inputs and exact owned container, or -recreates that container against the retained data. A foreign same-name object, -changed daemon or changed configuration fails before start/stop effects. -Application UUID binding is separate and can be sealed only after observing -that cluster. Default cleanup selects stopped recorded bridge and host-network -clusters and preserves a running cluster's containers, data and restart receipts. -Explicit cluster or all-cluster teardown can stop the selected containers. -Cleanup verifies the same ownership and removes containers, unused owned bridges, -node data and matching launch/application receipts. Bridge data is held in owned -Docker volumes. Host data deletion requires the retained storage declaration to match accepted media -and rejects symlinked roots, nested mounts, overlapping recorded data directories -and bind mounts referenced by another container in the selected runtime. Supplied -disk files, raw devices and external policy/password inputs are preserved; -cleanup never wipes their contents. Retaining data and restart receipts is an -explicit cleanup option. Failures leave ownership retryable until data deletion -and target-binding removal complete. Unscoped Docker cleanup also removes stopped -containers without receipts, unreferenced named and anonymous volumes, unused -networks and unused build cache. Retained cluster resources remain protected; -non-force runtime removal rejects concurrent restarts or new references. -Containerd-only cleanup requires no Docker daemon or CLI binary. - -Explicit monitor liveness/readiness probes report node health. A dedicated -systemd unit restarts the containerd daemon, and its restart plugin applies the -node container’s `unless-stopped` policy. Monitor supervises child services. -Readiness failure alone does not restart or reinitialize the node. Persisted UUID, -SSH keys and cluster binding survive recreation under either runtime. -Rootless production networking/device/resource support is deferred; rootless -build support does not imply rootless production deployment support. - -## 7. macOS and Kubernetes extension points - -macOS retains a build backend slot: the Pixi-managed client starts/connects to a -Linux VM builder (with Lima as the provisional VM provider). Compilation, Linux library collection and -image construction all execute inside that Linux environment. Native macOS -binaries cannot be staged into the Linux image. VM CPU architecture, repository -mounts, cache, artifact transfer and lifecycle require later design/acceptance; -this change does not add macOS implementation or multi-architecture promises. - -Kubernetes uses the same OCI image digest through its CRI runtime. Later manifests -or an operator supply persistent mounts, host networking where appropriate, -resource limits, credentials and monitor probes. Discovery and physical-host -identity require explicit scheduling/network policy. Kubernetes deployment -support is not established merely by OCI image publication. - -## 8. Validation and supported boundaries - -- Verify Docker-free full build on a clean Linux host, with the builder packages - resolved entirely through Pixi and no system container binaries required. -- Verify the same OCI digest under Docker and containerd: both startup modes, - persistent identity/data, health checks, SIGTERM and recreated containers. -- Verify host networking discovery/admission, actual resource enforcement and - rejection of unsupported hosts; preserve existing SSH and Group-0 contracts. -- Verify release publication preserves tested digests and includes signatures, - SBOM/provenance; failed gates never publish moving tags. -- Rootful build and isolated Linux containerd host acceptance use the pinned tool - matrix in the operational guide. Rootless preflight reports missing host UID/GID - helpers and never silently falls back to a privileged builder. -- CI publication, remote signing and protected-environment approval are configured; - live registry acceptance requires a release workflow dispatch. -- Keep Ubuntu 24.04 and Linux amd64 initially. ARM64, macOS VM execution and - Kubernetes orchestration are separate follow-up work. - -## 9. External references - -These documents define external formats/tools, not CROWDB membership semantics: - -- [OCI Image Specification](https://specs.opencontainers.org/image-spec/). -- [BuildKit: standalone workers, OCI output and macOS client](https://github.com/moby/buildkit). -- [BuildKit rootless prerequisites](https://github.com/moby/buildkit/blob/master/docs/rootless.md). -- [nerdctl CLI and resource/network options](https://github.com/containerd/nerdctl/blob/main/docs/command-reference.md). -- [nerdctl rootless limitations](https://github.com/containerd/nerdctl/blob/main/docs/rootless.md). -- [Skopeo image inspection and transfer](https://github.com/podman-container-tools/skopeo). diff --git a/doc/design/design-crowdb-access-dataset.md b/doc/design/design-crowdb-access-dataset.md new file mode 100644 index 000000000..b0038c5db --- /dev/null +++ b/doc/design/design-crowdb-access-dataset.md @@ -0,0 +1,64 @@ + + + +# CROWDB - Design: Dataset Storage + +CROWDB's Dataset storage is built around datasets, samples, and fields. A training application asks for selected samples from a fixed dataset version, with the fields and worker assignment it needs. The storage system carries that selection down to the payload reads, so a batch does not have to begin as a walk through a directory of small files. + +Iceberg integration implements an established table format and catalog interface. CROWDB Dataset defines its own sample-oriented storage contract: immutable input, field projection, Split selection, worker partitioning, and resumable reads. It is independent of Iceberg tables and S3 buckets, and does not require a training framework's data types to become storage types. + +`crowdb-access-dataset` owns that model and its publication rules. Chunk-KV stores the metadata, while Chunk storage holds larger field payloads. Trusted training processes can use the native SDK to read directly from storage nodes into host buffers over RDMA or TCP. Direct GPU-memory delivery is future work. + +## Small samples do not need small physical files + +A file-per-sample layout makes a batch perform repeated path resolution, file metadata access, open/close work, and small payload reads. Packing samples into archives can reduce that overhead, but moves indexing, selection, and update organization into another application layer. + +CROWDB gives samples and fields identities in its distributed metadata and aggregates their bytes in shared Chunks. Small fields can remain inline. Eligible uploaded payloads use the shared small-object writer, which batches many objects across a bounded set of pipelines and active chunks. Each field keeps its own location and integrity information while allocation and physical I/O are shared. + +An independently selectable sample therefore needs no filesystem inode or exclusive physical chunk. Dataset reads resolve sample metadata and payload ranges directly. + +The benefit extends beyond file counts. Field projection can skip an image when only a label is needed, Split selection can narrow the sample set before payload I/O, and read batches can aggregate the selected work. Small-object efficiency becomes part of the storage design rather than a packaging convention every training job must maintain. + +## Immutable input for repeatable work + +A Dataset groups immutable Snapshots of samples. Splits define selected sample sets within a Snapshot, with explicit revisions so a running job can retain the selection it chose. Each sample contains named fields; the Snapshot's Manifest records those samples and fields, whose values can be bounded inline data or references to stored payload. + +For example, a sample may have an image payload, a label, and preparation metadata. A read can select the label alone without transferring the image. A subsequent training job can use the same Snapshot with a different field projection or Split. The storage model captures identity and selection; decoding an image or constructing a tensor remains application work. + +Resource preparation and head publication are separate operations. A client can prepare a complete immutable Snapshot, inspect it, and then conditionally select it as the Dataset head. Another publisher changing the head produces a conflict that the application can resolve deliberately. Preparing a new version does not silently replace the input of a running job. + +## From a read plan to selected payloads + +![Dataset read binds a Snapshot and Split, selects and projects metadata, reads inline or uploaded fields, and returns a verified batch with a continuation.](../assets/dataset-read.svg) + +A read first resolves `latest` or `stable`, if used, to an immutable Snapshot ID and binds any selected Split revision. These names are mutable pointers; subsequent batches use the resolved IDs. The read plan fixes sample selection, projected fields, ordering or deterministic shuffle, worker partitioning, and batch policy. A continuation carries that binding and the current position, allowing another server to replay the same batch. + +Workers can restart with a saved continuation without keeping a server-side read session alive. The continuation does not retain storage: deletion rejects new reads and defers physical cleanup for a grace period. Already admitted transfers may finish during that period; a long-running job must keep its input available independently of its saved position. + +Selection inspects bounded metadata windows in Chunk-KV. Split filtering precedes deterministic shuffle and worker partitioning; field projection identifies the values the application actually requested. The reader does not load a complete Manifest or fetch every field merely to build one batch. + +Field locators connect the read path to the data prepared by the write path. An inline field is already in metadata. An uploaded field resolves its immutable completed-upload description and ordered part receipts, then reads the referenced Chunk ranges or streams through the Chunk client and DiskIO. Readers check part lengths and digests, and verify the complete field before reporting success. + +Delivery remains bounded through metadata selection, payload windows, and held buffers. HTTP streams a header, payload frames, field endings, and a batch ending; successful completion also requires EOF. The native path uses the same Dataset selection and storage semantics with retained host buffers. After processing the verified batch, the application saves the next continuation. Fetching, EOF, and buffer release do not acknowledge application progress. + +## Shared payloads need explicit retention + +Dataset versions and compositions can refer to existing payloads. Reuse avoids copying the same content merely to produce another logical view, but it makes deletion a storage concern. CROWDB tracks the owners and retained resources needed to reclaim data safely. + +The same principle applies to Split revisions. Changing the current Split does not rewrite field bytes. Removing historical membership metadata and reclaiming payload storage are different operations, with different owners. This lets dataset organization evolve without turning each edit into a bulk data movement job. + +## RDMA keeps the training data path direct + +In an internal training cluster, the native SDK embeds the Dataset access library in the application. It resolves metadata and reads Chunk payloads directly from DiskIO nodes over the configured transport. With RDMA enabled, registered buffers can carry the received bytes into the application. This removes the Access Server relay and its additional network transfer. + +Registered buffers and retained ownership carry the result into the application without an extra SDK copy on eligible paths. The useful optimization is the whole chain: select fewer bytes, aggregate small payload work, transfer directly, and keep the received allocation alive until its consumer finishes. + +The current native interface delivers host buffers that can stay valid through an asynchronous host-to-device copy. It does not expose GPU pointers or direct GPU delivery. See the [native SDK design](design-crowdb-sdk-native.md#extend-the-direct-path-toward-gpu-memory) for the proposed extension. + +## Portable access uses the same Dataset model + +The [HTTP SDK](../../sdk/README.md) packages portable Rust and Python clients. Rust owns transfers, integrity checks, retries where appropriate, and read progress; Python exposes synchronous and asynchronous interfaces. + +The native Python binding is a separate distribution for trusted deployments with internal storage access. HTTP and native clients share Dataset identities, selection, and publication semantics, so choosing a transport does not require rebuilding the dataset organization. + +Training frameworks retain responsibility for tensor decoding, augmentation, and accelerator scheduling. CROWDB supplies the part that benefits from storage-level coordination: repeatable sample selection, efficient small-payload organization, and direct bounded I/O. See the [Dataset source](../../lib/crowdb-access-dataset/src) and [SDK examples](../../sdk/README.md#python). diff --git a/doc/design/design-crowdb-access-iceberg.md b/doc/design/design-crowdb-access-iceberg.md new file mode 100644 index 000000000..f8f337082 --- /dev/null +++ b/doc/design/design-crowdb-access-iceberg.md @@ -0,0 +1,58 @@ + + + +# CROWDB - Design: Native Iceberg Storage + +CROWDB stores Iceberg tables, snapshots, manifests, and files directly on its distributed storage core. `crowdb-access-iceberg` provides the REST catalog and file interface; the Chunk client and DiskIO store the bytes. Analytical engines retain query planning and execution. There is no separate S3 storage backend in this path. + +A native upload returns a signed URI that carries the file's Chunk location. Once a manifest selects that URI, the file endpoint can reconstruct the storage descriptor without another Iceberg file-metadata lookup. The URI saves a namespace round trip while preserving standard manifests and table metadata; it does not bypass the file endpoint or grant access by itself. + +## Names can change without moving table data + +Catalogs contain namespaces; namespaces contain named tables. CROWDB gives these objects stable storage identities. A table name resolves to that identity and its published table metadata. + +A rename changes the name mapping while preserving the table's storage identity. File references do not need to follow a renamed directory across physical disks. This keeps administrative changes proportional to metadata work and protects the distinction between a readable name and the owner of stored data. + +Table metadata records snapshots; manifest lists and manifests identify the files that belong to a snapshot. Canonical Iceberg metadata defines the recoverable table state. Derived indexes can accelerate access, but do not become a second definition of the table. CROWDB can therefore change physical placement without changing the table format an engine reads. + +## From an engine write to a published table + +![Native Iceberg upload writes Chunk storage, returns a location-bearing URI, and publishes standard table metadata through a table-head CAS.](../assets/iceberg-commit.svg) + +The native loader streams Parquet bytes through the Iceberg upload interface into Chunk storage. Completion verifies the content and seals an immutable, owner-bound directory root. CROWDB signs that root descriptor into the returned URI and publishes the file record. The record supports ownership, listing, and cleanup; the URI already contains the information needed to reconstruct the read descriptor. + +The loader uses the returned URI as the file reference in standard manifests, then stores the manifest list and submits the table update to the REST catalog. The catalog validates the requirements, prepares immutable metadata JSON, and publishes the new table head with one compare-and-swap (CAS) in Chunk-KV. The head identifies the current table metadata. File transfers proceed independently; the atomic table decision stays small. + +S3-compatible FileIO requests terminate at CROWDB's Iceberg file interface. They reach native Chunk storage directly and retain Iceberg file ownership and lifecycle. The familiar protocol gives engines an integration point without inserting a separate object-storage service between the catalog and its files. + +## Read the location already in the URI + +![After table and manifest selection, the direct URI supplies an authenticated Chunk root; file reads bypass Iceberg file metadata lookup and stream through Chunk client and DiskIO.](../assets/iceberg-read.svg) + +A reader first selects a table snapshot and follows its manifest list and manifests. For a native direct file URI, the locator contains the file identity, length, integrity information, and Chunk directory root. The file endpoint verifies the request's grant and the locator signature, reconstructs the descriptor locally, and reads the requested byte range through the Chunk client and DiskIO. + +This is a useful optimization at the scale of an analytical scan. A URI selected from a manifest is already a storage reference, so opening each data file does not require another Iceberg KV query to discover its location or state. Large files keep their extent directory in Chunk storage, allowing a bounded locator and bounded range traversal even when the file spans many extents. + +The locator identifies stable storage, while the Chunk layer resolves the current replicated or erasure-coded layout. Replica repair and physical placement can therefore evolve without rewriting every manifest reference. The locator signature protects the descriptor's integrity; a separate request grant authorizes access. The endpoint also probes the owner-bound directory root, preventing an old URI from reading a different file after storage reuse. + +Ordinary SDK writers that choose a logical filename before upload keep that filename and resolve its file record on read. The direct-URI optimization applies to the native upload path, which returns the final URI before manifests are constructed. Both paths use CROWDB's own storage. + +## File ownership makes cleanup part of the design + +File cleanup uses table ownership and retained metadata to decide what can be reclaimed. That connection matters after interrupted writes and table retirement: the storage system can associate unfinished physical work with the table and file that created it. Cleanup still has to respect readers, retention, and authorized operations. + +## Small files share physical storage work + +Iceberg mixes large data files with small manifests, metadata documents, and incremental outputs. CROWDB gives them different physical treatment while preserving their independent file identities. Small metadata can stay inline in distributed KV storage. Eligible chunk-backed writes use shared small-object pipelines, allowing many files to share Chunk allocation and I/O; large writes use dedicated pipelines. + +This avoids making a small file pay for its own physical storage container. Batching and shared chunks reduce per-file allocation overhead, while manifests continue to describe ordinary immutable files. The table model remains convenient for engines, and the storage layer can aggregate work at a much larger granularity. + +## Why integrate the catalog and storage? + +Some deployments compose an independent catalog with an object store. CROWDB brings table publication and file storage under one design, making the returned URI, durable completion, and cleanup agree at their source. Direct locators are one concrete payoff: the write path prepares the information that makes later reads cheaper. + +Small control records can remain in distributed KV storage while larger immutable metadata and file content use Chunk storage. Catalog operations enforce metadata size and preparation limits; direct locators keep file-open work independent of a separate file-record lookup. + +The [access acceptance guide](../../app/crowdb-access-server/tests/README.md) and [ecosystem matrix](../../e2e-test/clients/ecosystem/README.md) describe the supported client and engine combinations. + +Implementation entry points are the [direct locator codec](../../lib/crowdb-access-iceberg/src/file/direct_location.rs), [file read handler](../../app/crowdb-access-server/src/iceberg/file_http/read.rs), and [table commit](../../lib/crowdb-access-iceberg/src/commit/direct.rs). The [architecture](design-crowdb-architecture.md) shows how metadata and file traffic reach the shared storage services. diff --git a/doc/design/design-crowdb-access-server.md b/doc/design/design-crowdb-access-server.md new file mode 100644 index 000000000..c7fb0d630 --- /dev/null +++ b/doc/design/design-crowdb-access-server.md @@ -0,0 +1,44 @@ + + + +# CROWDB - Design: Access Server and S3 + +Access Server hosts CROWDB's Iceberg, Dataset, and S3 network interfaces. It supplies the access libraries' runtime dependencies and manages listeners and process shutdown. The libraries implement storage operations and publication rules; the server's protocol adapters authenticate and translate network requests. + +This arrangement keeps the public interface close to the underlying storage. The S3 library can stream directly into the Chunk client and publish object metadata through Chunk-KV. Iceberg and Dataset use the same infrastructure with their own publication rules. + +## Buckets and objects + +`crowdb-access-s3` organizes objects by bucket and key. Each object record holds its metadata and the reference to its stored payload. Physical disk topology remains below the S3 interface. + +An upload prepares the payload, then publishes the complete object record under its bucket and key. Overwriting replaces that record; subsequent lookups find the new object data. Listing operates on the object namespace without scanning disks. + +## S3 I/O follows two direct paths + +![PUT streams bytes through Chunk client and DiskIO before publishing Chunk-KV metadata; GET resolves object metadata and streams the selected bytes back.](../assets/s3-publication.svg) + +**PUT:** the S3 library validates the request, resolves the bucket, and streams the body through the Chunk client to DiskIO. Once the payload completes and its checks pass, it publishes the object's locations, length, ETag, and attributes in Chunk-KV. The success response follows publication. Large transfers use dedicated writers; small writes share the Chunk client's aggregation pipelines. + +**GET / Range GET:** the S3 library resolves the object record by bucket and key, selects its stored locations and requested range, then streams checked payload frames from the Chunk client through the HTTP response. A delivered prefix does not establish success for the complete requested range. HEAD and listing use metadata without fetching the object payload. + +The S3 layer connects directly to the metadata and payload services. Namespace lookups remain small, bulk bytes go to the disk-serving nodes, and backpressure follows a slow consumer through the pipeline. Memory is governed by active buffers and outstanding work, so object size does not become a requirement to buffer the entire transfer. + +## Multipart upload keeps transfer and publication separate + +Multipart upload allows independent part transfers before the final object is selected. Complete resolves the chosen parts and publishes their combined object; abort and cleanup handle unfinished work. An uploaded part is useful preparation, but it does not make a completed object visible on its own. + +`crowdb-access-multipart` supplies shared multipart mechanisms. The S3 library retains the meaning of an S3 Complete request and its response. Iceberg and Dataset can reuse transfer machinery without inheriting S3 namespace or overwrite semantics. + +## Compatibility and the deployment boundary + +The S3 interface covers core bucket and object operations, streaming reads and writes, range access, copy, listing, deletion, and multipart transfer. It supports signed requests and presigned access. S3 object versioning is not supported. Unsupported operations return protocol errors; the interface is not the entire AWS S3 product. + +The configured listener currently shares a tenant namespace among its accepted credentials. Signature verification authenticates requests; per-user IAM policies and bucket ACL isolation are outside that interface. Deployments must choose their exposure and trust boundary accordingly. + +Standard clients are a major advantage of S3 access. The [container examples](../../container/single-node-container/README.md#tested-s3-client-recipes) show a concrete entry point, and the [access acceptance guide](../../app/crowdb-access-server/tests/README.md) describes the client and engine checks. HTTP remains the portable application boundary; trusted applications with direct service access can use the [native SDK](design-crowdb-sdk-native.md). + +## Why keep access libraries separate from the host? + +The host manages network adapters, runtime resources, and listener lifecycle. The libraries manage storage semantics. A shared host can run several interfaces without requiring table commits or Dataset reads to pass through the S3 object API. The Dataset and S3 libraries also serve trusted applications through the native SDK. + +This creates more explicit interfaces inside the codebase, but it preserves freedom where workloads differ most: authentication, naming, publication, and read selection. See the [host source](../../app/crowdb-access-server/src), the separate [Iceberg](design-crowdb-access-iceberg.md) and [Dataset](design-crowdb-access-dataset.md) designs, and the [S3 library](../../lib/crowdb-access-s3/src). diff --git a/doc/design/design-crowdb-architecture.md b/doc/design/design-crowdb-architecture.md new file mode 100644 index 000000000..c84420040 --- /dev/null +++ b/doc/design/design-crowdb-architecture.md @@ -0,0 +1,91 @@ + + + +# CROWDB - Design: Architecture + +CROWDB separates application metadata, payload placement, and physical I/O so each can scale and recover at the right granularity. Its access libraries connect those services into complete table, dataset, and object operations. The architecture follows the decisions a storage system must make: what data means, who may change it, where its bytes live, and how those bytes survive. + +## The component map + +![CROWDB access, metadata, payload, and control components.](../assets/crowdb-architecture.svg) + +Iceberg, Dataset, and S3 form one access layer with separate access libraries. Range metadata sits above the Chunk client: Tree pages and ChunkStream journals use its shared storage path, while access libraries can reach the Chunk client directly for payload I/O. Solid arrows show the main calls and data paths; dashed arrows show replicated metadata dependencies. + +The Rust projects own service coordination and application behavior. C++ implements the Tree engine, DiskIO server, and RPC engine, with scoped FFI interfaces exposing those capabilities to Rust. `crowdb-protocol` supplies shared identifiers and wire types; `crowdb-common` supplies common utilities. Their types support the owning services without taking over their decisions. + +## Keep each decision close to the work it controls + +Composing independent storage systems can make the same request pass through several namespaces, location indexes, and forwarding services. CROWDB's layers divide responsibilities inside one storage platform. The access library owns the table, sample, or object; Chunk-KV indexes application metadata; ChunkDB and DiskDB own physical placement and allocation; DiskIO moves the bytes. + +That division keeps coordination out of the bulk-data path once a location is known. An Iceberg direct URI supplies its native file descriptor. A Dataset batch selects fields before fetching payloads. A trusted native client sends I/O directly to storage nodes. Each operation uses the metadata it needs without making every byte traverse every service. + +Shared storage also separates serving capacity from storage capacity. A range can split or change owner while retaining its page backing, so a new metadata server can take on work without first receiving a private copy of the complete index. Adding disks and relocating physical Chunks remain separate operations. This lets growth in metadata traffic and growth in stored bytes be addressed at their respective layers. + +## Metadata and control + +[Paxos-KV](design-crowdb-paxos-kv.md) maintains replicated key-value (KV) state with Multi-Paxos, a write-ahead log (WAL), and the Tree engine. Group 0 is the system group, containing metadata such as service registrations and ownership information. Nonzero groups hold other replicated KV state, including storage-service metadata. Payload I/O reaches DiskIO through the Chunk client independently of this metadata path. + +Its write path runs multiple slots concurrently. Each can reach a durable quorum independently, and the Tree engine preserves per-key slot ordering when application completes out of order. Independent blind updates therefore use concurrency through replication and storage, while reads and recovery retain their required progress boundaries. + +[Chunk-KV](design-crowdb-chunk-kv.md) serves range partitions whose pages and journals live in shared Chunk storage. Its server manages the catalog, partition ownership, and range changes; its client routes requests to the appropriate server. A new owner can recover a partition from shared storage instead of requiring the old serving process to remain alive. + +The two models address different costs. Replicated control state supports coordination and bootstrap. Shared storage supports larger range metadata whose serving placement can change independently of its durable backing. Bootstrap must establish the control services needed to discover and operate the storage services; the design does not require Group 0 to bootstrap from the Chunk-KV service it will later coordinate. + +[Tree](design-crowdb-tree.md) supplies ordered storage to both models. It materializes the work selected by its caller without adding a separate mutation redo WAL. Each KV layer owns its journal, while Tree reports the durable coverage of its persisted state. Recovery combines that state with the required journal suffix; retention follows the caller's recovery obligations. The engine and its external log are designed together, avoiding repeated mutation logging inside Tree. + +[ChunkStream](design-crowdb-chunk-stream.md) supplies the durable byte stream used by Chunk-KV's journal. Ordered KV state, append streams, and Chunk payload storage are reusable foundations below the access models. A future stream interface could build on the same append, rollover, recovery, and trim machinery, with its own public API and record semantics. + +## Payload placement and movement + +Four projects divide the physical storage work: + +- [ChunkDB](design-crowdb-chunkdb.md) owns Chunk metadata, placement, protection layouts, and lifecycle transitions. +- [Chunk client](design-crowdb-chunkdb.md#two-write-paths) runs the payload pipeline, including buffering, encoding, streaming, and reads. +- [DiskDB](design-crowdb-diskdb.md) allocates and accounts for physical disk space through durable disk and zone records. +- [DiskIO](design-crowdb-diskio.md) executes byte reads, writes, and synchronization on the addressed storage devices. + +The split keeps placement decisions out of every data packet. Once a client has the required location and authority, it transfers payload to DiskIO. ChunkDB remains responsible for the layout's lifecycle, while DiskDB remains responsible for the underlying allocation. A table rename therefore has no reason to rewrite a storage block, and a physical repair has no reason to reinterpret an Iceberg commit. + +Logical key order also stays independent of physical byte placement. A prefix can occupy adjacent metadata ranges while its referenced payloads are spread across many nodes and disks. Range partitioning keeps listing efficient; Chunk placement distributes capacity, I/O, and protection. The storage layout can change without destroying the namespace order that clients use to find their data. + +## Representative operations through the architecture + +### Committing an Iceberg table + +An engine writes immutable files through the Iceberg file interface into Chunk storage. Native upload returns a signed URI containing the Chunk directory root; the loader places that URI in standard manifests. The catalog checks the table update's requirements and publishes the new head with one compare-and-swap (CAS). After snapshot and manifest selection, the authenticated file endpoint reconstructs the storage descriptor from a direct URI without another Iceberg file-record query. + +Catalog and FileIO coordinate identity, publication, small-file placement, and cleanup within CROWDB. This direct URI avoids a metadata lookup; the file endpoint still serves the HTTP transfer. See [Iceberg](design-crowdb-access-iceberg.md). + +### Reading a Dataset batch + +The access library binds a Snapshot and Split, selects sample metadata, and projects the requested fields. Inline values are already available; uploaded fields resolve to Chunk ranges. The Chunk client reads only those payloads, preserving bounded buffers through delivery. + +The native path runs that access-library work inside the training process and reads from storage nodes over RDMA or TCP. It returns retained host buffers and eliminates the Access Server relay. Sample identities and shared Chunk storage also avoid requiring one filesystem file per sample. See [Dataset](design-crowdb-access-dataset.md) and the [native SDK](design-crowdb-sdk-native.md). + +### Publishing an S3 object + +The access service authenticates the request and resolves the bucket and key. The Chunk client streams the body through bounded buffers to its storage layout. After the payload satisfies the required completion conditions, the S3 library publishes the object record under its bucket and key. Subsequent reads resolve that record and stream the object payload. + +Payload preparation can overlap across uploads. Publication remains a small metadata decision. An incomplete upload never becomes a complete object merely because some disk writes finished. The same separation makes multipart upload useful: parts can arrive independently, while Complete selects the object that readers will see. + +### Changing a Chunk-KV partition owner + +A candidate server opens a pinned shared Tree base and catches up through the WAL while the source continues serving. Once preparation is sufficient, fencing fixes the final source boundary and a higher epoch transfers authority through the catalog. The target completes the remaining catch-up, and clients follow the updated routing. + +Shared backing removes a full private range copy from the transfer's critical path. Local splits use the same advantage to create smaller serving ranges before balancing moves their owners. This makes the ordered range model useful for growing namespaces whose listing semantics should stay stable as placement changes. + +## Resource control reaches across the stack + +Large objects and training batches make buffer lifetime a system concern. The Chunk client bounds preparation, queued payloads, outstanding I/O, and erasure-coding (EC) work. RPC accounts for registered memory and transport queues; the native SDK also accounts for buffers held by the application after a read completes. These limits cover different resources and release them at their actual completion or ownership boundaries. + +For example, a training worker may finish receiving a batch while still holding it for decoding or a host-to-device copy. Those allocations remain charged until release. When capacity is occupied, later reads wait or reach their deadline, and backpressure limits further production. The design follows memory through useful application work, making sustained concurrency manageable even when the consumer is slower than the network. + +RDMA is the core high-performance RPC path; TCP provides the compatible fallback. Registered buffer ownership can continue through eligible storage and native-consumer handoffs. Encoding, reconstruction, HTTP adapters, and application conversions still have their own costs. The [RPC design](design-crowdb-rpc.md) explains how transport selection and memory ownership support this path. + +## Deployment and recovery + +An OCI image packages each node without requiring a particular container runtime or orchestrator. The recommended performance-oriented deployment uses host networking and host block devices; the runtime controls CPU and memory through cgroups. Persistent data and node identity remain outside the replaceable image. See [container deployment](design-crowdb-deployment.md). + +`crowdb-monitor` discovers nodes and supervises local processes. Group 0 stores system metadata. Shared console operations let the Web UI and CLI act on the same management model. Process observations describe what is running; authoritative records determine what may run and serve. + +Recovery follows those ownership boundaries. Consensus restores agreed metadata; partition recovery restores range state; Chunk repair restores physical protection; the monitor restores processes. This division makes a large failure understandable as related tasks with distinct owners. See [cluster management](design-crowdb-monitor.md). diff --git a/doc/design/design-crowdb-chunk-kv.md b/doc/design/design-crowdb-chunk-kv.md new file mode 100644 index 000000000..18686ff30 --- /dev/null +++ b/doc/design/design-crowdb-chunk-kv.md @@ -0,0 +1,68 @@ + + + +# CROWDB - Design: Chunk-KV + +Chunk-KV serves an ordered key-value index divided into range partitions. Partitions can split as the index grows and move between serving nodes. Their Tree pages and write-ahead logs (WALs) remain in shared Chunk storage, so a destination can reopen the durable state without first receiving a full private database copy. + +The design targets metadata behind table files, datasets, and object namespaces. These workloads repeatedly enumerate a prefix, resume from a key, or scan ordered records. Range partitioning keeps those operations in the primary index; split and transfer adapt the index's serving placement as demand changes. + +## Range partitioning fits a list-oriented system + +Hash distribution spreads independent point requests well, but neighboring keys need not share a server or storage order. An ordered listing can then require work across many shards and a merge of their results. Range partitioning preserves that neighborhood: a prefix maps to contiguous intervals, scans advance in key order, and clients cross partition boundaries as they progress. + +That fit matters throughout CROWDB. Object prefixes, table namespaces, and Dataset metadata all benefit from ordered traversal. A range tree lets the system build listing, pagination, and bounded scans on its primary index instead of maintaining a separate organization just to recover ordering. + +Ranges still need active management. Sequential inserts can concentrate on the newest range, and skewed access can make one partition hotter than its neighbors. Splitting creates smaller placement units; transfer distributes those units across nodes. A single hot key remains a single-key workload. CROWDB accepts this trade-off because ordered access is part of the product's core behavior, and invests in making range changes inexpensive and recoverable. + +## Ordered names do not constrain payload placement + +Range partitioning organizes the metadata keys. The referenced file, sample, or object bytes use Chunk layouts chosen independently by ChunkDB. A large object prefix can therefore retain an ordered index while its payloads occupy many nodes and disks. Logical adjacency does not require physical co-location. + +Consider objects named `photos/2026/0001`, `photos/2026/0002`, and `photos/2026/0003`. Their names are neighbors for listing, but their records may point to different Chunks with layouts spanning different storage nodes. Listing finds the next names through the ordered metadata ranges. Reading an object's body follows its locations through the Chunk client to DiskIO, without sending that body through the metadata range server. + +This gives listing and bulk I/O different ways to scale. The index routes a prefix scan through the relevant ordered ranges; payload clients reach the DiskIO nodes selected by each Chunk layout. Physical rebalance or protection changes can update those layouts without reordering the names. Metadata hot ranges still need split and transfer, but their boundaries do not force all referenced bytes through the range server or onto its disks. + +## A range-based tree across nodes + +Each partition covers a contiguous key range and owns its mutation order, Tree, and WAL. A split divides one range at a key boundary; the retained parent keeps one side and a new child owns the other. The child initially shares the parent's serving node. A later balance operation can move it to another node, so partition count and node count can evolve independently. + +![A range-based tree splits into partitions on multiple nodes; each partition stores WAL through ChunkStream and Tree pages through a Chunk backend.](../assets/chunk-kv.svg) + +`crowdb-chunk-kv` combines two storage paths. Its WAL uses [ChunkStream](design-crowdb-chunk-stream.md), which appends bytes across chunks. Its [crowdb-tree](design-crowdb-tree.md) index uses the native Chunk backend to store page data and durable roots in chunks. Recovery opens the stored Tree state and replays the required WAL suffix. Bounded admission controls queued mutations. Both paths use the shared Chunk layer, with separate responsibilities for recovery order and searchable state. + +`crowdb-chunk-kv-server` manages the catalog, serving ownership, and partition lifecycle. `crowdb-chunk-kv-client` maintains routing information and composes calls across ranges. A scan crossing a range boundary uses those routing responsibilities; it does not turn the entire cluster into one implicit transaction domain. + +## Smooth split reuses the tree already stored + +The retained parent keeps its identity, Tree, and WAL. A new child starts from a range-bounded view of a pinned shared base, with the required parent WAL suffix covering writes made during preparation. Parent reads and writes continue while the durable handoff is prepared. Local dispatch then directs each side to its proper writer, and one catalog publication replaces the old range with two adjacent serving ranges. + +This makes split primarily a change in range ownership and references. The system can reuse the immutable base, preserve the parent writer, and move the final independent materialization into background work. It avoids making a full rewrite of both halves a prerequisite for publishing the new ranges. Durable transition records let a restart resume the same split, including the writes accepted around its cutover. + +Both halves initially stay on the same node. Separating local split from later placement keeps a remote transfer out of the range-boundary change itself. Once the child is independently recoverable, balancing can choose where it should serve. + +## Transfer prepares the destination while the source serves + +A live transfer pins a shared Tree base and records a matching WAL cursor. The destination opens that exact base and catches up through the source's WAL while the source remains writable. Most preparation can therefore overlap ordinary service; the destination does not first need a private copy of every page on its own disks. + +Cutover closes the old owner's admission, drains accepted mutations, and fixes the final source cursor. A higher owner epoch and a published catalog assignment give the destination its authority. The target finishes the remaining WAL suffix before answering reads or conditional mutations. During that catch-up state, eligible unconditional writes can already enter its durable WAL; readers wait within bounded deadlines. + +Shared backing removes the full-range copy from cutover, advance catch-up reduces the remaining suffix, and explicit fencing prevents competing owners from acknowledging writes. Transfer still has a cutover interval: draining, remaining replay, and cache warming affect request latency. A shared root makes movement cheaper; it does not make it instantaneous. + +## Serving capacity and storage capacity + +Adding a metadata server supplies CPU and cache capacity; adding disks supplies storage capacity and physical I/O bandwidth. Split creates ranges that can be placed independently, and transfer lets a new server open those ranges from their shared roots and journal suffixes. Moving a range owner need not relocate its physical Chunks. + +For a namespace whose size changes little while listing and update traffic grows, a new server can take selected ranges using their existing backing. It still spends resources on replay and page reads, but its preparation does not require copying the entire persistent index. + +If the constraint is disk capacity or physical I/O, operators instead add storage resources and rebalance Chunks. Adding a range server alone cannot increase those disks' bandwidth. Keeping the two operations separate lets the deployment address the resource under pressure, while durable pages remain protected during serving changes. For a large namespace, this makes expansion less tightly coupled to the total size of its persistent index. + +## Why both Chunk-KV and Paxos-KV? + +Paxos-KV supplies replicated agreement and durable control state. Chunk-KV uses shared storage to separate a range's serving process from its durable pages and journal. These are complementary ways to pay for reliability. + +Paxos-KV stores records that manage shared storage, rather than another full copy of every Chunk-KV value. Chunk-KV's Tree pages and journal bytes live in Chunks protected by the storage layer. Its range server handles mutation ordering and index access, while replicated control metadata establishes who may serve the range. + +The trade-off is dependence on the shared Chunk services and their latency, availability, and repair behavior. CROWDB accepts that dependency because those services already protect the platform's payload and storage state. The design reuses a substantial storage investment instead of repeating it inside every metadata range. + +See the [partition implementation](../../lib/crowdb-chunk-kv/src/partition.rs), [native storage adapter](../../lib/crowdb-chunk-kv/src/partition/native.rs), and [server authority](../../app/crowdb-chunk-kv-server/src/server/authority.rs). diff --git a/doc/design/design-crowdb-chunk-stream.md b/doc/design/design-crowdb-chunk-stream.md new file mode 100644 index 000000000..47b75c699 --- /dev/null +++ b/doc/design/design-crowdb-chunk-stream.md @@ -0,0 +1,44 @@ + + + +# CROWDB - Design: ChunkStream + +ChunkStream turns finite mirrored Chunks into one durable, ordered byte stream. It provides stable stream identity, append and offset reads, writer fencing, rollover, recovery, and prefix reclamation. Journals can use that interface without implementing their own distributed physical log storage. + +`crowdb-chunk-stream` is a library over CROWDB's metadata and Chunk storage. Chunk-KV uses it for partition journals. The stream understands append ordering and byte validity; its caller understands the records stored in those bytes. + +## Logical order across physical chunks + +![Adjacent chunks cover continuous logical ranges [0, L1), [L1, L2), and [L2, tail).](../assets/chunk-stream.svg) + +A writer appends to the active tail. Adjacent chunks cover contiguous half-open logical ranges: `[0, L1)`, `[L1, L2)`, and `[L2, tail)`. One chunk's logical end is the next chunk's logical start. Rollover introduces no logical gap and does not reset the offset. Readers can request a logical range spanning several chunks without reconstructing the physical transitions themselves. + +Stream metadata records the structure needed to find and recover the tail. A Group 0 lease authorizes one writer, which checks its grant before admission and before acknowledging an append. An expired or fenced writer cannot report successful appends or publish a new tail. Recovery resolves previously submitted I/O and establishes the valid tail before a new writer continues in a fresh Chunk. + +An append succeeds after its framed bytes are synchronized to every required mirror and writer authority is rechecked. After an interruption, physical bytes alone do not prove that the complete logical append finished. Recovery validates the framed content to distinguish complete appends from an incomplete tail. + +## Journals retain their own meaning + +A Chunk-KV journal can encode mutations into the stream and replay them into Tree. ChunkStream does not interpret those mutations or decide when the partition may report a successful update. Its append result is one input to the caller's completion rules. + +The journal supplies the record format, mutation order, and the relationship between an applied mutation and a stream position. ChunkStream supplies the ordered byte range and its durable append boundary. Rollover can move later bytes into another physical Chunk while the journal continues using the same logical offset space. Record parsing and physical chunk-chain management therefore remain separate jobs. + +Likewise, old stream bytes can be reclaimed only after the owner has established that they are no longer needed. A partition may base that decision on durable Tree state and its recovery requirements. ChunkStream then manages the physical prefix cleanup. + +After a restart or owner transfer, the partition reopens its durable Tree state and reads the required journal suffix through logical stream ranges. As its durable recovery point advances and retained dependencies are released, it can authorize trimming an older prefix. Trim advances the logical lower bound first; physical reclamation releases eligible complete Chunks after retained readers and recovery dependencies no longer need them. + +## Why use shared chunks for a journal? + +A local append file is a natural choice when the journal belongs to one machine. A shared journal supports a serving process that may move: a new owner can recover the same stream through the storage services. + +Using CROWDB's Chunk layer also reuses its data protection and physical allocation. The stream needs to manage ordering and boundaries, while the underlying services manage where the bytes live. The cost is remote I/O and dependence on the metadata and storage services needed to open the stream. + +## A foundation for another access model + +A write-ahead log (WAL) is one application of a durable ordered stream. ChunkStream is currently an internal library, not a public streaming service. A future Stream API could reuse its storage machinery while adding public access, quotas, retry behavior, and delivery semantics. + +A record-oriented log adapter could build on that foundation with its own record offsets, producer identity, visibility, and retention rules. A record number would need a mapping to its byte range; a retried producer request would need an identity that distinguishes a repeat from a new append. Those meanings belong above the stream, just as mutation records belong to Chunk-KV's journal. + +The existing sequential reader captures a finite readable end. A future subscription or tail-following interface would also need waiting, cancellation, and retention behavior. ChunkStream supplies reusable storage for that work; it does not yet supply those public interfaces. + +The [stream implementation](../../lib/crowdb-chunk-stream/src) contains the authority, metadata, and append logic. Its main architectural consumer is [Chunk-KV](design-crowdb-chunk-kv.md). diff --git a/doc/design/design-crowdb-chunkdb.md b/doc/design/design-crowdb-chunkdb.md new file mode 100644 index 000000000..b8eb2dc08 --- /dev/null +++ b/doc/design/design-crowdb-chunkdb.md @@ -0,0 +1,77 @@ + + + +# CROWDB - Design: Chunk Storage + +CROWDB Chunk storage combines placement, data protection, and payload I/O. ChunkDB owns metadata, allocation coordination, and layout lifecycle. The Chunk client uses those layouts to move bytes directly through DiskIO. Together they provide dedicated large-write pipelines, shared small-write pipelines, replication or erasure coding (EC), and bounded reads. + +This puts the protection policy below Iceberg, Dataset, S3, and internal storage. Each can retain its own logical data model while reusing the same placement, repair, and capacity machinery. + +## From a Chunk to physical blocks + +A Chunk is a logical storage unit divided into strips. Each strip describes a logical byte interval and the physical segments holding its replicas or erasure-coded shards. Placement chooses those segments using the available topology and failure-domain policy; DiskDB supplies the underlying allocations. A Chunk ID identifies storage independently of the table file, Dataset field, or S3 object using it. + +`crowdb-chunkdb` manages those records and decisions. `crowdb-chunkdb-client` routes metadata operations to their owning service. `crowdb-chunk-client` performs payload I/O against the selected layout. The metadata service therefore does not need to proxy every byte of a large transfer. + +![Chunk client obtains a layout from ChunkDB, whose metadata is in Paxos-KV, then writes through DiskIO to disks on multiple nodes.](../assets/chunk-placement.svg) + +Chunk allocation follows a concrete sequence: + +1. The Chunk client requests a new layout from ChunkDB. +2. ChunkDB chooses node and disk-group targets, then asks DiskDB to allocate physical blocks. +3. ChunkDB persists the Chunk ID, strips, segment locations, and lifecycle state in the selected nonzero Paxos-KV store group. Group 0 supplies system metadata such as topology and service registrations. +4. The client receives the layout and sends payload through the target nodes' DiskIO services to their local disks. Later reads resolve the current layout through the Chunk client, using cached metadata or a refresh as needed. + +Each node can contain multiple disks. Placement combines segments from those disks according to the required mirror or EC protection and failure-domain policy. Metadata operations establish where the chunk lives; the bulk data path runs directly between the client and DiskIO. + +## Two write paths + +![Large writes use dedicated pipelines and chunks; small writes multiplex N objects over M pipelines and shared chunks, with distinct EC paths.](../assets/chunk-io.svg) + +A large write has a dedicated pipeline and an exclusive active chunk. Input is framed and buffered within bounded budgets. The pipeline writes the selected mirror or EC layout, seals the chunk, and returns its location. A write larger than one chunk rotates through additional dedicated chunks and returns their ordered locations. Those chunks are exclusive to that large-writer operation; a higher-level file or multipart upload can combine several such operations. + +This gives a sustained transfer its own progress and backpressure. Preparing the next chunk and submitting independent I/O can overlap with active work, without allocating storage or memory for the entire object in advance. + +Small writes share a bounded pool of pipelines, each with its own active chunk. The pool routes admitted objects to an available pipeline, which batches whole objects, frames their data, and writes their ranges to every required mirror. An eligible small object may cross strip boundaries within its chunk, but does not cross a chunk boundary. + +Each small object keeps its own location and completion result. A shared chunk may contain many objects, while allocation, physical writes, and chunk rotation are shared across them. The pool can adjust the number of active pipelines within configured limits. This reduces the cost of creating a separate physical container for every small payload while keeping concurrency and retained memory bounded. + +## EC follows the write path + +For a large write using EC, the strip layout supplies the data and parity widths. Data shards feed incremental parity computation and are submitted to DiskIO as they become available. At the end of a strip, the writer submits its parity shards. The next strip's data writes can overlap the previous strip's parity I/O. Chunk sealing joins the required data and parity completions before exposing the completed locations. Partial tails retain their exact valid byte length. + +The small-write path starts with mirrored strips. When automatic conversion is enabled, closed strips incrementally contribute to an EC group. The default conversion geometry is eight data strips and four parity shards, shown in the diagram. The client writes and synchronizes parity, then ChunkDB selects a healthy replica from each mirror set and publishes the EC layout. Those existing replicas become the data shards, so conversion reuses their stored bytes. Mirrors remain authoritative until publication; incomplete groups or failed foreground conversion remain mirrored and can be handled by background conversion. + +The two paths put EC at different points in the lifecycle. Large writes can establish their EC representation directly. Small writes first aggregate independently completing objects in mirrored storage, then reduce redundancy overhead across completed strips. + +## Completion stays per object + +Large-writer completion includes sealing every chunk used by that write. Shared chunks stay open across many small-object completions. Callers that immediately publish readable data use the shared writer's durable completion barrier: the chunk's acknowledged cursor must cover the returned locations. The lower-level mode without that barrier can return earlier and is insufficient on its own for immediately readable publication. + +Iceberg, Dataset, and S3 then perform their own metadata publication. Sharing physical storage does not merge their visibility or deletion rules. An open shared writer can retain buffers for repair, and reclamation must account for other objects still using the chunk. + +## Read flow + +![Locations resolve to current chunk layouts; mirror reads and direct or recovered EC reads feed verified bounded delivery.](../assets/chunk-read.svg) + +A reader maps the requested logical range through the object's locations and current ChunkDB metadata. It follows the stored strip intervals, exact valid lengths, and current physical segment identities. This allows a location to keep its meaning after a mirror-to-EC layout change. + +A mirror read fetches the needed interval from a usable replica and can try another replica when the failure permits a safe retry. A healthy EC read fetches only the data-shard intervals the request needs. If a required data shard is unavailable, the reader obtains enough surviving data and parity intervals to reconstruct that slice within its recovery budget. It does not decode the entire object merely to serve a small range. + +Frame checks, exact lengths, and layout validity govern delivery. A streaming reader returns bounded buffers as the consumer pulls; cancellation stops further production, while held buffers retain their owners. A readable prefix is useful for streaming, but complete success requires the validated end of the requested range. Repair remains a separate operation whose allocation and layout publication are owned by ChunkDB. + +## Repair is a metadata transition as well as a copy + +Rebuilding bytes is only part of repair. The system also needs to identify the work, select the current layout, publish the valid replacement, and eventually release obsolete allocations. ChunkDB owns that lifecycle. + +Tasks carry the authority needed to act on their Chunk. Routing and ownership changes must fence old workers, including workers that resume after a long pause. A completed physical copy cannot override a newer layout merely because its original task eventually finished. + +Keeping this coordination with Chunk metadata gives recovery a durable place to resume. It also lets foreground reads and writes use published layouts while background conversion or repair prepares their successors. + +## Placement and resource control + +Explicit layouts let CROWDB change physical protection and placement while retaining a file or object's logical data reference. ChunkDB durably records the exact allocations and pending conversion or repair work. Once the client has a usable layout, it transfers payload directly to DiskIO. + +Memory depends on active pipelines, queued objects, outstanding I/O, parity state, recovery scratch, and caller-held allocations. Each has an owner and a budget. Retained buffers can avoid staging copies across eligible handoffs, while framing, alignment, and EC still perform the transformations the storage format requires. Repair and conversion share real disk and network capacity with foreground traffic, so their work is budgeted as part of the same system. + +See [placement selection](../../app/crowdb-chunkdb/src/selector), [conversion](../../app/crowdb-chunkdb/src/conversion.rs), [repair](../../app/crowdb-chunkdb/src/repair.rs), the [large writer](../../lib/crowdb-chunk-client/src/writer/large_object.rs), [small writer pool](../../lib/crowdb-chunk-client/src/writer/small_pool.rs), and [reader](../../lib/crowdb-chunk-client/src/chunk/chunk_reader.rs). diff --git a/doc/design/design-crowdb-deployment.md b/doc/design/design-crowdb-deployment.md new file mode 100644 index 000000000..2a73a68a9 --- /dev/null +++ b/doc/design/design-crowdb-deployment.md @@ -0,0 +1,44 @@ + + + +# CROWDB - Design: Container Deployment + +CROWDB packages a storage node as an Open Container Initiative (OCI) image for Linux amd64. The image carries services and runtime dependencies; the host supplies the network, storage devices, and persistent node state. Operators can replace the software image while keeping node identity and data on the host. + +The recommended performance-oriented configuration uses host networking and explicitly attached host block devices. The container runtime manages CPU and memory through cgroups. Network and storage paths stay close to the host, while compute and memory consumption remain controlled. + +## An OCI image, without a required orchestrator + +`crowdb-node` is the deployable artifact. Its build can produce an OCI archive without a running Docker or containerd daemon; loading and launching that artifact are separate operations. An image digest identifies the exact software being deployed, independent of the command used to start it. + +Docker and containerd launch paths are available. CROWDB's architecture does not require either runtime or a Kubernetes control plane. An orchestration environment can manage node containers through the same image and host-resource requirements; that portability is an image and runtime contract, rather than a claim that every orchestrator integration is supplied. + +This keeps a useful operational boundary. Container tooling distributes software and establishes process resources. CROWDB manages storage topology, replica membership, partition ownership, and recovery through its own services. Moving between deployment environments does not require changing the storage model. + +## Host networking keeps the data path direct + +Host networking places service listeners on the host's network interfaces. Storage clients reach the node's advertised endpoints without a container bridge, container address translation, or container port forwarding in the payload path. RDMA also requires explicit device access and suitable memory-registration permissions; host networking alone does not enable it. TCP remains available according to transport policy. + +One node container per physical host is the natural production layout for the fixed service ports. It also makes the relationship between advertised addresses, physical topology, and the machine serving I/O easy to inspect. The launcher validates the effective host network namespace rather than assuming that the requested setting took effect. + +## Host devices keep storage outside the image + +Storage devices are attached explicitly to the node container. DiskDB manages allocation on the configured media, and DiskIO reads, writes, and synchronizes the addressed blocks. The container's writable image layer is not part of this storage path. + +Direct block-device access avoids adding a container filesystem layer between the storage engine and its media. Device permissions and mappings remain explicit deployment inputs. Disk files can also supply storage for suitable environments, including local testing; the performance-oriented deployment uses host block devices. + +Node identity and retained state live in a persistent host directory or volume, separate from the replaceable image. Recreating a container reopens the intended node and its data. Image replacement therefore does not imply reinitializing the disks or inventing a new storage identity. + +## CPU and memory are container resources + +The runtime applies the node's CPU and memory policy through cgroups. A CPU ceiling caps consumption; relative CPU shares allow the node to use otherwise idle capacity while controlling its weight under contention. A memory limit bounds the container's aggregate footprint. Launch validation checks the effective cgroup settings, so the accepted deployment describes actual limits. + +Inside that budget, CROWDB limits worker pools, registered memory, caches, and in-flight I/O. Operators must size those limits together with service overhead to fit the container's budget. Storage admission can delay or reject work under pressure; the cgroup limit bounds the whole container and can terminate processes if memory is exhausted. + +## The runtime starts the node; the monitor runs its services + +The container runtime owns creation, resource configuration, and container stop. Tini handles PID 1 duties, while `crowdb-monitor` owns the service processes and their shutdown order. Group 0 retains system metadata and the durable decisions that govern those processes. + +This separates replacing an executable environment from changing storage authority. A restarted container can recover accepted identity and deployment state; the monitor can restore the appropriate services under the existing cluster model. See [cluster management](design-crowdb-monitor.md) for discovery, readiness, and management operations. + +Container deployments have been exercised on physical Linux hosts. Build inputs, runtime options, persistent paths, and launch commands are documented in the [OCI image guide](../../container/oci-image/README.md); the [local node guide](../../container/single-node-container/README.md) covers a smaller setup. diff --git a/doc/design/design-crowdb-diskdb.md b/doc/design/design-crowdb-diskdb.md new file mode 100644 index 000000000..785f54960 --- /dev/null +++ b/doc/design/design-crowdb-diskdb.md @@ -0,0 +1,38 @@ + + + +# CROWDB - Design: Disk Allocation + +DiskDB turns disk capacity into managed allocations. It records which space is owned, which space can be reused, and how disks and zones recover their allocation state. Byte reads and writes are handled separately by DiskIO. + +This separation lets CROWDB make allocation a durable metadata operation without putting that operation in the path of every subsequent I/O. ChunkDB requests space for its layouts; payload clients then use the resulting locations to access DiskIO. + +## Durable records and working allocation state + +`crowdb-diskdb` manages disk groups across multiple nodes. A disk group belongs to one node and contains disks; a node can have several disk groups. Each disk is divided into logical zones and allocation blocks. These zones are allocator regions and do not require zoned hardware. + +Group 0 stores topology and owner bindings. Each disk group is bound to a nonzero Paxos-KV data group for its allocation records, and the client routes requests to the current DiskDB owner. Durable records establish ownership; in-memory bitmaps accelerate allocation and free-space queries and can be rebuilt from those records. + +That distinction is valuable after a restart. The server can rebuild the working representation from the recorded facts rather than treating a partially updated in-memory bitmap as the only account of disk ownership. Scanner and recovery work reconcile the durable and operational views. + +![DiskDB routes allocations across disk groups on multiple nodes, with durable records in Paxos-KV.](../assets/disk-allocation.svg) + +Freeing space first persists a free record. It does not immediately clear the allocation bitmap. Compaction matches the free record to the allocation it releases before clearing the corresponding bits for reuse. This conservative accounting prevents an old free request from releasing a newer allocation at the same address. + +## Capacity is more than a free-byte counter + +Operators need to understand disks, zones, allocated space, and the work that can return capacity. DiskDB maintains counts and observations at the appropriate levels; `crowdb-diskdb-client` routes calls and aggregates information for its consumers. + +These observations help placement and management, while the allocator remains responsible for the actual decision to grant space. A cached capacity report is useful for choosing candidates, but it cannot reserve the bytes it describes. + +Physical capacity, allocated capacity, and logical user-data size are different quantities. Replication and erasure coding add physical bytes; deferred cleanup can keep freed allocations counted as busy until compaction makes them reusable. Capacity reports describe allocator state, not simply the sum of visible object lengths. + +## Why allocation is a separate service + +Allocation changes durable ownership; reading an existing allocation normally leaves ownership unchanged. Keeping them separate lets payload clients reuse established locations without an allocation transaction for every I/O. + +DiskDB gives CROWDB a focused place for disk lifecycle, space accounting, and rebuildable allocation state. DiskIO can concentrate on efficient byte operations, while ChunkDB chooses how several allocations form a protected layout. + +The cost is a distributed dependency when obtaining or freeing space. Durable records, working allocation structures, and background recovery must agree. CROWDB accepts that coordination to keep ownership explicit and recoverable across process failures. + +See the [allocation model](../../app/crowdb-diskdb/src/model), [persistence](../../app/crowdb-diskdb/src/persistence.rs), and [recovery](../../app/crowdb-diskdb/src/recovery) code. The adjacent [DiskIO design](design-crowdb-diskio.md) covers actual I/O. diff --git a/doc/design/design-crowdb-diskio.md b/doc/design/design-crowdb-diskio.md new file mode 100644 index 000000000..c8c1e3ba4 --- /dev/null +++ b/doc/design/design-crowdb-diskio.md @@ -0,0 +1,36 @@ + + + +# CROWDB - Design: Disk I/O + +DiskIO is the node-level byte service. It accepts addressed reads, writes, and synchronization requests, validates their ranges, and submits them to the storage backend. Its narrow responsibility gives the rest of CROWDB a common physical I/O interface. + +`crowdb-diskio` is the C++ server. `crowdb-diskio-client` supplies the Rust client path. DiskDB owns allocation, and ChunkDB owns the layout that combines allocations into protected data. + +## A short path from request to device + +![DiskIO requests use RDMA or TCP, retaining buffers through server validation, device I/O, and response completion.](../assets/disk-io.svg) + +The client routes requests through `crowdb-rpc` over RDMA or TCP. RDMA uses registered memory, posted receives, and completion queues. On eligible aligned paths, the server passes registered receive storage into the disk engine without an extra payload staging copy. Reads can fill a retained registered buffer and send it back through the same transport. Alignment edges use explicit staging buffers. TCP exposes the same read, write, and synchronization operations. + +The server works with `Segment` addresses and byte ranges rather than bucket keys or table files. A Segment identifies physical storage selected by the layers above it. DiskIO validates the addressed disk and range; the access and metadata services retain naming, ownership, and publication decisions. + +At startup, the server tries io_uring when available and falls back to a blocking I/O thread pool if it cannot initialize that backend. Both expose asynchronous completion to the service. Queue depth, device behavior, and platform support still determine the useful level of concurrency. + +## Completion has a precise owner + +A read completes when the requested bytes are available to its caller. Write completion reports that the backend write finished; it does not by itself establish crash durability. A separate synchronization request completes the backend's durability barrier. Higher layers wait for the required writes and synchronization before acknowledging durable storage or publishing application metadata. + +An S3 object becomes visible through S3 metadata, for example; DiskIO cannot infer that event from a finished device write. Keeping the byte service's result focused makes it reusable across payload, journal, and page-storage users. + +Asynchronous I/O also gives buffers a lifetime beyond submission. The service and client retain the memory needed by accepted operations until the backend and transport are finished with it. Cancellation and shutdown must account for that accepted work before releasing resources. + +## Why keep this path under CROWDB's control? + +DiskIO connects device queues, RPC buffers, and the Chunk client's I/O schedule. Controlling those interfaces makes it possible to reduce unnecessary allocation and copying and to keep outstanding work bounded across the network/device boundary. + +The blocking backend keeps the service usable when io_uring is unavailable. Performance measurements must identify the selected backend and device configuration because those choices change queueing and execution costs. + +This focus comes with native-code and platform integration work. CROWDB keeps that work concentrated in the service and shared C++ utilities so the access models can remain independent of device details. + +The [server source](../../app/crowdb-diskio/src), [client source](../../lib/crowdb-diskio-client/src), and [shared C++ utilities](../../lib/crowdb-common/cpp) are the main implementation entry points. [RPC](design-crowdb-rpc.md) explains the transport side of the data path. diff --git a/doc/design/design-crowdb-monitor.md b/doc/design/design-crowdb-monitor.md new file mode 100644 index 000000000..8cdbf10fd --- /dev/null +++ b/doc/design/design-crowdb-monitor.md @@ -0,0 +1,38 @@ + + + +# CROWDB - Design: Cluster Management + +`crowdb-monitor` discovers nodes, supervises local service processes, and executes deployment work. Paxos-KV Group 0 stores the durable system metadata that governs membership, registrations, and ownership. The Web UI and CLI submit management operations through shared console logic. + +A process can be alive while it is recovering, fenced, or unable to serve its assigned data. The management design keeps those conditions visible instead of treating process existence as complete readiness. + +## Local execution and durable decisions + +Group 0 is the Paxos-KV system group. It stores system metadata, including hardware topology, service registrations, membership, and ownership. The monitor performs local work under that control model: preparing resources, starting services, observing them, and releasing resources when their lifecycle permits it. + +Discovery supplies candidates and observations. It does not by itself admit a machine, transfer ownership, or replace durable identity. This matters when a machine restarts with old local state or a previously reachable process appears again after a partition. + +Bootstrap establishes the initial control services and then the storage services that depend on them. A prepared deployment can advance as its dependencies become ready. Recovery can inspect existing state and resume the relevant work without treating every restart as a new cluster. + +## Management interfaces share operations + +`crowdb-console-shared` implements common management operations and validation. `crowdb-web` presents them through HTTP and the browser UI; `crowdb-cli` presents commands, output, and exit status. + +The interfaces can differ in interaction without inventing different meanings for a disk removal or service deployment. Business authority remains with the service that owns the underlying state. An operator's request enters that service's lifecycle rather than rewriting observed state in the UI. + +When an operation stalls, operators can inspect its requested state, durable ownership, observed process, and readiness separately. A running process alone does not prove that recovery or an ownership transfer has completed. + +## Service lifecycle inside a node + +The monitor owns the service processes inside a node container: it starts them, observes readiness, coordinates shutdown, and restores the appropriate processes after restart. Each storage service recovers its own data and validates its serving authority. The container runtime establishes the node's CPU and memory budget and attaches host resources. + +Retaining data and accepted identity across restart is essential. A recreated container reopens the intended node state; a fresh image does not authorize initialization over existing storage. The [container deployment design](design-crowdb-deployment.md) explains OCI packaging, runtime independence, host networking, block devices, and resource limits. + +## Why integrate management with storage? + +Deployment changes who can serve data and which physical resources exist. Keeping these operations connected to durable metadata makes it possible to recover their intent after a worker or management process fails. It also prevents an old process observation from becoming a new ownership decision. + +The cost is coordination between persistent state and external effects such as launching a process or attaching storage. Those effects still need observation, retry handling, and cleanup. CROWDB gives that work explicit owners instead of leaving each command to improvise recovery. + +Implementation entry points are the [monitor](../../container/crowdb-monitor/src), [shared console operations](../../lib/crowdb-console-shared/src), [Web application](../../app/crowdb-web), and [CLI](../../app/crowdb-cli). The [E2E guide](../../e2e-test/README.md) describes deployment and restart validation scenarios. diff --git a/doc/design/design-crowdb-paxos-kv.md b/doc/design/design-crowdb-paxos-kv.md new file mode 100644 index 000000000..12c5ccbb3 --- /dev/null +++ b/doc/design/design-crowdb-paxos-kv.md @@ -0,0 +1,46 @@ + + + +# CROWDB - Design: Paxos-KV + +Paxos-KV replicates key-value state through concurrent Paxos slots. A slot is one position in the group's logical operation order; a value becomes chosen when a quorum accepts it. Multiple slots can reach durable agreement independently, while Tree uses their slot numbers to preserve the correct value when engine application finishes out of order. Unrelated metadata updates can therefore progress without waiting for every earlier slot to complete. + +`crowdb-paxos-kv` implements replication and the KV state machine. `crowdb-paxos-kv-server` hosts groups and their lifecycle. `crowdb-paxos-kv-client` routes application requests and handles service discovery. Group 0 is the system group and stores system metadata. + +## Independent slots keep writes moving + +The leader assigns slots in order and runs multiple proposals concurrently. Each slot is a separate Paxos decision with its own quorum progress. If slot N is delayed, N+1 and N+2 can become chosen without first filling that gap. A bounded in-flight window keeps that parallelism within the capacity of the WAL, network, and engine; recovery has its own path for resolving missing slots. + +This is particularly useful for blind Put and Delete operations: writes that do not first test the previous value. Their effect is determined by the key and assigned slot, so the engine can apply chosen work out of order. Reads select the highest eligible slot for a key, including deletion tombstones. Two updates to different keys need not share a serial apply queue, and a late older update cannot replace a newer value. + +Durability remains part of the write path: accepted values are durable on a quorum before the write is acknowledged. The synchronous path also waits for local apply; the asynchronous apply mode can acknowledge a chosen write while owned engine work continues. Linearizable reads enforce the appropriate apply fence before returning data. + + + +## How this differs from an ordered-log write path + +In Raft's replicated state-machine model, commitment advances through a contiguous log prefix and commands are applied in log order. Batching and pipelining improve replication throughput while retaining that ordering model. See the [Raft paper, sections 5.3 and 9.3](https://raft.github.io/raft.pdf). + +CROWDB uses independent Paxos slots together with an engine that understands their ordering. Later blind writes can reach their own completion point while an earlier slot is still being resolved. This reduces head-of-line waiting where the operations do not require each other's results, and lets available network, WAL, and engine capacity serve useful work across many concurrent writes. + +The benefit depends on slot agreement, per-key version selection, and parallel application working together. Read-modify-write operations use additional coordination, and reads or maintenance that require a complete prefix still wait for it. + +## Stores, groups, and replicas + +A store contains multiple Paxos groups. Each group has its own replica set and consensus state; each replica maintains a write-ahead log (WAL) and Tree state. Replicas run on different nodes; one node can host replicas for several groups, and each group can elect a different leader. Clients route by store and group before choosing the appropriate replica. + +![A store contains multiple groups, each replicated across nodes with replica-local WAL, crowdb-tree, and local disk storage.](../assets/paxos-kv.svg) + +Each replica writes its WAL and `crowdb-tree` state to local disks. Replication protects the group across nodes. Group 0 stores system metadata, while nonzero groups hold other KV data, including the metadata owned by storage services. A group remains a consensus and ordering boundary; application work spanning groups needs explicit coordination above it. + +## Durable control state for the rest of the system + +The WAL records the information needed to recover replication and accepted work. Tree materializes selected operations without adding another mutation redo WAL. Its durable snapshot coverage tells the KV layer which applied state is recoverable from the engine; the retained consensus history supplies the required suffix. This avoids duplicate engine logging while coordinating persistence and recovery through explicit progress. The [Tree recovery design](design-crowdb-tree.md#recovery-is-part-of-the-engine-contract) explains that shared contract. + +The stages can advance at different rates: replica WALs establish durable quorum acceptance, parallel apply updates the ordered index, and a Tree checkpoint persists the materialized state. Paxos-KV retains the history needed between those stages. A write can therefore be recoverable before its values appear in a durable Tree root. + +Group 0 relies on this recovery path for service registration, ownership, membership, and deployment decisions. Those decisions must survive the process that made them. A restarted monitor or service can recover them from durable metadata instead of reconstructing them from whatever happens to be running. + +Recovery uses the same slot ordering as normal operation. Gap repair restores unresolved decisions; contiguous applied progress records which prefix is complete; snapshot-backed retention determines which history may be reclaimed. Those mechanisms preserve recovery information while the active write path exploits independent progress. + +The [learner](../../lib/crowdb-paxos-kv/src/paxos/learner.rs) and [Tree adapter](../../lib/crowdb-paxos-kv/src/kv/crowdb_tree_engine.rs) show the connection. [Chunk-KV](design-crowdb-chunk-kv.md) explains the complementary shared-storage model. diff --git a/doc/design/design-crowdb-rpc.md b/doc/design/design-crowdb-rpc.md new file mode 100644 index 000000000..94c2345a8 --- /dev/null +++ b/doc/design/design-crowdb-rpc.md @@ -0,0 +1,52 @@ + + + +# CROWDB - Design: Storage RPC + +`crowdb-rpc` carries internal requests and payloads between storage services and native applications. Its remote direct memory access (RDMA) backend uses registered buffers and asynchronous completion to reduce staging and copying. TCP supports ordinary networks and can serve as a connection-setup fallback when deployment policy allows it. + +A C++ engine supplies connections, framing, workers, and both transport backends; `crowdb-rpc-ffi` exposes them to Rust. Services use the same request and payload model on either path. Deployment policy enables RDMA and controls whether setup may fall back to TCP. + +## RDMA carries the storage data path + +The RDMA backend manages registered memory, posted receives, work requests, and completion queues. Payload views retain their allocations until the hardware and subsequent consumers have finished with them. Eligible buffers can keep their registration through DiskIO, Chunk reads, and native SDK delivery, reducing repeated allocation, registration, and intermediate copying. + +The value appears across the complete I/O path. A DiskIO read can fill a registered buffer for transport; a native consumer can retain the received allocation while processing its bytes. A direct native client also removes the Access Server payload relay. CROWDB can therefore optimize both the network transfer and the handoffs on either side of it. The [DiskIO design](design-crowdb-diskio.md#a-short-path-from-request-to-device) and [native SDK](design-crowdb-sdk-native.md) describe those paths. + + + +## Control fields and payload buffers stay separate + +A control message describes the operation. Payload views describe the bytes it transfers and retain their owners. The transport can keep those allocations alive until asynchronous send or receive work completes, without forcing every caller to serialize the payload into a new message buffer first. + +Request correlation and transport errors belong to RPC. The service handler defines business success, retry behavior, and durability. A successful send cannot establish that an object was published or a KV mutation committed. + +This division lets DiskIO, metadata services, and native consumers share the transport while keeping their completion rules explicit. + +## Backpressure follows the resources still in use + +Storage resource use continues beyond network transfer. A payload may be queued for DiskIO, retained during a write, involved in EC work, or held by the application after delivery. The RPC engine bounds registered memory and transport work; the Chunk pipeline and native runtime apply their own admission limits to the work and allocations they own. + +Consider a burst of writes that arrives faster than the disks can complete it. Receiving another payload may be easy for the network while its memory remains occupied behind earlier device operations. Transport flow control sees network progress; storage admission must also see queued and in-flight storage work. Bounding both requests and bytes matters because a small control operation and a large payload consume very different amounts of memory. + +When downstream capacity is occupied, those limits stop or delay further production. A slow disk or consumer therefore becomes visible as backpressure, rather than an invitation to accumulate more payloads. Separate budgets account for request concurrency, bytes, and processing resources; a count of completed RPCs alone cannot describe the system's available capacity. + +The receive pool, send queue, storage pipeline, and consumer each release capacity at a different point. Sharing a payload allocation preserves its owner across those stages; it does not make their completion events interchangeable. Registered memory stays occupied while any stage still retains it. + +Ownership also survives cancellation. If a write times out after device submission, the caller can stop waiting while the device still references the buffer. The retained owner keeps that allocation valid until the outstanding work finishes. On reads, the [native SDK](design-crowdb-sdk-native.md#completed-reads-still-consume-capacity) continues to account for application-held results after transport completion. + +## TCP provides a practical fallback + +TCP keeps the same storage services usable on ordinary hosts and networks. Its backend uses event-driven workers with epoll or kqueue, handling framing, send and receive progress, and connection shutdown. A deployment can select TCP explicitly; an RDMA-enabled client can use it when capability negotiation or connection setup cannot establish a supported RDMA path and fallback is allowed. + +Performance-sensitive deployments can disable fallback to require RDMA. This makes the chosen transport an explicit operational policy: a workload that requires RDMA fails setup visibly instead of silently running on another path. + +Fallback is a connection-setup decision. It does not switch an accepted RDMA operation to TCP after a transfer error. An uncertain outcome is reported to the caller, and the transport neither appends a TCP suffix to partial data nor transparently replays the operation. The service's request semantics determine whether a subsequent retry is safe. + +## Why own an RPC engine? + +A storage RPC engine can make registered memory and asynchronous I/O part of its basic interface. Control fields remain small, payload buffers keep their owners, and completion releases resources at the right stage. RDMA receives the design attention required by the fast path, while TCP preserves deployment flexibility through the same service boundary. + +Internal RPC operates within the cluster's trust boundary; externally authenticated applications use the access interfaces. Public protocol compatibility remains with those interfaces. + +See the [transport interface](../../lib/crowdb-rpc/include/crowdb-rpc/transport.h), [buffer interface](../../lib/crowdb-rpc/include/crowdb-rpc/buffer.h), and [FFI](../../lib/crowdb-rpc/ffi). The [native SDK](design-crowdb-sdk-native.md) shows how retained transport allocations become application-visible results. diff --git a/doc/design/design-crowdb-sdk-native.md b/doc/design/design-crowdb-sdk-native.md new file mode 100644 index 000000000..fb0e5225b --- /dev/null +++ b/doc/design/design-crowdb-sdk-native.md @@ -0,0 +1,64 @@ + + + +# CROWDB - Design: Native SDK + +`crowdb-sdk-native` lets trusted applications embed the Dataset and S3 access libraries and communicate directly with storage services. Bulk data travels between DiskIO nodes and the application over RDMA or TCP, without an Access Server relay. The SDK exposes retained host buffers; direct GPU-memory delivery is future work. + +The objective is efficiency across the whole system: fewer network hops, less protocol work, fewer staging buffers, and less CPU spent forwarding bytes. RDMA is most valuable when the architecture also removes the intermediate process that would otherwise receive, wrap, and retransmit the payload. + +## Make storage part of the application runtime + +An HTTP access path sends storage-node payloads through Access Server before they reach the application. The native runtime executes the access-library work inside the application process and uses the Chunk client and DiskIO connections directly. This removes the storage-node-to-gateway-to-application relay from bulk I/O. + +The same Dataset Snapshot, Split, and field-selection rules still apply. S3 operations still resolve bucket and key and use the owning library's publication rules. Metadata services retain their authority; the application gains a shorter data path rather than a second interpretation of stored data. + +The native SDK currently covers Dataset and S3; Iceberg's native upload interface is a separate integration described in the [Iceberg design](design-crowdb-access-iceberg.md). + +## One runtime for many operations + +`ClientRuntime` owns or borrows the underlying service clients. Dataset and S3 handles share its connection groups and resource budgets. Topology observations help locate services and data; current routing and serving authority still determine whether a request may execute. + +Sharing matters under concurrency. Creating independent connection pools and memory limits for every object or worker handle would multiply both resources and coordination. A common runtime gives the application one place to account for active requests, producer buffers, and held results. + +Operations carry deadlines and cancellation. Shutdown closes admission and drains owned work. Buffers already handed to an application retain their own lifetime, so closing a stream or runtime does not invalidate data that the caller still holds. + +## RDMA needs buffer ownership all the way to the consumer + +Read streams return `BufferLease` values: immutable views that retain the backing allocation. A consumer can process a view, pass it to another asynchronous operation, or retain it until a host-to-device copy finishes. Resource accounting follows the allocation until its final owner releases it. A lease contains a piece of the result; only the stream's verified end establishes complete-read success. + +Eligible registered receive buffers reach the caller without a second SDK payload copy. Producers can also fill native allocations and submit immutable views. Buffer ownership survives asynchronous I/O and application processing, so a fast transport does not immediately lose its benefit to another copy at the SDK boundary. + +Copies still have reasons to exist: ordinary input may need framing, erasure coding creates or reconstructs fragments, and language adapters may materialize their own values. For example, the native Python binding's `to_bytes()` explicitly creates Python-owned bytes. The benefit comes from making those conversions deliberate and keeping ownership intact between them. + +## Completed reads still consume capacity + +A training worker can finish a read and keep its buffers while decoding samples or waiting for a host-to-device transfer. The SDK continues to charge those allocations against the caller-held capacity budget until their final owner releases them. Completing the RPC or closing the stream does not erase that memory use. + +For example, prefetching a batch while decoding the previous one keeps both batches charged. Sharing or slicing a view retains the same allocation and charges its capacity once, even if a small slice keeps a much larger buffer alive. Releasing the final owner makes that capacity available to another read. + +When held capacity is exhausted, later reads wait for release or return a deadline or cancellation result. Producer allocations and active requests have their own limits, alongside the transport's registered-memory and queue budgets. Sharing a runtime makes Dataset and S3 handles share the appropriate SDK accounts, so adding handles does not silently multiply the application's allowance. + +Request concurrency controls how much work is active; held capacity controls how much delivered data remains in the application. A workload can have few active requests and still hold substantial memory from earlier reads, so those limits serve different purposes. Producer memory has a similar lifetime concern: shared small-write machinery can retain submitted owners for open-strip repair after an individual PUT completes. + +The held-buffer limit covers SDK-managed allocations after delivery. Copies into application-owned arrays, decoded objects, or Python `bytes` need the application's own memory policy. Retained-buffer accounting is not a limit on the process's total memory. + +## Transport policy follows the deployment + +The runtime selects transport policy for metadata and data connections. RDMA uses configured devices, registered memory, and bounded transport resources. TCP supplies the portable path for deployments without that hardware. Applications keep the same access semantics and retained-buffer model in either case. + +An operation that has already transferred data cannot be treated as a fresh attempt merely because another transport is available. The SDK preserves errors and uncertain mutation outcomes so the application can recover with the original identity and publication conditions. It does not turn an ambiguous write into an automatic new write. + +## Extend the direct path toward GPU memory + +A future GPU integration could extend the direct storage path into device memory and remove host staging. RDMA transport and retained host buffers provide useful foundations, but direct GPU delivery needs its own registration, ownership, and completion support. + +Today, applications must retain the host lease until an asynchronous host-to-device copy finishes. Neither GPU pointers nor GPUDirect Storage are part of the current SDK interface. + +## The deployment boundary is explicit + +The embedding application needs access to internal management and storage services. It has the same internal trust as an access service; this API does not perform S3 SigV4 authentication. HTTP remains the appropriate boundary for portable clients and externally authenticated access. + +Native integration brings native libraries and, for RDMA, host device and registration requirements. It fits an internal compute/storage cluster whose operators control both ends of the connection. + +The [native SDK README](../../lib/crowdb-sdk-native/README.md) contains Rust examples and runtime options. The [native Python README](../../lib/crowdb-sdk-native-python/README.md) describes packaging and its explicit copy boundary. The separate [Dataset HTTP SDK](../../sdk/README.md) provides the portable path. diff --git a/doc/design/design-crowdb-tree.md b/doc/design/design-crowdb-tree.md new file mode 100644 index 000000000..fe914ddbb --- /dev/null +++ b/doc/design/design-crowdb-tree.md @@ -0,0 +1,64 @@ + + + +# CROWDB - Design: Tree Storage Engine + +`crowdb-tree` combines concurrent skiplist write buffers, immutable pages behind a mapping table, and slot-aware version selection in one ordered key-value engine. Flush distributes work across independent subtrees and joins their results toward the root. A checkpoint persists those results to local disks or shared Chunk storage. + +These choices connect foreground concurrency, ordered reads, and background persistence. Paxos-KV uses the engine on each replica's local disks; Chunk-KV uses its Chunk backend to keep range state independent of the serving node. CROWDB can optimize the same engine for both storage models. + +## Concurrent skiplists feed a versioned page tree + +![Concurrent skiplist tables use CAS; a Bw-tree mapping table resolves immutable pages, with ordinary and persistent flush modes and local-disk or Chunk backends.](../assets/tree.svg) + +Each write buffer, or MemTable, is an ordered concurrent skiplist. Compare-and-swap (CAS) publishes new nodes and immutable version sets, while epoch protection delays reclamation until readers are finished. Multiple writers can update independent keys without putting every insertion behind a shared MemTable mutex. The skiplist also gives flush jobs ordered range cursors without sorting the entire input batch again. + +The page tree uses mapping indirection: a logical page ID resolves to an immutable page version through a versioned mapping table. This follows the indirection principle described in the [Bw-tree paper](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/bw-tree-icde2013-final.pdf); CROWDB combines it with its own MemTables, ordered overlays, and checkpoint flow. A page's overlays hold updates above its base data. Readers retain a consistent mapping and source view while writers prepare newer state, and a split can share base ranges without immediately copying every value. + +This keeps several expensive operations off the ordinary update path. Inserting a value need not rewrite a disk page; publishing a new page need not invalidate a reader's view; moving entries into the tree can transfer or share their existing allocation owners. Consolidation combines overlays and adjusts page boundaries when the structure needs it. + +## Logical versions survive out-of-order application + +Each key retains slot information and the versions needed by active read and recovery boundaries. A slot records the operation's logical order, supplied by the caller. An older update that finishes after a newer one cannot overwrite it merely by arriving later. Point reads and ordered scans select the highest eligible version, including tombstones that represent deletion, from the retained sources and pages. + +That makes concurrency useful all the way through the replicated write path: consensus establishes logical order, while independent application work can finish in a different physical order. Paxos-KV builds user-facing conditional CAS operations from versioned reads and coordinated chosen writes; Tree's own CAS operations publish its concurrent in-memory state. + +## Fork flush down the tree, join results toward the root + +![A fixed set of MemTables feeds root-to-leaf fork tasks; leaves consolidate independently and parents join child results, with memory-only or persistent output from the same flow.](../assets/tree-flush.svg) + +The flush design captures a finite set of skiplist tables and a fixed version frontier. Starting at the root, index separators divide the work into child ranges. Each leaf receives all captured updates belonging to its range, then consolidates as soon as its own inputs are ready. Independent subtrees can advance concurrently. + +Join follows the reverse dependencies. A parent consumes its children's ordered results, resolves sibling merges and index changes, and returns its result upward. The design avoids a whole-tree handoff barrier before consolidation can begin. A ready subtree can produce its final pages while another branch is still receiving updates. + +This is where the tree structure becomes a scheduling advantage. Parallelism follows key ranges and page dependencies, with bounded CPU and I/O work. Waiting for a child or storage completion yields the worker, allowing the shared runtime to advance other ranges and other Trees. + +## Flush and flush with snapshot share the same work + +Ordinary flush publishes the resulting pages and mapping in memory. Flush with snapshot selects persistent output for that same fork/join round; this is the checkpoint path. Final pages are encoded and written as their dependencies become ready, parents join those results, and changed mapping segments are persisted. The durable root commits only after the required output and storage barriers complete. + +The design shares consolidation, split/merge decisions, and final page layout between memory maintenance and persistence. Snapshot output therefore reuses the work already needed to organize the tree. Unchanged durable pages and mapping segments keep their existing backing, while a persistent round also includes dirty state left by earlier ordinary flushes. + +Incremental persistence lets unchanged backing be reused across checkpoints. Ordinary flush and page writeback can reduce memory pressure, but neither advances the durable replay point on its own. Only a committed checkpoint reports the covered operation prefix that the caller can use in its journal retention policy. + +## Recovery is part of the engine contract + +Tree relies on its caller's recoverable journal rather than creating a separate mutation redo write-ahead log (WAL). Paxos-KV supplies consensus-selected operations from its replica-local WAL; Chunk-KV supplies ordered mutations from its partition journal. The engine materializes those operations, preserves their version information, and reports the durable progress covered by a committed checkpoint. An in-memory Tree update alone is not crash-durable. + +This removes a second recording of the same mutation inside the engine. The larger benefit is that application, persistence, and recovery share explicit progress boundaries. If a snapshot covers a prefix and later writes exist only in memory, recovery replays the required external journal suffix. A high observed slot or a completed page write cannot substitute for a committed root and its coverage. + +For example, suppose the caller has completed operations through 120, while the last durable Tree root covers 100. Operations 101–120 must remain recoverable from the caller's journal. After a crash, the caller opens that root and supplies the valid suffix to Tree. In Paxos-KV, consensus determines which operations were chosen; Tree restores their materialized state using the supplied slots. Version selection, including tombstones, prevents an older replayed update from replacing a newer value. + +Now suppose a snapshot starts persisting state through 120. Its page writes can finish at different times, but the recoverable boundary advances only when the root and its dependencies are committed. Until then, the previous root and required journal history remain the recovery path. This makes checkpoint publication the point where the caller learns that more state can be recovered from Tree itself. + +The caller uses that durable coverage together with its replication and retention obligations to decide which journal history can be released. Tree manages the lifetime of its own pages and retained views. Each part supplies the evidence needed by the other, so memory maintenance, checkpointing, and log reclamation can be optimized together without confusing their completion points. + +## One engine, local disks or shared Chunks + +The local backend writes Tree pages and durable roots to local disks. Paxos-KV combines that local engine with replicated consensus and a local WAL on each replica. + +The Chunk backend writes page data and roots through CROWDB's shared storage services. Chunk-KV can reopen the same durable backing from another serving node, reuse page state during range splits, and recover its WAL through ChunkStream. The ordered KV model above the backend stays the same. + +This combination is the reason to own the engine. CAS-based ingestion, slot-aware reads, page indirection, subtree concurrency, and incremental persistence can be designed together with the distributed services that consume them. The same investment supports fast local replicas and movable shared-storage ranges. + +See the [skiplist implementation](../../lib/crowdb-tree/src/memtable), [mapping table](../../lib/crowdb-tree/include/crowdb-tree/maptable), [C++ interface](../../lib/crowdb-tree/include/crowdb-tree/crowdb-tree.h), and [benchmark guide](../../lib/crowdb-tree/bench/README.md). diff --git a/doc/design/design-crowdb.md b/doc/design/design-crowdb.md new file mode 100644 index 000000000..f14b39a50 --- /dev/null +++ b/doc/design/design-crowdb.md @@ -0,0 +1,54 @@ + + + +# CROWDB - Design: A Shared Storage Core for Tables, Datasets, and Objects + +CROWDB brings Iceberg tables, AI datasets, and S3 objects onto one distributed storage core. Each has its own identity, publication rules, and read model. They share the machinery that places bytes, protects them, tracks ownership, and recovers after failure. + +A table commit, a training batch, and a multipart object upload need different coordination. Beneath those differences lie the same expensive operations: moving buffers, allocating storage, recording durable state, and rebuilding data. CROWDB develops these pieces together so improvements to the storage path can benefit all three workloads. + +## The workloads that shape the design + +An analytical engine works with tables, snapshots, manifests, and files, using atomic commits to publish changes. It needs a catalog that understands table identity and a file path that can deliver large data files efficiently. CROWDB provides a native Iceberg catalog and storage integration; the analytical engine retains query planning and execution. + +A training application works with selected samples and fields from a fixed dataset version. It needs workers to divide the input, resume progress, and avoid fetching unused payloads. CROWDB organizes datasets into immutable Snapshots, with Splits selecting samples and read continuations recording progress. + +An object application wants a familiar bucket and key, streaming upload, range reads, and predictable behavior when a request fails. S3 compatibility lets existing tools use those operations without learning a new storage API. + +These workloads combine large sequential transfers with many small metadata operations. Disk bandwidth alone cannot remove catalog contention, buffer copies, or recovery work. CROWDB addresses each cost in the component that controls it. + +## Shared infrastructure, independent data semantics + +![Iceberg, Dataset, and S3 share distributed storage services.](../assets/crowdb-overview.svg) + +Iceberg controls when table metadata becomes the current table state. Dataset controls which immutable Snapshot a read selects. S3 controls when an object becomes visible under its bucket and key. Sharing storage does not turn these into aliases for the same namespace. That distinction gives each access model room to improve without forcing its rules onto the others. + +## The design advantages + +**Native Iceberg removes a storage-system relay.** Table files live directly in Chunk storage, with no separate S3 storage backend. Native uploads return signed URIs carrying the Chunk location, so a manifest-selected data file can be read without another Iceberg file-metadata lookup. Catalog publication and file lifecycle are designed together, reducing the repeated metadata work that appears when independent systems are stacked. See [Iceberg](design-crowdb-access-iceberg.md). + +**Small logical objects share large physical storage work.** Small files and field payloads retain independent identities while sharing Chunk allocation, batching, and write pipelines. Large writes get dedicated pipelines and chunks. This lets the system serve small metadata-heavy workloads and sustained bulk transfers without making every small item own a physical container. See [Chunk I/O](design-crowdb-chunkdb.md#two-write-paths). + +**Dataset storage understands samples before it reads bytes.** Snapshots, Splits, field projection, and worker assignment select the input in metadata before payload transfer. Samples do not each need a filesystem file, and an unused field does not need to enter the training pipeline. Shared chunks address small-payload organization while continuations make repeated work explicit. See [Dataset](design-crowdb-access-dataset.md). + +**Ordered names and distributed bytes work together.** Range metadata preserves the neighborhoods needed by prefix listing, pagination, and ordered scans. Chunk placement independently distributes the referenced payloads across nodes and disks. Names under one prefix can remain adjacent in the index while their data uses the storage cluster's capacity and bandwidth. See [Chunk-KV](design-crowdb-chunk-kv.md#ordered-names-do-not-constrain-payload-placement). + +**Metadata serving can grow without first copying the whole index.** Chunk-KV reuses shared Tree pages during split and prepares a transfer destination from a pinned base and WAL suffix while the source continues serving. Adding a serving node does not require a full private copy of each range it takes over. Smooth split and transfer let serving capacity respond to load independently of physical Chunk relocation. See [Chunk-KV](design-crowdb-chunk-kv.md#serving-capacity-and-storage-capacity). + +**Replicated metadata writes can complete across concurrent slots.** Paxos-KV lets each slot reach its own durable quorum, while per-key slot ordering keeps out-of-order Tree application correct. Independent blind writes can make progress before an earlier gap is resolved, reducing head-of-line waiting through the consensus and engine path. See [Paxos-KV](design-crowdb-paxos-kv.md). + +**The engine's concurrency follows its data structure.** Tree combines concurrent skiplist write buffers, slot-aware versions, and a mapping table that separates page identity from page contents. Flush distributes work across independent subtrees and joins their results toward the root; a checkpoint persists the resulting state. Local-disk and Chunk backends let Paxos-KV and Chunk-KV share that engine investment. See [Tree](design-crowdb-tree.md). + +**The journal and Tree form one recovery design.** Tree materializes work recorded by its caller without adding a separate mutation redo WAL. Slot-aware application, durable snapshot coverage, and retained journal history connect in one recovery path. This avoids duplicate mutation logging inside the engine and gives the caller the durable progress needed to decide what to replay and what history can be reclaimed. See [Tree recovery](design-crowdb-tree.md#recovery-is-part-of-the-engine-contract). + +**Native access removes a payload relay.** Trusted Dataset and S3 applications embed the access libraries and communicate with storage nodes directly, using RDMA or TCP according to deployment policy. This removes an intermediate network hop and its protocol and staging work. The SDK delivers retained host buffers today; direct GPU-memory delivery is future work. See the [native SDK](design-crowdb-sdk-native.md). + +These advantages reinforce one another. Ordered metadata selects the work, shared chunks aggregate it, direct I/O moves the bytes, and retained ownership carries them to the consumer. Resource budgets cover queued work, in-flight I/O, and buffers still held by an application, so slower consumers can apply backpressure through the path. Durable system metadata coordinates placement and recovery across it. + +## Scope and adoption + +CROWDB is a storage platform. Query execution, model training, and application transactions spanning unrelated data models belong to their respective engines and applications. Its Dataset fields hold application data; CROWDB does not prescribe tensor formats or execute a training graph. + +Access Server hosts the network interfaces. The [container deployment design](design-crowdb-deployment.md) packages a Linux amd64 node as an OCI image, with Docker and containerd launch paths. Host networking and host block devices keep the I/O path direct; container limits control CPU and memory. The [Dataset HTTP SDK](../../sdk/README.md) and [native SDK](../../lib/crowdb-sdk-native/README.md) provide separate client distributions. Use the [repository README](../../README.md#run-crowdb) for deployment entry points and compatibility boundaries. + +Continue with the [architecture](design-crowdb-architecture.md) or use the [component index](README.md) to explore a specific design. diff --git a/doc/design/diskdb/design-crowdb-diskdb-space-metrics.md b/doc/design/diskdb/design-crowdb-diskdb-space-metrics.md deleted file mode 100644 index 13d7469ff..000000000 --- a/doc/design/diskdb/design-crowdb-diskdb-space-metrics.md +++ /dev/null @@ -1,818 +0,0 @@ - - - -# CROWDB - Design: diskdb Space Metrics + Query API - -Depends on: [`design-crowdb-diskdb.md`](design-crowdb-diskdb.md) §3.4 (records are -source of truth), §7 (record model + recovery strategies), §3.7 (reuse -`crowdb-common` metrics), §8 (allocation algorithm); [`design-crowdb-kv-group0.md`](../kv/design-crowdb-kv-group0.md) -(hardware hierarchy + service registry). -Satisfies: `design-crowdb-diskdb.md` §11 (Space Metrics). - -Detailed design for diskdb's space-metrics component — the third major -component after the allocator (§8) and recovery (§7). Architecture -decisions and rationale live in the root design doc; this doc carries -the structs, algorithms, RPC shapes, and data flow. Reused surfaces -(named, not re-built): `DdbDiskGroupContainer`/`DdbDiskGroup`/`DdbDisk`, -`DdbKvClient`, `DiskdbMetrics`, `model/alloc.rs` two-phase -allocate/free, `RecoveryEngine` (`recover_zone_inner`, -`rebuild_zone_bitmap_full_scan`), `CompactionEngine`, -`HardwareClient` (hierarchy scans), `ServiceRegistryClient` -(`read_all_diskdb_instances`, `heartbeat_diskdb` accepting -`DiskGroupUsageSummary[]`), `DiskGroupUsageKey` + `prefix_all`. - -## Table of Contents - -- [1. Per-zone usage accessors](#1-per-zone-usage-accessors) -- [2. Per-disk + per-disk-group aggregation](#2-per-disk--per-disk-group-aggregation) -- [3. Per-disk hot-path counters](#3-per-disk-hot-path-counters) -- [4. QueryCapacityStats handler](#4-querycapacitystats-handler) -- [5. Recalculation path](#5-recalculation-path) -- [6. crowdb-common metrics integration](#6-crowdb-common-metrics-integration) -- [7. Reporting loop](#7-reporting-loop) -- [8. Keepalive usage piggyback](#8-keepalive-usage-piggyback) -- [9. Schema extension](#9-schema-extension) -- [10. kv-client space-usage aggregation](#10-kv-client-space-usage-aggregation) -- [11. diskdb-client — full client library](#11-diskdb-client--full-client-library) -- [12. E2E test migration](#12-e2e-test-migration) -- [Invariants](#invariants) -- [Module Structure](#module-structure) -- [Config Extensions](#config-extensions) -- [Server Wiring](#server-wiring) - -## 1. Per-zone usage accessors - -`DdbZone` tracks `used_count: AtomicU32`, `unit_capacity: u32`, and -`UsageBitmap::count_set()`, but exposes no busy/free/bytes/ratio -accessors. Every higher-level aggregation (disk, disk-group, query -handler, keepalive piggyback, reporting-loop gauges) bottoms out at the -zone, so the accessors live here. CROWDB reuses freed space immediately -via bitmap scan (no append-only `allocate_pos`, §3.4/§8), so -`free = capacity - busy`. There is no separate free cursor to read. - -```rust -/// Busy block units (live `used_count`). -pub fn busy_blocks(&self) -> u32 { self.used_count.load(Ordering::Acquire) } -/// Free block units = `unit_capacity - used_count`. -pub fn free_blocks(&self) -> u32 { self.unit_capacity.saturating_sub(self.busy_blocks()) } -/// Busy bytes = `busy_blocks * unit_size_bytes`. -pub fn busy_bytes(&self, unit_size_bytes: u32) -> u64 { u64::from(self.busy_blocks()) * u64::from(unit_size_bytes) } -/// Free bytes = `free_blocks * unit_size_bytes`. -pub fn free_bytes(&self, unit_size_bytes: u32) -> u64 { u64::from(self.free_blocks()) * u64::from(unit_size_bytes) } -/// Capacity bytes = `unit_capacity * unit_size_bytes`. -pub fn capacity_bytes(&self, unit_size_bytes: u32) -> u64 { u64::from(self.unit_capacity) * u64::from(unit_size_bytes) } -/// Usage ratio = `used_count / unit_capacity` as f64. -pub fn usage_ratio(&self) -> f64 { - let used = f64::from(self.busy_blocks()); - let cap = f64::from(self.unit_capacity); - if cap == 0.0 { 0.0 } else { used / cap } -} -``` - -- `busy_blocks` reads the live atomic; `free_blocks` derives from - capacity. `UsageBitmap::count_set()` is **not** used on the read path - — it is reserved for the recalc verifier (§5) which needs an - independent popcount of the bitmap bytes, not the cached - `used_count`. -- The `unit_size_bytes` argument is per-disk (from - `DiskValue.unit_size_bytes`), not a global shift — disks in one - disk-group may have different unit sizes in principle, so the caller - passes the owning disk's value. - -Edge cases: -- `unit_capacity == 0` (degenerate) → `usage_ratio` returns 0.0 - (guarded). -- Saturating sub on `free_blocks` guards against a transient - `used_count > unit_capacity` (should never happen; a CAS bug would - set it). -- These are read-only and lock-free; safe to call concurrently with - allocate/free. - -## 2. Per-disk + per-disk-group aggregation - -`DdbDisk` and `DdbDiskGroup` own the zone/disk collections but expose -no aggregated usage. The query handler, keepalive piggyback, and -reporting loop all need a single struct built from in-memory state (no -KV reads). The aggregation is a straightforward sum, but the struct -shape is reused in three places, so it is defined once. - -```rust -/// Per-zone brief usage (counts only — no bitmap bytes). -pub struct ZoneUsage { - pub zone_index: u32, - pub capacity_bytes: u64, - pub busy_bytes: u64, - pub free_bytes: u64, - pub busy_block_count: u32, - pub free_block_count: u32, - pub alloc_state: ZoneAllocationState, // derived_alloc_state() - pub zone_state: DdbZoneHealth, -} - -/// Per-disk usage (aggregated across zones). -pub struct DiskUsage { - pub disk_id: DiskId, - pub capacity_bytes: u64, - pub busy_bytes: u64, - pub free_bytes: u64, - pub zone_count: u32, - pub active_zone_count: u32, - pub busy_zone_count: u32, // zones with used_count == unit_capacity - pub free_zone_count: u32, // zones with used_count < unit_capacity -} - -/// Per-disk-group usage (aggregated across disks). -pub struct DiskGroupUsage { - pub disk_group_id: DiskGroupId, - pub capacity_bytes: u64, - pub busy_bytes: u64, - pub free_bytes: u64, - pub disk_count: u32, - pub allocatable_disk_count: u32, - pub disks: Vec, -} -``` - -`DdbDisk::usage(&self, unit_size_bytes: u32) -> DiskUsage`: -- Read `disk_value` + `zones` under their read locks. -- Sum `capacity_bytes`/`busy_bytes`/`free_bytes` via each zone's - accessors (§1) using the disk's `unit_size_bytes` (passed in, or read - from `disk_value.unit_size_bytes`). -- `zone_count` = `zones.len()`; `active_zone_count` = - `active_zone_context.read().len()` (RCU snapshot size). -- `busy_zone_count` = zones with `busy_blocks() == unit_capacity`; - `free_zone_count` = the rest. - -`DdbDiskGroup::aggregate_usage(&self) -> DiskGroupUsage`: -- Read `disks` under the read lock; read each disk's `disk_value` for - its `unit_size_bytes`. -- Sum across disks; `disk_count` = `disks.len()`; - `allocatable_disk_count` comes from the immutable RCU membership snapshot - shared by disk-ID lookup and allocation routing. Membership changes publish - both views together. -- Carry the per-disk `DiskUsage` breakdown in `disks`. - -`DdbDiskGroup::zone_usage(&self, disk_id: DiskId, zone_index: u32) -> Option`: -- Locate the disk in the immutable RCU membership snapshot; `None` if unknown. -- Locate the zone by `zone_index` (read lock on `disk.zones`); `None` - if out of range. -- Build `ZoneUsage` from the zone's accessors + - `derived_alloc_state()` + `*zone.zone_state.read()`. **Brief counts - only** — no bitmap bytes. - -Edge cases: -- Disk with zero zones → `DiskUsage` zeroes, `zone_count == 0`. -- A `Bad` disk is still summed (its zones carry their last-known - `used_count`); it is excluded from `allocatable_disk_count` but its - capacity still counts in the total. This matches "capacity is total; - allocatable is the live set." -- `zone_usage` for an out-of-range index → `None` (caller maps to - `NotFound`). -- Read locks make the snapshot best-effort-consistent across - zones/disks; a concurrent `disk_add_init`/`remove_disk` may miss a - transient disk this tick. Acceptable for metrics (§11: exact - consistency not required; recalc gives exact verification). - -## 3. Per-disk hot-path counters - -§11 calls for per-disk event counters (allocate/free count + bytes) on -the hot path. The capacity/busy/free **gauges** are derived from the -bitmap on the reporting tick (single source of truth). Maintaining -parallel capacity atomics would drift from the bitmap. So the per-disk -hot-path struct holds **only** period/total event counters; gauges are -computed in the reporting loop (§7). - -`metrics.rs` is a module dir: `metrics/mod.rs` (existing `DiskdbMetrics` -moves here) + `metrics/disk.rs`. `DiskMetrics` holds lock-free atomic -counters per `DdbDisk`: - -```rust -pub struct DiskMetrics { - // Period counters (swapped to 0 by the reporting loop each tick). - period_allocate_count: AtomicU64, - period_allocate_bytes: AtomicU64, - period_free_count: AtomicU64, - period_free_bytes: AtomicU64, - // Total counters (monotonic, never reset). - total_allocate_count: AtomicU64, - total_allocate_bytes: AtomicU64, - total_free_count: AtomicU64, - total_free_bytes: AtomicU64, -} - -impl DiskMetrics { - pub fn new() -> Self { /* all zero */ } - /// Bump period + total count/bytes. Called after the Phase 1 bitmap - /// CAS succeeds (exactly once per durable-bound allocation). - pub fn record_allocate(&self, unit_count: u32, unit_size_bytes: u32) { ... } - /// Bump period + total count/bytes. Called after the Phase 1 bitmap - /// clear succeeds. - pub fn record_free(&self, unit_count: u32, unit_size_bytes: u32) { ... } - /// Atomically swap period counters to 0, return the deltas. - pub fn swap_periods(&self) -> PeriodSnapshot { ... } -} - -pub struct PeriodSnapshot { - pub allocate_count: u64, pub allocate_bytes: u64, - pub free_count: u64, pub free_bytes: u64, -} -``` - -- All atomics use `Ordering::Relaxed` (counters, no cross-variable - ordering). -- `record_allocate`/`record_free` do `fetch_add(unit_count)` on count - and `fetch_add(u64::from(unit_count) * u64::from(unit_size_bytes))` - on bytes, on both period and total. -- `swap_periods` does `swap(0, Relaxed)` on each period counter; totals - are kept inline (also incremented in `record_*`) for crash-safe - monotonicity — the reporting loop does **not** add period deltas to - totals (they're already there); it only flushes period deltas into - the crowdb-common `allocate_total`/`free_total` counters. - -Wiring: a `metrics: Option>` field on `DdbDisk`, attached -during `disk_add_init` (keepalive) and `recover_disk_group` (recovery). -`None` in tests that don't care. The allocate/free paths in -`model/alloc.rs` call -`disk.metrics.as_ref().map(|m| m.record_allocate(range.unit_count, unit_size))` -after the Phase 1 CAS succeeds (before Phase 2 persist. The counter -reflects a durable-bound allocation; on Phase 2 failure the bitmap -rolls back but the counter over-counts by one, which is acceptable for a -best-effort event counter and avoids a second hot-path branch on the -persist result). - -Edge cases: -- `AtomicU64` overflow at 1M-unit blocks → extremely unlikely; - `swap_periods` saturating semantics avoid loss; no special handling. -- `DiskMetrics == None` (test disk) → record calls are no-ops; - aggregation unaffected. -- Counter over-count on Phase 2 rollback (above) — documented, - acceptable for an event counter; the `allocate_errors_total` counter - (§6) separately tracks persist failures. - -## 4. QueryCapacityStats handler - -`QueryCapacityStats` is the operator/console drill-down surface for live -per-disk/per-zone usage, the authoritative counterpart to the -kv-client's stale cluster-wide view. The request carries two drill-down -fields (`disk_id`, `zone_index`) to select three query shapes without -three separate RPCs. - -Proto (§9): `QueryCapacityStatsRequest` gains `optional uint64 disk_id = 2` -and `optional uint32 zone_index = 3` (0 = not set). `DiskInfo`/ -`DiskGroupInfo` gain usage fields + `repeated ZoneUsage zone_usages`. -`ZoneUsage` message added. - -The handler in `service/diskdb_service.rs` is read-only, allowed in any -lifecycle phase (like `GetDiskGroupInfo`); it does **not** check -`allows_mutating_rpcs` or `is_degraded`. - -- **Cluster/disk-group level** (`disk_group_id` only, `disk_id == 0`): - if `disk_group_id == 0`, iterate `container.disk_group_ids()` and - build one `DiskGroupInfo` per owned group; if set, look up that one - group (`NotFound` if not owned). Each `DiskGroupInfo` carries the - aggregated capacity/busy/free + member `DiskInfo`s with per-disk - usage + zone counts. **No `zone_usages` entries** at this level. -- **Disk level** (`disk_group_id` + `disk_id` set, `zone_index == 0`): - return the one `DiskInfo` with a `zone_usages` list of **brief** - per-zone `ZoneUsage` entries (counts only — no `usage_bitmap`). Built - via `DdbDisk::usage` + a loop over zones calling `DdbZone` accessors - + `derived_alloc_state`. `NotFound` if the disk is not in the group. -- **Zone level** (`disk_group_id` + `disk_id` + `zone_index` all set): - return the one `ZoneUsage` with the **full `usage_bitmap` bytes** via - `DdbZone::usage_bits.snapshot()`. Out-of-range `zone_index` → - `NotFound`. - -`disk_id == 0` sentinel: `DiskId` is a message (`{high, low}`); 0 means -`Some(DiskId{high:0,low:0})` is treated as "not set" only when the field -is `None`. The schema field is `disk_id: DiskId` so -"not set" = `None`; a real disk with all-zero id is pathological and -rejected at disk-add time. `zone_index == 0` is ambiguous (zone 0 is -valid), so the disk-level shape is selected by `disk_id` being set -**and** `zone_index` being absent. Use proto3 `optional` on `zone_index` -(field presence) so "not set" is distinguishable from 0. (proto3 scalar -presence requires `optional` keyword; flatbuffers generates `Option`.) - -Edge cases: -- `disk_group_id == 0` + zero owned groups → empty `disk_groups`, `Ok`. -- Unowned `disk_group_id` → `NotFound` (matches `GetDiskGroupInfo`). -- Disk with thousands of zones → disk-level returns brief entries (no - bitmap bytes); only zone-level serializes one bitmap. -- Out-of-range `zone_index` → `NotFound` (not silent empty bitmap). -- Zone with `used_count == unit_capacity` → `Full` alloc_state, - `free == 0`; reported as-is, not an error. -- Last zone smaller (word-aligned) → `unit_capacity` is the rounded - value; `capacity_bytes` reflects the real smaller capacity. - -## 5. Recalculation path - -§11 requires "accurate statistics with a recalculation path": replay -the journal into a **separate** bitmap and compare against the live -`DdbZone` to detect drift. This is the exact-verification counterpart to -the best-effort-consistent gauges. The background scanner (§12 of the -root doc) reuses `recalc_all`. v1 reports drift only (no auto-correct; -operator runs `RebuildZoneBitmap`). - -`RecalcEngine` in `metrics/recalc.rs`: - -```rust -pub struct RecalcEngine { - kv: Arc, - container: Arc, -} - -pub struct RecalcResult { - pub disk_id: DiskId, - pub zone_index: u32, - pub matches: bool, - pub live_busy_blocks: u32, - pub replayed_busy_blocks: u32, - pub live_snapshot_slot: u64, - pub replayed_snapshot_slot: u64, - pub drift_detected: bool, - pub fallback_used: Option, // None | JournalScanGcGap | SnapshotCrcFail -} - -pub enum FallbackReason { JournalScanGcGap, SnapshotCrcFail } - -pub struct DiskGroupRecalcResult { - pub disk_group_id: DiskGroupId, - pub zone_results: Vec, - pub drift_detected: bool, -} -``` - -`recalc_zone(bind, disk_id, zone_idx, unit_capacity, live_zone: &DdbZone) -> Result`: -- Call `recover_zone_inner(&self.kv, bind, disk_id, zone_idx, unit_capacity)` - (strategy 2) into a throwaway `DdbZone`. On `JournalScanGcGap` / - `SnapshotCrcFail`, fall back to `rebuild_zone_bitmap_full_scan` - (strategy 1); record `fallback_used`. -- `replayed_busy_blocks` = replayed zone's `usage_bits.count_set()` - (independent popcount, not its `used_count`). -- `live_busy_blocks` = live zone's `busy_blocks()` (its `used_count`). -- `matches = (replayed_busy_blocks == live_busy_blocks)`; - `drift_detected = !matches`. -- v1 does **not** mutate the live zone — drift is reported only. - -`recalc_disk_group(dg_id) -> Result`: iterate all -disks + zones in the disk-group, call `recalc_zone` per zone (bounded -concurrency via a semaphore, matching `recover_disk_group`), aggregate. - -`recalc_all() -> Result>`: iterate -`container.disk_group_ids()`, call `recalc_disk_group` per owned group. - -The `RecalcDiskUsage` admin RPC (§9) delegates to `RecalcEngine`; the -handler lives in `service/diskdb_service.rs` (mutating RPC, requires -`allows_mutating_rpcs`, since recalc does KV reads/journal scans and is -an admin operation, not a read-only query). - -Edge cases: -- `JournalScanGcGap` → strategy 1 fallback; - `fallback_used = Some(JournalScanGcGap)`. -- `SnapshotCrcFail` → strategy 1 fallback; the live bitmap is suspect; - `drift_detected = true` with `FallbackReason::SnapshotCrcFail` even - if counts happen to match (the snapshot was corrupt). -- Strategy 1 also fails → return a `RecalcResult` with - `matches = false`, `drift_detected = true`, and a KV error noted (do - not panic; the operator sees the failure in the response). -- Live `used_count` unchanged after recalc (v1 no auto-fix). - -## 6. crowdb-common metrics integration - -§11 specifies three metric categories (counters, gauges, latency -hierarchy) reusing `crowdb-common`'s `MetricsRegistry`. `DiskdbMetrics` in -`metrics/mod.rs` holds the full set. Gauges are derived snapshots -(bitmap-derived on the reporting tick), not hot-path writes. - -- **Gauges** (`Arc`): `disk_capacity_bytes`, `disk_busy_bytes`, - `disk_free_bytes`, `disk_active_zone_count`, `disk_total_zone_count`, - `dg_capacity_bytes`, `dg_busy_bytes`, `dg_free_bytes`, - `disk_group.imbalance.used_pct_spread`, - `disk_group.imbalance.used_pct_max`, - `disk_group.imbalance.used_pct_min`, `rebalance.plan_count`, - `rebalance.planned_blocks`, `owned_disk_group_count`, `degraded`, and - `last_sync_age_secs`. - (Per-disk/per-disk-group gauges are single instances updated each tick - with the summed value — v1 does not label per-disk-id in the registry; - the reporting loop sums across disks into the single gauge. The - console reads the live per-disk view via `QueryCapacityStats`, not the - metrics log.) -- **Counters** (`Arc`): `allocate_total`, `free_total` (flushed - from per-disk `DiskMetrics` totals by the reporting loop), - `allocate_errors_total`, `sync_success_total`, - `sync_failure_total`, `compaction_records_deleted_total`, - `rebalance.moves.total`, and `rebalance.errors.total`. -- **Latency histograms** (`Arc`, hot paths): - `allocate.rpc.latency_us`, `allocate.bitmap_scan.latency_us`, - `allocate.kv_persist.latency_us`, `free.rpc.latency_us`, - `free.persist.latency_us`, `free.kv_persist.latency_us`. -- **Latency summaries** (`Arc`, cold paths): - `allocate.zone_rotate.latency_us`, `sync.latency_us`, - `sync.read_group0.latency_us`, `sync.apply_changes.latency_us`, - `compaction.latency_us`, `compaction.scan_free.latency_us`, - `compaction.merge_bitmap.latency_us`, `compaction.kv_persist.latency_us`, - `recovery_duration_ms`. - -`register(registry: &mut MetricsRegistry) -> Self` registers all of the -above (plus the two existing R72 counters). `disabled()` constructs a -throwaway registry for tests. - -Instrumentation points (start-instant → observe-on-completion, -nanoseconds): -- `model/alloc.rs` `allocate_block`/`allocate_blocks`: observe - `allocate.rpc.latency_us` (total), `allocate.bitmap_scan.latency_us` - (around `dg.allocate_block(s)`), `allocate.kv_persist.latency_us` - (around `kv.persist_busy_batch`). On persist failure, - `allocate_errors_total.inc()`. -- `model/alloc.rs` `free_block`/`free_blocks`: observe - `free.rpc.latency_us`, `free.persist.latency_us` (around - `dg.free_block`), `free.kv_persist.latency_us`. -- `liveness/keepalive.rs` `tick`: observe `sync.latency_us`, - `sync.read_group0.latency_us` (around `observe_ownership` + - `observe_disks`), `sync.apply_changes.latency_us` (around - reconcile). `sync_success_total`/`sync_failure_total` per tick - outcome. -- `recovery/compaction.rs`: observe `compaction.*` summaries; - `compaction_records_deleted_total` per compaction. - -The latency handles are passed into the allocate/free paths via -`DiskdbMetrics` (already threaded through `BgCtx`; for the service -handler, `DiskdbService` gains an `Arc` field). To avoid -plumbing metrics through every `alloc::allocate_block` signature, wrap -the latency observation in the service handler around the `alloc::` call -(rpc + kv_persist measured together at the handler boundary) and add a -finer-grained `bitmap_scan` observation inside `alloc.rs` guarded by an -optional `&DiskdbMetrics` parameter. v1 measures rpc + bitmap_scan + -kv_persist at the handler/alloc boundary; full per-phase splitting is -best-effort. - -Edge cases: -- Histogram/summary `observe(0)` is valid (sub-microsecond ops). -- Metrics handle `None` in test paths → `disabled()` registry is used. - -## 7. Reporting loop - -The hot-path atomic counters (`DiskMetrics`) + bitmap state must be -bridged to the crowdb-common `Gauge`/`Counter` reporting layer on a -cadence. §11: gauges are derived snapshots updated on the reporting -interval, not hot-path writes. - -`ReportingTask` in `metrics/reporting.rs` implements `BackgroundTask`. -Registered in `BgRunner` alongside keepalive + compaction in `main.rs`. -Every 10 s (configurable via `reporting.interval_secs`, live-reload via -the shared config handle): - -- For each owned disk-group → each disk: - `DiskMetrics::swap_periods()` → flush period deltas into - `allocate_total`/`free_total` counters (`inc_by(period.allocate_count)` - etc.). -- Recompute gauges from the bitmap: - `DdbDiskGroup::aggregate_usage()` → set - `dg_capacity_bytes`/`dg_busy_bytes`/`dg_free_bytes` (summed across - groups); per-disk sums → `disk_*` gauges; - `disk_active_zone_count`/`disk_total_zone_count` summed. The imbalance - gauges expose the minimum, maximum, and spread across the disks observed by - the instance. -- `owned_disk_group_count.set(container.disk_group_ids().len())`; - `degraded.set(if container.is_degraded() {1} else {0})`; - `last_sync_age_secs.set(...)` (tracked from the last successful - keepalive tick — keepalive records a - `last_sync_at: Mutex>` on the container, or the - reporting task reads a shared `AtomicU64` epoch). - -`ReportingTask` owns an `Arc` + -`Arc` + the shared config handle (for the interval). -`run_cycle` does the flush; `trigger` returns `Trigger::TimerFn` reading -`reporting.interval_secs`. - -Edge cases: -- Tick during concurrent `disk_add_init`/`remove_disk` → read locks - make the gauge snapshot best-effort; a transient disk may be missed - this tick. Acceptable (§11). -- Zero owned groups → gauges set to 0, counters unchanged. - -## 8. Keepalive usage piggyback - -Keepalive sends one `DiskGroupUsageSummary` for each owned disk-group through -`heartbeat_diskdb`. Group 0 stores the latest summary at -`/hw/dg_usage/` for the cluster-wide view and placement ranking. The -summary is derived from the bitmap on every keepalive tick, not a source of -truth. - -In `liveness/keepalive.rs` `tick()`, after `observe_disks`, compute one -`DiskGroupUsageSummary` per owned disk-group from `aggregate_usage()`: - -```rust -let owned_dg_ids: Vec = ...; -let group_usages: Vec = container - .disk_group_ids() - .into_iter() - .map(|dg_id| { - let dg = container.get_disk_group(dg_id).unwrap(); - let u = dg.aggregate_usage(); - DiskGroupUsageSummary { - disk_group_id: dg_id, - capacity_bytes: u.capacity_bytes, - used_bytes: u.busy_bytes, - free_bytes: u.free_bytes, - disk_count: u.disk_count, - allocatable_disk_count: u.allocatable_disk_count, - allocatable_capacity_bytes: u.allocatable_capacity_bytes, - allocatable_used_bytes: u.allocatable_busy_bytes, - allocatable_free_bytes: u.allocatable_free_bytes, - sampled_at_ms: unix_time_ms(), - } - }) - .collect(); -``` - -`KeepAlive::tick` passes the owned IDs and summaries to -`heartbeat_diskdb`. The endpoint is the reachable DiskDB RPC address used by -client routing. The summary is recomputed each tick rather than cached. - -Edge cases: -- Zero owned groups → empty `group_usages` (not an error). -- `heartbeat_diskdb` fails (group 0 unreachable) → keepalive already - handles this (missed count → degraded); the summary is just not - delivered this tick. -- Group 0 stores **only disk-group-level** usage; per-disk/per-zone - live usage is never written to group 0. - -## 9. Schema extension - -`lib/crowdb-protocol/src/fbs/diskdb_type.fbs`: - -```fbs -table ZoneUsage { - zone_index: uint32; - capacity_bytes: uint64; - busy_bytes: uint64; - free_bytes: uint64; - busy_block_count: uint32; - free_block_count: uint32; - alloc_state: ZoneAllocationState; - // Populated only for a specific-zone query; omitted at disk level. - usage_bitmap: [ubyte]; -} -``` - -`DiskInfo` gains `busy_units`, `free_units`, `capacity_bytes`, -`busy_bytes`, `free_bytes`, `active_zone_count`, and -`[ZoneUsage] zone_usages`. `DiskGroupInfo` gains -`capacity_bytes`, `busy_bytes`, `free_bytes`, -`allocatable_disk_count`. - -`diskdb_op.fbs`: -- `QueryCapacityStatsRequest`: add - `disk_id: DiskId` and - `zone_index: uint32`. -- Add `RecalcDiskUsageRequest { disk_group_id: uint64; }` / - `RecalcDiskUsageResponse { results: [DiskGroupRecalcResult]; }` - + `DiskGroupRecalcResult { disk_group_id: uint64; drift_detected: bool; - zones: [ZoneRecalcResult]; }` + `ZoneRecalcResult { ... }` - (mirror `RecalcResult`). - -`diskdb_service.fbs`: add -`rpc RecalcDiskUsage(RecalcDiskUsageRequest) -> (RecalcDiskUsageResponse);` - -Edge cases: -- flatbuffers scalar presence → flatc generates `Option` so - `zone_index` absence is distinguishable from 0. -- `usage_bitmap` is `[ubyte]` → `Option>`; omitted at - disk level. - -## 10. kv-client space-usage aggregation - -No client reads `/hw/dg_usage/*` back. The console needs a -cluster/rack/node/disk-group capacity view built from the piggyback -summaries joined with the hardware hierarchy. This is the stale (≤1 -sync interval) cluster-wide view; live per-disk/per-zone drill-down is -via the diskdb-client (§11). - -`SpaceUsageClient` in `lib/crowdb-kv-client/src/space_usage.rs` wraps -`HardwareClient` (which already has the hierarchy scans + -`scan_prefix`). Re-exported from `lib.rs`. - -```rust -pub struct SpaceUsageClient { hw: HardwareClient } - -impl SpaceUsageClient { - pub fn new(hw: HardwareClient) -> Self { Self { hw } } - - /// Prefix-scan `/hw/dg_usage/` → one summary per disk-group. - pub async fn list_disk_group_usages(&self) - -> Result>; - - /// Join summaries with the hardware hierarchy → cluster-level totals. - pub async fn cluster_usage(&self) -> Result; - - /// Scoped aggregations down the hierarchy. - pub async fn rack_usage(&self, rack_id: RackId) -> Result; - pub async fn node_usage(&self, rack_id: RackId, node_id: NodeId) -> Result; - pub async fn disk_group_usage(&self, dg_id: DiskGroupId) -> Result>; -} -``` - -- `list_disk_group_usages`: `scan_prefix::` over - `DiskGroupUsageKey::prefix_all()` text path (`/hw/dg_usage/`). The - values are JSON-encoded (group 0 stores text-path keys with JSON - values, matching `HardwareClient`'s convention). -- `cluster_usage`: read all summaries + `list_racks`/`list_nodes`/ - `list_disk_groups`/`list_disks_in_group`; sum capacity/used/free + - disk-group count. Per-disk live busy/free is **not** available here - (group 0 has only disk-group-level usage + per-disk static - `DiskValue.capacity_units`); the disk-level view carries capacity - only. -- `rack_usage`/`node_usage`/`disk_group_usage`: scope the join to the - hierarchy subtree. - -`ClusterUsage`/`RackUsage`/`NodeUsage` structs hold capacity/used/free -bytes + disk-group/disk counts. Mirror `crowdb-kv-client`'s retry pattern -(the underlying `HardwareClient` already retries). - -Edge cases: -- No summaries written yet (fresh cluster) → empty vec / zero totals. -- Stale summary (≤1 sync interval behind) → documented; not an error. -- Per-disk busy/free absent → `NodeUsage.disk_capacity_bytes` only, no - busy/free. - -## 11. diskdb-client — full client library - -`crowdb-diskdb-client` is the primary client surface for all diskdb rpc -operations (allocate/free/query with retry + endpoint caching), -mirroring `crowdb-kv-client`'s pattern. Without it, callers must use raw -rpc stubs and do endpoint discovery manually. - -`DiskdbClient` in `lib/crowdb-diskdb-client/src/client.rs`: - -```rust -pub struct DiskdbClient { - svc: ServiceRegistryClient, // endpoint discovery from group 0 - routing: Arc, // shared generated RCU routes - rpc_transport: Arc, - retry: RetryConfig, -} - -impl DiskdbClient { - pub fn new(svc: ServiceRegistryClient) -> Self; - /// Eager warm: read all diskdb instances, populate cache. - pub async fn refresh_endpoints(&self) -> Result<()>; - /// Lazy refresh on cache miss for one dg_id. - async fn refresh_for(&self, dg_id: DiskGroupId) -> Result<()>; - - pub async fn allocate_blocks(&self, req: AllocateBlocksRequest) -> Result; - pub async fn free_blocks(&self, req: FreeBlocksRequest) -> Result; - pub async fn query_capacity_stats(&self, req: QueryCapacityStatsRequest) -> Result; - pub async fn query_disk_group(&self, dg_id: u64) -> Result; - pub async fn query_disk(&self, dg_id: u64, disk_id: DiskId) -> Result; - pub async fn query_zone(&self, dg_id: u64, disk_id: DiskId, zone_index: u32) -> Result; - pub async fn get_disk_group_info(&self, dg_id: u64) -> Result; - pub async fn get_disk_info(&self, dg_id: u64, disk_id: DiskId) -> Result; - pub async fn recalc_disk_usage(&self, req: RecalcDiskUsageRequest) -> Result; -} -``` - -- **Endpoint discovery + cache**: `refresh_endpoints` calls - `svc.read_all_diskdb_instances()`, reads each - `InstanceValue.rpc_endpoint` + `DiskdbExtra.owned_dg_ids`, and publishes one - complete `dg_id -> endpoint` snapshot. Called on startup (eager), on cache - miss (lazy `refresh_for`), and on retryable routing failure. Client clones - share this state; incrementally learned disk routes carry the endpoint - generation and are reconciled after refresh. -- **Connection pool**: `ConnectionPoolIndex` holds immutable generated pools - per endpoint. Connections are reused across calls, and late failures can - invalidate only the generation used by that request. -- **`allocate_blocks`**: look up endpoint for `req.disk_group_id` - (refresh on miss), open channel, call `AllocateBlocks`, retry on - transient errors (`Unavailable`, deadline-exceeded). - `ResourceExhausted` (no space) → return to caller (not retryable). -- **`free_blocks`**: the request carries `Segment`s (each with - `disk_id`); route by looking up which diskdb instance owns the - `disk_id`'s disk-group. If segments span multiple disk-groups, split - the request per-group and issue one `FreeBlocks` per group (v1: no - cross-group batch). Retries on transient errors. To map - `disk_id -> dg_id`, the client needs the disk→dg mapping from group - 0 (`HardwareClient::list_disks_in_group` per dg, or a cached reverse - map built during `refresh_endpoints` from the hardware hierarchy). v1 - builds a `disk_id -> dg_id` reverse map during `refresh_endpoints` by - reading the hardware hierarchy once; refreshes on cache miss. -- **`query_*`**: convenience helpers building the right - `QueryCapacityStatsRequest` shape (§4). - `query_disk_group(dg_id)` → `disk_id=None, zone_index=None`; - `query_disk(dg_id, disk_id)` → `disk_id=Some, zone_index=None`; - `query_zone(dg_id, disk_id, zi)` → all set. -- **`recalc_disk_usage`**: wraps the admin RPC. -- **Error model**: `DiskdbClientError` with `Rpc(crowdb_rpc::Status)` / - `NoSpace` / `NotFound` / `InvalidArgument` mappings. `RetryConfig` - mirrors `crowdb-kv-client` (max retries, backoff). - -Edge cases: -- Cache miss (unknown `dg_id`) → lazy refresh + retry; if still not - found → `Unreachable`. -- Endpoint moved (instance restarted on new port) → cached endpoint - stale → next call `Unavailable` → refresh → retry → succeeds. v1 - reactive (proactive refresh needs group-0 notify/watch, tracked as a - use case). -- `free_blocks` spans multiple disk-groups → split per-group; partial - failure returns per-group results (v1: returns the first error; a - follow-up can aggregate). Documented. -- v1 no proactive refresh — no background refresh task. - -## 12. E2E test migration - -New rpc-level e2e tests in `lib/crowdb-diskdb-client/tests/` using -`DiskdbClient` against an in-process diskdb rpc server + `KvCluster` -(mirroring `crowdb-kv-client/tests/` pattern): -- `allocate_free_e2e.rs` — allocate → verify busy record → free → - verify free. -- `query_e2e.rs` — allocate → - `query_disk_group`/`query_disk`/`query_zone`. -- `endpoint_cache_e2e.rs` — endpoint discovery, cache miss → refresh → - retry. -- `recalc_e2e.rs` — allocate → recalc → matches; corrupt live bitmap → - drift. - -The existing in-process `app/crowdb-diskdb/tests/diskdb_e2e_test.rs` -(calls `alloc::allocate_block` directly) stays as a server-internal -integration test. - -## Invariants - -- **I1 — Free = capacity − busy**: CROWDB reuses freed space immediately - via bitmap scan (no append-only `allocate_pos`, §3.4/§8); there is no - separate free cursor. `free_blocks = unit_capacity - used_count`. -- **I2 — Gauges are derived snapshots**: capacity/busy/free gauges are - recomputed from the bitmap on the reporting tick (single source of - truth), not maintained as parallel hot-path atomics that could drift. -- **I3 — Recalc uses an independent popcount**: the recalc verifier - reads `usage_bits.count_set()` on the replayed zone (not its - `used_count`) and compares against the live `used_count`, so drift in - either the bitmap or the cached counter is detected. -- **I4 — Group 0 carries disk-group-level usage only**: per-disk and - per-zone live usage are never written to group 0 (a disk can have - thousands of zones; group 0 is not a per-zone registry). Per-disk/ - per-zone drill-down is served by diskdb directly. -- **I5 — Counters increment once per durable-bound op**: - `record_allocate`/`record_free` fire after the Phase 1 bitmap CAS - succeeds, so a CAS-failed retry does not bump the counter. A Phase 2 - rollback over-counts by one — documented, acceptable for a - best-effort event counter. - -## Module Structure - -``` -app/crowdb-diskdb/src/ - model/ - zone.rs # + accessors, ZoneUsage - disk.rs # + DiskUsage, DdbDisk::usage, metrics field - disk_group.rs # + DiskGroupUsage, aggregate_usage, zone_usage - usage.rs # (optional) ZoneUsage/DiskUsage/DiskGroupUsage structs - alloc.rs # + DiskMetrics record calls + latency observation - metrics/ - mod.rs # DiskdbMetrics (extended §11 set) - disk.rs # DiskMetrics (period/total counters) - recalc.rs # RecalcEngine + RecalcResult - reporting.rs # ReportingTask (BackgroundTask) - service/diskdb_service.rs # QueryCapacityStats + RecalcDiskUsage handlers - liveness/keepalive.rs # usage piggyback + endpoint + sync metrics - ddb_config.rs # + ReportingConfig - main.rs # wire ReportingTask + RecalcEngine + DiskdbMetrics -lib/crowdb-protocol/src/fbs/ - diskdb_type.fbs # ZoneUsage, extend DiskInfo/DiskGroupInfo - diskdb_op.fbs # extend QueryCapacityStatsRequest, RecalcDiskUsage* - diskdb_service.fbs # RecalcDiskUsage RPC -lib/crowdb-kv-client/src/ - space_usage.rs # SpaceUsageClient + ClusterUsage/RackUsage/NodeUsage - lib.rs # re-export -lib/crowdb-diskdb-client/src/ - lib.rs # DiskdbClientError (extended) - client.rs # DiskdbClient (allocate/free/query/recalc + cache) -app/crowdb-diskdb/tests/ - space_flow_test.rs # whole-flow single + multi-thread -lib/crowdb-diskdb-client/tests/ - allocate_free_e2e.rs - query_e2e.rs - endpoint_cache_e2e.rs - recalc_e2e.rs -``` - -## Config Extensions - -- `ddb_config.rs`: `ReportingConfig { interval_secs: u32 }` (default - 10), added to `DdbConfig` as `pub reporting: ReportingConfig`. - `validate()`: `interval_secs > 0`. Live-reloadable (dynamic field), - read by `ReportingTask::trigger` via the shared config handle. - -## Server Wiring - -1. `main.rs`: build `DiskdbMetrics::register(&mut metrics_registry)` - (extended set), already constructed; the §11 handles are now - populated. -2. Construct `RecalcEngine::new(Arc::clone(&dg_kv), Arc::clone(&container))`; - pass `Arc` + `Arc` into - `DiskdbService::new` (signature gains two fields). -3. Construct `ReportingTask::new(container, metrics, config_handle)`; - register in `BgRunner` alongside keepalive + compaction. -4. `keepalive` gains the real `server.listen_addr` endpoint for the - piggyback (passed via `with_endpoint` or read from config in `tick`). -5. Recovery (`run_recovery`) attaches `DiskMetrics` to each recovered - disk (matching `disk_add_init`). diff --git a/doc/design/diskdb/design-crowdb-diskdb-zone-management.md b/doc/design/diskdb/design-crowdb-diskdb-zone-management.md deleted file mode 100644 index b920af0fe..000000000 --- a/doc/design/diskdb/design-crowdb-diskdb-zone-management.md +++ /dev/null @@ -1,929 +0,0 @@ - - - -# CROWDB - Design: diskdb Zone Management - -Depends on: [`design-crowdb-diskdb.md`](design-crowdb-diskdb.md) §3.3 (no CAS, -exclusive ownership), §3.9 (unit-based sizes, disk-id key routing), §6 -(hierarchy); -[`design-crowdb-protocol-key.md`](../protocol/design-crowdb-protocol-key.md) -(binary key encoding); -[`design-crowdb-kv-state-machine.md`](../kv/design-crowdb-kv-state-machine.md) -(slot tracking, apply semantics). -Satisfies: `design-crowdb-diskdb.md` §7 (Zone Records and Crash Recovery), -§8 (Allocation Algorithm), §9 zone allocation state, §14 zone-level -concurrency. - -Detailed design for diskdb's zone-management component — the zone -allocator, record model, free path, compaction, rotation, crash -recovery, and zone-level concurrency. Architecture decisions and -rationale live in the root design doc; this doc carries the structs, -algorithms, record layouts, and coordination rules. Reused surfaces -(named, not re-built): `DdbZone`, `DdbDisk`, `DdbDiskGroup`, -`DdbDiskGroupContainer`, `DdbKvClient`, `CompactionEngine`, -`RecoveryEngine`, `UsageBitmap`, `ZoneValueExt`. - -## Table of Contents - -- [1. Scope](#1-scope) -- [2. Zone as a Logical Concept](#2-zone-as-a-logical-concept) -- [3. Record Model](#3-record-model) - - [Record key layout](#record-key-layout) - - [Value schemas](#value-schemas) - - [Record model](#record-model) - - [Current state determination](#current-state-determination) -- [4. Allocation Algorithm](#4-allocation-algorithm) - - [Zone-level allocate (sync, in-memory)](#zone-level-allocate-sync-in-memory) - - [Disk-level allocate (sync) — rotating active-zone-set](#disk-level-allocate-sync--rotating-active-zone-set) - - [Round-robin across disks within the named disk-group (sync)](#round-robin-across-disks-within-the-named-disk-group-sync) - - [Two-phase async allocation](#two-phase-async-allocation) - - [Free (persist-only)](#free-persist-only) -- [5. Zone Rotation and Compaction](#5-zone-rotation-and-compaction) - - [Compaction-on-rotation](#compaction-on-rotation) - - [Preparatory thread](#preparatory-thread) - - [Periodic compaction fallback](#periodic-compaction-fallback) - - [Compaction algorithm](#compaction-algorithm) - - [Ready-zone tracking](#ready-zone-tracking) -- [6. Crash Recovery](#6-crash-recovery) - - [Three recovery strategies](#three-recovery-strategies) - - [How the strategies work together](#how-the-strategies-work-together) - - [Crash-safety invariants](#crash-safety-invariants) - - [Recovery and compaction engines](#recovery-and-compaction-engines) - - [Recovery triggers](#recovery-triggers) -- [7. Zone Allocation State (derived)](#7-zone-allocation-state-derived) -- [8. Concurrency Model](#8-concurrency-model) - - [Allocate (lock-free)](#allocate-lock-free) - - [Free (persist-only, no lock)](#free-persist-only-no-lock) - - [Zone-level lock for non-allocate operations](#zone-level-lock-for-non-allocate-operations) - - [Common methods on DdbZone](#common-methods-on-ddbzone) - - [Free-side lookup structures](#free-side-lookup-structures) - - [Monotonic timestamp source](#monotonic-timestamp-source) -- [9. Background Scanner Coordination](#9-background-scanner-coordination) -- [10. Invariants](#10-invariants) -- [11. Tunables and Defaults](#11-tunables-and-defaults) - -## 1. Scope - -This doc covers everything zone-level: the in-memory zone allocator -(bitmap-scan with per-bit CAS), the durable record model (busy/free/ -snapshot records), the free path (persist-only, bitmap untouched on -free), compaction (the sole bit-clearer), zone rotation (compaction- -before-rotation with a preparatory thread), crash recovery (three -strategies), and the zone-level concurrency model (lock-free allocate, -zone-level lock for non-allocate operations). - -What is **not** here: disk-level status management (root §10), space -metrics (root §11 / space-metrics sub-design), background scanner design -(root §12 / scanner sub-design), group-0 sysdata schema (root §5). Those -docs reference this one for zone-level behavior. - -## 2. Zone as a Logical Concept - -Not all zones on a disk must be the same size. The last zone may be -smaller (disk capacity is rarely an exact multiple of the zone size). -Zone is a **logical concept** defined for easier implementation; it can -later adopt to native zoned-namespace SSD or SMR HDD zone APIs, but -currently no such devices are targeted. The zone is a rough mapping. - -**Word alignment rule:** each zone's `unit_capacity` must be a multiple -of 64 (the 64-bit bitmap word size). All zones except the last have -`unit_capacity = zone_size / unit_size`. The last zone has -`unit_capacity = remaining_capacity / unit_size`, rounded down to a -multiple of 64; the sub-64-unit tail (at most 63 units) is unallocated. -Only the last zone may have a different size; all other zones on a disk -are uniform. No bitmap masking, no padding bits. Each zone's bitmap is -sized to its own `unit_capacity`, which is word-aligned. - -## 3. Record Model - -### Record key layout - -Three types of KV entries on the disk-group's bound data group. Keys -use the cross-component binary encoding -(`doc/design/protocol/design-crowdb-protocol-key.md`): each is a flat -struct with a `magic | type_tag` header and big-endian fixed-width -fields. Keys are disk-id-based (globally unique → reverse-lookup to the -data group). No `node_id`/`disk_group_id` in record keys. - -``` -# Busy block (on allocate; deleted by matching compaction) -BusyBlockKey { disk_id, zone_index, unit_offset } -> BusyBlockValue - -# Immutable free fact (on free; transient — deleted by compaction) -FreeBlockKey { disk_id, zone_index, unit_offset, allocation_ts } - -> FreeBlockValue - -# Zone (compacted snapshot, periodic) -ZoneKey { disk_id, zone_index } -> ZoneValue -``` - -`disk_id` is 16 bytes (`high:u64 BE | low:u64 BE`); `zone_index` is -`u32 BE`; `unit_offset` and `allocation_ts` are `u64 BE`. Big-endian fixed width gives -lexicographic byte order == numeric order, so a prefix scan of one -zone's busy (or free) keys returns blocks in `unit_offset` order -without deserialization. Exact byte layouts and prefix constructors -(e.g. `BusyBlockKey::prefix_for_zone(disk_id, zone_index)`) are in -`doc/design/protocol/design-crowdb-protocol-key.md`; value field details -are in `diskdb_type.fbs`. - -### Value schemas - -- **`BusyBlockValue`**: - - `allocation_ts: u64` — monotonically increasing allocation incarnation, - also returned in the `Segment`. It is an identity token, not a KV commit - timestamp or a compaction-ordering value. - - `unit_count: u32` — number of units in this allocation (≥ 1; - multi-unit allocations span consecutive offsets). - - `unit_size: u32` — size of one unit in bytes (e.g. 1 MB). Carried - per-block so the data-IO layer knows the block's granularity - without a separate lookup. - - `owner_chunk: ChunkId` (192-bit) — reverse reference to the - block's owner (the chunk that holds this block's data). Needed for - recovery and other logic. - - `state: BlockState` — per-block I/O-behavior state enum: - `Ok` (normal; default on allocate), `Suspect` (data may be - unreadable; data-IO layer tries with a timeout, falls back to - mirror/EC rebuild), `Corrupt` (data confirmed unreadable; data-IO - layer skips read, rebuilds from EC/mirror). Updated by background - paths (sync, health probe, scanner) or via `MarkBlockSuspect` / - `MarkBlockCorrupt` admin RPCs — **not** by the allocate hot path. -- **`FreeBlockValue`**: - - `pre_allocation_ts: u64` — the exact busy incarnation released by this - immutable free fact. It equals the `allocation_ts` in the request segment. - - `unit_count: u32` — number of units freed (matches the - corresponding `BusyBlockValue.unit_count`). - - `previous_owner: ChunkId` (192-bit) — the `owner_chunk` from the - `BusyBlockValue` that was freed (carried in the `Segment` on the - free request; no KV read needed). Carried for audit / scanner - cross-check. No `state` field — a free block has no data. - - `free_ts: u64` — diagnostic time at which free was submitted. It is useful - for logs and age metrics but never controls correctness or compaction. -- **`ZoneValue`**: - - `bitmap_codec: u8` — `1` identifies a size-prefixed LZ4 block. Unsupported - codecs and unversioned records are rejected; old records are not supported. - - `usage_bitmap: bytes` — LZ4-compressed zone bitmap (one bit per unit; - bit set = busy, bit clear = free). Decoded length must match the zone's - `unit_capacity`, rounded to complete 64-bit words. The decoded length is - checked before allocating a decompression buffer. A 4 KiB all-zero bitmap - uses a shared 31-byte compressed constant. - - `snapshot_slot: u64` — the slot at which this snapshot was written. - Strategy 2 (journal scan replay) replays operations after this slot. - - `compact_slot: u64` — the last successfully persisted fixed - contiguous-applied scan cutoff. It records scan progress, but does not - exclude a surviving immutable free fact at or below an earlier cutoff; - such a fact must still be reconciled. Initialized to zero for a baseline - snapshot. - - `crc32: u32` — CRC32 checksum over `bitmap_codec` + compressed - `usage_bitmap` + `compact_slot` - for integrity verification (§9 scanner coordination). - -### Record model - -- A busy/free entry can span multiple units (`unit_count` ≥ 1). -- On free, DiskDB blindly writes `FreeBlockKey(..., allocation_ts)` and does - not delete `BusyBlockKey`. Retrying the same segment rewrites the same - logical fact; a stale retry targets only its old incarnation. -- Compaction matches `FreeBlockValue.pre_allocation_ts`, `unit_count`, and - `previous_owner` against the current `BusyBlockValue`. Only a complete - match clears the bitmap and deletes the busy record. A mismatch is stale or - invalid and cannot change the busy state. -- Re-allocation occurs only after compaction has cleared the bitmap and - deleted the prior busy incarnation. It writes a higher `allocation_ts`. -- Current durable state is derived from the current busy record plus any - matching incarnation-qualified free fact. Before compaction both remain - durable and the bitmap conservatively stays busy; after compaction neither - remains for a freed incarnation. -- `ZoneValue` carries a CRC32 checksum for integrity verification. - -### Current state determination - -The full `ZoneValue` is a **compacted snapshot** written periodically. -The ideal approach on free would be to update the bitmap in the -`ZoneValue` and write the whole `ZoneValue` to KV. But `ZoneValue` is -large (full bitmap), and frees are random across all zones and disks, -so a per-free `ZoneValue` write is too expensive. Instead, each free -writes one small immutable incarnation-qualified fact. Later, compaction -bounded-scans busy and free records, validates matching incarnations, clears -only matched bitmap ranges, and atomically persists the updated snapshot plus -the exact record deletions. - -On crash/restart, diskdb reconstructs the in-memory zone state using -three complementary strategies (§6). - -**Note on scan ordering:** a key prefix scan returns records in -**lexicographic key order** (= `unit_offset` order), not slot order. -`min_slot` on the ordinary crowdb-kv scan request is a read-freshness floor, -not a record-slot filter. Compaction uses the R132 fixed-cutoff bounded scan, -which returns each current record's `commit_slot`. Strategy 2 uses -`JournalScan` for slot-ordered replay. - -## 4. Allocation Algorithm - -The algorithm uses bitmap-scan with a rotating cursor, per-bit CAS, -and disk-level zone rotation. The only difference from a local-file -allocator is persistence (CROWDB writes `BusyBlockValue`/ -`FreeBlockValue` to the KV journal per-allocation instead of saving the -full bitmap to a local file). - -### Zone-level allocate (sync, in-memory) - -Bitmap-scan with per-bit CAS (no zone-level lock): -1. Check zone health is Healthy and `used_count < unit_capacity`. -2. Scan the bitmap from `last_pos_64` (rotating cursor), wrapping - around. For each 64-bit word, use `countr_one` to find the first - zero bit (hardware-optimized). -3. CAS-set the bit via `compare_exchange` on the 64-bit word. On CAS - failure (another thread set the same bit), re-scan the same word. - **CAS retry bound:** per-bit CAS is capped at `cas_retry_limit` - retries (default 100); on exhaustion, fall through to the next bit / - word / zone. This prevents indefinite spinning under heavy - contention. The `cas_retry_count` atomic (and the wired crowdb-common - counter) is incremented on each retry as the key operational signal - for lock-free allocator contention. -4. For `unit_count > 1`: find `unit_count` consecutive zero bits (may - span words), CAS-set each; on any CAS failure, clear bits already - set in this attempt and continue scanning. -5. On success: update `last_pos_64`, increment `used_count`, return - `AllocatedRange { unit_offset, unit_count }`. No zone-level state - change. Other threads can allocate concurrently. - -### Disk-level allocate (sync) — rotating active-zone-set - -1. Load the current `ActiveZoneContext` (RCU read, `Arc` clone, no - lock held). This is a small set of `zone_rotate_count` zones. -2. Round-robin over the active set via `pos_v_zone_ctx.fetch_add(1)`. -3. Call `zone.allocate(unit_count)` on each zone. On success, return. -4. If all zones in the active set fail, call `rotate_active_zones()`: - publish the next set of **ready** (pre-compacted) zones using RCU - compare-and-swap. If too few ready zones exist, include other allocatable - zones; preparatory or periodic compaction handles their retained free facts. - Advance the scan cursor past the last selected position in either pass. - A failed large allocation can leave smaller free ranges in the current - zones; rotation must continue to later zones rather than repeat that set. -5. Retry with the new context (up to `zone_num / zone_rotate_count + 2` - loops). - -### Round-robin across disks within the named disk-group (sync) - -The `AllocateBlocks` request carries `disk_group_id` (root §3.2); -allocate is scoped to one disk-group. diskdb never round-robins across -disk-groups. Within that named disk-group, an `AtomicU64` iterator -(`pos_v_disk_ctx`) round-robins across disks: each allocation increments -and selects `iterator % num_disks` from the `AllocateDiskContext` (RCU -context of allocatable disks within the named disk-group). Multi-block: -`fetch_add(count)` distributes across disks within the group. -`exclude_disks` (anti-affinity, per-disk, skip a disk that just failed) -is applied within the named disk-group. Multi-block uses one -`batch_write` on the disk-group's bound paxos data group; atomic within -the group. No cross-group multi-block allocate in v1. - -`allow_disk_reuse` is reserved for a caller that already validated a joint -placement plan, such as ChunkDB's mirror-to-EC reservation. When enabled, -allocation resets the per-request used-disk set only after a full -anti-affinity pass, so each pass still spreads across distinct disks. The -complete set of claims is persisted in the same single `batch_write` and is -rolled back together on allocation or persistence failure. - -### Two-phase async allocation - -1. **Phase 1 (sync)**: bitmap-scan allocate (nanoseconds). Bits are set - in `usage_bits` via per-bit CAS. No zone-level lock. Other threads - can allocate concurrently. -2. **Phase 2 (async)**: `.await` on crowdb-kv `batch_write` that Puts - the `BusyBlockValue` with a new `allocation_ts` at `BusyBlockKey { - disk_id, zone_idx, unit_offset }`. On success, return a `Segment` - carrying the same incarnation. On failure: - `zone.rollback_allocate(unit_offset, unit_count)` (clear the bits - that were set in Phase 1), return error. - -For multi-block: Phase 1 allocates all blocks (sync, may span multiple -zones/disks within one disk-group), Phase 2 uses one `batch_write` per -data group (one async round-trip per group). - -### Free (persist-only) - -The free path is **persist-only**: blindly put one incarnation-qualified -`FreeBlockValue`. It does not read or delete `BusyBlockKey`. The in-memory bitmap is **not** -touched. The bit stays set, `used_count` is not decremented. The block -is freed on disk but still shows busy in memory until compaction clears -the bit (§5). This is the data-safety principle: the bitmap is a -conservative over-estimate that never shows freed space as available -until compaction reconciles it from records. - -After durable persistence, the free path increments -`uncompacted_free_record_count` (an atomic counter per zone) so compaction -knows there is work to do. The bitmap reconciliation remains deferred to -compaction. - -Free steps: -1. **Persist**: put `FreeBlockValue` at `FreeBlockKey { disk_id, - zone_index, unit_offset, allocation_ts }`. The value carries - `pre_allocation_ts = allocation_ts` from the segment and diagnostic - `free_ts`. No read-before-write validation is required for correctness. -2. **Defer validation**: compaction later compares allocation timestamp, unit - count, and owner with the current busy value. A forged or stale free cannot - clear the bitmap or delete a newer busy incarnation. -3. **Post-persist (in-memory)**: increment - `uncompacted_free_record_count` on the zone (lookup via zone-index → - zone vec). No bitmap mutation, no `used_count` decrement. -4. Record per-disk event counter after the durable free. - -Persist-before-bitmap-clear is not needed here because the bitmap is -never cleared on free. The persist is the entire free operation. If the -persist fails, the block is still busy on both disk and memory. The -caller can retry safely. If the persist succeeds, the block is free on -disk; the in-memory bitmap still shows it busy (conservative over- -estimate) until compaction reconciles. - -When concurrent-free coalescing is enabled, preparation first deduplicates and -captures each request's disk-group bind without changing tentative or zone -state. Enqueue uses a lock-free MPSC queue and an atomic single-drainer claim. -The drainer starts immediately—there is no minimum threshold or timer—and -combines whole requests for the same bind up to the smallest captured maximum. -One oversized request is persisted alone rather than split. Requests arriving -while a KV write is in flight form the next natural batch. - -Every covered waiter resolves only after its combined `batch_write` completes. -Success applies tentative removal, compaction-backlog increments, and free -metrics once per distinct incarnation across the batch. Failure resolves all -covered waiters without accounting changes or automatic re-enqueue; an -idempotent caller retry writes the same incarnation-qualified fact. Shutdown -closes admission, drains both direct and coalesced accepted frees, and only -then stops the RPC server. The disabled path retains one durable KV batch per -request under the same lifecycle tracker. - -**`rollback_allocate` — allocate-only bitmap clear:** the -`DdbZone::rollback_allocate` method (CAS-clear bits, decrement -`used_count`) is used **only** by the allocate Phase 2 failure path to -undo Phase 1's bitmap claim. It is **not** called by the free path. This -method replaces the former `DdbZone::free` that was used by both paths. - -## 5. Zone Rotation and Compaction - -The disk maintains a small active zone set (`zone_rotate_count` zones, -default 4). When all zones in the set are exhausted (no free bits), the -set rotates: a new set of allocatable zones is selected from a rotating -scan position and published via RCU. This spreads wear across zones and -avoids bias toward low-index zones. - -### Compaction-on-rotation - -Before a zone enters the active set, it **must be compacted**: freed -bits cleared, `used_count` recomputed, snapshot written, free records -deleted. This ensures the allocator sees accurate free space in active -zones. Without compaction, a rotated-in zone would have stale set bits -for freed-but-not-compacted blocks, and the allocator would skip them -(appearing fuller than it is). - -Compaction runs on **non-active zones only**, no concurrent allocate. -The zone-level lock (§8) ensures compaction, scanner, and health checks -do not conflict with each other on the same zone. - -### Preparatory thread - -A separate background thread pre-compacts the next batch of -`zone_rotate_count` zones in advance, so rotation is instant. The next -active set is already compacted and ready to publish via RCU. The -preparatory thread runs continuously: - -1. Identify the next `zone_rotate_count` zones in the rotation order - (starting from `pos_v_zone + zone_rotate_count`, wrapping around). -2. For each zone that is not already ready and not in the current active - set: acquire the zone-level lock, compact it (§5 Compaction - algorithm), mark it as ready. -3. Sleep briefly, then re-check (new zones may need compaction after - frees, or the rotation position may have advanced). - -When rotation is triggered, the ready zones are published immediately -as the new active set. Their ready flag is cleared. The preparatory -thread then starts building the next ready set. - -**Fallback when no ready zones exist:** if the preparatory thread has -not finished pre-compacting (e.g., on startup, or under heavy churn -where zones exhaust faster than compaction can keep up), rotation falls -back to **synchronous compaction**: compact the next batch inline, then -publish. This is slower but correct. The allocator waits for -compaction to finish before allocating from the new zones. - -### Periodic compaction fallback - -A background compaction task (`CompactionEngine`) polls at most every five -seconds for the configured `uncompacted_free_record_count` threshold. Independently, -the full sweep cadence (default 300s) admits every nonempty backlog, including -small zones that can never reach the threshold. Threshold polls do not reset -the full sweep deadline. Both paths skip active zones and use the zone-level -lock. A disk whose zones are all active must rotate a zone out before background -compaction can reclaim its free records. Managed files use -`zone_size_bytes = min(capacity_bytes, 32 GiB)`; capacities above 32 GiB contain -whole 32 GiB zones. Allocator and compaction policy do not change this geometry. - -### Compaction algorithm - -`compact_zone` (strategy 3) is the background maintenance task. It uses a -fixed contiguous-applied `compact_slot` boundary and allocation-incarnation -matching, then writes the new snapshot plus exact record deletions in one -atomic `batch_write`. - -1. **Bounded scan** free and busy records at one fixed - contiguous-applied cutoff. Retain each record's `commit_slot`; an - incomplete scan cannot advance the watermark. -2. **Match incarnations** by physical range. Reconcile every scanned free - fact whose record revision is at or below the fixed cutoff, including a - surviving fact whose revision is below the prior `compact_slot`. A free is - effective only when its `pre_allocation_ts`, unit count, and previous owner - equal the current busy value. Mismatches are stale or invalid facts and - never clear bits. -3. **Acquire the zone-level lock**, clear only ranges with a complete match, - and recompute `used_count = popcount`. Release the lock before KV I/O. -4. Build a new `ZoneValue` from the merged bitmap with `snapshot_slot` and - `compact_slot` set to the fixed scan cutoff and compute its checksum. -5. **One atomic `batch_write`**: Put the new `ZoneValue`, delete busy records - for matched frees, and delete every processed free fact at or below the - cutoff. This is a single - `batch_write` on the bound data group, atomic via crowdb-kv paxos. - The snapshot and the free-record deletion succeed or fail together; - there is no window where the snapshot is written but the free - records survive (the race that caused double-free in the two-op - design). -6. Decrement `uncompacted_free_record_count` by the number of free - records deleted (both stale and new). - -Busy records without a matching free incarnation remain untouched. - -**Why incarnation matching prevents stale free:** - -Suppose allocation 1001 is freed and later the same physical range is -allocated as incarnation 1002. A delayed retry can only recreate -`FreeBlockKey(..., 1001)`. Compaction compares it with the current busy value, -sees `1001 != 1002`, and leaves both the bitmap bit and busy record unchanged. -The atomic exact-key deletion normally removes every observed fact before -`compact_slot` advances. A surviving or delayed lower-revision fact is still -processed on a later complete scan; the watermark is never used as a lower -record filter. - -### Ready-zone tracking - -Each `DdbZone` carries: -- `compacted_ready: AtomicBool` — `false` when the zone needs - compaction before it can enter the active set (or it is currently - active, or has been modified since last compaction); `true` when the - zone has been compacted and its bitmap is accurate (eligible for - rotation into the active set). -- `compact_ts: AtomicU64` — the in-memory mirror of - `ZoneValue.compact_ts`. Used by compaction to partition free records - by watermark without a separate KV read. Updated when compaction - writes a new `ZoneValue`; loaded from the snapshot during recovery. - -The `compacted_ready` flag is set to `true` by the preparatory thread -after compaction completes. It is set to `false` when the zone is -published into the active set (it will need re-compaction after being -allocated from and freed). `rotate_active_zones` picks zones where -`compacted_ready == true` and not in the current active set. If fewer -than `zone_rotate_count` ready zones exist, it falls back to -synchronous compaction for the remainder. - -## 6. Crash Recovery - -### Three recovery strategies - -diskdb uses three complementary strategies for crash recovery and -maintenance. All three belong in the design, each in its role. - -**Strategy 1 — full scan rebuild (on-demand, via RPC/API).** - -Scan all live `BusyBlockKey`s for a zone and rebuild the bitmap from -scratch. No snapshot needed. For each offset: if a `BusyBlockKey` -exists, the bit is set (busy); otherwise the bit is clear (free). No -slot ordering needed. The busy record's existence is the indicator. -(`FreeBlockKey`s carry `previous_owner` audit info but are not needed -for state determination.) **Not in the common code flow**, provided -as an on-demand operation via the `RebuildZoneBitmap` RPC and API, used -for consistency checks (verify the in-memory bitmap matches the -records) or a full rebuild (e.g. after corruption, or when no -`ZoneValue` snapshot exists). Works with the existing `scan` API. Cost -= O(all live busy records per zone). Too slow for regular restart with -many zones, but correct and always available. - -**Strategy 2 — journal scan replay (fast restart, primary path).** - -Load the `ZoneValue` snapshot, then replay only the operations (Put / -Delete) written after `snapshot_slot`, in slot order, and apply them to -the snapshot bitmap. One scan per disk-group (or per disk) covers all -its zones, batch recovery, low overhead. **Requires a `JournalScan` -crowdb-kv RPC** (slot-range + key-prefix filter, returns ops in slot -order). This is the sole crowdb-kv extension diskdb needs. Fast because -compaction (strategy 3) keeps the uncompacted record set small. - -**Strategy 3 — compaction (ongoing maintenance).** - -Periodically (or when the free-record count for a zone exceeds a -threshold), match free facts to busy incarnations, merge valid frees into the -`ZoneValue` bitmap, and delete the matched busy plus processed free records in -one `batch_write`. The fixed-cutoff bounded scan is the only KV extension. - -### How the strategies work together - -- **Steady state**: allocate writes `BusyBlockValue` with a new - `allocation_ts`. Free blindly writes an incarnation-qualified free fact and - leaves the busy record intact. The bitmap is **not touched on free** — the bit stays set, - `used_count` is not decremented. Compaction (strategy 3) runs - periodically (and before rotation), merging free records into - `ZoneValue`, clearing the freed bits, recomputing `used_count`, and - deleting the free records. -- **Restart**: load `ZoneValue` snapshot → journal scan (strategy 2) - replays post-`snapshot_slot` operations in slot order → apply to - bitmap. Fast because compaction kept the record set small. -- **On-demand (RPC/API)**: full scan (strategy 1) rebuilds the bitmap - from all live records. Triggered by an operator or the scanner for a - consistency check or full rebuild — not in the common code flow. - -### Crash-safety invariants - -The in-memory bitmap is a **conservative over-estimate** of busy -blocks. It is never cleared on the free path. Free is an immutable blind put; -the bitmap bit stays -set until compaction clears it (§5). This means the bitmap may show a -block as busy when it is actually freed on disk. This is intentional -(data-safety principle: never show a block as free until compaction has -confirmed it from records). The durable state is the set of -`BusyBlockKey` / `FreeBlockKey` / `ZoneValue` records on the bound data -group. Current state is derived from the busy incarnation plus a matching free -fact. The invariants: - -- **Allocate ordering** — Phase 1 (bitmap CAS, set bit) happens before - Phase 2 (`BusyBlockValue` persist). If diskdb crashes between Phase 1 - and Phase 2, the bit is set in memory but no `BusyBlockKey` exists on - disk. On restart, strategy 1 full scan rebuilds the bitmap from - records — the bit is clear (no busy record), so the block is - correctly free. This is a **ghost-busy** (bit set in-memory, no - record) that is self-correcting on restart; the scanner also detects - this drift during live operation. -- **Free = persist only** — free puts one immutable `FreeBlockKey` containing - the segment's `allocation_ts`. It neither reads nor deletes the busy record - and never touches the bitmap. Retries are the same blind logical event. -- **Compaction reconciles the bitmap** — compaction (§5) is the sole - mechanism for clearing freed bits in the bitmap. It partitions free - records through a fixed `compact_slot` cutoff, matches each free fact to the - current busy incarnation, `range_clear`s only complete matches, - recomputes `used_count = popcount`, and writes the new `ZoneValue` - with the updated `compact_slot`, deletes matched busy records, and deletes - processed free facts in - **one atomic `batch_write`**. After compaction, the bitmap - accurately reflects the durable state. Compaction runs on non-active - zones (before they enter the active set via the preparatory thread, - or periodically as a fallback) — no concurrent allocate. -- **Compaction crash safety** — the snapshot write and free-record - deletion are one atomic `batch_write`; they succeed or fail together. - If diskdb crashes during compaction, either both the new `ZoneValue` - (with `compact_slot` advanced) and the record deletions are - durable, or neither is. There is no window where the snapshot is - written but its corresponding deletions do not. -- **Re-allocate after compaction** — once compaction has cleared the - bit and deleted the matching busy/free records, the block is available for - re-allocation. The re-allocate is a normal allocate (CAS-set bit + - persist `BusyBlockValue`) with a higher `allocation_ts`. A delayed old free - creates only its old incarnation key; later compaction rejects the mismatch. - -The bitmap is never persisted on the allocate/free hot path. Only the -`ZoneValue` snapshot (written by compaction) carries a serialized -bitmap, with `snapshot_slot` (replay start point), `compact_slot` -(compaction watermark), and `crc32` (integrity check over -`bitmap_codec` + compressed `usage_bitmap` + `compact_slot`). The baseline -`ZoneValue` is written during disk-add initialization (root §8). A disk without -snapshot, busy or free records uses three bounded prefix reads to prove it is -fresh, then publishes initial zone records in batches of at most 128 through -its ownership fence. Its baseline `snapshot_slot` captures the applied frontier -before the disk becomes allocatable. Existing records use ordinary recovery. -Empty zones keep only geometry and allocation counters; bitmap words are -materialized by atomic publication on first allocation. The standard empty -snapshot is copied directly from its compressed constant. Baseline write failure -keeps the disk unavailable; restart loads already published records and fills -missing baselines. all subsequent state changes are `BusyBlockValue` / -`FreeBlockValue` records until compaction merges them into a fresh -snapshot. - -**Hot-path error handling:** - -- **Allocate persist failure** (Phase 2 `batch_write` fails): rollback - the bitmap (clear the bits set in Phase 1 via - `zone.rollback_allocate`) and return error. No record was written, so - the record set is consistent (no ghost). `rollback_allocate` is the - bitmap CAS-clear, used only by allocate rollback, not by the free - path. -- **Free persist failure** (`batch_write` fails): the bitmap was - never touched. Return error; the block is still busy on both disk - and memory. The caller can retry safely. -- **Degraded mode** (group-0 / data-group unreachable): allocate/free - RPCs return `Unavailable` before touching the bitmap. No partial - state. - -### Recovery and compaction engines - -The recovery and compaction logic is implemented by two engines owned -by the diskdb server: - -- **`RecoveryEngine`** — owns a `DataGroupClient`. Runs during startup - and on ownership transfer. For each owned disk-group it builds an - empty `Node` and calls `RecoveryEngine::recover_node`, which - reconstructs each zone via `recover_zone`. -- **`CompactionEngine`** — owns a `DataGroupClient`. Runs as a - background `compaction_loop` over the owned nodes in - `NodeContainer`. Also drives the preparatory thread for - compaction-on-rotation. - -`recover_node` creates one `ZoneDisk` per disk, recovers each zone in -parallel (bounded by `recovery_concurrency`), adds the zones to the -disk, rebuilds the active zone set (picking ready / compacted zones), -and returns the reconstructed `Node`. - -`recover_zone` (strategy 2) is the primary restart path: - -1. Load the latest `ZoneValue` for the zone with a single point get. If - the CRC32 fails or no snapshot exists, proceed from an empty bitmap - or fall back to strategy 1. -2. Initialize the bitmap from the snapshot, or from empty. -3. Issue two narrow `JournalScan`s from `snapshot_slot + 1` to the - applied frontier: one over `BusyBlockKey` and one over `FreeBlockKey` - prefixes. Merge the two op lists by slot. `JournalScan` supports - slot-range, key-prefix, and transparent pagination; a - `KV_ERROR_JOURNAL_SCAN_GC_GAP` response falls back to strategy 1. -4. Apply only `Put BusyBlockKey` ops in slot order: each sets the bit - range using `BusyBlockValue.unit_count`. `Delete BusyBlockKey` ops - (frees) are **ignored**. The bitmap stays as a conservative over- - estimate (persist-only recovery, consistent with the free path - during normal operation). `Put`/`Delete FreeBlockKey` are no-ops for - the bitmap. The free records on disk are the source of truth for - what's freed; the common compaction flow (§5) will `range_clear` - their bits naturally. This eliminates the double-free risk that - would arise if recovery cleared bits for frees while the free - records still exist on disk. -5. Return the rebuilt `Zone`, resetting the rotating cursor. Mark the - zone as `compacted_ready = true` (it was just recovered from - records). `compact_ts` stays at `snapshot.compact_ts`, no - advancement needed. The free records on disk have `freed_ts > - snapshot.compact_ts` (they were written after the snapshot), so the - next compaction will classify them as "new" and `range_clear` their - bits, correct, since those blocks ARE free (no `BusyBlockKey` on - disk). Count the net free records (Put FreeBlockKey = +1, Delete - FreeBlockKey = -1 from re-allocate) to initialize - `uncompacted_free_record_count` so the compaction engine knows the - backlog. The per-disk-group monotonic timestamp source is - initialized to `max(now(), max(freed_ts of all scanned free - records) + 1)` after all zones in the disk-group are recovered - (§8 Monotonic timestamp source). - -`rebuild_zone_bitmap_full_scan` (strategy 1) is the on-demand fallback: - -1. Read all `BusyBlockKey`s for the zone via `read_zone_records` (key - order equals `unit_offset` order). `FreeBlockKey`s are ignored for - state determination. -2. Set the bit range for each busy record using `unit_count` and build - the `Zone`. -3. Optionally write a fresh `ZoneValue` snapshot so the next restart can - use strategy 2. -4. Return the rebuilt `Zone` plus derived stats. Mark the zone as - `compacted_ready = true`. - -`compact_zone` (strategy 3) is the background maintenance task. See -§5 Compaction algorithm. - -### Recovery triggers - -- **Startup**: after the blocking `sync_once` fetches owned - disk-groups, `RecoveryEngine::recover_node` runs for each one. The - rpc server does not accept RPCs until recovery completes. -- **Ownership transfer**: when `SyncLoop` detects a disk-group newly - assigned to this instance, it checks whether `ZoneValue` snapshots - already exist. If they do, the new owner runs - `RecoveryEngine::recover_node` and discards any stale in-memory state; - otherwise it initializes the disk-group via `disk_add_init`. - -## 7. Zone Allocation State (derived) - -`ZoneAllocationState` is a **derived** enum for reporting only: -`Active` (`used_count == 0`), `Available` (0 < `used_count` < -`unit_capacity`), `Full` (`used_count == unit_capacity`). There is no -CAS state machine. Allocation concurrency is handled by per-bit CAS on -the usage bitmap. A zone is allocatable when it is Healthy (inheriting -the disk's `HwStatus`) and `used_count < unit_capacity`. - -In the persist-only model, `used_count` is only decremented by -compaction (never by free). So after a free, `used_count` still reflects -the pre-free count. The zone may show `Full` even though it has freed -blocks on disk. This is correct: the bitmap is a conservative over- -estimate, and the derived state reflects the bitmap, not the durable -records. After compaction clears the freed bits and recomputes -`used_count`, the derived state accurately reflects the durable state. -A zone that was `Full` before compaction may become `Available` after -compaction, and then it is eligible for rotation back into the active -set. - -## 8. Concurrency Model - -All public and inter-module APIs are `async`. Runtime is `tokio` -(multi-threaded for production). - -### Allocate (lock-free) - -Per-bit CAS on the usage bitmap (`compare_exchange` on 64-bit words). -Multiple threads allocate from the same zone concurrently; no zone-level -lock held across `.await`. CAS losers re-scan the same word or try the -next zone; no thread blocking. Per-bit CAS is capped at `cas_retry_limit` -retries (default 100), then falls through to the next bit / word / zone. -Allocate runs only on active zones (in the `active_zone_context`). - -### Free (persist-only, no lock) - -Blindly put the incarnation-qualified `FreeBlockValue`; validation is deferred -to compaction, which matches it against the current busy incarnation. No -bitmap touch, no `used_count` decrement, and no zone-level lock. -Free can run on any zone (active or not); successful frees increment -`uncompacted_free_record_count`. The bitmap is reconciled later by compaction. - -### Zone-level lock for non-allocate operations - -Compaction, scanner, and health checks acquire a zone-level lock -(`RwLock<()>` on `DdbZone`) to coordinate with each other. These -operations run on **non-active zones only** (no concurrent allocate), so -the lock does not contend with the allocate hot path. The lock protects -only the in-memory bitmap mutation. It is **not** held across `.await` -on the KV client. The KV read is done before acquiring the lock, and the -KV write is done after releasing the lock. - -### Common methods on DdbZone - -Common methods on `DdbZone` encapsulate the lock + operation: - -- `compact_zone_inner()` — bounded-read busy/free records (KV read, no lock), - match allocation incarnations, acquire the zone lock, `range_clear` only - matched free records, recompute `used_count`, release the lock, then write - the new `ZoneValue` and exact deletions in one atomic `batch_write`. Used by `CompactionEngine` - (preparatory thread + periodic fallback). -- `scan_zone_inner()` — replay journal (KV read, no lock), acquire zone - lock, compare bitmap (in-memory), release lock. Used by the scanner. -- `health_check_zone_inner()` — verify zone records (KV read, no lock), - acquire zone lock, verify CRC + snapshot integrity (in-memory), - release lock. Used by the health probe. -- Each method acquires the zone lock only for the in-memory bitmap - mutation/verification, and releases it before any KV write. -- Allocate does **not** acquire this lock — it uses per-bit CAS - (lock-free). The lock only coordinates non-allocate operations on - non-active zones (no concurrent allocate). - -Disk-level active zone set uses RCU publish (`Arc` swap) for lock-free -reads; rotation takes a brief write lock. - -### Free-side lookup structures - -RCU-published alongside the allocate context on add/remove/status-change: - -- **disk-id → disk**: a hash map (O(1) average). Used by free to find - the in-memory disk from the `disk_id` in the `Segment`. -- **zone-index → zone**: a vec indexed by zone-index (O(1) direct - index). Used by free to find the in-memory zone from the - `zone_index` in the `Segment`. (Lookup is for - `uncompacted_free_record_count` increment; no bitmap mutation.) -- **KV free path:** the incarnation-qualified `FreeBlockKey` is constructed - directly from the `Segment`. `owner_chunk` and `allocation_ts` are carried - in the segment and become `previous_owner` and `pre_allocation_ts`. Free - validates the full-engine busy value and revision before the guarded batch; - compaction consumes only the resulting self-consistent fact. - -Disk membership changes acquire the disk-list write lock only while rebuilding -the immutable membership snapshot. Allocation and free load that snapshot and -do not acquire the disk-list lock. - -### Monotonic allocation incarnation source - -`BusyBlockValue.allocation_ts` uses a per-disk-group monotonic source. The -source is an `AtomicU64` initialized above every durable busy and free -incarnation discovered during recovery. It is an identity token and does not -order compaction. `FreeBlockValue.free_ts` is independently diagnostic. - -Ownership is currently immutable. Any future ownership transfer must -reconstruct or transfer the allocation high-water mark before enabling the new -owner. - -### Bounded tentative allocation cache - -Each disk group keeps recently allocated `TentativeBlock` values in a -lock-free ordered index keyed by `allocation_ts`. The default capacity is -262,144 unique incarnations. A per-entry pending/count/removed state makes the -atomic size budget exact when duplicate publication races removal. One atomic -trim owner removes the oldest entries until the cache is within capacity. - -Commit uses an exact matching tentative-cache entry without a read. A miss -after retry, restart, or eviction reads and validates the authoritative busy -incarnation. Tentative values are changed to Committed in one ordinary batch -write; an already Committed value is idempotent success. Commit and free remove -only exact incarnation cache entries. - -## 9. Background Scanner Coordination - -The background scanner (root §12) is a periodic consistency check that -detects live-state drift, catches record corruption early, and gives -operators visibility into cluster health during uptime. It coordinates -with compaction via the zone-level lock (§8): - -- **Zone skipping and locking** — the scanner skips zones in the disk's - `active_zone_context` (the allocator is actively handing blocks from - them — transient drift from the allocate Phase 1→2 window is - expected). For non-active zones, the scanner acquires the zone-level - lock (§8) to coordinate with compaction and health checks. Skipped - zones are checked on a later cycle when they rotate out of the active - set (and have been compacted). -- **Compaction coordination** — compaction and the scanner share the - zone-level lock (§8). They run on non-active zones only. The scanner - skips zones locked by compaction (or waits briefly, since compaction - is fast). Common methods on `DdbZone` (§8) encapsulate the lock + - operation, used by both compaction and the scanner. -- **Drift detection in the persist-only model** — the free path does - not touch the bitmap (§4 Free), so "bit set, no `BusyBlockKey`" is - **normal** for freed-but-not-compacted blocks (a `FreeBlockKey` - exists). The scanner distinguishes: - - **Real ghost-busy** (drift): bit set, no `BusyBlockKey`, no - `FreeBlockKey` — the block was never freed and never allocated - (crash between allocate Phase 1 and Phase 2, or a bug). Records - are authoritative → block is free → safe to clear the bit. - - **Normal uncompacted**: bit set, matching `BusyBlockKey` and `FreeBlockKey` - exist — the block was freed (persist-only) but compaction hasn't - cleared the bit yet. This is **not drift** — it's the expected - state. The scanner does not report it. (If the zone is not active - and has a high `uncompacted_free_record_count`, the scanner may - log a hint that compaction is lagging, but it's not a drift - finding.) - - **Ghost-free** (drift): bit clear, `BusyBlockKey` exists — should - not happen in the persist-only model (free never clears bits, and - only allocate/compaction touch the bitmap). If detected, it - indicates a bug or hardware error. Records are authoritative → - block is busy → set the bit back. Data may be written. - -## 10. Invariants - -- **I1 — Bitmap is a conservative over-estimate**: the in-memory - `usage_bits` never shows a freed block as free until compaction clears - the bit. Free is persist-only (no bitmap touch); compaction is the - sole bit-clearer. `used_count` is only decremented by compaction - (never by free). -- **I2 — Records are the source of truth**: the current busy record and an - incarnation-matching free fact determine whether a block is logically - busy. The bitmap is derived from records and may lag conservatively. -- **I3 — Compaction is the sole bit-clearer**: no code path except - compaction (and allocate rollback, which only clears Phase 1 claims - that were never persisted) clears bits in `usage_bits`. The free path - never touches the bitmap. -- **I4 — Compaction runs on non-active zones only**: no concurrent - allocate. The zone-level lock coordinates compaction, scanner, and - health checks on the same zone. -- **I5 — Compaction-before-rotation**: a zone must be compacted (freed - bits cleared, `used_count` recomputed, snapshot written) before it - enters the active set. The preparatory thread pre-compacts the next - batch; rotation publishes ready (pre-compacted) zones. -- **I6 — Atomic snapshot + delete**: compaction writes the new - `ZoneValue` with its advanced `compact_slot`, deletes matched busy records, - and deletes processed free facts in one atomic `batch_write`. -- **I7 — Incarnation match prevents stale free**: compaction clears a range - only when `pre_allocation_ts`, unit count, and owner match the current busy - value. An old retry cannot clear or delete a newer allocation. -- **I8 — `rollback_allocate` is allocate-only**: the bitmap CAS-clear - method is used only by the allocate Phase 2 failure path to undo - Phase 1's claim. It is never called by the free path. -- **I9 — Zone lock not held across `.await`**: the zone-level lock - protects only in-memory bitmap mutation. KV reads happen before - acquiring the lock; KV writes happen after releasing it. -- **I10 — Persist-only recovery**: recovery replays busy records - conservatively and leaves matching free facts for the common compaction - flow. It never makes a range reusable from an unmatched or incomplete - record view. - -## 11. Tunables and Defaults - -- `zone_rotate_count` — number of zones in the disk-level active zone - set (default 4). The disk round-robins over this many zones at a time; - when all are exhausted, the set rotates to a new batch of pre-compacted - zones. -- `cas_retry_limit` — per-bit CAS retry cap in the zone bitmap-scan - allocator (default 100). On exhaustion, the allocator falls through to - the next bit / word / zone. -- `zone_size_bytes` — default zone size in bytes (default 16 GB). -- `block_size_bytes` / `allocate_granularity` — block unit size (default - 1 MB; configurable 512 KB–2 MB; must be power of 2). v1 enforces - `allocate_granularity == block_size_bytes`. -- `compaction_cadence_secs` — periodic compaction interval (default - 300). The compaction loop sleeps this long between cycles. -- `snapshot_compaction_threshold` — compact a zone when its - `uncompacted_free_record_count` exceeds this (default 4096). Cadence - OR threshold — whichever fires first for a given zone. -- `validate_owner_on_free` — removed; compaction validates the free fact - (default false). When true, the free path reads the `BusyBlockValue` - from the data group first and validates `owner_chunk` (one extra paxos - round-trip, doubles free latency). -- `free_batch_enabled` — free batching toggle (default false). When - false, each request immediately issues one durable `batch_write`. When true, - the first queued request still drains immediately, while concurrent requests - accumulated during an in-flight write are coalesced by bind with no timer. -- `free_flush_max_batch` — maximum record operations in one coalesced KV - proposal (default 256). Requests are never split; one oversized request is - persisted alone. -- `recovery_concurrency` — max concurrent zone recoveries in - `recover_node` (default 16). diff --git a/doc/design/diskdb/design-crowdb-diskdb.md b/doc/design/diskdb/design-crowdb-diskdb.md deleted file mode 100644 index 2dc994d4e..000000000 --- a/doc/design/diskdb/design-crowdb-diskdb.md +++ /dev/null @@ -1,1066 +0,0 @@ - - - -# CROWDB - Design: diskdb (Overview) - -This is the root design document for the diskdb component area. It -defines **what diskdb is**, **why key choices were made**, and **how the -component is structured**. Field-level details live in the schema files -and Rust source; this doc covers decisions and architecture only. - ---- - -## Table of Contents - -- [1. Overview](#1-overview) -- [2. Non-Goals (Design Envelope)](#2-non-goals-design-envelope) -- [3. Key Design Decisions](#3-key-design-decisions) - - [3.1 Group 0 is the centralized sysdata store](#31-group-0-is-the-centralized-sysdata-store) - - [3.2 disk-group → paxos group binding via a table (not hash)](#32-disk-group--paxos-group-binding-via-a-table-not-hash) - - [3.3 Incarnation-safe lifecycle transitions](#33-incarnation-safe-lifecycle-transitions) - - [3.4 Records are the source of truth; bitmap is derived](#34-records-are-the-source-of-truth-bitmap-is-derived) - - [3.5 Zone is a logical concept; sizes may vary](#35-zone-is-a-logical-concept-sizes-may-vary) - - [3.6 Common protocol crate; crowdb-rpc transport](#36-common-protocol-crate-crowdb-rpc-transport) - - [3.7 Reuse crowdb-common metrics](#37-reuse-crowdb-common-metrics) - - [3.8 Schema types used directly; no Rust type duplication](#38-schema-types-used-directly-no-rust-type-duplication) - - [3.9 Unit-based sizes; disk-id key routing](#39-unit-based-sizes-disk-id-key-routing) -- [4. Architecture Overview](#4-architecture-overview) -- [5. Group-0 Sysdata Schema](#5-group-0-sysdata-schema) -- [6. Hierarchy](#6-hierarchy) -- [7. Zone Management](#7-zone-management) — detailed design in [`design-crowdb-diskdb-zone-management.md`](design-crowdb-diskdb-zone-management.md) -- [8. Disk Status Management](#8-disk-status-management) -- [9. Space Metrics](#9-space-metrics) — detailed design in [`design-crowdb-diskdb-space-metrics.md`](design-crowdb-diskdb-space-metrics.md) -- [10. Background Scanner](#10-background-scanner) -- [11. Crate Layout](#11-crate-layout) -- [12. Concurrency Model](#12-concurrency-model) -- [13. Non-Gaps (Good Fits with CROWDB)](#13-non-gaps-good-fits-with-crow) -- [14. Configuration](#14-configuration) -- [15. Implementation Scope](#15-implementation-scope) -- [16. References](#16-references) -- [17. DiskdbClient Scanner / Rebuild Wrappers](#17-diskdbclient-scanner--rebuild-wrappers) -- [18. Group-0 Status Write-Back: Init → Offline](#18-group-0-status-write-back-init--offline) -- [19. Init-state zone load](#19-init-state-zone-load) - -## 1. Overview - -diskdb is a **distributed disk-block allocator** that runs on top of -CROWDB's KV cluster. It is a **lightweight, stateless server**: a diskdb -instance takes ownership of some disk-groups, manages the disks and -space inside them, and persists all durable state to CROWDB KV. It holds -no state that cannot be reconstructed from KV. On crash or restart it -rebuids in-memory structures from the KV records and group-0 metadata. - -Multiple diskdb instances run across a cluster, each managing a subset -of disk-groups. diskdb provides fast block allocation/deallocation on -physical disks using per-zone bitmap-scan allocators with per-bit CAS -and a rotating cursor for O(1) block tracking. The free path is -persist-only (the bitmap is a conservative over-estimate; compaction -reclaims freed space). All state changes are durably persisted to CROWDB -KV before being acknowledged to callers. - -diskdb allocates blocks and does not serve foreground block contents. Callers -write their own allocated blocks. For background relocation only, DiskDB -orchestrates an exact source-to-target copy through DiskIO before asking the -ChunkDB owner to publish the new layout. - -**Language:** Rust. **Runtime:** tokio (async everywhere). - -**Core goals:** -- **Fast allocation** — per-zone bitmap-scan allocators with per-bit - CAS (no zone-level lock); compaction reclaims freed space. No - KV-level CAS on the hot path. -- **Durability via CROWDB KV** — the paxos journal is the sole durable - store; diskdb has no local WAL and is stateless on disk. -- **Crash safety via record replay** — busy/free records in the paxos - slots are the source of truth; the bitmap is derived and rebuilt on - restart. -- **Accurate space metrics** — per-disk / per-disk-group / per-zone - capacity and busy/free, with a recalculation path to verify - correctness. - -**Design philosophy:** "diskdb is a thin, stateless client of crowdb-kv." -All consensus, replication, and durability are delegated to crowdb-kv. -diskdb's job is allocation policy, disk health, and space accounting — -nothing more. One crowdb-kv extension is required: a `JournalScan` RPC -(slot-range + key-prefix filter, returns ops in slot order) for fast -crash recovery (zone-management §6, strategy 2). This is the sole extension; all other -diskdb ↔ crowdb-kv interaction uses the existing scan / get / put / -batch_write API. - -## 2. Non-Goals (Design Envelope) - -- **No direct device I/O.** diskdb does not serve foreground block contents or - implement a device engine. Background relocation delegates copy and fsync to - DiskIO and persists only allocation and relocation metadata. -- **No local WAL.** CROWDB KV's WAL is the sole durability mechanism. -- **No consensus code.** diskdb is a client of crowdb-kv, with one - extension: a `JournalScan` RPC for fast crash recovery (zone-management §6). All - other interaction uses the existing crowdb-kv API. -- **No native SMR / zoned-namespace SSD support (v1).** Start with - conventional HDD/SSD (`ZoneBlockDisk`); the zone is a logical concept - that can later adopt to native zone APIs. -- **No leak-detection scanner (v1).** Leak detection needs caller - registries that do not exist yet; ship ghost/drift/integrity only. -- **No rack/data-center aggregated dashboards (v1).** Per-disk and - per-node metrics first; higher-level aggregation later. -- **No console UI (v1).** Minimal HTTP management API first; full - console integration (web + CLI) as a follow-up. - -## 3. Key Design Decisions - -### 3.1 Group 0 is the centralized sysdata store - -Group 0 (store 0, group 0) holds the basic metadata for the diskdb -subsystem: nodes, disks, disk-groups, and the two maps (disk-group → -diskdb instance, disk-group → paxos data group). diskdb fetches all -metadata from group 0 on startup and on sync. Any local status change -(disk found, disk bad, disk added/removed) is written to group 0 first, -then reflected locally. - -**Zones are NOT maintained in group 0** — a zone belongs to a disk, is -created when the disk is added, and is maintained as a separate zone -record on the disk-group's bound paxos data group. - -### 3.2 disk-group → paxos group binding via a table (not hash) - -A disk-group's zone records all live on **one paxos data group** so -multi-block allocation can use a single `batch_write` (atomic within a -group). The disk-group → paxos-group mapping is a **bind table stored in -group 0**, not a hash. A table (vs hash) enables dynamic scaling: new -paxos data groups can be added and disk-groups rebound without -rehashing the whole keyspace. - -**Allocation routing:** the `AllocateBlocks` request carries -`disk_group_id` (not `node_id`); CROWDB uses disk-groups instead of nodes -as the allocation routing unit. Allocate is scoped to one disk-group; -multi-block uses one `batch_write` on that group's bound paxos data -group; atomic within the group. No cross-group multi-block allocate in -v1. The caller issues separate `AllocateBlocks` calls per group if -needed. A joint placement caller may set `allow_disk_reuse` to execute -multiple distinct-disk passes in one atomic per-group request. The caller is -responsible for validating that roles requiring mutual anti-affinity fit in -the first pass. The caller, normally ChunkDB placement, picks the disk-group. - -Within the selected disk-group, allocation remains lock-free and applies the -configured load-aware policy before cursor fallback. `free_bytes` chooses the -eligible disk with the most absolute free space; `inverse_used_pct` chooses the -lowest normalized utilization. Equal scores retain rotating cursor order, -`exclude_disks` remains a hard constraint, and disabling load awareness restores -pure round-robin selection. This affects new allocations only; existing blocks -move through the fenced relocation path in §10. - -### 3.3 Incarnation-safe lifecycle transitions - -Each disk-group is owned by exactly one diskdb instance at a time (map in group -0). Allocation and free use batchable writes guarded by the ownership fence; allocation -incarnations and compaction-time validation make delayed frees safe. Commit -uses an exact matching tentative cache entry when present; otherwise it reads -the authoritative BusyBlock and validates its allocation incarnation. It -changes Tentative to Committed with a conditional batch that preserves both -the observed record revision and ownership generation. An already -Committed record is idempotent success. -In-memory concurrency within one instance is handled by **per-bit CAS** -on the usage bitmap (`compare_exchange` on 64-bit words), not a -zone-level lock. Multiple threads can allocate from the same zone -concurrently. The `ZoneAllocationState` enum is a **derived** field -for reporting (Active/Available/Full based on `used_count`), not a CAS -state machine. - -### 3.4 Records are the source of truth; bitmap is derived - -For each allocate or free, diskdb **cannot** update the full zone -bitmap in KV, as that would be a large write for the paxos group on every -block. Instead, each allocate writes a small **`BusyBlockValue`** carrying a -monotonic `allocation_ts` at the `BusyBlockKey`. Each free writes an immutable -**`FreeBlockValue`** at an incarnation-qualified `FreeBlockKey`; it does not -delete the busy record. The bitmap is **derived** from the records, never -written directly as a full bitmap on the hot path. The free path is -**persist-only**: the bitmap is not touched on free (the bit stays set, -`used_count` is not decremented); compaction is the sole mechanism for -clearing freed bits after matching the free fact to the current busy -incarnation. This makes the bitmap a conservative over-estimate -that never shows freed space as available until compaction reconciles -it from records. - -Record key layout, value schemas, the record model, current-state -determination, and scan-ordering notes are in -[`design-crowdb-diskdb-zone-management.md`](design-crowdb-diskdb-zone-management.md) §3. - -### 3.5 Zone is a logical concept; sizes may vary - -Zone is a **logical concept** — a rough mapping that can later adopt to -native zoned-namespace SSD or SMR HDD zone APIs. The last zone on a disk -may be smaller; the word-alignment rule (each zone's `unit_capacity` is -a multiple of 64) ensures no bitmap masking or padding. Details in -[`design-crowdb-diskdb-zone-management.md`](design-crowdb-diskdb-zone-management.md) §2. - -### 3.6 Common protocol crate; crowdb-rpc transport - -A new `lib/crowdb-protocol` crate holds flatbuffers definitions for all CROWDB -components. diskdb uses it first; crowdb-kv's existing schemas stay where -they are (unchanged). The flatbuffers messages are reused across the -crowdb-rpc transport. - -### 3.7 Reuse crowdb-common metrics - -diskdb reuses `crowdb-common`'s metrics module (no parallel metrics -system). Per-disk atomic counters stay as hot-path counters that flush -into the crowdb-common registry at reporting intervals. - -### 3.8 Schema types used directly; no Rust type duplication - -Schema types (`DiskId`, `ChunkId`, `Segment`, `HwStatus`, `DiskType`, -`ZoneAllocationState`, value types) are defined in `crowdb_protocol` -and used directly in Rust, with no field duplication. Extension traits add -domain methods. Internal types (metas, bitmap, CRC logic) are Rust -structs not exposed via rpc. - -**Keys are not flatbuffers.** KV keys use the cross-component binary -encoding defined in `doc/design/protocol/design-crowdb-protocol-key.md` (flat -per-kind Rust structs, two-byte `magic | type_tag` header, big-endian -fixed-width fields). Flatbuffers-serialized `*Key` messages are never -used as KV key bytes. The tag bytes break prefix scans. RPC -responses/requests use `**Info` schema messages that flatten key and -value fields into one message; see §5 and zone-management §6. - -### 3.9 Unit-based sizes; disk-id key routing - -All sizes are in units (block unit size defined per-disk, default 1M). -Record keys are disk-id-based (globally unique → reverse-lookup to the -data group). No `node_id`/`disk_group_id` in record keys or in -`Segment`. The `AllocateBlocks` request carries `disk_group_id` for -routing (§3.2); it is not stored in record keys or `Segment`. - -## 4. Architecture Overview - -``` - CROWDB Cluster - ┌────────────────────────────────────────────────────────────────┐ - │ diskdb-A diskdb-B diskdb-C │ - │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ - │ │ disk-grp1│ │ disk-grp2│ │ disk-grp3│ │ - │ │ disk-grp4│ │ disk-grp5│ │ disk-grp6│ │ - │ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ - └───────┼─────────────────────┼─────────────────────┼────────────┘ - │ crowdb-kv-client │ │ - v v v - ┌────────────────────────────────────────────────────────────────┐ - │ CROWDB KV (Multi-Paxos) │ - │ group 0 (sysdata) group-1..N (diskdb data groups) │ - │ ┌──────────────┐ ┌──────────────────────────────────┐ │ - │ │ nodes │ │ zone records (busy/free/snapshot)│ │ - │ │ disk-groups │ │ │ │ - │ │ ownership map│ │ │ │ - │ │ binding map │ │ │ │ - │ └──────────────┘ └──────────────────────────────────┘ │ - └────────────────────────────────────────────────────────────────┘ - ^ - │ HTTP mgmt API (axum) - ┌───────┴────────┐ - │ Console/CLI │ (follow-up: disk/disk-group management) - └────────────────┘ -``` - -- **diskdb instance** — one process per node (or per disk-group set). - Owns disk-groups assigned via group 0. Stateless on disk. -- **disk-group** — logical container of disks on one node; unit of - ownership and paxos-group binding. A disk-group belongs to exactly - one node; a node can have multiple disk-groups. -- **group 0** — centralized sysdata: node/disk/disk-group metadata + - ownership map + binding map. -- **data groups (group-1..N)** — paxos groups holding zone records and - snapshots. Each disk-group is bound to one data group. - -### Protocol - -`diskdb.fbs`, `chunkdb.fbs` and `diskio.fbs` define the storage services, -with shared identities and message types. Rust and C++ use generated definitions -from the same current schemas. - -The storage path uses three RPC services: -- **`DiskdbService`** (served by the diskdb server): `AllocateBlocks` - (carries `disk_group_id`, not `node_id`; carries `exclude_disks` for - anti-affinity), `FreeBlocks`, `QueryCapacityStats`, - `GetDiskGroupInfo`, `GetDiskInfo`, `RebuildZoneBitmap` (on-demand - full-scan rebuild, strategy 1, zone-management §6), `MarkBlockSuspect`, - `MarkBlocksCorrupt` (exact committed BusyBlock incarnation transition). A - caller uses this operation only after frame parse or write-frame checksum - proves corruption; the server checks owner, geometry, allocation timestamp, - and committed state before a revision-fenced `Corrupt` update. Ordinary I/O - failures do not change the block state. Allocation and ownership-qualified - free update the durable allocator records. -- **Hardware admin** (no rpc surface): rack/node/disk-group/disk - add/remove and `set_*_status` are writes to group-0 sysdata - performed through `HardwareClient` in `crowdb-kv-client`, invoked by - the console (`crowdb-web` / `crowdb-cli`). Hardware reads use - `HardwareClient` prefix scans; liveness uses `ServiceRegistryClient.heartbeat`. - The diskdb server reads hardware state - from group 0 via `HardwareClient` in its sync loop; it does not - serve hardware admin. See `doc/design/kv/design-crowdb-kv-group0.md` - §2.8. -- **`ChunkdbService`**: `AllocateChunk`, - `AppendChunk`, `QueryChunk`, `SealChunk`, `DeleteChunk`, - `DeleteChunkRange`, `ReplaceChunkStripRange`, `ListChunks`, exact-segment owner - queries, and relocation handoff. -- **`DiskioService`**: `DiskWrite`, `DiskRead`, and `DiskFsync`. - -Key protocol decisions: integer IDs throughout (no string UUIDs); -`DiskId` is globally unique (no `node_id`/`disk_group_id` in `Segment` -or record keys. `disk_group_id` is in the `AllocateBlocks` request for -routing only); `Segment.owner_chunk` (192-bit `ChunkId`) replaces the -former `tag`; all sizes are unit-based; errors are returned via rpc -status codes with `ErrorInfo` details (`error_code.fbs`), not -`bool ok + string error` in response bodies. - -## 5. Group-0 Sysdata Schema - -Group 0 stores the diskdb subsystem metadata as KV entries. The -authoritative schema is defined in -[`doc/design/kv/design-crowdb-kv-group0.md`](../kv/design-crowdb-kv-group0.md) -(text-path keys + JSON values, owned by `crowdb-kv-client`'s -`HardwareClient` / `ServiceRegistryClient` / `KVClusterMetaClient`). -This section summarizes the diskdb-relevant parts; see the group-0 -design doc for the full key/value layout and scan patterns. - -### Key layout (text-path encoding in group 0) - -Group-0 keys use the `TextKey` encoding (`/magic/type/…`), -JSON-encoded values. The hardware hierarchy embeds the full path in -each key for scan narrowing: - -``` -/hw/rack/ -> RackValue -/hw/node// -> NodeValue -/hw/dg/// -> DiskGroupValue -/hw/disk//// -> DiskValue -/hw/dg_owner/// -> OwnerMapValue -/hw/dg_bind/// -> BindMapValue -/srv/diskdb/ -> InstanceValue -/srv/kv-server/ -> InstanceValue -``` - -`rack_id`, `node_id`, `dg_id` (`DiskGroupId`), and `instance_id` are -`u64`; `disk_id_hex` is the 32-char hex of the 128-bit `DiskId` -(`high:u64 BE | low:u64 BE`, lowercase). v1 ships flat (no DC layer); -`dc_id` is dropped from `RackKey` and the Info types. The key concept -structs live in `lib/crowdb-protocol/src/key/` and implement both -`BinaryKey` (kept for data-group records) and `TextKey` (group 0). See -`doc/design/protocol/design-crowdb-protocol-key.md` §5 and -`doc/design/kv/design-crowdb-kv-group0.md` §3. - -diskdb's own data groups (zone records) keep the `BinaryKey` encoding: -`ZoneKey`, `BusyBlockKey`, `FreeBlockKey` (binary + flatbuffers, unchanged). - -Value types live in `crowdb-protocol` (flatbuffers `*Value` types used -directly; new sysdata values `OwnerMapValue`, `BindMapValue`, -`InstanceValue` added as schema messages; Entry return types -`DiskGroupEntry`/`DiskdbOwnerEntry`/`KVGroupBindEntry` are plain serde -structs). See `design-crowdb-kv-group0.md` §3.3. - -**Note**: zones are NOT stored in group 0. A zone is created when its -disk is added, and its state (records + snapshot) lives on the -disk-group's bound data group. - -### Map semantics - -- **Ownership map** (`/hw/dg_owner/...`) — the Group 0 leader's - `diskdb-ownership` monitor assigns bound disk-groups to live, fencing-capable - DiskDB instances. Missing or dead owners are replaced; count balancing moves - at most one healthy group per tick. Updates compare the owner record revision. - Before a live capable owner's one-hour placement lease expires, the monitor - renews it through the same revision-checked publication path without changing - the instance or process epoch. The DiskDB observes the new map generation and - reacquires its data-group fence. Fresh leases are not rewritten each tick; - renewals defer count balancing until the next snapshot. -- **Binding map** (`/hw/dg_bind/...`) — the console recommends the ordinary KV group with the fewest - bound DiskGroups cluster-wide and allows an explicit override at creation and publishes the binding, hardware record, - and node membership atomically. Missing selections, Group 0, and unknown - groups are rejected. Existing bindings are fixed; adding KV groups does not - migrate DiskGroups. The monitor skips unbound groups without repairing them. -- **Ownership fence** — before loading a group's allocation state, the target - claims `/diskdb/ownership-fence///` in its bound data group, - using the Group 0 owner revision as generation. Metadata requests carry the - expected owner identity; the KV leader validates its applied fence locally - and admits concurrent writes without changing that fence. Business-record - revision CAS remains independent. Only an ownership CAS closes admission, - drains previously admitted writes through apply, and persists the new fence. - Admission survives request cancellation and topology snapshot replacement; - unknown outcomes disable readiness until the leader recovery barrier resolves - prior slots. Background tasks retain their exact group's generation, so stale - work cannot write after a newer generation claims the fence. Ordinary writes - cannot mutate the fence namespace. All participating KV and DiskDB servers - must support owner fencing before automatic handover is enabled; older KV - servers reject the versioned owner request rather than silently bypassing it. -- **Service registry** (`/srv//`) — written by - each service instance on heartbeat (diskdb, kv-server); used by the - console and other components for discovery. - -## 6. Hierarchy - -CROWDB v1 has rack and node (no data-center layer). The physical -hierarchy and the logical disk-group layer: - -- **Physical**: rack → node → disk. These are real hardware - containers. v1 ships with rack and node; a data-center layer is not - in v1 (the schema drops `dc_id`). -- **Logical**: disk-group sits between node and disk. A disk-group - belongs to exactly one node; a node can have multiple disk-groups. A - disk-group is the unit of ownership (assigned to one diskdb instance) - and the unit of paxos-group binding (a disk-group's zone records all - live on one paxos data group). - -``` -Rack (rack_id, u64) → Node (node_id, u64) → Disk-Group (dg_id, u64) [logical] - → Disk (DiskId, 128-bit) → Zone (index, variable size) → Disk-Block (1 MB aligned) -``` - -## 7. Zone Management - -diskdb's second major component. The zone is the unit of allocation: -each disk is divided into zones, and each zone has a bitmap-scan -allocator with per-bit CAS (lock-free allocate). The free path is -**persist-only**: the bitmap is not touched on free; compaction is the -sole bit-clearer. Zone rotation (compaction-before-rotation with a -preparatory thread) ensures the allocator always sees accurate free -space in active zones. Crash recovery uses three strategies: full scan -rebuild, journal scan replay, and compaction. - -Detailed design (record model, allocation algorithm, free path, -compaction, rotation, crash recovery, zone-level concurrency, and -invariants) lives in -[`design-crowdb-diskdb-zone-management.md`](design-crowdb-diskdb-zone-management.md). - -## 8. Disk Status Management - -### Node / disk-group / disk HwStatus - -Shared enum `HwStatus`: `Init`, `Up`, `Maintenance`, `Suspect`, -`Missing`, `Bad`, `Offline` (ordered by severity). Effective status = -`max(node, group, disk)`, a three-level check (node + disk-group + -disk). The reference impl checks two levels (node + disk); CROWDB adds -the disk-group layer, which is new. Allocations require effective `Up`; -frees allow `Up`, `Maintenance`, or `Suspect`. - -Transitions: -- Init → {Up, Offline, Maintenance} on startup (load from group 0). -- Up → Suspect (3 missed syncs). -- Up → Offline / Maintenance (operator). -- Suspect → Up (sync recovers) or → Missing (cannot probe) or → Offline. -- Missing → Bad (confirmed) or → Up (rediscovered). **Missing is - detected by absence from a group-0 sync response** — a disk/node - absent from the sync response is transitioned to `Missing` (then to - `Bad` after confirmation or `Up` if rediscovered). -- Offline ↔ Maintenance (operator). -- Offline → Up (operator). - -An unknown or cancelled durable metadata write closes its `DiskWriteGate`. -The local disk then reports Offline even if Group 0 still records Up; this -is unresolved persistence, not evidence of exhausted capacity. `/ready` returns -503 with the affected DiskGroup, disk and `metadata-write-outcome-unknown` reason. -Health reads only observe this state. - -The process recovery watcher detects that uncertainty, drains the service and -stops. The monitor supervises the existing Start/Restart intent and launches a -new process. Its process session advances the owner generation and claims the -data-group fence before reconstructing allocation state from durable records. -Old process writes remain fenced and old disk snapshots retain their closed -gates. Recovery never clears an uncertain gate or treats uncertain occupied -space as free. Administrative Offline without an unknown write does not trigger -this process recovery; its explicit Online authorization remains required. - -Usage reports derive allocatable disk count and bytes from current disk gates, -independently of the cached allocation membership. Allocation failures with no -currently usable disks return Unavailable, including group/disk state, unknown -write state and capacity counters. NoSpace describes insufficient eligible -space or placement on usable disks. - -diskdb's first major component: -- **Sync with group 0** (fixed 10 s interval, same on success and - failure — no error back-off in v1): fetch latest metadata; update - group 0 first when local status changes (disk found, disk bad, disk - added/removed). A disk/node - **absent from a sync response** is transitioned to `Missing` (then to - `Bad` after confirmation or `Up` if rediscovered) — this is how - `Missing` is detected (§8). -- **Disk discovery + health probing**: discover local disks (config- - driven for v1; `/dev` scan later); probe health (existence, size, - basic read/write test). Source of truth for disk identity/capacity is - group 0; live health is probed locally. -- **Disk failure detection + bad-disk handling**: on disk failure, - transition to `Suspect`, update group 0. When a disk transitions to - `Bad` (via `Missing → Bad` after confirmation, §8), its busy blocks - are no longer readable. diskdb does **not** rebuild or relocate them - inline on the sync path — relocation/rebuild is a follow-up - requirement. The bad-disk handling on the sync path: - - Mark the `ZoneDisk` and all its `Zone`s as `Bad` - (`zone_state = Bad`; `allocatable()` returns `false`). No new - allocates touch the disk. Free is also blocked — - `allows_free(Bad)` is `false` (free allows `Up`/`Maintenance`/ - `Suspect` only, §8). - - Scan the zone records for the bad disk (`read_zone_records` per - zone, zone-management §6) and collect all live `BusyBlockValue`s — these are the - impacted blocks. Each carries `owner_chunk` (the chunk that owns - the allocation) so the caller / data-IO layer can be notified. - - Emit the `disk.bad.impacted_blocks` gauge (§9) and log the hand-off. The - dedicated bad-disk recovery flow still needs to turn this list into - ChunkDB repair work. The ordinary relocation journal is not authority to - reconstruct data that is already unreadable. - - The disk stays `Bad` — its records are read-only until an operator - removes the disk or marks it `Up` after repair (which triggers - strategy-1/2 recovery, zone-management §6). -- **Disk-add initialization flow**: the operator adds a disk in group 0 - (via `HardwareClient.add_disk` through the console), which writes - `DiskValue` to group 0. On the next sync tick, diskdb fetches the - updated `DiskValue` and sees a disk not yet in its in-memory state. - diskdb then initializes the disk: creates the in-memory `ZoneDisk` - with one `Zone` per zone (zone count = `capacity / zone_size`, last - zone sized per - zone-management §2's word-alignment rule), and writes baseline `ZoneValue` records - (empty bitmap, `snapshot_slot = 0`) to the bound data group at - `ZoneKey { disk_id, zone_index }` in one `batch_write`. These are the - replay baselines (zone-management §6); subsequent allocates write `BusyBlockValue` - records on top. Group 0 holds disk metadata only; zone records live - on the bound data group. - -**Follow-up — group-0 notify/watch:** the watch/notify mechanism is a -**client-pulled `WatchNotify` bidi stream**: diskdb opens the stream to -the group-0 leader and subscribes to the prefixes it cares about -(`/hw/dg_owner/`, `/hw/dg_bind/`, `/hw/disk/`); the leader pushes -hw-status-change and ownership-change notifications over that stream. -No separate notify-endpoint registration is needed for notify delivery. -The `rpc_endpoint` arg of `heartbeat_diskdb` is the diskdb rpc -service address for service-registry discovery (so clients can route -`allocate_blocks`), already populated from `config.server.listen_addr`. -This replaces polling for status changes as the primary -change-detection mechanism; polling stays as a safety net with an -increased interval. The detailed design (schema, registry, apply-path -trigger, client, diskdb handler, configuration) lives in -[`design-crowdb-kv-watch-notify.md`](../kv/design-crowdb-kv-watch-notify.md). - -## 9. Space Metrics - -diskdb's third major component. Detailed design (usage accessors, -`QueryCapacityStats` handler, per-disk counters, keepalive piggyback, -recalc verifier, reporting loop, schema, kv-client aggregation, and the -full `crowdb-diskdb-client` library) lives in -[`design-crowdb-diskdb-space-metrics.md`](design-crowdb-diskdb-space-metrics.md). -This section carries the architecture and the metric categories. - -Metrics must show **internal status** -(gauges reflecting current state) and a **latency hierarchy** (where -time is spent, broken down by layer) so operators can diagnose both -capacity problems and performance bottlenecks. - -**Space accounting:** -- **Per-disk** — capacity, busy, free, active zone count. -- **Per-disk-group** — aggregated from disks. -- **Per-zone** — dive into busy/free blocks (for the console zone - visualization). -- **Accurate + recalculation** — statistics are derived from the - in-memory bitmap (which is derived from the records). A recalculation - path replays the records to verify the bitmap matches the derived - statistics, detecting drift. - -**Disk-group-level usage summary on keepalive:** diskdb piggybacks a -per-disk-group usage summary (`capacity_bytes`, `used_bytes`, `free_bytes`, -allocatable capacity/used/free bytes, disk counts, and sample time) on the -keepalive message sent to group 0 on each sync tick. Group 0 maintains this at -the disk-group level (`DiskGroupUsageKey { disk_group_id }`). The -console reads this for cluster-wide overview; per-disk/per-zone -drill-down is via the `QueryCapacityStats` API. The summary is -derived (recomputed from the in-memory bitmap on each tick), not a -source of truth. - -**Metrics categories:** - -Metrics reuse `crowdb-common`'s metrics module. Per-disk hot-path -counters (atomics) flush into the crowdb-common registry at reporting -intervals. - -**1. Counters (events, monotonically increasing):** -- per-zone: `allocate.count`, `free.count`, - `allocate.retry.cms.bit.count` (contention signal — ties to §8 CAS - retry bound) -- per-disk: `allocate.count`, `free.count` -- per-disk-group: `allocate.count`, `free.count` -- per-instance: `sync.count`, `sync.error.count`, - `compaction.count`, `compaction.error.count`, and free-batch input request/ - record, output KV batch/record, oversize, and failure counters - -**2. Gauges (internal status, current state snapshot):** -- per-disk: `capacity_bytes`, `used_bytes`, `free_bytes`, `used_pct`, - `zone_count`, `active_zone_count` -- per-zone: `unit_capacity`, `used_count`, `free_count`, - `alloc_state` (derived: Active/Available/Full), `hw_status` - (inherited from disk) -- per-disk-group: `disk_count`, `allocatable_disk_count`, - `capacity_bytes`, `used_bytes`, `free_bytes` -- per-instance: `owned_disk_group_count`, `degraded` (0/1), - `free_batch.queue_depth` and `free_batch.coalescing_ratio_x1000`, - `uncompacted_free_record_count` (per zone — compaction backlog), - `last_sync_slot` (group-0 sync frontier), - `last_sync_age_secs` (time since last successful sync), active relocation - journal and planned-block counts - -Relocation counters record completed moves and errors. Capacity summaries and -the current instance-wide disk utilization spread/minimum/maximum gauges are -observations only; they influence ranking and admission but never authorize -publication or source release. - -**3. Latency hierarchy (where time is spent, per layer):** - -The allocate/free paths are two-phase (sync bitmap claim + async KV -persist). The latency hierarchy breaks down each phase so operators -can see whether the bottleneck is the in-memory allocator or the KV -persist round-trip. - -- **Allocate path:** - - `allocate.rpc.latency_us` — total RPC latency (handler entry → - response). Top of the hierarchy. - - `allocate.bitmap_scan.latency_us` — Phase 1 sync: time in the zone - bitmap-scan + per-bit CAS (includes CAS retries). Nanoseconds in - the common case; spikes indicate contention. - - `allocate.kv_persist.latency_us` — Phase 2 async: time awaiting - the `batch_write` of `BusyBlockValue` to the data group. Dominant - latency component (one paxos round-trip). - - `allocate.zone_rotate.latency_us` — time in `rotate_active_zones` - when the active set is exhausted. -- **Free path:** - - `free.rpc.latency_us` — total RPC latency. - - `free.bitmap_clear.latency_us` — time in the per-bit CAS clear. - - `free.kv_persist.latency_us` — time awaiting the `batch_write` of - `FreeBlockValue` (immediate in v1; batch flush when batching enabled). -- **Sync path:** - - `sync.latency_us` — total `sync_once` latency. - - `sync.read_group0.latency_us` — time reading from group 0. - - `sync.apply_changes.latency_us` — time applying changes to - in-memory state. -- **Compaction path:** - - `compaction.latency_us` — total compaction latency per zone. - - `compaction.scan_free.latency_us` — time scanning free records. - - `compaction.merge_bitmap.latency_us` — time merging free records - into the `ZoneValue` bitmap (in-memory). - - `compaction.kv_persist.latency_us` — time awaiting the - `batch_write` (new `ZoneValue` + delete free records). - -Latency metrics use `LatencyHistogram` (percentile precision) for hot -paths (allocate/free bitmap scan, KV persist) and `LatencySummary` -(count + sum + max + avg) for cold paths (sync, compaction, zone -rotate), matching crowdb-kv's convention. Gauges are derived snapshots -updated on the reporting interval, not hot-path writes. `degraded` and -`last_sync_age_secs` are the key health indicators for alerting. - -## 10. Background Scanner - -The scanner is a periodic consistency check. It detects and reports -live-state drift, catches record corruption early, and gives operators -visibility into cluster health during uptime. It is **not** a safety -mechanism (the free path is persist-only, zone-management §6, and the bitmap is a -conservative over-estimate; freed blocks stay busy until compaction); -it is an operational health mechanism for early corruption detection, -defense-in-depth against unknown bugs or hardware errors, and operator -visibility. - -### Integrity scanning - -- **Data-safety principle** — a busy block may have data written to - it. The scanner's first priority is to never free a block that might - have data. When drift is detected and the true state is uncertain - (corrupt records, conflicting signals), the scanner defaults to - **busy** (keep the bit set, keep the block allocated). Wasting space - is always preferable to freeing a block with data. The scanner only - clears a bit when records confidently say "free" (no `BusyBlockKey`, - no `FreeBlockKey`, records intact and readable). -- **Drift detection in the persist-only model** — the free path does - not touch the bitmap (zone-management §6), so a bit set with matching busy - and free records is **normal** for freed-but-not-compacted blocks. The scanner - distinguishes: - - **Real ghost-busy** (drift): bit set, no `BusyBlockKey`, no - `FreeBlockKey` — the block was never freed and never allocated - (crash between allocate Phase 1 and Phase 2, or a bug). Records - are authoritative → block is free → safe to clear the bit. - - **Normal uncompacted**: bit set, matching `BusyBlockKey` and - `FreeBlockKey` exist — the block was freed (persist-only) but compaction hasn't - cleared the bit yet. This is **not drift** — it's the expected - state. The scanner counts it as `uncompacted_lag` (not drift) so - operators can see compaction lag, but does not auto-correct it. - - **Ghost-free** (drift): bit clear, `BusyBlockKey` exists — should - not happen in the persist-only model (free never clears bits, and - only allocate/compaction touch the bitmap). If detected, it - indicates a bug or hardware error. Records are authoritative → - block is busy → set the bit back. Data may be written. -- **Ghost scan implementation** (`scanner/ghost.rs`) — `scan_ghosts` - iterates owned disk-groups → disks → zones. For each non-active, - unlocked zone, it replays the journal into a throwaway `DdbZone` - via `recover_zone_inner` (strategy 2) with - `rebuild_zone_bitmap_full_scan` (strategy 1) fallback — same logic - as `RecalcEngine::recalc_zone`. It then does a bit-by-bit diff - (`diff_bitmaps`) between the live and replayed bitmaps, classifying - each differing bit by checking `ZoneRecords` busy/free offset sets - (built into `HashSet` for O(1) lookup). The zone lock is not - held across awaits (I9); it is re-acquired (try_write) only for the - synchronous classify + auto-correct step. -- **Re-verify step** — if real drift is detected and - `reverify_delay_ms > 0`, the scanner sleeps, re-snapshots the live - bitmap, and re-checks each drifted bit. If the drift disappeared - during the delay (a zone rotated out with an in-flight allocate - whose Phase 2 completed), it was transient and is skipped. If - persistent, it is reported. Setting `reverify_delay_ms = 0` - disables re-verify (report immediately, may include false - positives from in-flight operations). -- **Auto-correct** — when `ghost.auto_correct` is enabled and - re-verify confirms persistent drift, the live bitmap is corrected: - real ghost-busy → `cas_bit(off, false)` + `used_count` decrement; - ghost-free → `cas_bit(off, true)` + `used_count` increment. Normal - uncompacted bits are never auto-corrected (compaction's job). - Fallback (CRC fail / GC gap) suppresses auto-correct (corruption - signal — the scanner reports but does not correct when the - record-derived bitmap itself may be unreliable). -- **Record integrity** (`scanner/integrity.rs`) — `scan_integrity` - re-checks CRC on the `ZoneValue` snapshot via - `ZoneValueExt::verify_checksum`, detects records that - `read_zone_records` silently skips (undecodable keys/values) by - doing its own prefix scans with `BusyBlockKey::from_bytes` + - `bincode::deserialize` on each item, and optionally validates - `owner_chunk` well-formedness (non-zero) on each `BusyBlockValue`. - Corrupt records are reported; the block is kept busy (key exists, - data may be written); no auto-correction frees a block with a - corrupt record. -- **Leak detection** (`scanner/leak.rs`) — deferred (needs caller - registries). The scaffold returns `LeakScanResult { status: - "deferred" }` so the scanner loop calls it unconditionally. -- **Scanner task** (`scanner/task.rs`) — `ScannerTask` implements - `BackgroundTask` and runs on `BgRunner` with a `TimerFn` trigger - reading `scanner.scan_interval_secs` from the shared config handle. - `ScanState` holds the last `ScanSummary` (shared between the task - and the rpc service handlers) + an `AtomicBool` scan-requested - flag (set by `TriggerScan`, consumed at the start of the next - `run_cycle`) + an `AtomicBool` in-progress flag (prevents overlap). -- **Admin RPCs** — `TriggerScan` sets the scan-requested flag and - returns the current summary + `scan_in_progress` flag; - `GetScanStatus` returns the last summary + `has_run` flag. The - `ScanSummary` schema message carries all scan counts (ghost_busy, - ghost_free, uncompacted_lag, corrupt_snapshots, corrupt_records, - owner_mismatches, leak_status) + timing. -- **Scanner metrics** — `scanner_runs_total` (counter), - `scanner_duration_ms` (latency summary), `scanner_ghosts_found` - (gauge), `scanner_drift_found` (gauge = ghost_busy + ghost_free), - `scanner_corrupt_records` (gauge = corrupt_snapshots + - corrupt_records). -- **Zone coordination** — the scanner skips active zones, acquires the - zone-level lock for non-active zones, and coordinates with compaction - via the shared lock. Details in - [`design-crowdb-diskdb-zone-management.md`](design-crowdb-diskdb-zone-management.md) §9. - -### Tentative owner reconciliation and relocation - -Tentative allocation cleanup is independent from bitmap drift scanning. -`BusyBlockOwnerScanner` reads only tentative durable BusyBlock records after -the disk-group reaches lifecycle `Up`. It queries ChunkDB with the owner chunk -and exact disk/zone/offset/allocation incarnation. `Referenced` confirms the -block, `TaskPending` retains it, and `Absent` persists a first-observed grace -record. Only a repeated authoritative `Absent` after the 24-hour default grace -writes the normal free record. Transport failures retain the block. - -The scanner visits disk-groups, disks, and zones in sorted serial order and -awaits an async configurable delay after every zone. The default is three -seconds. This pacing bounds KV and owner-RPC pressure without blocking a tokio -executor thread. - -Active movement uses a durable relocation journal keyed by the exact source -incarnation. The worker reserves a tentative target, asks DiskIO to copy and -fsync it, then delivers the exact source/target pair to ChunkDB. Journal phases -are `Reserved`, `Copied`, `Accepted`, `Published`, `TargetConfirmed`, -`SourceFreed`, and `Discarded`; startup resumes every non-terminal phase. -ChunkDB alone conditionally publishes the new layout. DiskDB confirms the -target and writes the source free record only after `Published`. - -The intra-disk-group planner selects one committed source from an over-used -disk and one target on an under-used disk when their utilization-point spread -meets the configured threshold. It resumes existing journals before selecting -new work, limits admissions per cycle, and uses serial per-zone pacing. For -cross-group moves, ChunkDB applies its own skew hysteresis and protection gate, -allocates the target, and sends `ExecuteRelocation` to its owning DiskDB, which -adopts that exact target into the same journal state machine. Source -finalization is local when the source disk belongs to another owned group and -otherwise routes through `DiskdbClient`. - -The relocation journal, not an in-memory job set, is the recovery authority. -Operation identity includes the source allocation incarnation, so offset reuse -cannot alias an earlier move. `Stale` discards only the tentative target; -`Rejected` and transient failures retain both sides for reconciliation. No age, -usage observation, or planner timeout is permission to free either block. - -## 11. Crate Layout - -``` -lib/crowdb-protocol (common flatbuffers for all CROWDB components; diskdb first) -app/crowdb-diskdb (server: lib + binary — types, allocator, records, sync, rpc + HTTP, CLI) -lib/crowdb-diskdb-client (client library for callers: allocate/free/query; crowdb-rpc transport) -``` - -- **`lib/crowdb-protocol`** — flatbuffers definitions for diskdb rpc services - (allocate/free/query) + extension traits for schema types - (`diskdb_type_util.rs`: `DiskIdExt`, `HwStatusExt`, - `ZoneAllocationStateExt`, `ZoneValueExt`, `effective_status`). - crowdb-kv's existing schemas stay in `crowdb-kv` (unchanged). The - flatbuffers messages are reused across the crowdb-rpc transport. -- **`app/crowdb-diskdb`** — server crate (package name `crowdb-diskdb`). - Combined library + binary: contains all diskdb logic (types, zone - allocator, record persistence, scanner, ownership/sync, rpc + - HTTP handlers, CLI, config). The library target enables integration - tests without spawning a separate process; the binary target - (`crowdb-diskdb`) is the server executable. Proto types and their - extension traits are re-exported from `crowdb_protocol`; internal types - (metas, key layout, `AllocatedRange`, `ActiveZoneContext`, bitmap) - are Rust structs. - - Liveness and RPC use pure index modules with private implementation - children: - - ```text - liveness/keepalive.rs - └── keepalive/ - ├── scheduler.rs KeepAlive facade, tick, and BackgroundTask - ├── heartbeat.rs service-registry heartbeat publication - ├── observation.rs group-0 ownership, bind, and disk observation - ├── reconciliation.rs runtime ownership and disk reconciliation - └── loading.rs generation-fenced asynchronous zone loading - - service/diskdb_rpc_service.rs - └── diskdb_rpc_service/ - ├── service.rs DiskdbRpcService state and handler registration - ├── mutations.rs allocate, commit, and free handlers - ├── queries.rs capacity and inventory query handlers - ├── admin.rs recovery, compaction, and scanner handlers - └── wire.rs FlatBuffer parsing and response construction - ``` - - `KeepAlive::tick` is the liveness orchestrator and preserves the order - heartbeat → observation → reconciliation. `DiskdbRpcService` remains the - public registration facade; all mutations pass through `mutation_gate`. -- **`lib/crowdb-diskdb-client`** — client library for easy access to the server - (allocate/free/query), mirroring `crowdb-kv-client`'s retry + - topology-cache pattern. crowdb-rpc transport; flatbuffers framing. - -### Dependencies - -``` -crowdb-common (metrics, logging, time) - ↑ -crowdb-kv-client (sole durable store — group 0 + data groups) - ↑ -crowdb-diskdb (server) ──depends──> crowdb-kv-client, crowdb-common, protocol -crowdb-diskdb-client ──depends──> protocol, crowdb-kv-client (for group-0 discovery) - -protocol ──> (no internal deps beyond flatbuffers/crowdb-rpc) -``` - -## 12. Concurrency Model - -All public and inter-module APIs are `async`. Runtime is `tokio` -(multi-threaded for production). - -- No blocking calls in business-logic paths. -- **Allocate** — lock-free per-bit CAS on the usage bitmap; no - zone-level lock. Allocate runs only on active zones. Details in - [`design-crowdb-diskdb-zone-management.md`](design-crowdb-diskdb-zone-management.md) §4, §8. -- **Free** — persist-only: one `batch_write`, no bitmap touch, no - zone-level lock. Free can run on any zone without coordination. - Details in [`design-crowdb-diskdb-zone-management.md`](design-crowdb-diskdb-zone-management.md) §4, §8. -- **Zone-level lock for non-allocate operations** — compaction, - scanner, and health checks acquire a zone-level lock on non-active - zones only. Common methods on `DdbZone` encapsulate the lock + - operation. Details in - [`design-crowdb-diskdb-zone-management.md`](design-crowdb-diskdb-zone-management.md) §8. -- Disk-level active zone set uses RCU publish (`Arc` swap) for - lock-free reads; rotation takes a brief write lock. -- Disk-group allocation context retains its read lock. Allocation and usage - clone one `Arc` while holding it, and configuration replacement is rare; - `ArcSwap` publication is reserved for evidence of contention here. -- **Free-side lookup structures** — disk-id → disk hash map and - zone-index → zone vec for O(1) lookups on the free path. Details in - [`design-crowdb-diskdb-zone-management.md`](design-crowdb-diskdb-zone-management.md) §8. -- v1 free is immediate (no `FreeBatch`, no background flush loop). When - free batching ships, `FreeBatch` will be protected by a `Mutex` held - only for the append, not for the KV flush; the flush is triggered by - batch size (no timer). -- Node-level `add_disk` / `remove_disk` acquire a write lock on the - disk list; allocation/free acquire a read lock (concurrent with each - other, exclusive with add/remove). - -## 13. Non-Gaps (Good Fits with CROWDB) - -These design assumptions map cleanly onto CROWDB and need no design -work, just implementation: - -- **Durability model**: crowdb-kv's WAL is the sole durable log. DiskDB's blind - allocation/free writes and conditional commit batches become durable through - the same Paxos/WAL path. -- **Consensus semantics**: Multi-Paxos retains parallel blind allocation and - free throughput. Conditional commit serializes only on its one busy key - before proposing an ordinary batch. -- **Async runtime**: tokio multi-threaded; diskdb's two-phase - async allocation (sync bitmap-scan claim + async KV persist) maps - directly. - -## 14. Configuration - -diskdb requires a typed TOML file at startup. Omitted fields use typed -defaults, and explicitly supplied CLI fields override file values according to -the shared [`service configuration design`](../config/design-crowdb-config.md). -`server.rpc_workers` controls inbound RPC concurrency, defaults to 2, must be -positive, and requires restart when changed. - -All settings that control flow behavior live in a config class (no -hardcoded tunables in business logic). Defaults: - -- **Sync** — sync interval (10 s, fixed — same on success and failure), - degraded miss threshold (3), temp-failure timeout (900 s) -- **Allocator** — `zone_rotate_count`, CAS retry limit (100) -- **Allocation policy** — `allocator.load_aware` (true) and - `allocator.load_aware_weight` (`free_bytes`) -- **Free** — `free_batch_enabled` (default false — one durable KV batch per - request; immediate concurrent-request coalescing when true), - `free_flush_max_batch` (256, maximum records in one coalesced KV proposal; - one oversized request remains atomic) -- **Compaction** — snapshot compaction threshold (record count or - time), compaction cadence (periodic interval for strategy 3) -- **Disk** — block / unit size (default 1 MB), zone size -- **Free validation** — no read on the free path; compaction validates the - immutable free fact against the current busy incarnation -- **Scanner** — `scan_interval_secs` (600), `ghost.detect` (true), - `ghost.auto_correct` (false — manual review first; enable for - self-healing), `integrity.verify` (true), - `integrity.detect_owner_mismatch` (false — piggybacks on - `read_zone_records`), `reverify_delay_ms` (1000 — set 0 to disable - re-verify), tentative-owner interval (600 s), per-zone delay (3 s), and - absent-owner grace (86,400 s) -- **Rebalance** — enabled, planning interval (300 s), utilization-point - threshold (20), one new job per cycle, and a three-second per-zone delay - -## 15. Implementation Scope - -The diskdb implementation is organized by functional scope. Each area -below covers a coherent slice of the component; together they make up -the full diskdb server. - -- **Protocol + core types** — flatbuffers services, core types, record key - layout, config validation, bitmap utilities, CRC integrity. -- **Group-0 sysdata schema + sync** — disk status management, group-0 - read/write, ownership/binding maps, heartbeat, disk-add initialization - flow, keepalive usage summary. -- **Zone allocator + record persistence** — bitmap-scan allocator, - rotating active-zone-set, busy/free records, two-phase async - allocation, persist-only free (no `FreeBatch`, no timer), - `disk_group_id` routing, load-aware disk selection, `exclude_disks`, CAS - retry bound. -- **Crash recovery + snapshot compaction** — three strategies: full - scan rebuild, journal scan replay via `JournalScan` RPC, compaction - that deletes only free records. -- **Space metrics + query API** — per-disk/group/zone metrics, - recalculation path, `query_disk_usage`, three-category metrics: - counters, gauges, latency hierarchy. -- **Background reconciliation and relocation** — ghost/drift/integrity - detection, tentative-owner reconciliation, paced intra-disk-group planning, - durable DiskIO relocation, and restart-safe ChunkDB publication handoff. -- **Disk discovery + health probing** — config-driven disk list, health - probe, disk failure detection. -- **Console + CLI integration** — disk/disk-group management UI, zone - busy/free visualization, CLI command design. -- **Group-0 notify/watch** — replace polling with watch/notify; see - [`design-crowdb-kv-watch-notify.md`](../kv/design-crowdb-kv-watch-notify.md). - Polling stays as safety net. -- **Free batch** — size-threshold batching, no timer; graceful-shutdown - drain + flush. - -The design doc (this file and future sub-designs under -`doc/design/diskdb/`) is kept permanently — it is the root design for -the diskdb component area. - -## 16. References - -- CROWDB root KV design: `doc/design/kv/design-crowdb-kv.md` -- CROWDB WAL design: `doc/design/kv/design-crowdb-kv-wal.md` -- CROWDB metrics design: `doc/design/kv/design-crowdb-kv-observability.md` -- CROWDB console design: `doc/design/console/design-crowdb-console.md` - ---- - -## 17. DiskdbClient Scanner / Rebuild Wrappers - -`DiskdbClient` wraps allocate/free/query/recalc/compact and also the -scanner RPCs (`TriggerScan`, `GetScanStatus`) and -`RebuildZoneBitmap`. The schema + server handlers exist; the client -wrappers make them reachable from the REST proxy and CLI. - -```rust -use crowdb_protocol::diskdb::rpc::{ - TriggerScanRequest, TriggerScanResponse, - GetScanStatusRequest, GetScanStatusResponse, - RebuildZoneBitmapRequest, RebuildZoneBitmapResponse, -}; - -impl DiskdbClient { - /// Trigger a scan on all owned groups (or one group if `dg_id` - /// is set). Returns the last `ScanSummary` + `scan_in_progress`. - pub async fn trigger_scan(&self, dg_id: Option) - -> Result - - /// Get the last scan summary + `has_run` flag. - pub async fn get_scan_status(&self, dg_id: Option) - -> Result - - /// Rebuild one zone's bitmap on a disk. Routes via `dg_for_disk`. - pub async fn rebuild_zone_bitmap( - &self, disk_id: DiskId, zone_index: u32, - ) -> Result -} -``` - -- `trigger_scan` / `get_scan_status` route to `first_cached_dg()` - when `dg_id` is `None` (mirrors `recalc_disk_usage`); otherwise - route to the specified dg. Both go through `with_retry`. -- `rebuild_zone_bitmap` routes via `dg_for_disk(disk_id)` (mirrors - `compact_zone`), then `with_retry`. -- All three are admin/debug calls; transient `Unavailable` is retried - per `RetryConfig`; `NotFound` (unknown disk/zone) is returned - immediately. - -Edge cases: -- `trigger_scan` while a scan is running → server returns - `scan_in_progress: true`; client returns the response as-is (no - error, no stacking). -- `rebuild_zone_bitmap` on unknown disk → `dg_for_disk` returns - `Unreachable`; surfaced to the caller. -- Empty endpoint cache → `first_cached_dg()` returns `Unreachable`; - caller sees "no cached endpoints; call refresh_endpoints". - -## 18. Group-0 Status Write-Back: Init → Offline - -`background_zone_load` publishes `Offline` before its local `Init → Offline` -transition when zone loading or baseline persistence fails. This keeps a -failed disk unavailable across sync ticks. - -- The task retains its exact `OwnershipFence`. Group-0 write-back requires - the same owner instance and owner-record revision. -- A conditional batch preserves the owner value and writes the disk status - atomically. Advancing the owner revision makes the next observation reclaim - the data-group fence. A concurrent handover rejects the whole batch. -- A stale loader cannot mark its successor's healthy disk Offline. Rejected - write-back is reported; its retired in-memory disk cannot allocate. - -## 19. Init-state zone load - -New disks start in `Init` state with no zones. A background task -loads zones from the bind before transitioning `Init → Up` (or -`Init → Offline` on load failure or operator-set offline). This -prevents allocations against a disk with existing KV records from -overwriting used blocks with empty zones. `reconcile_absent_disk` -removes `Offline`/`Maintenance`/`Init` disks from memory when they -disappear from group-0 sync (the disk is gone); `Bad` disks are kept -(recovery scan running). - -- **I1 — Init before allocatable**: A disk is never allocatable - while in `Init` state. Zones must be loaded from the bind before - the disk transitions to `Up`. diff --git a/doc/design/diskdb/diskdb-allocate-flow-analysis.md b/doc/design/diskdb/diskdb-allocate-flow-analysis.md deleted file mode 100644 index d0ce7fd99..000000000 --- a/doc/design/diskdb/diskdb-allocate-flow-analysis.md +++ /dev/null @@ -1,203 +0,0 @@ - - - -# CROWDB - Design: DiskDB Allocation Flow Analysis - -Allocation flow, observability, tuning controls, and reference performance for -the durable DiskDB path. - -Depends on: [DiskDB overview](design-crowdb-diskdb.md), -[zone management](design-crowdb-diskdb-zone-management.md), -[RPC design](../rpc/design-crowdb-rpc.md), and -[KV write flow](../kv/kv-write-flow-analysis.md). - -## Table of Contents - -- [1. Flow](#1-flow) -- [2. Metrics](#2-metrics) -- [3. Tunables](#3-tunables) -- [4. Reference Results](#4-reference-results) -- [5. Bottleneck](#5-bottleneck) -- [6. Retained Logs](#6-retained-logs) - -## 1. Flow - -```text -crowdb-cli workload task - -> DiskdbRpcTransport connection pool - -> crowdb-rpc DiskDB server worker - -> Tokio allocation task - -> lock-free bitmap claim - -> busy-record construction - -> DdbKvClient::persist_busy_batch - -> KvRpcTransport connection pool - -> KV batch_write - -> proposal coalescing and inflight admission - -> Paxos accept quorum - -> WAL and engine apply - -> DiskDB response construction and crowdb-rpc submission - -> client completion -``` - -One allocation RPC produces one KV `batch_write`. Multiple blocks in an -allocation RPC become multiple busy records in that batch. Independent -allocation RPCs can be combined by KV proposal coalescing; DiskDB does not add -a second flow-level batcher. - -## 2. Metrics - -Every DiskDB request method has this family: - -```text -request.{method}.lh -request.{method}.inflight.g -request.{method}.errors.c -``` - -The histogram count and rate identify the completed workload mix. The gauge -identifies the current live mix, including slow requests. The error counter is -the non-success response population. Timing begins at Rust handler dispatch -and ends at response submission, including async task scheduling. - -Allocation adds bitmap scan, record build, KV persist, response build, -claimed-unit, rollback-unit, partial-batch, and error-reason metrics. The -DiskDB-to-KV boundary reports batch-write latency, record magnitude, inflight, -and errors. Each process's `cpp-rpc` block supplies transport scheduling, -writev/read/parse, bandwidth, queue pressure, connection, and transport-error -metrics without duplicate Rust counters. - -KV-server logs supply `kv.batch_write.lh`, proposal inflight wait, Paxos phase -latencies, WAL append/fsync, engine apply/flush, inter-replica RPC, CPU, RSS, -and network counters. - -## 3. Tunables - -The three connection and worker layers remain independent: - -- benchmark client: workload concurrency, connections per DiskDB endpoint, - and client RPC workers; -- DiskDB: server RPC workers, KV connections per endpoint, and KV-client RPC - workers; -- KV: server RPC workers, peer pool, proposal inflight window, coalesce size, - drain threshold, event-write mode, WAL, and storage backend. - -The high-throughput memory profile starts from the KV write sentinel. The -current matrix enables event-write and uses inflight 32 and coalesce 32 without -an explicit drain-threshold override. The 1/16-thread cases use two -connections and RPC workers on every layer; 128/256-thread cases use four. -Connection count is recorded explicitly because a per-process, per-endpoint -pool is not comparable to the direct KV benchmark's total count. - -## 4. Reference Results - -AMD Ryzen 9 5950X, 16c/32t, Linux 6.8, x86_64, memory KV/WAL, one block per -request, and 4 TiB per logical disk. Every case ran for 20 seconds, stopped at -its deadline with zero errors, and reported exact busy-space accounting. - -| Workload | Grp | Thread | Block | Client connection | DiskDB connection | KV internal connection | Epoll worker | Window | Coalesce | ops/s | p50 us | p99 us | Duration | Errors | Space | -| --- | ---: | ---: | ---: | ---: | ---: | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | -| allocate | 3 | 1 | 1 | 2 | 2 | 2 | 2 | 32 | 32 | 2,481 | 410 | 504 | 20 s | 0 | exact | -| allocate | 3 | 16 | 1 | 2 | 2 | 2 | 2 | 32 | 32 | 42,959 | 362 | 582 | 20 s | 0 | exact | -| allocate | 3 | 128 | 1 | 4 | 4 | 4 | 4 | 32 | 32 | 130,863 | 930 | 1,863 | 20 s | 0 | exact | -| allocate | 3 | 256 | 1 | 4 | 4 | 4 | 4 | 32 | 32 | 159,418 | 1,514 | 3,294 | 20 s | 0 | exact | -| allocate | 1 | 256 | 1 | 4 | 4 | 4 | 4 | 32 | 32 | 171,206 | 1,423 | 2,883 | 20 s | 0 | exact | -| mix | 3 | 1 | 1 | 2 | 2 | 2 | 2 | 32 | 32 | 2,494 | 405 | 500 | 20 s | 0 | exact | -| mix | 3 | 16 | 1 | 2 | 2 | 2 | 2 | 32 | 32 | 43,106 | 361 | 568 | 20 s | 0 | exact | -| mix | 3 | 128 | 1 | 4 | 4 | 4 | 4 | 32 | 32 | 128,013 | 948 | 1,957 | 20 s | 0 | exact | -| mix | 3 | 256 | 1 | 4 | 4 | 4 | 4 | 32 | 32 | 154,624 | 1,556 | 3,456 | 20 s | 0 | exact | -| mix | 1 | 256 | 1 | 4 | 4 | 4 | 4 | 32 | 32 | 171,419 | 1,419 | 2,915 | 20 s | 0 | exact | - -`Grp` is the number of KV data groups bound round-robin to DiskDB disk groups; -three groups in the default three-node topology gives each node's DiskDB disk -group a distinct KV group. `Epoll worker` applies uniformly to the client, -DiskDB server, DiskDB KV-client, and KV-server RPC layers. The fixture uses -4 TiB per logical disk so a 20-second high-concurrency run does not approach a -single node's available space. - -The active regression matrix runs allocate and 70/30 allocate/free mix at 1, -16, 128, and 256 threads. The 1/16-thread cases use two connections and two -epoll workers on every RPC layer; 128/256 threads use four of each. A separate -256-thread case binds all DiskDB disk groups to one KV data group to measure -the single-group ceiling. Other cases bind one KV data group per node. -Environment overrides can replace the group, connection, or worker count for -experiments. - -The 256-thread three-group allocation completed 3,191,195 allocations and -reported an exact 3,346,210,488,320-byte busy-space delta. The one-group case -completed 3,427,099 allocations with an exact 3,593,573,761,024-byte delta. - -The direct KV write sentinel peaks near 264K writes/s at 512 tasks with the -larger 64/64 window/coalesce profile. The closest 32-slot direct-KV references -reach about 181–191K writes/s at 128/256 threads. Against that range, the -one-group DiskDB allocation result is within about 6%, while the three-group -result is within about 12% of the 256-thread KV result. - -## 5. Bottleneck - -At 256 threads, DiskDB request latency is dominated by KV persistence. In -representative saturated allocation windows, whole-handler average latency is -about 1.25–1.76 ms while KV persistence is about 1.08–1.50 ms. Bitmap scan and -record construction average at or below one microsecond. KV batch-write -service averages about 0.5–0.7 ms, and KV processes consume several cores in -both user and system time. Inflight slots remain around one to three, so the -32-slot proposal window is not saturated. - -Three data groups are about 7% slower than one at the same 256-thread settings -for both allocate and mix. All groups are replicated across the same three KV -processes, so they contend for shared accept-round CPU and consensus transport; -adding groups does not provide independent hardware capacity. - -The current bottleneck is KV accept-round and RPC CPU capacity, not DiskDB -bitmap allocation, the 32-slot inflight window, or disk capacity. The durable -one-unit path cannot reach 400K operations/s through DiskDB tuning alone while -it requires one KV client write per operation and the measured direct KV -ceiling is about 264K operations/s. Reaching a higher rate requires raising the -underlying KV ceiling or changing the number of DiskDB operations represented -by one KV client write. Any such batching must preserve durable acknowledgement -and exact rollback semantics. - -### Concurrent-free coalescing reference (2026-09-14) - -The immediate-drain free coalescer was compared with the direct path using: - -```bash -DISKDB_BENCH_DURATION=10 DISKDB_BENCH_CASES='free_batch_off_mem free_batch_on_mem' pixi run -- bash tools/benchmark/bench-diskdb-regression.sh -``` - -Reference host: Intel Core i9-7960X (16 cores / 32 threads), x86_64, -Linux 6.11.0-29, XFS on `/dev/nvme1n1p1`. Both cases used one KV data group, -three KV nodes, three DiskDB instances, 128 workload tasks, one block per RPC, -four connections/workers at each RPC layer, memory KV/WAL, and a 10-second -70/30 allocate/free workload. - -| Free batch | ops/s | Avg us | p50 us | p99 us | Free requests | KV proposals | Requests/proposal | Errors | Space | -| ---------- | ----: | -----: | -----: | -----: | ------------: | -----------: | ----------------: | -----: | ----- | -| off | 112,319 | 1,138 | 1,110 | 1,766 | 337,596 | 337,596 | 1.00 | 0 | exact | -| on | 118,794 | 1,076 | 1,013 | 1,953 | 356,984 | 40,209 | 8.88 | 0 | exact | - -Enabled mode reduced KV proposals per free record by 88.7%, improved aggregate -throughput by 5.8%, and reduced average latency by 5.4%. Both paths completed -with zero errors and exact post-compaction space accounting. The deterministic -coalescer tests additionally feed the same incarnation-qualified records -through direct and combined persistence and verify whole-request atomicity, -cross-request duplicate suppression, failure fan-out, and explicit retry. - -Open observation: enabled-mode p99 was 10.6% higher in this single comparison -even though average and p50 improved. A repeated latency-focused run is needed -to separate normal run variance from the expected queueing tail of requests -that join an in-flight KV batch; this does not affect durability or proposal -reduction. - -## 6. Retained Logs - -Reference run roots: - -- `bench-log/diskdb-regression-20260905-133409`: complete current allocation - and mixed-workload matrix. -- `bench-log/diskdb-regression-20260914-010652`: direct versus enabled - concurrent-free coalescing comparison. - -Each root retains command output and configuration plus three KV metrics/RPC -log pairs, three DiskDB metrics/RPC log pairs, and one CLI metrics/RPC pair per -case. `results.tsv` records the resolved parameters and correctness fields. -Generated logs remain untracked. diff --git a/doc/design/diskio/design-crowdb-diskio.md b/doc/design/diskio/design-crowdb-diskio.md deleted file mode 100644 index a69986ab2..000000000 --- a/doc/design/diskio/design-crowdb-diskio.md +++ /dev/null @@ -1,885 +0,0 @@ - - - -# CROWDB - Design: diskio (Overview) - -diskio is the per-node data I/O engine. diskdb allocates disk blocks; -chunkdb manages chunk metadata; diskio reads and writes the block -contents. It is a C++ binary (`crowdb-diskio`) that runs on each storage -node, using io_uring on Linux for asynchronous I/O and a thread-pool -`pwrite`/`pread` fallback on macOS and non-liburing Linux builds. The -server uses the crowdb-rpc C++ engine directly; Rust callers -(chunkdb writers, recovery, rebalance) use a typed client crate that -wraps `crowdb-rpc-ffi`. - -The io_uring engine (`DiskIOUring`) is a multi-pipeline io_uring -wrapper that also serves the crowdb-tree btree page store. It lives in -`crowdb-common` and is shared by both consumers. - -## Table of Contents - -- [1. Overview](#1-overview) -- [2. Non-Goals (Design Envelope)](#2-non-goals-design-envelope) -- [3. Key Design Decisions](#3-key-design-decisions) - - [3.1 C++ server, Rust client](#31-c-server-rust-client) - - [3.2 io_uring on Linux; thread-pool pwrite/pread fallback](#32-io_uring-on-linux-thread-pool-pwritepread-fallback) - - [3.3 Control + data separation in the RPC frame](#33-control--data-separation-in-the-rpc-frame) - - [3.4 No disk status tracking](#34-no-disk-status-tracking) - - [3.5 Partial writes are errors, not retried internally](#35-partial-writes-are-errors-not-retried-internally) - - [3.6 Shared DiskIOUring in crowdb-common](#36-shared-diskiouring-in-crowdb-common) -- [4. Architecture Overview](#4-architecture-overview) -- [5. IoEngine Abstraction](#5-ioengine-abstraction) - - [5.1 AlignedWriter](#51-alignedwriter) - - [5.2 UringEngine](#52-uringengine) - - [5.3 BlockingEngine](#53-blockingengine) - - [5.4 DummyDiskEngine](#54-dummydiskengine) - - [5.5 NullDisk and MemDisk](#55-nulldisk-and-memdisk) -- [6. Disk Model](#6-disk-model) -- [7. DiskIOUring](#7-diskiouring) - - [7.1 Architecture](#71-architecture) - - [7.2 fd → Pipeline Routing](#72-fd--pipeline-routing) - - [7.3 Polling Modes](#73-polling-modes) - - [7.4 Batched SQE Submission](#74-batched-sqe-submission) - - [7.5 Pipeline Topology](#75-pipeline-topology) - - [7.6 Bad-Disk Cancel by fd](#76-bad-disk-cancel-by-fd) - - [7.7 Lock-Free Design](#77-lock-free-design) - - [7.8 FFI Multi-Eventfd Pump](#78-ffi-multi-eventfd-pump) -- [8. RPC Service](#8-rpc-service) -- [9. Client Library](#9-client-library) -- [10. Invariants](#10-invariants) -- [11. Configuration](#11-configuration) -- [12. References](#12-references) - ---- - -## 1. Overview - -diskio is a **per-node data I/O server**. Each storage node runs one -`crowdb-diskio` process that owns a set of local disks and serves -read/write/fsync RPCs over crowdb-rpc. Callers (chunkdb's large-object -writer, small-object writer, chunk read flow, recovery, rebalance) -send a `Segment`-based address — `{disk_id, zone_index, zone_offset, -size}` — and diskio translates it to a physical offset and performs -the I/O. - -**Language:** C++ server, Rust client. **I/O:** io_uring (Linux), -thread-pool `pwrite`/`pread` (macOS, non-liburing Linux). - -**Core goals:** -- **Low-latency data path** — io_uring on Linux with no - `spawn_blocking`, no thread hop, no Rust→C++ round-trip on the data - path. The entire I/O path is async: RPC receive → DiskIOUring submit → - CQE completion → RPC response. -- **Cross-platform dev/test** — thread-pool `pwrite`/`pread` fallback - on macOS gives the same semantics at lower performance, so the test - suite runs everywhere. -- **Shared DiskIOUring** — the io_uring engine lives in `crowdb-common` - and is shared by the btree page store and the diskio engine. One - implementation, two consumers. -- **Testability** — `MemDisk` (drop-write + rule-based read) for - throughput benches, `SimulatedDisk` (latency + error injection) for - fault-path tests. Both run without real hardware. - -## 2. Non-Goals (Design Envelope) - -- **No disk allocation.** diskdb allocates blocks; diskio reads and - writes their contents. diskio does not track which blocks are busy - or free. -- **No allocation-ownership fencing.** diskio does not validate allocation - incarnations or decide whether an extent may be reused. The owning lifecycle - layer prevents reuse while an earlier writer can still retry, for example by - retaining freed extents beyond the maximum write retry window. -- **No disk status tracking.** diskio does not maintain disk health - state. If an I/O fails, the engine returns the error to the caller; - the top layer (chunkdb/diskdb) handles the failure and stops - allocating new blocks on that disk. -- **No chunk metadata.** chunkdb manages chunk/strip metadata; diskio - is unaware of chunks, strips, or redundancy. It reads and writes - raw block contents at the `Segment` granularity. -- **No internal retry on partial writes.** A short write (fewer bytes - than requested) is an error, not retried internally. The caller - decides whether to retry the whole write or fail the operation. -- **No RDMA or SPDK (v1).** v1 uses Linux io_uring on block devices - and regular files. RDMA and SPDK are future additions. -- **No streaming RPC.** v1 supports request-response only. The data - payload (up to the max block size, default 2 MB) fits in one frame. - -## 3. Key Design Decisions - -### 3.1 C++ server, Rust client - -The diskio server is C++ because the io_uring engine and the -crowdb-rpc server engine are C++. Running the server in C++ means the -data path has no FFI boundary: RPC receive, frame decode, I/O submit, -CQE completion, and response submit all happen in C++ without a -Rust→C++ round-trip. The Rust client (`crowdb-diskio-client`) wraps -`crowdb-rpc-ffi` with typed `DiskIoClient` methods, following the -existing `crowdb-diskdb-client` / `crowdb-chunkdb-client` pattern. - -### 3.2 io_uring on Linux; thread-pool pwrite/pread fallback - -io_uring is the Linux asynchronous I/O interface. It provides -submission queues (SQ) and completion queues (CQ) in shared memory, -allowing batched I/O submission and completion polling with minimal -syscalls. For `O_DIRECT` on block devices, I/O completes inline -during SQ processing in most cases — no kernel thread pool needed. - -macOS has no io_uring. POSIX `aio` exists but is weak: internally -thread-pool-based, limited queue depth, inconsistent across -filesystems. `libaio` is Linux-only. A dedicated C++ thread pool with -`pwrite`/`pread` and configurable `fdatasync` is the pragmatic -cross-platform fallback. It also serves as the Linux non-liburing -production path (if liburing is not available at build time). - -The `IoEngine` virtual base abstracts over both backends so the RPC -layer is backend-agnostic. - -Read, write, and fsync latency is measured from engine submission through -completion with the same `diskio.engine.*.lh` histograms for both backends. -The io_uring backend additionally publishes current in-flight operations and -SQ saturation as `diskio.uring.inflight.g` and `diskio.uring.sq_full.c`. -RPC request and response send-queue pressure remains owned by crowdb-rpc's -transport latency and rejection counters rather than being duplicated here. - -### 3.3 Control + data separation in the RPC frame - -A diskio RPC frame carries a small flatbuffer control message -(`{disk_id, zone_index, zone_offset, size}`) and a raw data payload -in separate buffers. crowdb-rpc's 12-byte header carries both -`msg_size` (control) and `data_size` (data), so the parser knows the -full frame shape after 12 bytes. The data payload is written directly -to / read directly from the I/O buffer with no intermediate -serialization. This mirrors crowdb-rpc's control+data separation design -(see `design-crowdb-rpc.md` §2). - -### 3.4 No disk status tracking - -diskio does not track whether a disk is Good, Bad, or Suspicious. It -opens disk handles at startup and serves I/O until a disk fails. When -an I/O fails, the engine returns the error to the caller. The top -layer (chunkdb/diskdb) is responsible for marking a disk as bad and -stopping new allocations on it. After all in-flight I/O to a failing -disk drains, no further I/O is sent to it. - -Administrative Offline is enforced by Group-0 discovery. A complete valid -membership observation detaches Offline disks from the active map, rejecting -new requests while existing handles drain through descriptor retirement. Offline -disks still contribute their physical DiskGroup identity to the service's -observation. Bad, Suspect and maintenance states do not by themselves close a -handle needed for recovery reads. Failed or malformed membership observations -retain the previous map; missing status or duplicate disk identities are errors. - -This keeps diskio simple — it does one thing (data I/O) and reports -results. Disk health policy lives in diskdb, where it belongs. - -### 3.5 Partial writes are errors, not retried internally - -When `pwrite` or io_uring returns fewer bytes than requested, the -engine returns `IoError::PartialWrite` immediately. No internal retry. - -For `O_DIRECT` block devices (the primary path), partial writes -essentially do not occur for correctly aligned I/O. If they do, it -usually indicates a hardware error — retrying the same range is -futile. The caller (chunkdb writer) has strip-level retry logic and -can decide whether to retry the whole strip or fail it. Internal -retry would complicate the completion callback path (resubmit for -remaining bytes, track partial progress, handle retry-failure) for no -benefit. - -### 3.6 Shared DiskIOUring in crowdb-common - -`DiskIOUring` is a multi-pipeline io_uring engine: it maps file -descriptors to pipelines (one `io_uring` instance each), shares -polling threads across CQs, batches SQE submission, and cancels -in-flight I/O by fd via `IORING_ASYNC_CANCEL_FD`. It is not specific -to the btree page store. It lives in `crowdb-common` so both the btree -page store (`crowdb-tree`) and the diskio engine share one -implementation. The WAL's io_uring integration later builds Rust -async submit wrappers over the same C ABI. - -## 4. Architecture Overview - -``` -Caller (Rust) crowdb-diskio Server (Node X, C++) - │ │ - │ DiskIoClient::write(seg, data) │ - │ ──► crowdb-rpc-ffi frame: │ - │ [hdr][DiskWriteReq][data] │ - │ ──────────────────────────────────►│ - │ │ - │ crowdb-rpc reader decodes frame - │ handler: resolve disk_id → Disk - │ compute phys_offset = zone_base + zone_offset - │ │ - │ ▼ - │ IoEngine::write(disk, phys_offset, data, on_complete) - │ │ - │ ┌─────────┴────────────────────┐ - │ │ UringEngine (Linux) │ BlockingEngine (macOS) - │ │ crowdb::common::DiskIOUring │ thread pool - │ │ fd→pipeline, SQE submit │ pwrite → callback - │ │ → CQE completion │ - │ └─────────┬────────────────────┘ - │ │ (also: DummyDiskEngine wrapper for dummy disks) - │ │ - │ on_complete → crowdb_rpc_server_submit_response - │ crowdb-rpc response: [hdr][DiskWriteResp] - │ ◄──────────────────────────────────│ - │ write() returns Ok(()) │ -``` - -Components: -- **DiskSet** — holds `HashMap>`, opened at - startup from the node's disk list. Resolves `disk_id` to a `Disk`. -- **Disk** — virtual base with subclasses: `BlockDisk` (real block - device, `O_DIRECT`), `NullDisk` (`/dev/zero`, drop-write + pattern read), - `MemDisk` (memfd, store + read-back). - Each `Disk` shares the node's `IoEngine` instance; dummy disks wrap - it with a `DummyDiskEngine` for read-content hack and optional fault - injection. -- **Zone** — `{zone_index, base_offset, capacity, state}`. The - physical offset for an I/O is `zone.base_offset + zone_offset`. -- **IoEngine** — virtual base: `submit_write`/`submit_read`/ - `submit_fsync` taking a disk handle, physical offset, buffer, size, - and a completion callback. Implementations: `UringEngine`, - `BlockingEngine`, `DummyDiskEngine` (wrapper for dummy disks). -- **DiskIOUring** — the shared multi-pipeline io_uring engine in - `crowdb-common`. Maps fds to pipelines (one `io_uring` instance per - pipeline); polling threads drain CQs and submit pending SQEs. - Lock-free on the hot path (SQE claim via atomic CAS, fd→pipeline - lookup via direct-indexed array). -- **RPC handler** — dispatches `DiskWriteRequest`/ - `DiskReadRequest`/`DiskFsyncRequest` to the `IoEngine`; the - completion callback builds the response and submits it via - `crowdb_rpc_server_submit_response`. -- **DiskIoClient** — Rust client wrapping `crowdb-rpc-ffi` with typed - methods. Routes to the correct node's diskio server based on - `segment.node_id`. - -## 5. IoEngine Abstraction - -`IoEngine` is a C++ virtual base: - -``` -class IoEngine { - public: - virtual ~IoEngine() = default; - virtual void submit_write(Disk *disk, off_t phys_offset, - const uint8_t *data, size_t size, - std::function on_complete) = 0; - virtual void submit_read(Disk *disk, off_t phys_offset, - uint8_t *buf, size_t size, - std::function on_complete) = 0; - virtual void submit_fsync(Disk *disk, - std::function on_complete) = 0; -}; -``` - -`on_complete` is invoked exactly once with the raw result: `>=0` bytes -transferred, `<0` negative `-errno`. The RPC handler's callback -resolves the request and calls `crowdb_rpc_server_submit_response`. - -### 5.1 AlignedWriter - -`AlignedWriter` is the server-owned stage between the write RPC handler and a -disk's `IoEngine`. Every write traverses it. `NullDisk` and `MemDisk` use a -block size of one, so the same path delegates their writes without padding. - -For disks with larger blocks, the stage aligns the physical range and retains -one aligned buffer through backend completion. It preserves every byte outside -the requested range. Partial head and tail blocks come from valid cache entries -or exact boundary-block reads; a range spanning both uncached boundaries needs -at most two reads. Request bytes are overlaid only after those reads complete. -The RPC reports the logical count only after the complete aligned write succeeds. - -The stage caches the last partial physical block by disk and block offset. -The bounded cache is an optimization: eviction, invalidation and process restart -do not change range-write semantics. Failed writes invalidate affected entries; -full-block writes invalidate prior partial images. An aligned start with a -partial tail also preserves the rest of that physical block, so a delayed -immutable retry cannot erase bytes written by a later request. - -Writes are assigned to 64 shards by disk and segment ordering base. Each shard -has a bounded lock-free MPSC queue and one active consumer through write -completion. Related partial-block writes remain ordered without a mutex. -Independent segment bases and disks can execute separately apart from hash -collisions. Queue saturation fails the request with `-EAGAIN`. - -Byte-addressed writes use the same ordering queue. A zero-length ordered request -waits for preceding write completions without issuing a media write; when its -sync flag is set, the RPC handler then flushes the disk before proving completion. -This is a segment drain after the caller has excluded future producers. A flush -alone, or a drain while new producers remain authorized, cannot establish a final -tail or authorize physical reuse. - -At admission, a write timestamp must fall within the bounded request age and -future clock-skew allowance. The server derives a conservative monotonic deadline -and checks it when dequeuing and before submitting an aligned merge after boundary -reads. An expired queued request returns `OldRequest` without a media write. An -already submitted backend operation retains its queue position through completion, -so a later drain waits for it even when its kernel execution outlasts request age. -The shared admission ceilings are 30 seconds of request age and one second of -clock skew. A server may use shorter windows; startup rejects larger values so -it cannot extend the late-write interval assumed by recovery and reclamation. - -Concurrent overlapping writes are not a supported caller contract. As with -raw `pwrite`, their final contents would otherwise depend on completion order. -The chunk writers submit sequential updates to a segment and wait for each -response before issuing the next overlapping update. - -### 5.2 UringEngine - -Linux only (`CROWDB_HAVE_LIBURING`). Owns a `crowdb::common::DiskIOUring` -instance (one engine per diskio server). `O_DIRECT` aligned writes -(configurable, default on for data blocks). The entire I/O path is -async — no `spawn_blocking`, no thread hop. - -`UringEngine` is a thin adapter: it validates the disk handle and -`O_DIRECT` alignment, then delegates to `DiskIOUring::submit_*` with -the disk's fd. `cancel_disk` looks up the disk's fd via `DiskSet` and -calls `DiskIOUring::cancel_fd` — one SQE cancels all in-flight I/O on -that fd via `IORING_ASYNC_CANCEL_FD` (kernel 6.0+). No per-disk -in-flight tracking map; the kernel does the cancel lookup. See §7.6. - -### 5.3 BlockingEngine - -macOS + non-liburing Linux. A dedicated C++ thread pool (configurable -size, default 4 threads per disk) with `pwrite`/`pread` and -configurable `fdatasync`/`fsync`. Each I/O operation is submitted to -the pool; the worker thread performs the blocking syscall and invokes -the completion callback. Correct semantics, lower performance (thread -hop per I/O). Also used for `BlockDisk` without `O_DIRECT` (pwrite to -a block device at an offset). - -### 5.4 DummyDiskEngine - -The dummy-disk wrapper. `DummyDiskEngine` wraps a real `IoEngine` -(`UringEngine` or `BlockingEngine`) and provides two features: - -- **Read-content hack** (NullDisk): after the inner engine completes - a `pread`, the wrapper overwrites the buffer with deterministic - pattern data generated from `disk_id` + `phys_offset` (xorshift64 - PRNG). The full uring/blocking `pwrite`/`pread` syscall path - executes against `/dev/zero` — no real disk I/O or retained page-cache - data, but the ring mechanics, syscall cost, and DiskIOUring batching all run. - Used for benchmark tests that measure uring overhead without - storage. -- **Fault injection** (NullDisk + MemDisk): per-I/O random latency - (uniform draw from `latency_min_ms`..`latency_max_ms`, sleep before - completing) and errors (return `-EIO` at the configured - `error_rate`). Configured via `DiskProperties` on the dummy disk. - Merged from the former `SimulatedEngine` — no separate wrapper - engine needed. - -### 5.5 NullDisk and MemDisk - -Two dummy disk types use different memory semantics: - -- **NullDisk** — the default dummy disk, backed by `/dev/zero`. Writes are - discarded without retaining page-cache data. Reads execute the full - `pread` path, then the `DummyDiskEngine` wrapper overwrites the buffer - with pattern data. For benchmark tests: measures uring/blocking overhead - without real disk I/O, storage capacity limits, or memory growth. -- **MemDisk** — backed by `memfd_create` (tmpfs), stores written data and - reads it back. For end-to-end correctness tests that verify I/O data - integrity. The full `pwrite`/`pread` path executes against the memfd; - no real disk I/O. - -Both dummy disks share the node's `IoEngine` instance (uring or -blocking, auto-detected). Optional `DiskProperties` enable fault -injection on either type. - -## 6. Disk Model - -`Disk` is a C++ virtual base with subclasses: - -- **BlockDisk** — real block device, opened with `O_DIRECT | O_DSYNC | O_RDWR` - (Linux). Aligned I/O only. The primary production disk type for - NVMe/SATA SSDs and HDDs. The device path comes from the - `device_path` field in `DiskValue` (group-0 sysdata). -- **NullDisk** — `/dev/zero`-backed dummy disk. Writes are discarded - without retained memory; reads return deterministic pattern data via - `DummyDiskEngine` wrapper. - Default dummy disk type. For benchmark tests. -- **MemDisk** — memfd-backed dummy disk. Stores written data and - reads it back. For end-to-end correctness tests. - -Each `Disk` shares the node's `IoEngine` instance (auto-detected: -uring on Linux with liburing, blocking otherwise). Dummy disks -(`NullDisk`, `MemDisk`) wrap the shared engine with a -`DummyDiskEngine` for read-content hack and optional fault injection. - -Every valid descriptor used by `UringEngine`, including dummy disks discovered -after startup, is registered with `DiskIOUring`. Removal unregisters the -descriptor after draining in-flight work. The first group-0 disk sync runs -immediately so a restarted service does not advertise an empty `DiskSet` for -one refresh interval. - -Detachment removes the disk from new request lookups first. The synchronization -owner retains its handle until old map snapshots and all read/write/fsync -callbacks have released theirs, then unregisters and closes the descriptor. -Rediscovery waits for this retirement before reopening the same disk identity. - -`Zone` holds `{zone_index, base_offset, capacity, state}`. The -physical offset for an I/O is `zone.base_offset + zone_offset`. Zone -records come from diskdb's allocation. - -`DiskSet` holds `HashMap>`, opened at startup -from the node's disk list (via `HardwareClient` / diskdb). `DiskId` -is the 128-bit id from diskdb. - -## 7. DiskIOUring - -`DiskIOUring` is a multi-pipeline io_uring engine. It maps file -descriptors to pipelines (one `io_uring` instance per pipeline), -shares polling threads across CQs, batches SQE submission, and is -fully lock-free on the hot path. Both diskio and btree use -`DiskIOUring` as their I/O interface. - -`DiskIOUring` lives in `crowdb-common` (`crowdb::common::DiskIOUring`), -guarded by `CROWDB_HAVE_LIBURING`. It is Linux-only. On macOS and -non-liburing Linux builds, `DiskIOUring` is absent and -`BlockingEngine` is used instead. - -### 7.1 Architecture - -``` - DiskIOUring - ┌─────────────────────────────────────────────┐ - │ fd_table[fd] → pipeline_index (read-only) │ - │ │ -submit_read(fd) ──► │ Pipeline 0 Pipeline 1 ... │ -submit_write(fd) ─► │ ┌──────────┐ ┌──────────┐ │ -submit_fsync(fd) ─► │ │ io_uring │ │ io_uring │ │ -cancel_fd(fd) ────► │ │ SQ + CQ │ │ SQ + CQ │ │ - │ └────┬─────┘ └────┬─────┘ │ - │ │ │ │ - │ ┌────┴────────────────┘ │ - │ │ PollThread 0 (busy-poll or epoll-wait) │ - │ │ drains CQ-0 + CQ-1, submits pending SQEs │ - │ └───────────────────────────────────────────┘ - └─────────────────────────────────────────────┘ -``` - -Components: -- **Pipeline** — one `io_uring` instance: one SQ + one CQ in shared - memory. The low-level uring wrapper (lock-free SQE claim, CQE - drain). Internal to `DiskIOUring`; not exposed to callers. -- **PollThread** — one OS thread that polls a group of pipelines' CQs - and submits their pending SQEs. Multiple CQs share one polling - thread (no one-thread-per-CQ requirement). Configurable grouping. -- **fd_table** — direct-indexed `std::vector` mapping fd → - pipeline_index. Populated at `register_fd` time, read-only during - I/O. No synchronization needed (concurrent reads of an immutable - array). -- **CallbackEntry** — allocated per submit, freed after CQE dispatch - via deferred-delete (polling-thread-only). Holds the callback; no - atomics, no cancel flags, no hazard pointers. Callback suppression - is client-side (§7.7): the caller wraps the callback with a shared - cancel flag. - -### 7.2 fd → Pipeline Routing - -`fd_table` is direct-indexed by fd number. fds are small integers -(typically < 1024). The vector is sized once at construction to -`ulimit -n` (capped at 4096) and never grows — no reallocation, no -data race on concurrent reads. Each slot defaults to `SIZE_MAX` -(sentinel: "not registered"). Looking up `fd_table[fd]` is a single -memory load — no hash, no collision, no atomic, no lock. - -Two registration modes: -- **Auto-assignment** (`register_fd(fd)`) — `DiskIOUring` picks the - pipeline with the lowest in-flight count and sticks the fd to it. - Used by btree (single pipeline) and uniform-disk diskio topologies. - Sticky: once assigned, all future submits for that fd go to the - same pipeline (required for `cancel_fd` correctness — cancel targets - the ring where the fd's SQEs were submitted). -- **Explicit assignment** (`register_fd(fd, pipeline_index)`) — caller - pins the fd to a specific pipeline. Used when the caller built a - disk-class-aware topology (NVMe → pipeline 0, HDD → pipeline 2) and - needs deterministic routing. - -### 7.3 Polling Modes - -A `PollingMode` config per pipeline: - -- **Classic** — event-wait on the poll thread's private eventfd with a - bounded timeout. Both submissions and kernel completions wake it; - no pipeline waits on an empty CQ while another pipeline has ready work. - Used by crowdb-tree's single-pipeline topology. -- **Hybrid** `{ busy_poll_budget }` — busy-poll the CQ ring's shared - memory via `io_uring_peek_cqe` with no syscall while I/O is active. - After `busy_poll_budget` consecutive empty peeks, transition to - `epoll_wait` on the poll thread's private eventfd. Any CQE resets the counter, - returning to busy-poll. Gives sub-µs CQE dispatch during I/O bursts, - sleeps when idle, no core burned at idle. Default for `UringEngine`. -- **Sqpoll** `{ sq_thread_idle }` — opt-in for sustained high-IOPS: - `IORING_SETUP_SQPOLL` eliminates submit syscalls via a kernel - SQ-poll thread. Requires root/CAP_SYS_NOPRIV. Useful for recovery - scan and bulk rebalance. - -When multiple Hybrid-mode pipelines share a poll thread, the thread -uses the **minimum** `busy_poll_budget` across its assigned Hybrid -pipelines — the most conservative budget wins. - -Each pipeline registers its owning poll thread's private eventfd with -`io_uring`. The poll thread drains ready CQEs from every assigned pipeline -and then signals the separate pipeline eventfds consumed by external -reactors. External notifications therefore follow callback completion; -external consumers cannot steal the poll thread's wakeup. - -### 7.4 Batched SQE Submission - -Client threads fill SQE slots (lock-free CAS on shadow tail + per-slot -ready flags) and set a `pending_submit` atomic flag. The polling -thread submits all pending SQEs in one `io_uring_enter` per iteration -— batching N SQEs into one syscall. Clients never call -`io_uring_enter`; only the polling thread does. - -In busy-poll mode (I/O active), SQEs are submitted within <1µs of -being filled. The first client changing `pending_submit` from false to -true writes the owning poll thread's private eventfd. Subsequent clients -in the same pending batch skip that write. The eventfd remains readable -if the thread has not entered event-wait yet, so a submission cannot lose -its wakeup. The thread publishes newly pending SQEs after waking and -before draining completions. - -### 7.5 Pipeline Topology - -io_uring does not support sharing a CQ ring across separate -`io_uring` instances — each `io_uring_queue_init` creates its own SQ + -CQ pair. But one ring can submit I/O for any number of fds, so the -answer is fewer rings, not shared CQs. Topology by disk type: - -- **NVMe SSD** (100k+ IOPS/disk): one pipeline per disk. High IOPS - needs SQ headroom; per-disk isolates SQ/CQ backpressure so one busy - disk's full SQ doesn't block another. -- **SATA SSD** (10k-100k IOPS/disk): one pipeline per 4-8 disks - (grouping). Medium IOPS; grouping reduces polling thread count - while keeping SQ headroom. For 24-30 SSDs this gives 3-8 rings, not - 24-30. -- **HDD** (100-200 IOPS/disk): one shared pipeline for all HDDs. Low - IOPS; one ring's SQ handles 30 HDDs trivially. `entries = 2048`. - -Poll thread grouping: -- **1 thread, all pipelines** (default, simple): one thread - busy-polls all CQs. Best for low pipeline count. -- **1 thread per pipeline** (max isolation): each CQ has its own - thread. Best for high-IOPS NVMe where CQ drain latency matters. -- **Grouped by disk class** (balanced): `{ {0,1,2}, {3,4,5} }`. 2 - threads for 6 pipelines. Best for SATA SSD farms. -- **Grouped by NUMA node**: pipelines whose disks are on NUMA node 0 - → thread 0 (pinned to node 0 core); node 1 → thread 1. - -`IORING_SETUP_ATTACH_WQ` (Linux 5.18+): when `Topology::attach_wq` is -true and there are ≥2 pipelines, subsequent pipelines share the -kernel's io-wq pool (8 kernel threads total instead of N×8). For -`O_DIRECT` on block devices, I/O almost always completes inline and -io-wq is rarely involved; `attach_wq` matters most for non-`O_DIRECT` -file I/O. Default: `false` (opt-in). - -### 7.6 Bad-Disk Cancel by fd - -`cancel_fd(fd)` cancels all in-flight I/O for an fd with one SQE: -`IORING_OP_ASYNC_CANCEL` with `flags = IORING_ASYNC_CANCEL_FD | -IORING_ASYNC_CANCEL_ALL` and `addr = fd` (kernel 6.0+). The kernel -cancels all matching ops and posts CQEs (`-ECANCELED` for each -cancelled op, or the original result if already completed). No -per-fd in-flight tracking needed — the kernel does the lookup. Zero -overhead on the hot path. - -On a shared pipeline, a bad disk's in-flight I/O holds SQ slots. If -the SQ fills, good disks' I/O is rejected with `-ENOMEM`. -`cancel_fd` frees SQ slots immediately by cancelling all of the bad -fd's in-flight ops. The SQ ring's capacity check is -`tail - head >= entries`, where `head` is the kernel's consumed SQ -head — in-flight I/O is in the block layer, not the SQ ring, so the -SQ has space for the cancel SQE even when all of the fd's I/O is -in-flight. - -**Kernels < 6.0** (no `IORING_ASYNC_CANCEL_FD`): `cancel_fd` returns -`-ENOSYS`. The caller (diskio's `cancel_disk`) logs a warning and -falls back to waiting for in-flight I/O to complete naturally (the -bad-disk path has a timeout). macOS never compiles `DiskIOUring`; -`BlockingEngine` is the non-uring path. - -CQEs are independent — the polling thread drains all ready CQEs each -iteration, so a slow disk's CQE arriving late doesn't block CQ drain -for others. - -### 7.7 Lock-Free Design - -The hot path (submit + CQE completion) uses no mutexes, no locks, no -blocking: - -- **SQE claim**: atomic CAS on shadow tail + per-slot ready flags. - Multi-threaded clients claim slots concurrently with no contention - beyond CAS retries. -- **fd → pipeline lookup**: direct array index, read-only. No - synchronization. -- **Batch submit coordination**: `pending_submit` atomic flag, - `pending_count` atomic counter, `thread_sleeping` atomic flag. All - single-word atomics. -- **CQ drain**: polling thread only (single-threaded per CQ). No - concurrency, no synchronization. -- **Callback dispatch**: polling thread only. `CallbackEntry` holds - just the callback — no `cancelled`/`dispatched` atomics. The - dispatch path calls `entry->cb(res)` unconditionally; if the client - wants to suppress the callback, it checks a shared cancel flag - inside the callback itself (client-side cancel). The CQE always - arrives, the callback always fires, the SQ slot is always freed — - only the callback body is skipped via the flag. A `shared_ptr` keeps - the cancel state alive until the CQE arrives (no UAF). -- **CallbackEntry lifetime**: deferred-delete on the polling thread. - The polling thread is the sole toucher of `CallbackEntry` after - `submit_*` returns — no concurrent access, no hazard pointers, no - generation counters. - -**Kernel-internal locks** (not controllable from userspace): - -| I/O path | Lock | Hold duration | Contention scope | -| --- | --- | --- | --- | -| SQE fill / CQE drain | None | — | — | -| `O_DIRECT` block device (diskio `BlockDisk`) | `blk-mq` per-HW-queue spinlock | µs (queue insertion) | per-HW-queue | -| `O_DIRECT` regular file write (btree block files) | `i_rwsem` exclusive | full I/O latency | per-inode | -| `O_DIRECT` regular file read | `i_rwsem` shared | full I/O latency | per-inode (concurrent reads OK) | - -io_uring never blocks the userspace submitter on `i_rwsem` — when an -inline execution would block, io_uring punts to `io_wq` (kernel worker -pool). The design's per-fd pipeline isolation aligns with the kernel's -per-device / per-inode lock scope: if one fd's I/O contends on a -kernel lock, it is on its own pipeline with its own polling thread. -Both consumers avoid same-inode write contention by design: btree -uses one file per extent (different inodes), diskio writes to block -devices (no `i_rwsem`). - -### 7.8 FFI Multi-Eventfd Pump - -Each pipeline registers its eventfd via -`IORING_REGISTER_EVENTFD`. The Rust FFI layer -(`lib/crowdb-tree/ffi/src/reactor.rs`) spawns one `EventfdPump` per -pipeline eventfd, all sharing one `Arc`. Any pipeline's CQE -completion wakes all waiting `drive_ct_future` calls. The C ABI -function `ct_uring_eventfds` returns the pipeline eventfds; the Rust -wrapper spawns one pump per fd. - -## 8. RPC Service - -The diskio server uses crowdb-rpc's `RpcServer::register_handler` to -handle three message types. Message type IDs are in the diskio range -(3600s) of crowdb-rpc's `msg_type` enum. - -Each request's control message is a flatbuffer with -`{disk_id, zone_index, zone_offset, size}` (read also has -`test_pattern_offset`); the write request additionally carries an -`ordering_zone_offset`, `write_create_time_ms`, a `sync` option, and a raw data -payload of exactly `size` bytes. The write handler validates the FlatBuffer, -zone bounds and payload length before IO. The handler: - -Async callbacks retain the disk, connection, buffer and shared transport -independently of the server facade. Exact registered payload views remain -DiskOwned through write and synchronization completion; a closed connection -suppresses response submission without releasing active disk data. -1. Resolves `disk_id` to a `Disk` via `DiskSet`. -2. Computes the physical offset: `zone.base_offset + zone_offset`. -3. Calls `AlignedWriter::submit_ordered` for writes or `IoEngine::read`/`fsync` for - the other operations, with the disk handle, physical offset, buffer, and a - completion callback. -4. A synchronized write submits its disk flush only after a complete successful - write. Its sole response follows that flush; buffered writes reply after the - write. Failed or short writes never flush or report synchronized success. -5. Completion retains the disk, shared connection and request buffer through - both operations, then submits the response through the transport. Rejected - submission releases the caller-owned response frame. - -The data payload is passed from the crowdb-rpc frame decoder to -`AlignedWriter`. Block-size-one writes remain zero-copy on the server; padded -writes use one aligned server buffer. The read response includes the raw data -payload. - -The negotiated RDMA control/data layout supplies a separately aligned 1 MiB -data region. Eligible aligned writes hand this registered receive storage to -the disk engine and retain DiskOwned state until completion. A READ reuses its -request's reserved receive data allocation as output, retaining it through disk -and response SEND completion. Client response leases stay caller-owned until -their final release. These retained receive allocations propagate disk and -consumer pressure through the bounded receive window. Alignment edges retain -the existing explicit staging path; TCP keeps the same result/error mapping. - -Related writes are serialized by `ordering_zone_offset` within -`AlignedWriter`. This ordering protects partial-block read-modify-write state; -it does not validate allocation ownership. Write requests carry only the -current addressing, ordering, request-age and synchronization contract. - -Flatbuffer schemas (`diskio.fbs`): -- `DiskWriteRequest { disk_id, zone_index, zone_offset, size, - ordering_zone_offset, write_create_time_ms, sync }` -- `DiskWriteResponse { ret_code, sync_completed }` -- `DiskReadRequest { disk_id, zone_index, zone_offset, size, test_pattern_offset }` - (`test_pattern_offset` used by NullDisk for deterministic content; - default 0 = start of disk/object) -- `DiskReadResponse { ret_code }` (data payload follows the control - message) -- `DiskFsyncRequest { disk_id }` -- `DiskFsyncResponse { ret_code }` - -## 9. Client Library - -`DiskioClient` (Rust, in `crowdb-diskio-client`) is the routed semantic -boundary for every production Rust caller. Callers provide an allocated -`SegmentTarget`, a segment-relative byte range, a normal or priority traffic -lane, one total deadline, and (for writes) buffered or fsync durability: - -``` -async fn write(target, offset, data: Bytes, durability, options) -> Result<()> -async fn read(target, offset, length, options) -> Result -async fn fsync(disk_id, options) -> Result<()> -``` - -The client constructs its control-plane handles from group-0 management seeds, -or accepts injected `ServiceRegistryClient` and `HardwareClient` handles. It -joins live DiskIO registrations with hardware disk-group ownership and publishes -one immutable route generation mapping every `DiskId` to rack, node, disk group, -DiskIO instance, and endpoint. Missing or duplicate ownership, missing hardware -identity, an identity mismatch, a malformed endpoint, or either failed -control-plane read rejects the entire candidate. There is no fallback endpoint -and no partial generation. - -Each endpoint has independently bounded normal and priority connection groups. -Unchanged refreshes reuse those groups. An endpoint or instance change creates -a new generation before publication; operations retain the selected old -connection through completion, and exact generation checks prevent a delayed -old failure from invalidating its replacement. Removing a pool drops the -client's references, while the final in-flight reference controls when its C++ -connection is closed. Selection and route/status snapshots use immutable RCU -state and atomics, so request routing does not acquire a caller lock. - -Readiness uses the priority group for bounded 4096-byte reads of existing disk -headers. Normal bulk responses and readiness headers therefore have independent -TCP response ordering while sharing the same validated disk identity and route. -Priority selection does not change physical read semantics, error propagation, -route refresh rules or the caller's total deadline. - -The internal wire transport retains the caller's owned `Bytes` allocation in -the RPC buffer until completion. This avoids a client-side `Bytes` to `Vec` -payload copy. On the server, block-size-one and already aligned payloads can -continue directly into `IoEngine::write`. Other writes are copied once into -the server-owned aligned buffer that preserves boundary bytes and satisfies the -`O_DIRECT` address-alignment requirement. - -`write_views` splits oversized view lists into bounded RPC frames without -copying their payload. Frames within one strip's disk block complete serially -in byte order, including aligned boundaries. Connections can reorder arrivals; -serial completion preserves a contiguous prefix before the final flush. Separate -caller operations on that block must obey the same ordering. Independently -owned strip blocks and different disks holding EC shards or mirror replicas -may run concurrently. Only the final frame requests synchronization, after -preceding frames have completed. Its flush covers the whole preceding range; -a frame error stops the remaining submissions. - -Input arithmetic, segment bounds, and wire-size limits are checked before RPC -admission. Reads retry transient transport outcomes within the original -deadline. Writes can retry only the same immutable bytes at the same allocated -location and retain their original creation time; partial writes and permanent -disk errors return immediately. Expired operations submit no new request. -Fresh-storage initialization disables automatic retries, and an unresolved -admitted write is reported as ambiguous. Durable writes require -`sync_completed=true` after write and flush, using one request/reply. Missing -proof fails directly. A failed -flush returns `FsyncFailed`, exposed as `DurabilityFailure`; no failure is a -proof of synchronization. Independent fsync remains available and can be -repeated after an ambiguous response. The lifecycle no-reuse window is what -makes an identical write retry safe. - -Failure of an accepted RDMA read, write or fsync is terminal for that operation: -the client does not reconnect and replay it through a TCP fallback connection. -It returns the typed transport error, ambiguous write outcome or durability -failure. Known pre-submit rejections retain the existing retry handling. - -Status exposes route generation and age, topology and pool counts, healthy -connections, operation/retry/backpressure/ambiguity counters, and operation -latencies without exposing raw transport handles. The only production escape -hatch is an opaque retained priority-lane route set for the native crowdb-tree -page store. Wire-level methods remain available only to protocol tests through -the crate's `test-util` feature. - -## 10. Invariants - -- **I1 (data integrity)**: A successful `write(segment, data)` - followed by a `read(segment, None)` on the same `Segment` returns - the same bytes. Holds for `UringEngine`, `BlockingEngine`, and - `DummyEngine` (with rule-based read content). -- **I2 (durability)**: A successful synchronized write, or a write followed - by successful explicit fsync, retains the written data across process restart - when its backing media remains available. A synchronized response follows - both write and flush, with exactly one completion. -- **I3 (offset correctness)**: The physical offset for an I/O is - `zone.base_offset + zone_offset`, where `zone` is looked up by - `zone_index` from the `Disk`'s zone records. -- **I4 (no silent drop)**: `DiskIOUring` does not drop I/O requests. - If the SQ is full, the lock-free SQE claim enters bounded retry - waiting for a slot. If the ring is invalid, the completion callback - is invoked synchronously with `-EIO`. -- **I5 (completion guarantee)**: `on_complete` is invoked exactly - once per `submit_*` call — either from the polling thread (CQE - dispatched) or synchronously (submission failure). `cancel_fd` - replaces the original callback with a `-ECANCELED` CQE. -- **I6 (bad-disk isolation)**: A bad disk's in-flight I/O does not - permanently block good disks' I/O on a shared pipeline. - `cancel_fd(fd)` cancels all in-flight ops on the bad fd with one - SQE, freeing SQ slots immediately. -- **I7 (partial write is error)**: A short write returns - `IoError::PartialWrite` immediately. No internal retry. The caller - decides the next action. -- **I8 (partial-block preservation)**: Sequential writes to the same physical - block complete in submission order. An unaligned continuation merges with - the cached acknowledged block image. Each uncached partial boundary is read - before the request bytes are overlaid. Alignment preserves all bytes outside - the requested range, including tails after aligned starts and late retries. -- **I9 (backend alignment)**: Every backend write to a disk with block size - greater than one has aligned address, offset, and length. - -- **I10 (strip-block write ordering)**: Only one write frame may be in flight - for a strip's disk block. Its successful completion precedes submission of - the next frame, regardless of alignment. Parallel writes require independent - strip blocks or different disks holding shards or replicas. - -## 11. Configuration - -diskio accepts an optional typed TOML file through `--config`; explicit CLI -options override file values according to the shared -[`service configuration design`](../config/design-crowdb-config.md). Its -sections are: - -- **`[server]`** — bind address, listen port, inbound RPC workers (default 4), - node ID, dummy-disk type, `O_DIRECT`, and optional fault injection. Worker - count must be positive and changes require restart. -- **`[engine]`** — blocking thread-pool size and io_uring SQ entry count. The - backend remains auto-detected: `UringEngine` when available and - `BlockingEngine` otherwise. -- **`[group0]`** — KV management seeds, instance/rack/disk-group identity, - synchronization interval, and disk auto-discovery. -- **`[metrics]`** — log directory and collection interval. -- **`[[disk]]`** — ordered disk ID, path, and single-zone capacity entries. - An empty path selects a `NullDisk` or `MemDisk`. - -Fault latency requires both minimum and maximum values with minimum not above -maximum. Fault error rate is in `0.0..1.0`. Loading is transactional and the -complete candidate is validated before the RPC listener or I/O engine starts. - -The server registers with the group-0 service registry on startup, -reporting the diskio service is alive. Other services use this for -health detection. - -## 12. References - -- [`design-crowdb-diskdb.md`](../diskdb/design-crowdb-diskdb.md) §2 (Non-Goals: "No - data I/O") — diskdb allocates blocks; diskio reads/writes contents. -- [`design-crowdb-chunkdb.md`](../chunkdb/design-crowdb-chunkdb.md) §5.1 - (Disk block — `Segment { node_id, disk_id, zone_index, - zone_offset, size, tag }`) — the addressing unit for diskio. -- [`../rpc/design-crowdb-rpc.md`](../rpc/design-crowdb-rpc.md) §2 (Key - Design Decisions: control + data separation, native buffer, C ABI + - oneshot FFI) — the RPC engine diskio builds on. -- [`design-crowdb-tree.md`](../tree/design-crowdb-tree.md) — the btree - page store that shares `DiskIOUring`. diff --git a/doc/design/kv/design-crowdb-kv-group0.md b/doc/design/kv/design-crowdb-kv-group0.md deleted file mode 100644 index 7d02c0e18..000000000 --- a/doc/design/kv/design-crowdb-kv-group0.md +++ /dev/null @@ -1,551 +0,0 @@ - - - -# CROWDB - Design: System Group (Group 0) - -Depends on: [`design-crowdb-kv.md`](design-crowdb-kv.md) §3.3 -[`design-crowdb-kv-server.md`](design-crowdb-kv-server.md), -[`design-crowdb-kv-reconfiguration.md`](design-crowdb-kv-reconfiguration.md) -Satisfies: [`design-crowdb-kv.md`](design-crowdb-kv.md) §3.3 - ---- - -## Table of Contents - -- [1. Overview](#1-overview) -- [2. Design Decisions](#2-design-decisions) - - [2.1 External and in-process group-0 API surfaces](#21-external-and-in-process-group-0-api-surfaces) - - [2.2 `crowdb-kv-server` mgmt API is internal](#22-crowdb-kv-server-mgmt-api-is-internal) - - [2.3 Unified key concept with two encodings](#23-unified-key-concept-with-two-encodings) - - [2.4 All cross-component protocol types live in `crowdb-protocol`](#24-all-cross-component-protocol-types-live-in-crowdb-protocol) - - [2.5 ID types defined in `crowdb-protocol`](#25-id-types-defined-in-crowdb-protocol) - - [2.6 Two monitoring models: push (services) and pull (infrastructure)](#26-two-monitoring-models-push-services-and-pull-infrastructure) - - [2.7 kv-server keep-alive to group 0 (revised)](#27-kv-server-keep-alive-to-group-0-revised) - - [2.8 Hardware admin via kv-client (no admin crowdb-rpc service)](#28-hardware-admin-via-kv-client-no-admin-crowdb-rpc-service) -- [3. Group-0 Sysdata Schema](#3-group-0-sysdata-schema) - - [3.1 Key layout (text-path encoding)](#31-key-layout-text-path-encoding) - - [3.2 Text magic namespaces](#32-text-magic-namespaces) - - [3.3 Value encoding](#33-value-encoding) - - [3.4 Scan patterns](#34-scan-patterns) - - [3.5 `DiskGroupId` widening: u32 → u64](#35-diskgroupid-widening-u32--u64) - - [3.6 `dc_id` removal](#36-dc_id-removal) -- [4. Service Registry](#4-service-registry) - - [4.1 Registration and keep-alive](#41-registration-and-keep-alive) - - [4.2 Services registered](#42-services-registered) - - [4.3 Liveness and expiry](#43-liveness-and-expiry) -- [5. Bootstrap and Cutover](#5-bootstrap-and-cutover) - - [5.1 Two-phase bootstrap](#51-two-phase-bootstrap) - - [5.2 No `/topology/ready` flag](#52-no-topologyready-flag) - - [5.3 Greenfield migration](#53-greenfield-migration) - - [5.4 Leader readiness before writing](#54-leader-readiness-before-writing) -- [6. Relationship to Existing Design Docs](#6-relationship-to-existing-design-docs) -- [7. Cascading cleanup](#7-cascading-cleanup) - ---- - -## 1. Overview - -The **system group** (store 0, group 0) is a designated Paxos group -that stores cluster-wide system data ("sysdata") as regular KV entries. -Because it is a Paxos group, sysdata is replicated, consistent, and -highly available by the same mechanism that protects user data. No -external coordinator is needed. - -Group 0 holds several categories of sysdata: - -- **Hardware hierarchy** — racks, nodes, disk-groups, disks. -- **Per-disk-group maps** — ownership map (which diskdb instance owns - a disk-group), binding map (which paxos data group a disk-group's - zone records live on). -- **KV-cluster topology** — stores, groups, replicas (the persistent - record of the KV cluster's structure, for disaster recovery). -- **Service registry** — live service instances (diskdb, kv-server, - future services) with their endpoints and heartbeat timestamps. - -This document defines the architecture of group 0: who owns the -schema, who writes what, how services register and keep alive, and -how the circular-dependency between kv-server and group 0 is handled. - ---- - -## 2. Design Decisions - -### 2.1 External and in-process group-0 API surfaces - -External processes use `crowdb-kv-client` for group-0 sysdata. Code already -running inside `crowdb-kv-server` uses `KvGroupOperations` and the fixed -`Group0ControlPlane` facade instead of opening a loopback RPC connection to -itself. RPC handlers also delegate to `KvGroupOperations`, so internal and -external paths share the same read fences, conditional-write semantics, and -error mapping without coupling a monitor to the RPC transport. - -`crowdb-kv-client` provides service classes wrapping a client pinned to group -0: - -- **`HardwareClient`** — hardware hierarchy + per-disk-group maps - (rack/node/disk-group/disk CRUD, ownership map, binding map). -- **`ServiceRegistryClient`** — service instance registry + keep-alive - (register, heartbeat, unregister, discover). -- **`KVClusterMetaClient`** — KV-cluster topology records - (store/group/replica metadata read/write). -- **`KVClusterAdmin`** — control-plane surface for cluster management - (lifecycle HTTP calls to kv-server + delegates metadata writes to - `KVClusterMetaClient`). Contains a `KVClusterMetaClient` internally. - -Values are small and JSON-encoded. Conditional mutations use revision CAS -where concurrent ownership or lifecycle changes must be preserved. Disk status -setters require an existing record, read it linearly and compare its revision -when writing the new status; missing disks are never recreated by a status -update. A matching status is a read-only no-op. Conflicts and unknown outcomes -require reconciliation before retry. See §3 for the key/value schema. - -### 2.2 `crowdb-kv-server` mgmt API is internal - -The kv-server HTTP management API (lifecycle endpoints: `add_store`, -`add_group`, `add_remote_replicas`, `step_down`, `system_init`) -stays in kv-server. These create/destroy actual `PxKvStore`/`PxGroup` -objects with WALs and election drivers, which cannot move to a client. -But the API is now **internal**: only `crowdb-kv-client`'s -`KVClusterAdmin` calls it. All other code (`crowdb-web`, `crowdb-cli`) -goes through `KVClusterAdmin`. - -Removed from kv-server: `POST /topology/finalize`, -`GET /topology/ready`. Persistent topology-record management moves -to `KVClusterMetaClient` / `HardwareClient` in `crowdb-kv-client`. - -Retained in kv-server (internal): lifecycle endpoints, `GET /topology` -(live runtime state export), `GET /health`, `GET /metrics`. - -### 2.3 Unified key concept with two encodings - -CROWDB keys use a single key concept (struct + fields) with two encoding -traits: `BinaryKey` (binary, for data groups) and `TextKey` -(text-path, for group 0). The encoding protocol (rules, frozen -layouts, evolution policy) is defined in -[`design-crowdb-protocol-key.md`](../protocol/design-crowdb-protocol-key.md) §5. Group 0 -uses text keys + JSON values; the full group-0 key/value schema is in -§3 below. - -### 2.4 All cross-component protocol types live in `crowdb-protocol` - -`crowdb-protocol` is the single home for all cross-component protocol -types: data structures, RPC messages, key types, and HTTP mgmt API -contracts. When `ServerClient`'s mgmt methods are absorbed into -`KVClusterAdmin`, the -HTTP request/response types (`AddStoreRequest`, `AddGroupRequest`, -`RemoteReplicaInfo`, `StepDownRequest`, `StoreSummary`, -`TopologyResponse`) move to `crowdb-protocol` under a new `mgmt` -module. - -### 2.5 ID types defined in `crowdb-protocol` - -All identifier types are defined in `crowdb-protocol` and used -consistently across all crates (console config, kv-client, kv-server, -diskdb, chunkdb). No crate uses `String` for an ID that is -fundamentally numeric. ID aliases are defined **once** in -`crowdb-protocol` (`RackId`, `NodeId`, `DiskGroupId`, `StoreId`, -`GroupId`, `ReplicaId`, `InstanceId`) and imported by every other -crate, with no per-crate redefinition. Console config -(`crowdb-console-shared`) widens every rack-id and node-id reference to -`RackId`/`NodeId` (not just `RackEntry.id`/`NodeEntry.id`/ -`NodeEntry.rack_id`, also `ServerEntry.node_id`, `StoreEntry.nodes`, -`ReplicaEntry.node_id`, and the persisted BTreeMap keys). The -console-side `ServerEntry.id` label stays `String` (it is a console -handle, not a numeric cluster ID). - -- **`RackId`** — `u64`. Console config today uses `String` for - `RackEntry.id`; the values are numeric strings (e.g. "1"). Console - config schema changes to `u64`; `http_cluster_init` passes the value - directly to `HardwareClient.add_rack(rack_id: u64)`. -- **`NodeId`** — `u64`. Console config today uses `String` for - `NodeEntry.id`; the values are numeric strings. Console config - schema changes to `u64`. -- **`DiskGroupId`** — `u64`. (This is a widening; see §3.5.) -- **`StoreId`** — `u64`. Already u64 in kv-server and console config. -- **`GroupId`** — `u64`. Already u64 in kv-server and console config. -- **`ReplicaId`** — `u64`. Already u64 in kv-server and console config. -- **`InstanceId`** — `u64`. Service instance identifier (diskdb - instance, kv-server instance). -- **`DiskId`** — 128-bit: two `u64` (`high`, `low`). Already defined - as a flatbuffer message in `common_type.fbs`. Globally unique. -- **`ChunkId`** — 192-bit: three `u64` (`high`, `mid`, `low`). - Already defined as a flatbuffer message in `common_type.fbs`. - -The simple integer IDs (`RackId`, `NodeId`, `DiskGroupId`, `StoreId`, -`GroupId`, `ReplicaId`, `InstanceId`) are type aliases (`pub type -RackId = u64;`) in `crowdb-protocol`, not newtypes. They exist for -documentation and API clarity (function signatures read -`rack_id: RackId` rather than `rack_id: u64`), not for type-safety -enforcement. The composite IDs (`DiskId`, `ChunkId`) are already -flatbuffer structs. - -### 2.6 Two monitoring models: push (services) and pull (infrastructure) - -There are two distinct monitoring models in the cluster, determined -by the relationship to group 0: - -**Push model (service keep-alive):** Application services (diskdb, -chunkdb, future services) are **guests** on the platform. They -register themselves in group 0 under `/srv//` -and heartbeat periodically. Other components discover them by -scanning group 0. This works because the service is not the host of -group 0. It's a client. - -**Pull model (infrastructure health):** The console (crowdb-web) polls -kv-server nodes via `GET /health` and `GET /topology` to observe -runtime state. This is the existing on-demand pattern; it can be -extended to periodic polling. - -### 2.7 kv-server keep-alive to group 0 (revised) - -**Yes, kv-server instances do keep-alive to group 0**, same push -model as application services. Each kv-server instance registers -under `/srv/kv-server/` and heartbeats with its hosted -stores/groups and health status. This is implemented (a -background keep-alive loop in `crowdb-kv-server` writing via -`ServiceRegistryClient`); it is not deferred to a follow-up. - -**Circular-dependency analysis:** The concern is that group 0 is -hosted by kv-server, so kv-server writing to group 0 is "writing to -itself." This is not actually harmful: - -- **Group 0 is replicated.** If one node hosting a group-0 replica - goes down, the remaining replicas maintain quorum. The down node's - heartbeat expires, which is the desired behavior. -- **kv-server is already a client of group 0.** `KVClusterMetaClient` - writes topology records to group 0; `KVClusterAdmin` delegates to - it. A heartbeat is just another client write. -- **If ALL group-0 replicas are down, the cluster is down.** Heartbeats - don't matter in that scenario. No component can read group 0 - anyway. -- **A node hosting group 0 also hosts other groups.** When it goes - down, its heartbeat expires (visibility), and each of its groups - elects a new leader independently (Paxos failover). The heartbeat - doesn't trigger failover; it provides visibility into which nodes - are alive and what they host. - -The keep-alive pattern for kv-server: - -``` -/srv/kv-server/ -> KvServerInstanceValue { - instance_id: u64, - mgmt_endpoint: String, - hosted_stores: Vec, - hosted_groups: Vec<(u64, u64)>, // (store_id, group_id) - health: HealthSummary, // aggregate of per-store/per-group status - last_heartbeat_ms: u64, -} -``` - -Other components (console, diskdb) can scan `/srv/kv-server/` to -discover live kv-server instances and their health. This complements -the pull model — push provides self-registration and discovery; pull -provides detailed runtime state. - -**The group-0 leader writing its own heartbeat is fine.** If the -group-0 leader is also a kv-server instance (which it usually is), it -writes its heartbeat to group 0 locally (as a blind put to its own -group). This is no more circular than a kv-server instance writing -user data to a group it hosts. It's a normal client write that goes -through the Paxos consensus path. - -### 2.8 Hardware admin via kv-client (no admin crowdb-rpc service) - -Hardware admin operations (add/remove rack/node/disk-group/disk, -set `*_status`) are writes to group-0 sysdata. They are performed -through `HardwareClient` in `crowdb-kv-client`, invoked by the console -(`crowdb-web` / `crowdb-cli`). There is **no `DiskdbAdminService` crowdb-rpc -surface**: the previous `diskdb_sys_service.fbs` / -`diskdb_sys_op.fbs` admin RPCs (`AddRack`, `SetDiskStatus`, -`FetchHardware`, `Keepalive`) are removed. `FetchHardware` is -replaced by `HardwareClient` prefix scans, `Keepalive` by -`ServiceRegistryClient.heartbeat`, and the add/remove/status ops by -`HardwareClient` methods. The diskdb server serves only -`DiskdbService` (allocate/free; stubbed `Unimplemented`) and -reads hardware state from group 0 via `HardwareClient` in its sync -loop. It does not own or serve hardware admin. - ---- - -## 3. Group-0 Sysdata Schema - -### 3.1 Key layout (text-path encoding) - -``` -# Service registry — scan all instances of a service -/srv/diskdb/ -> InstanceValue -/srv/kv-server/ -> KvServerInstanceValue -# Future: /srv/chunkdb/, /srv// - -# KV-cluster topology — persistent records for disaster recovery -/kv/store/ -> StoreValue -/kv/group// -> GroupValue -/kv/replica/// -> ReplicaValue - -# Hardware hierarchy — full path in each key for scan narrowing -/hw/rack/ -> RackValue -/hw/node// -> NodeValue -/hw/dg/// -> DiskGroupValue -/hw/disk//// -> DiskValue -# disk_id_hex = 32-char hex (128-bit DiskId, high:low, lowercase) - -# Per-disk-group maps — same hierarchy path as the dg they refer to -/hw/dg_owner/// -> DiskdbOwnerEntry -/hw/dg_bind/// -> KVGroupBindEntry -``` - -### 3.2 Text magic namespaces - -- `/hw` — hardware namespace (rack/node/disk-group/disk + maps). -- `/srv` — service registry namespace (all service instances). -- `/kv` — KV-cluster topology namespace (store/group/replica records). - -The encoding rationale (text vs binary magic independence, per- -namespace choice) is in -[`design-crowdb-protocol-key.md`](../protocol/design-crowdb-protocol-key.md) §5. - -### 3.3 Value encoding - -All group-0 values are JSON-encoded (`serde_json::to_vec` / -`serde_json::from_slice`). Value types live in `crowdb-protocol` (the -single home for cross-component data structures) and are used by -`crowdb-kv-client` directly, with no per-crate redefinition: - -- **Existing flatbuffer `*Value` types** (`RackValue`, `NodeValue`, - `DiskGroupValue`, `DiskValue`, `HwStatus`, `DiskId`) — used directly. - `crowdb-protocol`'s `build.rs` already derives - `serde::Serialize`/`Deserialize` on them. -- **New group-0 sysdata value types** added as flatbuffer messages in - `crowdb-protocol` (parallel to the hardware values): `StoreValue`, - `GroupValue`, `ReplicaValue`, `InstanceValue` (service-registry - value, generic across services), `OwnerMapValue` - (`{ instance_id, lease_expiry_ms }`), `BindMapValue` - (`{ store_id, group_id }`). `build.rs` derives serde on each. -- **Entry return types** (key fields parsed from the path + value, - produced by `HardwareClient`/`KVClusterMetaClient` reads) are plain - `#[derive(serde)]` Rust structs in `crowdb-protocol`: `DiskGroupEntry` - (`{ rack_id, node_id, dg_id, value: DiskGroupValue }`), - `DiskdbOwnerEntry` (`{ rack_id, node_id, dg_id, instance_id, - lease_expiry_ms }`), `KVGroupBindEntry` (`{ rack_id, node_id, dg_id, - store_id, group_id }`). These are not stored as-is; they are the - decoded form of a path + its JSON value. - -### 3.4 Scan patterns - -- `/hw/rack/` → all racks. -- `/hw/node/` → all nodes; `/hw/node//` → nodes in one rack. -- `/hw/dg/` → all disk-groups; `/hw/dg//` → disk-groups in - one rack; `/hw/dg///` → disk-groups of one node. -- `/hw/disk/` → all disks; narrowing by rack/node/disk-group. -- `/hw/dg_owner/` → entire ownership map; narrowing by rack/node. -- `/hw/dg_bind/` → entire binding map; narrowing by rack/node. -- `/srv/diskdb/` → all live diskdb instances. -- `/srv/kv-server/` → all live kv-server instances. -- `/kv/store/` → all stores; `/kv/group//` → groups in one - store; `/kv/replica///` → replicas in one group. - -### 3.5 `DiskGroupId` widening: u32 → u64 - -Key structs (`DiskGroupKey`, `DiskKey`, `OwnerMapKey`, -`BindMapKey`) use `u32` for `disk_group_id`. The flatbuffer type -`NodeValue` also uses `u32` for `disk_group_ids` (`repeated uint32`) -and `last_used_dg_id` (`uint32`). (`DiskGroupValue` has no -`disk_group_id` field; the id is in the key.) This requirement -widens `DiskGroupId` to `u64` for consistency with all other integer -IDs and to remove the artificial 4-billion limit. The `NodeValue` -flatbuffer fields change from `uint32` to `uint64`; the key struct fields -change from `u32` to `u64`. The `disk_group_id` fields in -`diskdb_op.fbs` (`AllocateBlocksRequest`, `QueryCapacityStatsRequest`, -`GetDiskGroupInfoRequest`, `GetDiskInfoRequest`) and the Info response -types (`NodeInfo`, `DiskInfo`, `DiskGroupInfo`) widen from `uint32` to -`uint64` as well. This is a breaking schema change, acceptable because -diskdb is greenfield. - -### 3.6 `dc_id` removal - -The schema drops `dc_id` from `RackKey { dc_id, rack_id }` (struct + -binary layout), `RackInfo`, `NodeInfo`, and `NodeValue.dc_id` (v1 -ships flat, no DC layer). The text-path schema has no `dc_id` -(`/hw/rack/`). The binary `RackKey` layout changes -accordingly (greenfield). - ---- - -## 4. Service Registry - -### 4.1 Registration and keep-alive - -Every long-running service instance registers itself in group 0 -under `/srv//` and heartbeats periodically. -`ServiceRegistryClient` in `crowdb-kv-client` is **generic across -services** (diskdb and kv-server now; chunkdb and future services -later). It takes a `service` name that selects the path namespace -(`/srv/diskdb/`, `/srv/kv-server/`, …) and the value shape. The API: - -- `register(service, instance_id, value: &InstanceValue) -> Result<()>` -- `heartbeat(service, instance_id, value: &InstanceValue) -> Result<()>` -- `unregister(service, instance_id) -> Result<()>` (clean shutdown) -- `read_instance(service, instance_id) -> Result>` -- `read_all_instances(service) -> Result>` (prefix scan) - -`InstanceValue` is the generic service-registry value -(`{ instance_id, rpc_endpoint, last_heartbeat_ms, extra: ServiceExtra }`). -`ServiceExtra` is a per-service enum (diskdb carries `owned_dg_ids: -Vec`; kv-server carries `hosted_stores`, `hosted_groups`, -`health`). Diskdb convenience wrappers -(`register_diskdb`/`heartbeat_diskdb`/`read_all_diskdb_instances`) and -kv-server wrappers delegate to the generic methods. - -DiskIO physical identity and configured hardware scope are immutable once -accepted. Physical identity is pinned with instance-revision CAS before scope -publication. The first configured scope requires an existing Rack record and -is published atomically with an advance of that Rack's revision, fencing -concurrent disk membership changes. Revalidate the instance after reading the -Rack revision so another first-scope publication cannot be overwritten. If the -Rack is absent, retain the physical pin and reject scope publication; an -instance-only write cannot substitute for the Rack fence. Ordinary heartbeats -with the same accepted scope update liveness without advancing the Rack. - -### 4.2 Services registered - -- **diskdb** — `/srv/diskdb/`. Heartbeat includes - `owned_dg_ids` (which disk-groups this instance owns). Used by the - diskdb sync loop and by a future notify mechanism for instance discovery. -- **kv-server** — `/srv/kv-server/`. Heartbeat includes - `hosted_stores`, `hosted_groups`, and aggregate health. Used by - the console and other components for kv-server discovery and - health visibility. **Implemented** (a background keep-alive - loop in `crowdb-kv-server` writing via `ServiceRegistryClient`). -- **Future services** (chunkdb, etc.) — same pattern under - `/srv//`. - -### 4.3 Liveness and expiry - -A service instance is considered live if its `last_heartbeat_ms` is -within a configurable TTL (default: 3× heartbeat interval). Readers -filter expired entries when scanning. There is no active eviction. -Expired entries are ignored and eventually overwritten or deleted on -clean shutdown. - -### 4.4 Client discovery - -`ServiceDiscoveryClient` (in `crowdb-kv-client`) wraps -`ServiceRegistryClient` with a per-service state map and TTL-based -refresh. Each state contains an atomically published cached result, an atomic -round-robin cursor, and an async single-flight refresh gate. The sharded-map -guard is dropped immediately after cloning the state and never crosses I/O. -Clients call `discover_all(service)` or -`discover_one(service)` to find living service instances by service -name without hardcoding addresses. The cache is poll-on-demand: the -first call queries group-0, subsequent calls within the cache TTL -(default 5 seconds — one heartbeat cycle) return the cached result. -`invalidate(service)` forces a re-query on the next call; called -after known topology changes (e.g. `cluster_init`, `deploy_diskdb`). - -Simultaneous expiry for one service produces one group-0 refresh; waiters -recheck the freshly published result after acquiring the gate. Instance -selection uses the service state's atomic cursor to round-robin among live -instances. The -`DiskdbClient` and `ChunkdbClient` retain their per-disk-group / -per-range routing caches (they have routing requirements that -round-robin doesn't cover); the generic discovery client handles the -"pick any living instance" case. - -The bootstrap flow: a client knows group-0's mgmt port (10000, from -R118) → constructs `CrowdbKvClient` with that seed → -`ServiceDiscoveryClient::from_shared_kv` → `discover_all("diskdb")` -queries `/srv/diskdb/` via group-0 → cache → client connects to the -discovered RPC endpoint. No per-service address configuration needed -beyond group-0's bootstrap address. - -Explicit address overrides (`--diskdb-ip`, `--chunkdb-ip` on the -CLI; `Option<&str>` explicit endpoint on ops functions) bypass -discovery for that one call. This is the test exception (unit tests -pass addresses directly) and the operator escape hatch. When the -registry is empty and no override is provided, the client fails with -a clear "no living instances" error. - ---- - -## 5. Bootstrap and Cutover - -### 5.1 Two-phase bootstrap - -Phase 1 uses console TOML as the topology source of truth -(operator-managed). Phase 2 cuts over to group 0 authoritative: -`http_cluster_init` writes hardware records via `HardwareClient` and -KV-cluster topology records via `KVClusterMetaClient` into group 0, -and `crowdb-kv-server` restores its stores/groups from local disk on -restart with group 0 as the verification and fallback source for -remote-replica wiring (see -[`design-crowdb-kv-server.md`](design-crowdb-kv-server.md) §2.2). The -topology content in TOML is bootstrap-only: `--root` is required on every -start, and restart with `--root` alone rebuilds every local store/group from -WAL and rejoins the cluster. An optional `--config` remains the local authority -for process tunables in both modes; deleting it intentionally restores typed -defaults. Group 0 does not publish process configuration. The node -root is persisted to group 0 via `KvServerExtra.data_root` on each -keep-alive tick so the cluster/console knows each node's data -location. - -### 5.2 No `/topology/ready` flag - -The old schema had a `/topology/ready` flag key indicating group 0 -is authoritative. The text-path schema has no equivalent. v1 does -not need it. diskdb's sync loop treats an empty group 0 as "nothing -assigned yet" and retries. If a readiness signal is needed later, -add a derived condition (e.g. "group 0 has ≥1 `/hw/node/` key"). - -### 5.3 Greenfield migration - -diskdb is greenfield (no production diskdb). The -old `/topology/...` records are replaced by `/hw/...` and -`/kv/...`. Treat as greenfield: require a fresh cluster init. Old -`/topology/...` keys are orphaned harmlessly. - -### 5.4 Leader readiness before writing - -`HardwareClient` and `KVClusterMetaClient` write via crowdb-rpc to the -group-0 leader. `http_cluster_init` can only write sysdata after a -group-0 leader is elected and reachable. For single-node init this -is immediate (self-elect). For multi-node, the init flow must wait -on leader election before writing. Add a readiness poll if the -existing flow doesn't already wait. - ---- - -## 6. Relationship to Existing Design Docs - -- **`design-crowdb-kv.md` §3.3** — defines the system group concept. - This doc expands it with the sysdata schema, service registry, and - keep-alive model. -- **`design-crowdb-kv-server.md` §2.4** — defines the HTTP management - API. This doc clarifies that the API is internal (only - `crowdb-kv-client` calls it) and that `topology_finalize` / - `topology_ready` are removed. -- **`design-crowdb-kv-reconfiguration.md`** — group 0 membership - evolves using the shipped Model B reconfiguration. No new consensus - primitive required. -- **`design-crowdb-protocol-key.md` §5** — documents the unified key concept - (one struct, two encoding traits: `BinaryKey` + `TextKey`). - -## 7. Cascading cleanup - -`HardwareClient` provides `remove_*_cascade` methods that delete an -entity and all its children in group 0: `remove_rack_cascade` → -nodes → disk-groups → disks + owner/bind/usage records. The -single-record `remove_*` methods are kept for internal use. - -- **I1 — ID reuse safety**: After a rack/node/disk-group/disk is - removed, its ID can be safely reused. The group-0 sysdata deletion - is the fence against resurrection. - -Deployment nodes use the [deployment architecture](../deploy/design-crowdb-deploy.md) -for discovery, numeric ID allocation, SSH admission and system-resource ownership. -The KV server durably accepts one `SystemBootstrapIdentity` before store 0 creation; -identified initialization and snapshot join share this exclusion. Cleanup retires -the operation before deleting store 0 and releasing bindings, fencing delayed -commands. Published topology and node/service operations require current Group 0 -authority. Its unavailability is an error, never an empty ownership set. Other data -groups retain their own Paxos authority; ownership-dependent services obey their -existing expiry/fencing rules. diff --git a/doc/design/kv/design-crowdb-kv-leader-election.md b/doc/design/kv/design-crowdb-kv-leader-election.md deleted file mode 100644 index 5566f47aa..000000000 --- a/doc/design/kv/design-crowdb-kv-leader-election.md +++ /dev/null @@ -1,373 +0,0 @@ - - - -# CROWDB - Design: Leader Election, Term, and Lease - -Depends on: [`design-crowdb-kv.md`](design-crowdb-kv.md) §3, §4.2, §6.2, §7 -Satisfies: [`design-crowdb-kv.md`](design-crowdb-kv.md) §3, §4.2, §6.2, §7 - -This document specifies leader election, term management, the `PxBallot`/`PxTerm` separation, and the leader lease used for fast linearizable reads. The design follows Raft very closely; only the per-slot Paxos parts differ. - -## Table of Contents - -- [1. Why Raft-Style Election on a Paxos Log](#1-why-raft-style-election-on-a-paxos-log) -- [2. PxTerm vs PxBallot](#2-pxterm-vs-pxballot) -- [3. Election Protocol](#3-election-protocol) -- [4. New-Leader Bulk Phase 1](#4-new-leader-bulk-phase-1) -- [5. Heartbeats and Liveness](#5-heartbeats-and-liveness) -- [6. Leader Lease](#6-leader-lease) -- [7. ReadIndex Fallback](#7-readindex-fallback) -- [8. Step-Down Triggers](#8-step-down-triggers) -- [9. Safety Argument](#9-safety-argument) -- [10. Tunables and Defaults](#10-tunables-and-defaults) - ---- - -## 1. Why Raft-Style Election on a Paxos Log - -Classical Multi-Paxos elects a "distinguished proposer" through Paxos itself: an aspiring leader runs Phase 1 over the entire log. This works but conflates election with proposal and complicates clean step-down, lease management, and observability. - -Raft's contribution was to factor election out of the log — a separate election RPC, a per-term vote, randomized timeouts. CROWDB adopts that factoring exactly, and lets per-slot Paxos handle just the per-slot work. The separation maps onto two distinct identifiers: `PxTerm` for elections, `PxBallot` for slot proposals. - -This design choice is consistent with *How to Build a Highly Available System Using Consensus* (Lampson, 1996) and is the same pattern used in Spinnaker, Spanner's Paxos groups, and TiKV's region leadership. - ---- - -## 2. PxTerm vs PxBallot - -The two identifiers serve disjoint purposes; conflating them is a common bug source. - -### 2.1 `PxTerm` - -- Type: monotonic non-negative integer per group. -- Persistence: durable. Every term increment is fsynced before being acted on. -- Scope: one term covers all writes by a single leader, across many slots. -- Used by: - - Election RPCs (`RequestVote`, `Vote`). - - Heartbeats and `Accept` messages (carried for fencing). - - Lease validity (a lease is associated with a term). - -### 2.2 `PxBallot` - -- Type: lexicographically ordered pair `(round, leader_id)`. -- Persistence: durable on every acceptor (in WAL with the `PxLogEntry`). -- Scope: one ballot covers proposals at a single slot. Different slots may simultaneously be at different ballots. -- Used by: - - Phase 1 (`Prepare`) and Phase 2 (`Accept`) messages of classical Paxos. - - Repair: a higher round at the same leader_id supersedes a lower round at that slot. - -### 2.3 The bridge - -A new leader at term `T` proposes new slots with ballot `(round=0, leader_id=me)`. The implicit ordering is "this leader's ballots are 'newer' than any leader before it" because acceptors fence on `term` first: an `Accept` carrying `term=T` is rejected by any acceptor whose `current_term > T` regardless of ballot. Conversely a same-term `Accept` is compared by ballot rules. - -This means: - -- Acceptors keep `(current_term, slot_state[slot])` where `slot_state` records `(promised_ballot, accepted_ballot, accepted_value)`. -- Election bumps `current_term`, invalidating any in-flight `Accept` from older terms. -- Repair within a term bumps `round` for a single slot, invalidating any in-flight `Accept` for that slot only. - -Both fences are necessary, and they are independent. - ---- - -## 3. Election Protocol - -The protocol matches Raft's leader election with adaptations for the bulk Phase-1 step (§4). - -### 3.1 Roles - -Each member is exactly one of: - -- **Follower** — accepts heartbeats and `Accept`s from a leader. -- **PreCandidate** — running a `PreVote` round (see §3.2a); has not yet bumped `current_term`. -- **Candidate** — has bumped `current_term` and is collecting votes. -- **Leader** — won the latest election; serves writes. - -### 3.2 Election trigger - -A follower starts an election when its **election timer** expires. The timer is reset on every legitimate heartbeat or `Accept` from the current leader. The timeout is randomized in `[election_min, election_max]` (defaults 4000 ms – 8000 ms; see §10), avoiding split votes. - -When the timer fires: - -1. (PreVote, enabled by default) Follower transitions to PreCandidate and sends `PreVote(proposed_term, candidate_id, accepted_log_tip_slot, accepted_log_tip_term)` to all peers *without* bumping `current_term`. If a quorum grants, proceeds to step 2. If a peer reports a higher term, reverts to follower. If quorum is not reached, reverts to follower and waits for the next deadline. -2. Follower transitions to Candidate, bumps `current_term` to `proposed_term`, persists (`VoteGranted` WAL record). -3. Votes for itself. -4. Sends `RequestVote(term, candidate_id, accepted_log_tip_slot, accepted_log_tip_term)` to all peers. - -> **PreVote** (Raft optimization, ON by default): avoids spurious term increments from a partitioned-and-rejoined node. A node that was partitioned can rejoin and ask "would you vote for me?" without disrupting the current leader's term. - -### 3.3 Vote rules (matching Raft) - -A peer grants its vote iff: - -- The request's `term` is ≥ peer's `current_term`. If higher, peer adopts the new term and reverts to follower first. -- The peer has not already voted in this term, or already voted for the same candidate. -- The candidate's acceptor log is **at least as up-to-date** as the peer's, where up-to-date is the lexicographic comparison `(accepted_log_tip_term, accepted_log_tip_slot)`. -- The peer's `vote_lockout_until` has passed (no active lease promise from a heartbeat). - -The "log up-to-date" check is the safety property that prevents a stale leader from being elected. - -### 3.4 Outcome - -- **Win:** received votes from a quorum within the election timeout. Becomes leader; immediately runs the bulk Phase 1 (§4) and then sends initial heartbeats. -- **Loss / split:** sees a higher term, or election timer fires again with no quorum. Reverts to follower and waits for a new randomized timer. -- **Conversion:** if the candidate sees a `Heartbeat` or `Accept` with `term ≥ current_term`, it accepts that node as leader and reverts to follower. - ---- - -## 4. New-Leader Bulk Phase 1 - -A freshly elected leader does not immediately serve client writes. It first runs **one** Phase 1 over the open slot prefix to discover any in-flight values from the previous leader. - -### 4.1 What is the open prefix? - -When the leader takes office, it computes: - -- `floor = its own contiguous_chosen` (deliberately NOT maxed with peers' `contiguous_chosen`). After a restart, a replica can win election while missing committed slots it never received (the value lived only as a `ChosenNotice` watermark, which carries no payload). Maxing with peers' commit point would skip exactly those slots, leaving the new leader serving a stale value for a committed key. Sweeping from the leader's own frontier forces Phase 1 to re-derive every higher slot from the quorum. -- `ceiling = max(local highest_seen_slot, next_slot - 1, peers' highest_seen_slot from election replies)`. - -The open prefix is `[floor + 1, ceiling]`. If a previous leader had assigned slots beyond what any current acceptor knows about, those slots are simply not in this set; their values are unrecoverable but also un-acked, so no client expects them to exist. - -### 4.2 Bulk Prepare - -The leader runs Phase-1 `Prepare` **per slot** over `[floor+1, ceiling]`, batched by `bulk_prepare_window` (default 1024) with a `yield_now()` between batches to avoid starving other tasks. For each slot, `Prepare(ballot=(0, me), term=T)` is sent to all peers. Each acceptor responds with `Promise` carrying the highest `(accepted_ballot, accepted_value)` if any. - -The ballot used here is `(0, me)`. Because acceptors fence on `term` first, this ballot is "fresh" for the new term; we don't need a higher round. - -### 4.3 Adopt and re-Accept - -For each slot in the range: - -- If any peer's `Promise` returned an accepted value, choose the value with the highest `accepted_ballot` and re-Accept it at `(0, me)` under term T. -- If no peer returned a value, propose `NoOp` and re-Accept at `(0, me)` under term T. - -These re-Accepts are pipelined; the new leader does not wait for any one to be chosen before proposing the next. - -### 4.4 Steady state begins - -Once the bulk Phase 1 has been *issued* (not necessarily completed), the leader is free to start assigning new slots starting at `ceiling + 1`. The repair work for `[floor+1, ceiling]` proceeds in parallel and reuses the same machinery as routine gap repair (see [`design-crowdb-kv-slot.md`](design-crowdb-kv-slot.md) §9). - ---- - -## 5. Heartbeats and Liveness - -Heartbeats serve two purposes: liveness signaling (reset followers' election timers) and lease maintenance (extend the leader's lease). - -### 5.1 Heartbeat content - -Every heartbeat carries: - -- `term`, `leader_id`. -- `committed_safe_slot` (latest known safe-slot). -- `lease_grant_until` (monotonic-time deadline; see §6). -- `prev_log_slot, prev_log_term` (Raft-style consistency check). -- `t_send_ms_mono` (monotonic timestamp for lease grant calculation). - -`Chosen(slot)` notifications and `Accept`s are sent through separate `learner_stream` paths, not piggy-backed on heartbeats. - -### 5.2 Heartbeat cadence - -- Default `heartbeat_interval = 500 ms` (see §10). -- A follower's election timer must be ≫ `heartbeat_interval` to allow for occasional jitter; with defaults, election timeouts are 8–16× the heartbeat interval. - -### 5.3 Heartbeat response - -Followers respond with their own `term`, `success` (false if the follower's term is higher), `contiguous_chosen`, `last_chosen_term`, `contiguous_applied`, `highest_seen_slot`, and `durable_snapshot_slot`. The leader uses these to: - -Election term, vote, and lockout deadline remain one mutex-protected state -because transitions must publish them consistently. Observation-only paths use -the atomic current-term snapshot when staleness is safe; election decisions -take the full state lock. - -- Detect a stale leader's continued existence (if any response carries a higher term, the leader steps down (§8)). -- Maintain peer state (used by replicator and gap detection). -- Refresh the safe-slot computation. - -### 5.4 Follower catch-up and background apply - -When a follower's `contiguous_applied` lags behind the leader's `committed_safe_slot`, the heartbeat round drives catch-up in two phases: - -- **Phase 1 — `BatchChosenNotice` (value-less).** The leader sends a single fire-and-forget `BatchChosenNotification` frame over the peer RPC connection covering `[follower_applied+1, catchup_end]`, carrying only `(start_slot, end_slot, term, leader_id)`, no per-slot payload. The follower checks its local acceptor for each slot in the range: present slots advance the chosen frontier via `update_chosen_frontier`; missing slots remain gaps. This lets the follower's apply loop start processing present slots immediately, without waiting for the full-accept round-trip. -- **Phase 2 — full accepts.** The leader sends `send_accept` RPCs with full payloads for the same range. Slots the follower already has are re-accepted (idempotent CAS, cheap); slots the follower is missing get the real value. In steady state (no election churn) every slot is present, so Phase 1 does all the work and Phase 2 is a no-op. - -Engine apply is decoupled from the heartbeat handler via a **background apply loop**: - -- `handle_heartbeat` stores `committed_safe_slot` via `fetch_max` into `known_commit_slot` and signals `apply_notify`, then returns immediately with the current `contiguous_applied` (which may lag). The heartbeat reply is not blocked on engine apply. -- The background apply loop (lazily spawned, cancelled on shutdown) reads `known_commit_slot`, collects entries from the acceptor, and applies them via the learner. Missing slots are skipped (skip-and-continue) so a single gap doesn't block the entire apply backlog. `contiguous_applied` stays at the gap; subsequent available slots are applied out-of-order via `advance_applied_frontier`. -- Accept persists accepted state but does not advance the chosen or applied - frontier. A matching Chosen notification advances the chosen frontier and - wakes the background apply loop; missing or stale accepted values are - recorded as gaps for recovery. - ---- - -## 6. Leader Lease - -A lease lets the leader serve `Get(mode=Linearizable)` without a per-read quorum round-trip ([§6.1 of design-crowdb-kv.md](design-crowdb-kv.md#61-linearizable-leader-read)). The lease is the standard Raft-style approach with a clock-skew bound. - -### 6.1 What the lease grants - -While its lease is valid, a leader may serve linearizable reads from local state under two assumptions: - -- No other node was elected leader during the lease window. -- The leader's `contiguous_applied` slot reflects all writes acked through this leader (which is true by Invariant I3 from [`design-crowdb-kv-slot.md`](design-crowdb-kv-slot.md) §2). - -The first assumption is enforced by: - -- Acceptors **promise not to vote for any other candidate** for at least `lease_duration` after granting a heartbeat (this is the Raft "PreVote + lease" pattern). -- The leader rejects both PreVote and RequestVote while its quorum-renewed - effective lease remains valid, even after the original self-vote lockout - expires. Renewing this protection uses the existing atomic lease deadline; - a lagging minority cannot recruit the healthy leader as its extra voter. -- The leader treats its lease as expired at `lease_grant_time + lease_duration - max_clock_skew` for safety (see §6.3). - -### 6.2 Lease grant and renewal - -Each heartbeat round-trip is also a lease grant: - -1. Leader sends heartbeat at monotonic time `T_send`. -2. Follower receives, records "I will not vote for any candidate before `T_recv + lease_duration`", and replies. -3. Leader receives the response at `T_recv_reply`. The lease is valid through `T_send + lease_duration` on the leader's clock. The leader uses `T_send` (not `T_recv_reply`) as the start so that any clock skew works in its favor. -4. The leader treats the lease as effective until `T_send + lease_duration - max_clock_skew`. This is conservative; it gives a safety margin equal to the assumed skew bound. - -With the default `heartbeat_interval = 500 ms` and `lease_duration = 9 × heartbeat_interval = 4500 ms` (see §10), the leader is essentially always within an active lease in steady state. A short network blip costs the leader its fast-read privilege but not its leadership. - -### 6.3 Clock-skew assumption - -design-crowdb-kv.md §3](design-crowdb-kv.md) caps clock skew at `max_clock_skew` (default `500 ms`, see §10) per heartbeat interval. The lease formula is: - -``` - effective_lease = lease_duration - max_clock_skew -``` - -Concretely with defaults: `4500 ms - 500 ms = 4000 ms` of "fast-read" coverage between successful heartbeat round-trips. If the round-trip exceeds the effective lease, the lease has *technically* expired and the leader downgrades to ReadIndex for the next linearizable read until the next successful heartbeat refreshes the lease. - -### 6.4 Why monotonic-only - -All lease math uses the **monotonic clock**, not wall-clock. Wall-clock can jump (NTP step, manual operator change); monotonic cannot. This is the same discipline as Raft's reference implementation. - -### 6.5 What goes wrong if lease misuse occurs - -If a leader serves a linearizable read after its lease has *truly* expired and a new leader has been elected, the read can return stale data. The read would not have observed a write committed by the new leader. This is the precise correctness reason for the conservative `effective_lease`. - -The fallback is ReadIndex, which gives the same correctness guarantee without the clock assumption. - ---- - -## 7. ReadIndex Fallback - -ReadIndex is the lease-free path for linearizable reads. Always available; used automatically when the lease is not effective; available to clients on a per-read basis. - -### 7.1 Procedure - -1. The leader records its current `contiguous_applied` slot, call it `R`. -2. The leader broadcasts a heartbeat to a quorum, awaits responses. -3. As soon as a quorum has responded *with the leader's own term*, the leader is confirmed to still be leader at this point in time. -4. The leader serves the read from local state, ensuring the engine's `contiguous_applied >= R`. - -The cost is one network round-trip per ReadIndex (or per batch of reads). No fsync, no extra log records. - -### 7.2 Batching - -Concurrent ReadIndex barriers coalesce onto a single in-flight heartbeat round so a burst of N expired-lease linearizable reads costs one quorum round-trip, not N. - -The leader holds a pending-barrier batch (`PxGroup::pending_read_barrier`) for the duration of one ReadIndex round: - -1. The first read to arrive (the *round leader*) captures `R = contiguous_chosen`, registers the batch, and runs `run_heartbeat_round`. -2. Reads that arrive while the round is in flight enqueue a `oneshot` waiter on the batch instead of starting their own round. -3. When the round completes, the round leader drains the batch and resolves every waiter with the same outcome: `Ready { read_slot: R }` on quorum ack, `NotLeader` on step-down (higher term), `NoQuorum` when quorum is unreachable. A waiter whose round leader is cancelled (dropped sender) receives `NoQuorum` and retries. -4. The lease fast path is unchanged: lease-valid reads still serve immediately without queueing. - -The mutex over the batch serializes enqueue and drain, so no waiter is lost. A read either joins the in-flight batch or, after the leader drains, starts a fresh batch. - -**Correctness** is identical to the single-read procedure in §7.1. The heartbeat quorum at the leader's term confirms no higher-term election displaced this leader during the round, so every committed write is reflected in the leader's local state. The engine get returns the *latest* applied value for the key (single-version, highest-slot-wins), not a value pinned to `R`; each batched read performs its own `engine_get_bytes` after the barrier resolves and observes the freshest local state at its serve time. The shared `R` reported in the response is therefore a conservative freshness floor (the pre-round `contiguous_chosen`), never an over-estimate. A write that commits *during* the round has slot > `R` and is returned by the engine get, so late-arriving batched reads are not stale. - -**Metric.** `read.barrier.l` count increments once per read that enters the ReadIndex barrier. For a batched burst of N reads coalesced onto a single heartbeat round, all N reads observe the same (amortized) barrier latency — `read.barrier.l` drops toward one RTT amortized across the batch. - -### 7.3 When to choose ReadIndex over lease - -Per-read settings: - -- **Default:** lease if effective, ReadIndex if not. -- **Force ReadIndex:** for environments without a clock-skew guarantee, or for workloads requiring belt-and-suspenders correctness. Higher latency but no clock dependence. - -The choice is exposed as a per-request option in the read RPC and as a default at the server config. - ---- - -## 8. Step-Down Triggers - -A leader steps down (transitions to follower) on any of: - -- **Higher term seen.** Any RPC response carrying `term > current_term` causes the leader to adopt the new term and revert to follower. -- **Lease unrenewable.** The leader has been unable to obtain a heartbeat-quorum response for longer than `lease_duration`. It cannot refresh its lease and assumes a partition or majority loss. -- **Admin step-down RPC.** Operator forces it for testing, planned maintenance, or rolling upgrade. -- **Removed from group.** Designed but not yet implemented (`ConfigChange` log entries are not yet supported). - -On step-down: - -1. Cancel the per-tenure `CancellationToken`, aborts in-flight bulk Phase 1 and any tenure-bound work. -2. Stop heartbeats (the leader-state loop returns on step-down). -3. Set `role = Follower`; adopt the higher term if the trigger was `HigherTerm`. -4. Expire `LeaseState` (`become_follower` clears leader id). -5. In-flight proposals see `NotLeader` via the propose leadership gate (proposing_term check). - ---- - -## 9. Safety Argument - -The election + bulk-Phase-1 + lease design must satisfy three properties. - -### 9.1 At most one leader per term per group - -Standard Raft argument: - -- A candidate needs majority votes within a term. -- A peer votes at most once per term. -- Two majorities over the same membership intersect (by quorum overlap). -- Therefore at most one candidate can collect a majority in any one term. - -CROWDB inherits this directly. - -### 9.2 No chosen value is lost across leader change - -By Paxos's classical safety: if a value `v` was *chosen* at slot `s` (≥ majority of acceptors `Accepted` it), then any future Phase-1 over slot `s` will see `v` from at least one acceptor and re-propose it. - -The new-leader bulk Phase-1 (§4) executes this exact procedure over the open prefix, so any chosen value is preserved. - -A value that was `Accepted` by some but not a majority of acceptors may be either re-chosen or lost, but it was never acknowledged to the client (Invariant I3); the client knows its outcome is unknown and retries. - -### 9.3 No stale leader serves a stale linearizable read - -Two enforcement mechanisms run in series: - -- **Term fencing:** every RPC carries `term`. Any acceptor with a higher term refuses the request and informs the sender, which steps down. -- **Lease conservatism:** even when no fencing has happened, the leader self-expires its lease at `effective_lease = lease_duration - max_clock_skew`. If the clock-skew assumption holds, no other leader can have been elected within this window (because acceptors won't vote against the lease). - -If the clock-skew assumption is *violated*, lease-based reads can return stale data. This is documented and the operator can force ReadIndex for stronger guarantees. - ---- - -## 10. Tunables and Defaults - -| Parameter | Default | Range | Notes | -| --- | --- | --- | --- | -| `heartbeat_interval` | 500 ms | 10 ms – 30 s | Should be ≪ `lease_duration` | -| `lease_duration` | 4500 ms | 100 ms – 60 s | Should be ≫ `heartbeat_interval` + `max_clock_skew` (rule of thumb: `9 × heartbeat_interval`) | -| `effective_lease` | derived | — | `= lease_duration - max_clock_skew` | -| `max_clock_skew` | 500 ms | 1 ms – 5 s | Architectural bound from design-crowdb-kv.md §3](design-crowdb-kv.md) | -| `election_min` | 4000 ms | ≥ 8 × `heartbeat_interval` | Avoid spurious elections | -| `election_max` | 8000 ms | ≤ 60 s | Bounds time to elect after leader loss | -| `prevote_enabled` | true | bool | Reduces disruption from rejoining nodes | -| `bulk_prepare_window` | 1024 | 1 – 65536 | Slots per yield batch in bulk Phase-1 repair | -| `maintenance_tick_ms` | 30 000 | 1 000 – 300 000 | Engine-durability + WAL-GC maintenance loop interval | - -Notes: - -- **PreVote** (Raft optimization, ON by default): a candidate first asks "would you vote for me?" without bumping the term. This avoids spurious term increments from a partitioned-and-rejoined node. -- **Pre-emptive step-down on heartbeat loss** is governed by `lease_duration`, not `election_min`. A leader that loses contact with quorum gives up its lease at `lease_duration` and stops serving fast reads even before any follower starts an election. -- **Default choice rationale.** The defaults above are tuned for general-purpose deployments (mix of single-DC and modest cross-AZ latency). They give ~5–8 s failover detection, which matches the operational expectations of most online KV workloads while keeping heartbeat chatter low. See [`design-crowdb-kv.md`](design-crowdb-kv.md) §13 (Open Design Questions) for the analysis. -- **Operational profiles:** - - *Low-latency single datacenter:* `heartbeat_interval = 100 ms`, `election_min/max = 800/1500 ms`, `lease_duration = 900 ms`, `max_clock_skew = 100 ms`. Sub-second failover; higher message rate. - - *Cross-region / WAN:* `heartbeat_interval = 3 s`, `election_min/max = 24/48 s`, `lease_duration = 27 s`. Matches CockroachDB-style geo-replicated tunings; tolerates wider RTT variance at the cost of failover latency. - - *Tests:* `PxElectionConfig::for_tests()` produces `heartbeat = 5 ms`, `election = 30–60 ms`, `lease = 25 ms` for use under `tokio::time::pause()`. Not exposed on `crowdb-kv-server` CLI. diff --git a/doc/design/kv/design-crowdb-kv-observability.md b/doc/design/kv/design-crowdb-kv-observability.md deleted file mode 100644 index 5e4231632..000000000 --- a/doc/design/kv/design-crowdb-kv-observability.md +++ /dev/null @@ -1,505 +0,0 @@ - - - -# CROWDB - Design: Observability - -Depends on: [`design-crowdb-kv.md`](design-crowdb-kv.md) §16 -Satisfies: [`design-crowdb-kv.md`](design-crowdb-kv.md) §16 - -## Table of Contents - -- [1. Mandatory Signals](#1-mandatory-signals) -- [2. Metrics Module](#2-metrics-module) - - [2.1 Metric Types](#21-metric-types) - - [2.2 Registry and Lifecycle](#22-registry-and-lifecycle) - - [2.3 Naming Convention](#23-naming-convention) - - [2.4 Instrumentation Points](#24-instrumentation-points) - - [2.5 System Metrics Collector](#25-system-metrics-collector) - - [2.6 Metrics Log File](#26-metrics-log-file) - - [2.7 In-Memory Access](#27-in-memory-access) - - [2.8 FFI Boundary](#28-ffi-boundary) - - [2.9 Design Principles](#29-design-principles) - - [2.10 Read Path Metrics](#210-read-path-metrics) - - [2.11 Write Path Metrics](#211-write-path-metrics) - - [2.12 Client Metrics](#212-client-metrics) - - [2.13 C++ Registry Ownership](#213-c-registry-ownership) - - [2.14 Rust/C++ Metric Deduplication](#214-rustc-metric-deduplication) - - [2.15 C++ flush_to Format Alignment](#215-c-flush_to-format-alignment) - - [2.16 Shared Column Width](#216-shared-column-width) -- [3. Logging](#3-logging) - - [3.1 Log Directory Convention](#31-log-directory-convention) - - [3.2 Rotation and Compression](#32-rotation-and-compression) - - [3.3 Per-Stack Files](#33-per-stack-files) - - [3.4 Metrics Log](#34-metrics-log) - - [3.5 Level Unification](#35-level-unification) - - [3.6 C++ Logger Additivity](#36-c-logger-additivity) - - [3.7 No Separate ops_log](#37-no-separate-ops_log) - - [3.8 CLI Logging](#38-cli-logging) - - [3.9 Log Content Guidelines](#39-log-content-guidelines) - - [3.10 Extension Path](#310-extension-path) - -## 1. Mandatory Signals - -Per-group leader/term/max-slot/safe-slot/in-flight/gap count; per-node WAL -flush latency and throughput; per-RPC rate/latency/error breakdown; structured -logs with `node_id`, `group_id`, `slot`, `term` on consensus events. Tracing -hooks reserved but not required in the initial design. - -## 2. Metrics Module - -A lightweight metrics system with five metric types and periodic flush to a -dedicated metrics log file. Rust owns the registry for consensus/RPC/WAL -metrics; C++ owns its own registry for storage-engine metrics. Rust drives -the flush cycle and triggers C++ to emit its section via FFI — no metric -handles cross the FFI boundary at runtime. - -### 2.1 Metric Types - -- **Counter** (`AtomicU64` x 2) — monotonic, tracks window delta + total. - `inc()` / `inc_by(n)`. Flush shows `count`, `tps`, `total`. Use cases: - puts, gets, deletes, errors, WAL records, elections, step-downs. -- **Gauge** (`AtomicU64`) — current state, can go up or down. `set(v)`. - Flush shows last value. Use cases: buffer pool resident/dirty pages, - in-flight slots. -- **Bandwidth** (`AtomicU64` x 3) — monotonic bytes, tracks count + sum + - total_bytes. `observe(bytes)`. Flush shows `count`, `tps`, `avg_size(KB)`, - `rate(MB/s)`. Use cases: KV bytes in/out. -- **LatencyHistogram** (13 buckets + 2 `AtomicU64`) — fixed-bucket percentile - distribution. Bucket boundaries: `0, 1us, 10us, 100us, 500us, 1ms, 5ms, - 10ms, 50ms, 100ms, 500ms, 1s, infinity`. `observe(ns)` does binary search + - `fetch_add`. Flush computes p50/p99 from cumulative distribution. Use cases: - KV put latency, KV get latency. -- **LatencySummary** (`AtomicU64` x 4) — lightweight latency tracking - (count + sum + max + total_count). `observe(ns)`. Flush shows `avg(us)`, - `max(us)`. Use cases: scan, snapshot, WAL append, RPC, apply. -- **PreciseHistogram** (`lib/crowdb-common/rust/src/metrics/precise.rs`) — - HDR-style logarithmic histogram delivering ≥3 significant digits of - percentile precision, the precise counterpart to the fixed-bucket - `LatencyHistogram`. `1024` linear sub-buckets per power-of-2 magnitude - (relative error ≤0.1%); pre-allocated range `0..=2^32 µs` (~71 min) - makes `auto(true)` a no-op (no resize logic). `&mut self` methods - (`record`, `add`, `reset`); every call site has exclusive access - (`Mutex`-guarded in the client library, per-worker owned in the bench - `OpStats`, owned by the flusher task for `CumulativeLatency`), so a - lock-free impl is unnecessary. Tracks `count`/`sum`/`min`/`max` exactly - (only percentiles are bucketed). Use cases: bench-client per-op - latency (`OpStats`), client-library window latency - (`WindowLatencySnapshot`), bench cumulative run-wide percentiles - (`CumulativeLatency`). Replaces the external `hdrhistogram` crate. - -### 2.2 Registry and Lifecycle - -Each language has a `MetricsRegistry` that owns all metric instances. The -registry has `start(interval_secs)` (spawns flush thread/task), `stop()` -(final flush + join), and `flush()` (iterate all metrics, snapshot, format, -reset window state). Interval is typically 5s or 10s. - -- Rust (`lib/crowdb-kv/src/metrics/mod.rs`): `MetricsRegistry` with type-grouped - `Vec` collections, `Arc`-shared, metric handles stored on service/store - structs. `MetricsRunner` spawns a tokio interval task, computes real - elapsed `window_secs` per tick, flushes Rust metrics, then invokes a - post-flush callback that calls C++ `flush_metrics_str()` for each engine - and writes the `cpp-tree` block. Also provides `snapshot(prefix)` for - in-memory access without resetting window state. - The pre-flush engine collector loads immutable `ArcSwap` snapshots of its - per-group and per-store handles. Cumulative delta cursors live beside those - handles as atomics. Sampling therefore takes no handle-list or delta-state - mutex; discovery registers new metric handles in a separate bounded registry - phase and publishes a replacement snapshot. -- C++ (`lib/crowdb-tree/include/lib/crowdb-tree/metrics.h`, `lib/crowdb-tree/src/metrics.cpp`): - Same type-grouped pattern. `Crowdbtree` owns its own `MetricsRegistry` - internally (`init_metrics(prefix)` called from `open()`). Metric handles - are raw pointers (registry owns lifetime). `flush_metrics_str()` captures - `flush_to()` output via `open_memstream` for FFI return. The C++ - `start()`/`stop()` sleep-loop is retained for standalone/test use but not - called from the server's production flush path. - -### 2.3 Naming Convention - -Dot-separated hierarchical paths: `s.{store_id}.g.{group_id}.{module}.{metric}`. -Type suffix on every metric name: `.c` (Counter), `.g` (Gauge), `.bw` -(Bandwidth), `.lh` (LatencyHistogram), `.l` (LatencySummary). Dynamic suffix -`@{peer_endpoint}` for per-peer metrics. System metrics use `sys.` prefix -with no type suffix. - -Prefix-based snapshot: `registry.snapshot("s.1.")` returns all metrics for -store 1; `snapshot("")` returns all. This is the foundation for future GUI -integration. - -### 2.4 Instrumentation Points - -- Rust KV service (`kv_service.rs`): put/get latency histograms, scan summary, - delete counter, bytes in/out bandwidth, read bytes in/out bandwidth, - error/no-leader counters, get-forwarded / get-forward-failed counters. -- Rust WAL (`wal_engine.rs`, `pipeline_writer.rs`): append latency summary, - fsync latency summary (thinnest-layer disk IO), write bandwidth. -- Rust cluster (`local_replica.rs`): election/step-down counters, in-flight - slots gauge. Paxos slot watermarks (gauges bridged from `LocalReplica`). - Engine-apply latency summary (registered here, observed in - `learner.rs apply_entry`). -- Rust cluster (`group.rs`): read-path handles (`ReadRegistryHandles`) — - lease/ReadIndex path counters, read barrier latency summary, engine_get - latency summary, MinSlot-fallback counter, read-state gauges (lease valid, - contiguous applied, safe slot). Write-path handles - (`WriteRegistryHandles`) — propose-e2e, prepare-phase, accept-phase, - accept-quorum-RPC latency summaries. -- Rust RPC (`remote_replica.rs`): per-peer RPC latency summary + error counter - with dynamic names. -- C++ buffer pool (`buffer_pool.cpp`): hits/misses/evictions/writebacks - counters, resident/dirty gauges. -- C++ engine (`crowdb-tree.cpp`): apply latency summary, demand-load latency - summary + page-read bandwidth. -- C++ persist (`persist.cpp`): snapshot latency summary, snapshot-apply - latency summary, per-page write IO latency + cache-hit counter + write - bandwidth, metadata write bandwidth. -- C++ flush (`crowdb-tree.cpp`): flush latency summary, page-build (in-memory - mutation) latency summary, flush drain/entries magnitude counters. - -### 2.5 System Metrics Collector - -Special collector type polled at flush time (not increment-on-event). Reads -TCP retransmits/lost (Linux `/proc/net/snmp`, macOS no-op), CPU user/sys -and memory RSS via `getrusage(RUSAGE_SELF)`. Computes CPU% as delta over -flush window. - -### 2.6 Metrics Log File - -Dedicated file `metrics-{timestamp}-{pid}.log` in the log directory, separate -from application log. Each flush cycle produces one shared timestamp header -`[{ISO8601} window={N.NN}s]`, then section blocks: `rust` (Rust metrics + -misc system metrics), `cpp-tree` (C++ per-engine), and `cpp-rpc` (C++ global / -crowdb-rpc). Blocks are followed by type-grouped sections in order: -LatencyHistogram, LatencySummary, Bandwidth, Counter, Gauge, then System. -Names sorted alphabetically within each section, padded to `max_name_len` for -alignment. Zero-suppression: counters/histograms/summaries/bandwidths with -zero window activity are skipped; gauges always printed. C++ `flush_to()` -output is format-aligned to Rust's column layout (same units, columns, -precision). Format designed for both human reading and script parsing (split -on whitespace, parse as numbers). - -### 2.7 In-Memory Access - -`registry.snapshot(prefix)` returns current values without resetting window -state. Enables future `/metrics` HTTP endpoint and GUI integration. No need -to parse log files to get metric values. - -Management status is a curated view, not a second metrics system. Each status -field names one registry metric and reads it through exact-name lookup. Missing -or not-yet-registered names are omitted. The status path never registers a -metric or retains duplicate hot-path counters; `/metrics` remains the endpoint -for the complete registry. - -### 2.8 FFI Boundary - -C++ owns its own `MetricsRegistry` per `Crowdbtree` instance. Rust triggers C++ -to flush its metrics section into the same log file via FFI -(`ct_flush_metrics_str`). No metric handles cross FFI at runtime, only a -formatted string. Three log blocks per flush cycle: `rust` (Rust), -`cpp-tree` (C++ per-engine), and `cpp-rpc` (C++ global). The existing -`ct_get_stats` FFI call (used by `/topology` and the one remaining -`snapshot.pages.c` delta bridge) is unaffected. - -### 2.9 Design Principles - -- **Counter/Summary Non-Redundancy** — A `LatencySummary` (or - `LatencyHistogram`) already carries `count` (window delta) and - `total_count` (cumulative). Do not register a separate Counter for the - same event. A Counter next to a Summary is justified only for: (a) a - magnitude (e.g. entries moved, not calls made), (b) a different - population/outcome (e.g. errors vs. all attempts), or (c) a cache-hit - path where no latency is meaningful. A `Bandwidth` next to a `Latency` is - not redundant; they measure different dimensions (bytes vs. nanoseconds) - of the same event. -- **Latency Hierarchy** — For every flow, add latency at the thinnest layer - (actual syscall / disk IO / RPC boundary) and at the feature layer above - it (the meaningful unit of work a human debugs against). Children should - roughly sum to the parent; a persistent gap indicates an unaccounted code - path. Skip a layer if it isn't a distinct decision point. -- **Bandwidth Hierarchy** — Parallel to latency: measure bytes moved at the - IO boundary, domain-separated (page data vs. metadata, read vs. write). - Skip layers with no distinct IO. -- **Real Window-Time** — The flush window is the real elapsed time, not the - nominal configured interval. Rust computes `window_secs` once per tick - and passes the same value to both its own `reg.flush()` and the C++ FFI - call, guaranteeing identical windows across both sections. - -### 2.10 Read Path Metrics - -The read path mirrors the write path's latency-bandwidth-counter hierarchy. -Handles live in two places: `KvMetrics` (per store, group; in -`kv_service.rs`) for RPC-layer metrics, and `ReadRegistryHandles` (per -store, group; on `PxGroup` via `OnceLock`, mirroring -`ElectionRegistryHandles` on `PxLocalReplica`) for consensus- and -engine-layer metrics. - -- **Latency hierarchy** (feature layer → thinnest layer): - - `kv.get.lh` — get RPC end-to-end (existing). - - `read.e2e.l` — `LatencySummary` for the full server-side read path - (barrier + engine_get), measured in `kv_get` handler. - - `read.barrier.l` — `LatencySummary` for - `linearizable_read_barrier` (near-zero for lease path, one heartbeat - RTT for ReadIndex). - - `read.apply_fence.l` — `LatencySummary` for the R35 apply fence - wait (fast path is a single atomic load). - - `read.engine_get.l` — `LatencySummary` for `KVEngine::get_bytes` - (isolates engine cost from consensus barrier cost). - - `kv.scan.l` — scan RPC end-to-end (existing). -- **Bandwidth hierarchy** (read vs. write separation; combined kept for - backward compat, read subset lets operators derive write by subtraction): - - `kv.read_bytes_in.bw` / `kv.read_bytes_out.bw` — read traffic - separated from the combined `bytes_in/out.bw`. -- **Counters** (outcome / population separation): - - `kv.get_forwarded.c` — reads forwarded to leader (server-side). - - `kv.get_forward_failed.c` — forward attempts that failed. - - `read.minslot_fallback.c` — MinSlot reads redirected to leader - because the local replica hasn't caught up. -- **Gauges** (state, bridged on-demand at `resolve_read_point`): - - `read.safe_slot.g` — current `group_safe_slot`. - -`read.barrier.l` count equals the total linearizable get count in the -same window (lease fast path + ReadIndex path combined). - -### 2.11 Write Path Metrics - -The write path mirrors the read path's latency hierarchy. Handles live in -`WriteRegistryHandles` (per store, group; on `PxGroup` via `OnceLock`, -mirroring `ReadRegistryHandles`), except `engine_apply` which is registered -on `PxLocalReplica` (alongside the WAL handles) and observed in -`PxLearner::apply_entry`. - -- **Latency hierarchy** (feature layer → thinnest layer): - - `paxos.propose.e2e.l` — `propose_inner` entry → return (the full - client-observed proposal latency, including retries). - - `paxos.classic.prepare.l` — `run_prepare_phase` entry → return - (classic Paxos only; leader path skips prepare). - - `paxos.accept.quorum_rpc.l` — accept-phase start → first-quorum - reached (the k-th-fastest remote reply latency; recorded only on the - quorum short-circuit success path, not on the failure path). - - `paxos.learn.apply.l` — `PxLearner::apply_entry` entry → return - (isolates engine apply cost from consensus phase cost). - -All five are `LatencySummary` (count + sum + max), matching the read-path -`barrier` / `engine_get` pattern. Percentile precision is not needed for -phase-level attribution. The bench client already has `PreciseHistogram` -for client-observed p99. The `accept_quorum_rpc` timer is meaningful only -after the quorum short-circuit (§6.1 of `design-crowdb-kv-rpc.md`); it records the -quorum-th-fastest remote latency, not the full fan-out tail. - -### 2.12 Client Metrics - -`crowdb-kv-client` exposes its own `ClientMetrics` (lock-free `AtomicU64` -counters, snapshotted via `ClientMetricsSnapshot`) for retry, topology, -and read-distribution observability. They are not part of the server's -`MetricsRegistry`. The client is a separate process and has no C++ -handle. Two counters cover follower-read distribution: - -- `read_endpoint_distributed` — incremented each time the `AnyReplica` - selector picks a replica from the cached list for a `MinSlot` read - (i.e. the distribution branch fired). Pairs with the server-side - `read.minslot_fallback.c` to confirm the distribution rate and the - fallback rate. -- `read_endpoint_fallback` — incremented each time a distributed - `MinSlot` read fell back to the leader via `NotLeaderHint` (get) or - the scan `"not leader; retry scan at "` parse. A high - `read_endpoint_fallback / read_endpoint_distributed` ratio means - followers are lagging and the policy is providing little benefit. - -The client library also maintains per-op-kind window latency histograms -(`WindowLatency` / `WindowLatencySnapshot`). Recording threads select one of -64 stable shards and contend only with recorders assigned to that shard; -`drain_window` locks and merges one shard at a time for periodic flushing and -feeds the bench runner's cumulative -`bench.*.lh` histograms. These use `crowdb-common`'s `PreciseHistogram` -(not the server's fixed-bucket `LatencyHistogram`) because the bench -needs p50/p90/p99/p999/max at ≥3 significant digits, the same precision -the bench's own `OpStats` and `CumulativeLatency` require. The bench -runner's per-worker `WorkerCounters` use `crowdb-common`'s `Counter` -(window delta via `flush().count`, cumulative total via -`snapshot().total`), unifying the bench and client on the project's own -metrics primitives and removing the external `hdrhistogram` dependency. - -### 2.13 C++ Registry Ownership - -C++ `Crowdbtree` creates its own `MetricsRegistry` internally via -`init_metrics(prefix)`, called from `Crowdbtree::open()`. The external -`set_metrics(MetricsRegistry*, prefix)` API is removed. C++ metrics -(counters, gauges, summaries, bandwidths) are registered and observed -entirely in C++, then flushed to string via `flush_metrics_str()` and -written to the log by the Rust `MetricsRunner` post-flush callback. - -### 2.14 Rust/C++ Metric Deduplication - -Once C++ owns its registry, the Rust-side bridge (`engine_collector.rs`) -no longer polls C++ cumulative counters and gauges. Those metrics appear -natively in `cpp-tree`. The Rust `rust` section keeps only -Rust-native metrics (KV service, RPC, Paxos, WAL). The one exception is -`snapshot.pages.c`, a magnitude counter with no paired latency in the C++ -registry, which remains bridged via `ct_get_stats` delta polling. - -### 2.15 C++ `flush_to` Format Alignment - -C++ `flush_to()` output is aligned to Rust's column layout: `tps(/s)` -column on all windowed types, latency in `us` (not `ns`), bandwidth in KB, -Histogram column order matching Rust, and `window=%.2fs` precision. The -section header label is parameterized (`"metrics"` for standalone, -`"cpp-tree"` for per-engine FFI-driven flush, `"cpp-rpc"` for global -FFI-driven flush). - -### 2.16 Shared Column Width - -Both `rust` and `cpp-tree` sections use the same column width -for metric names. Before each flush tick, Rust queries each C++ engine's -max name length via `ct_max_name_len()`, computes -`shared_width = max(rust_max, max(cpp_maxes))`, and passes it to both its -own `flush()` and C++ `flush_metrics_str()`. This adapts automatically as -handles are added dynamically. - -## 3. Logging - -CROWDB has two logging stacks — a Rust `tracing` stack -(`crowdb-common/rust/src/logging.rs`) and a C++ `spdlog` stack -(`crowdb-common/cpp/src/log.cpp` + `compressing_sink.cpp`). Both -produce rotating, gzip-compressed log files with the same naming -convention. This section anchors the unified scheme across all -server binaries. - -### 3.1 Log Directory Convention - -Every server binary accepts a `--log-dir

` CLI flag. All Rust -and C++ logs for a process land under that directory. Defaults: - -- `crowdb-kv-server`, `crowdb-diskdb`, `crowdb-chunkdb` — `"log"` - (relative to CWD). `crowdb-kv-server` also falls back to - `CrowDBConfig.log_dir` (config field, default `root/log`) when - `--log-dir` is not given. -- `crowdb-web` — `~/.crowdb-kv/log`. -- `crowdb-cli` — multi-service lifecycle, server deployment, and benchmark - commands use `/cli-log/cli-{command-slug}-{timestamp}/`; short - inspection and data operations are console-only and create no file log. - -The test harness passes `--log-dir ` so child-process -file logs land in `test-logs/` alongside redirected stderr. - -### 3.2 Rotation and Compression - -Both stacks use the same defaults: - -- Max file size: 30 MiB (`DEFAULT_LOG_MAX_FILE_MB`). -- Max rotated files: 5 (`DEFAULT_LOG_MAX_FILES`). -- Tunable via `--log-max-file-mb` and `--log-max-files` CLI flags. - -Rotation: when the current file exceeds the size cap, it is closed, -gzip-compressed to `.log.gz`, and a new file is opened. The oldest -`.log.gz` files beyond the retention count are deleted. - -File naming: `{prefix}-{YYYYMMDD-HHMMSS.mmm}-{pid}.log`. The PID -suffix prevents collision when two processes with the same prefix -start in the same second. - -### 3.3 Per-Stack Files - -Rust and C++ write to separate files in the same directory (D1). -Prefixes: - -- `{server}` — Rust tracing log (e.g. `crowdb-kv-server`). -- `{server}-tree` — C++ crowdb-tree spdlog (e.g. - `crowdb-kv-server-tree`). -- `{server}-rpc` — C++ crowdb-rpc spdlog (e.g. - `crowdb-kv-server-rpc`). -- `{server}-metrics` — metrics log (Rust, via `MetricsRunner`). - -The shared timestamp + PID suffix lets the operator interleave files -from one process by sort order; no cross-FFI write coordination -needed. The merged-file option is rejected — per-stack files keep the -two stacks independent and avoid cross-FFI write contention. - -### 3.4 Metrics Log - -The metrics log is the one combined file per process (D2). Rust -queries the metrics registry (including C++ metrics via FFI) and -writes the content via `MetricsRunner`. The C++ stack does not write -to the metrics log directly. - -### 3.5 Level Unification - -Both Rust and C++ stacks default to `info` (D3). The `--log-level` -CLI flag drives both stacks. If `RUST_LOG` is set, the Rust -`EnvFilter` uses it directly; for the C++ side, the first global -directive is extracted via `cpp_level_from_rust_log()` (e.g. -`RUST_LOG=debug` → C++ `"debug"`, `RUST_LOG=crowdb_kv=info` → C++ -`"info"`). Valid levels: `trace`, `debug`, `info`, `warn`, `error`, -`off`. - -The `--log-stderr ` flag mirrors C++ log lines at the given -level or above to the process stderr (via `ct_add_log_stderr` / -`crowdb_rpc_ffi::add_log_stderr`). This lets operators tail stderr -for failures without grepping the file. `crowdb-web` defaults this to -`"warn"`; other servers default to off. - -### 3.6 C++ Logger Additivity - -When `ct_init_logging` runs first (as in `crowdb-kv-server`), -`crowdb_rpc_init_logging` calls `add_log_file` — it adds a second -file sink to the same `crowdb::common` logger rather than creating a -new one. The rpc sink inherits the tree logger's level. The -`--log-level` flag sets the tree logger's level, which the rpc sink -inherits. This is the expected behavior: both C++ stacks run at the -same level and write to separate files via separate sinks on the same -async logger. - -When `ct_init_logging` has NOT run (as in `crowdb-web`, -`crowdb-diskdb`, `crowdb-chunkdb`, `crowdb-cli`), -`crowdb_rpc_init_logging` creates a fresh logger with the given level. - -### 3.7 No Separate ops_log - -The former `ops_log` module has been deleted (D4). Operations lines -(HTTP/RPC/SSH call records) are folded into the normal tracing log -via `log_ops_http` in `crowdb-console-shared/src/clients.rs`. The -server log's rotation covers ops lines — no separate unbounded file. - -### 3.8 CLI Logging - -`crowdb-cli` writes file logs for all commands to a per-invocation -directory `cli-log/{command-slug}-{timestamp}/` (D5, revised). The -`bench` subcommand additionally opens a metrics log via -`BenchMetrics::new`. Other commands can opt into additional logging -later if a detached use case appears. - -### 3.9 Log Content Guidelines - -Server and client logs use the same context field names: - -- `s`: store ID when known. -- `g`: group ID when known. -- `replica`: the replica executing or receiving an operation. -- `leader`: the server's current leader or the client's currently believed - leader. Omit it when unknown; do not copy a merely targeted replica into it. - -Stable `s`, `g`, and `replica` context belongs in tracing spans at task and RPC -boundaries. Spawned futures are explicitly instrumented because Tokio tasks do -not inherit the caller's active span. Leadership is dynamic, so `leader` or -`role` is recorded on the event whose interpretation needs it instead of being -captured in a long-lived span. - -C++ thread names are published as immutable thread-id/name maps. The spdlog -formatter atomically loads a snapshot keyed by the originating log message's -thread ID; it does not use formatter-thread TLS and takes no shared mutex. - -- Every line should carry enough context to locate the code position and the - surrounding state. -- Consensus-event logs must carry `replica`, `g`, `slot`, and `term` where - applicable (per §1 mandatory signals). -- Machine-parseability is a plus — AI agents do the real log-digging. - -### 3.10 Extension Path - -Any future server (e.g. `diskio` server) follows the same scheme: - -- `--log-dir` / `--log-level` / `--log-max-file-mb` / `--log-max-files` - / `--log` / `--log-stderr` CLI flags. -- Rust tracing init via `init_file_logging` or - `init_file_and_console_logging_split`. -- C++ FFI init for each linked C++ library (`ct_init_logging` for - crowdb-tree, `crowdb_rpc_ffi::init_logging` for crowdb-rpc). -- Per-stack file prefixes: `{server}`, `{server}-tree`, `{server}-rpc`. diff --git a/doc/design/kv/design-crowdb-kv-reconfiguration.md b/doc/design/kv/design-crowdb-kv-reconfiguration.md deleted file mode 100644 index 3009e747f..000000000 --- a/doc/design/kv/design-crowdb-kv-reconfiguration.md +++ /dev/null @@ -1,260 +0,0 @@ - - - -# CROWDB - Design: Reconfiguration - -Depends on: [`design-crowdb-kv.md`](design-crowdb-kv.md) §9.1, §9.2, [`design-crowdb-kv-leader-election.md`](design-crowdb-kv-leader-election.md) -Satisfies: [`design-crowdb-kv.md`](design-crowdb-kv.md) §9.1, §9.2 - -This document specifies how a CROWDB group safely changes its membership while preserving consensus safety. The mechanism is direct per-node HTTP mutation of each replica's remote-replica list, persisted to the local `GroupConfigStore`, with a `membership_epoch` exact-match fence. This model applies to all groups including the system group (group 0, which stores cluster topology metadata, see `design-crowdb-kv.md` §3.3). - -## Table of Contents - -- [1. Scope and Supported Transitions](#1-scope-and-supported-transitions) -- [2. Direct Per-Node Mutation Model](#2-direct-per-node-mutation-model) -- [3. New-Member Bootstrap](#3-new-member-bootstrap) -- [4. Member Removal](#4-member-removal) -- [5. Leader Transfer](#5-leader-transfer) -- [6. The `membership_epoch` Fence](#6-the-membership_epoch-fence) -- [7. Group-0 Special Cases](#7-group-0-special-cases) -- [8. Safety Argument](#8-safety-argument) -- [9. Failure During Reconfiguration](#9-failure-during-reconfiguration) -- [10. Tunables and Defaults](#10-tunables-and-defaults) - ---- - -## 1. Scope and Supported Transitions - -CROWDB supports membership changes within a single group. Specifically: - -- **Add or remove voting members.** Adding a member when going 3 → 5 → 7 or removing a member when going 7 → 5 → 3. -- **Replace a member.** Implemented as add-then-remove (or vice versa), each as a single-member change. -- **Change the leadership of the group.** Triggered as a side effect when removing the current leader. - -Out of scope ([design-crowdb-kv.md §2](design-crowdb-kv.md)): - -- Changing `num_groups` (the total number of groups in the cluster) — fixed at cluster creation. -- Splitting or merging groups — not supported. -- Reconfiguring a protected group below 3 voting members — not supported. The explicit single-node test deployment starts with one voter and has no fault tolerance; it is not entered through membership reconfiguration. -- Going above 7 voting members — not in the initial scope; quorum size grows linearly with membership and the marginal availability gain past 7 is small. - -**Granularity:** every reconfiguration moves *exactly one member* in or out at a time. To go 3 → 5, do two single-member additions in sequence. Under the exact-match `membership_epoch` fence (§6) this single-member-at-a-time rule is no longer required for safety, but it is still recommended because it minimizes the propagation window during which writes stall. - ---- - -## 2. Direct Per-Node Mutation Model - -The design does not use `ConfigChange` log entries or a joint configuration. Instead, the operator (or `crowdb-console`) mutates the remote-replica list on each node independently through the HTTP management API: - -- `POST /stores/:sid/groups/:gid/remotes` — add one or more remote replicas. -- `DELETE /stores/:sid/groups/:gid/remotes/:rid` — remove a remote replica. - -Each call is atomically persisted to the local `GroupConfigStore` config file (atomic `tmp` + `fsync` + rename) and applied synchronously to the in-memory `PxGroup`. `membership_epoch` is bumped by exactly one when the call changes the voting set. A non-voting member addition/removal does **not** change the epoch or the quorum size. - -Because there is no cluster-wide `ConfigChange` log, the quorum view is local to each node. This is safe because the `membership_epoch` fence (§6) forces all `Prepare`/`Accept` RPCs to be evaluated at the same epoch before their replies can count toward a quorum. A leader whose local epoch differs from a peer's rejects that peer's reply and converges upward (§6.3). - -### 2.1 Propagation window and write stalls - -A membership mutation is applied to one node per HTTP call. Between the first node's call and the last node's call, different nodes transiently disagree on the `membership_epoch`. During this window a leader that has already adopted the new epoch will see `epoch_mismatch` from nodes that have not yet adopted it. Writes are rejected by the acceptors and the leader's `propose` loop self-heals by adopting the responder's higher epoch (up to `max(own, peer)`) and retrying the same slot. - -This is a deliberate availability/safety trade-off: writes stall for the cluster-wide duration of the fan-out (one HTTP round-trip per node), then resume immediately. The stall is bounded, self-healing, and loudly visible in the `epoch_mismatch` response bit rather than silently using a stale quorum. - -### 2.2 Group-0 complete membership authority - -Group 0 keeps one authoritative JSON record at `/kv/members/{store}/{group}`. -The record contains the complete ordered member list, a positive monotonic -epoch, and either `Installing { previous_members }` or `Ready`. A console -membership change first CAS-publishes the complete successor at -`current_epoch + 1` in `Installing`; peer wiring and consensus fencing then -use that exact successor before the record is CAS-completed to `Ready`. - -The management endpoints carry `x-crowdb-membership-epoch`. A successor -request is accepted only for the authorized next epoch. A replay at the -already-installed epoch succeeds only when its remote payload matches the -existing configuration exactly; a different payload returns HTTP 409 before -the config file or running group is changed. This protects persistence and -publication as one guarded operation while allowing a lost response to be -retried safely. - ---- - -## 3. New-Member Bootstrap - -A new member joining a group has empty WAL and empty engine state. It must be brought up to date before it can vote. - -### 3.1 Phases - -``` - 1. Console ensures the new node hosts the store and places the group on it. - 2. The new member is added as `voting: false` on every existing node. - 3. The new member is added as `voting: false` on itself (so it learns remotes). - 4. Snapshot install: a caught-up peer streams a snapshot to the new member. - 5. WAL catch-up: the new member follows the leader via normal replication. - 6. The new member is re-registered with `voting: true` on every node. - 7. `membership_epoch` is bumped on every node; the new member is now a voting peer. -``` - -### 3.2 Why pre-load before voting - -If a new member were added directly with `voting: true` while still empty, then a quorum could form with its empty-state vote. A subsequent leader election under that quorum could elect a leader that has not seen older committed writes — violating Paxos safety. - -By requiring catch-up while the member is non-voting, we ensure that when it first becomes a voting peer it already has all chosen values up to the current `max_chosen`. - -### 3.3 Snapshot install protocol - -Defined in [§8.4 of design-crowdb-kv.md](design-crowdb-kv.md#84-snapshot-and-install) and `design-crowdb-kv-state-machine.md` §6 (snapshot import). Resumable, throttled, end-to-end CRC. - -### 3.4 Catch-up termination criterion - -The leader continues streaming WAL until the new member's `contiguous_applied` is within `catchup_slack` (default 100 slots) of the leader's `max_chosen`. At that point the operator re-adds the member as `voting: true` and the new member is expected to keep up via normal replication once it becomes voting. - -If the new member cannot keep up (e.g. slow disk), the operator can abort by removing it before flipping the `voting` flag. - ---- - -## 4. Member Removal - -Removing a member is simpler than adding one because there is no catch-up to wait for. - -### 4.1 Procedure - -``` - 1. If the removed member is the current leader, ask it to StepDown first. - 2. Delete the removed replica as a remote from every surviving peer. - 3. Delete the local `PxGroup` on the removed member's node. - 4. The `membership_epoch` is bumped on every node where the voting set changed. -``` - -### 4.2 If the removed member is the leader - -The console's `DELETE /api/stores/:sid/groups/:gid/replicas/:rid` handler calls `POST /stores/:sid/groups/:gid/step-down` on the leader node first. The leader rejects further proposals, the survivors run a normal election, and only after a new leader is observed does the removal proceed. If the step-down call fails (network partition, crashed process), the console proceeds anyway and the survivors' lease-unrenewable logic eventually elects a new leader. - -### 4.3 If the removed member is unreachable - -Removal works regardless of reachability. The surviving peers delete the dead replica from their remote lists, and the dead replica's local group is removed if the node is still reachable. The `membership_epoch` bump on each survivor ensures the remaining voters cannot be confused by delayed heartbeat/accept traffic from the removed node once it comes back. The removed node is then absent from every survivor's `remote_replica_info()`. - ---- - -## 5. Leader Transfer - -Transferring leadership is needed when: - -- The current leader is being removed by reconfiguration. -- An admin wants to drain a node for maintenance. - -### 5.1 Procedure - -1. Admin or console calls `POST /stores/:sid/groups/:gid/step-down` on the leader node. -2. The leader strictly-fences the request (`self.id == target_leader_id && current term`) and calls `handle_step_down`, becoming a follower. -3. The old leader stops accepting new client writes; in-flight requests are handled by the new leader once elected. -4. Followers run a normal randomized-timeout election. The first up-to-date candidate to reach a quorum of the current voting set wins. -5. The old leader, on observing the new term in any RPC response, steps down. - -This is not Raft's `TimeoutNow` fast transfer. It avoids the complexity of target-precondition checking and is accepted as sufficient because membership changes are operator-driven. - -### 5.2 During reconfiguration - -When removing the current leader: - -1. The console calls step-down on the leader before issuing the removal. -2. The console waits (bounded, 5 s) for a survivor to report a leader *other than* the removed replica. -3. The removal proceeds; the new leader's `membership_epoch` now governs the group. - ---- - -## 6. The `membership_epoch` Fence - -Every `Prepare`/`Accept` RPC carries the proposer's current `membership_epoch`. The acceptor compares it to its own local `membership_epoch`. If they do not match exactly, the acceptor rejects the request with `epoch_mismatch: true` and attaches its own epoch. The proposer then adopts `max(own_epoch, responder_epoch)` and retries the same slot. - -### 6.1 Why exact match, not "within 1" - -Single-degree membership changes (old/new configs differing by exactly one voting member) have a proven quorum-overlap guarantee: any majority of `N` and any majority of `N±1` always share at least one member. That guarantee does **not** chain transitively across two sequential changes unless the first one fully converges everywhere. A counter-example: `C1 = {1,2,3}` (quorum 2) and `C3 = {1,2,3,4,5}` (quorum 3) have majorities `{2,3}` and `{1,4,5}` that do not intersect. So "one epoch behind is safe" is false in general. The only airtight rule is **exact epoch match**. - -### 6.2 What exact match buys - -Since a quorum decision can never straddle two epochs, the single-degree-at-a-time restriction stops being a safety requirement. A batch membership change (same voting set delta applied to multiple nodes in one fan-out) is just as safe as a single-member change; it only takes longer to converge. - -### 6.3 Convergence and self-healing - -Both the proposer and the acceptor converge upward to `max(own, peer)`: - -- **Proposer (`PxGroup::propose`):** on `MembershipEpochMismatch`, adopts `responder_epoch` and retries the same slot. -- **Acceptor (`PxLocalReplica::on_prepare`/`on_accept`, heartbeat catch-up):** on `epoch_mismatch`, adopts `max(own, proposer_epoch)` before returning the rejection. - -This bidirectional convergence ensures that even if the leader starts behind, it catches up within the same `propose()` call and the write succeeds as soon as the last straggler adopts the new epoch. The only cost is the bounded fan-out stall (§2.1). - -### 6.4 Non-voting members must not count toward quorum - -A non-voting catch-up member physically accepts and promises so it can follow the log, but its reply must not count toward the voting quorum. `PxGroup::run_prepare_phase` and `PxGroup::run_accept_phase` only increment `promised`/`accepted` for `RemoteReplicaKind::Real(remote)` when `remote.voting` is true. The local self-count is gated on `replica.voting()`. `PxGroup::propose` and `run_bulk_phase1` reuse `self.quorum()` (voting-only) rather than computing a voting-agnostic threshold. - ---- - -## 7. Group-0 Special Cases - -Group 0 (store 0, group 0) is the system group that stores cluster -topology metadata. It uses the same direct HTTP mutation + -`membership_epoch` fence reconfiguration model (§2) as all other -groups. No joint-consensus primitive is needed. Group 0 stores -cluster topology as KV entries under text-path keys (`/hw/rack/...`, -`/hw/node/...`, `/kv/store/...`, `/kv/group/...`) with JSON -values, written by `HardwareClient` and `KVClusterMetaClient` in -`crowdb-kv-client`. It is created via `POST /system/init`. See -`design-crowdb-kv.md` §3.3, `design-crowdb-kv-group0.md`, and -`../console/design-crowdb-console.md` §4.3 for details. - -A system embedding `crowdb-kv`'s primitives may choose to build a -joint-consensus layer on top of the `membership_epoch` fence. - ---- - -## 8. Safety Argument - -The safety property we must preserve through reconfiguration is: - -> **No two values can be chosen at the same slot.** - -### 8.1 The single-epoch case - -For a single fixed `membership_epoch`, standard Paxos safety applies: any two majorities of the voting set intersect. Therefore no two distinct values can both be chosen at the same slot. - -### 8.2 The transition case - -The only way two different values could be chosen at the same slot is if two disjoint quorums from different epochs both accepted. The `membership_epoch` fence prevents this: a `Prepare`/`Accept` reply is only counted toward quorum if the request's epoch exactly matches the acceptor's epoch. Two quorums can never be formed at different epochs for the same slot, because at least one of them would have required an epoch-mismatched reply and that reply would be rejected. - -### 8.3 Why the fan-out window is safe - -During the fan-out, nodes may transiently disagree on the epoch. A leader whose epoch is ahead sends `Prepare`/`Accept` with the higher epoch. Acceptor nodes still on the old epoch reject with `epoch_mismatch` and adopt the higher epoch. The leader's quorum count is therefore stalled until enough nodes have adopted the new epoch. The moment a quorum with the new epoch is reached, all nodes in that quorum share the same membership view, and Paxos safety holds. - -### 8.4 Asymmetric transitions - -The single-degree-at-a-time recommendation is not a safety requirement under the exact-match fence. Even a batch change (multiple voting members added or removed in one fan-out) is safe because the fence only permits quorums within one epoch. The trade-off is a longer propagation window and a correspondingly longer write stall. - ---- - -## 9. Failure During Reconfiguration - -Because membership is persisted per node and not via a consensus log, failure recovery is simpler: - -- **Leader crashes after the first node has been mutated but before the fan-out completes.** The remaining nodes may have different epochs. The new leader, once elected, adopts the highest epoch it observes from any peer and continues the fan-out from the console. Writes self-heal as epochs converge. -- **A new member fails during catch-up.** The operator removes it before flipping `voting` to `true`. Because the member is non-voting, its failure cannot affect quorum safety. -- **Network partition during reconfiguration.** The partition side that cannot reach a quorum of the current voting set cannot choose values. The side with the newer epoch may stall until enough nodes adopt it. The partition heals; no split-brain is possible because the epoch fence prevents cross-epoch quorums. -- **Console crashes mid-fan-out.** The operator re-runs the same mutation on the remaining nodes. The `membership_epoch` bump is idempotent because it is a synchronous local increment; applying the same mutation twice leaves the epoch at the same value. - ---- - -## 10. Tunables and Defaults - -| Parameter | Default | Range | Notes | -| --- | --- | --- | --- | -| `catchup_slack` | 100 slots | 10 – 10 000 | How close to leader's `max_chosen` the new member must be before flipping `voting` | -| `catchup_timeout` | 10 min | 1 min – 24 h | Operator abort threshold for catch-up | -| `lease_duration_ms` | election-profile dependent | — | How long a removed/unreachable leader can remain leader before survivors re-elect | -| `step_down_timeout` | 5 s | 1 s – 60 s | How long the console waits for a new leader after step-down before proceeding | - -**Operational guidance:** - -- Add members one at a time. Wait for the catch-up to finish and `voting` to flip before starting the next member. -- For 3 → 5: do two single-add reconfigurations. -- For 7 → 3: do four single-remove reconfigurations. -- Always step-down a leader before removing it. If the leader is unreachable, proceed and rely on the lease-unrenewable fallback. -- Monitor `epoch_mismatch` responses and `membership_epoch` values during reconfiguration — a sustained `epoch_mismatch` is a sign that the fan-out is still in progress. diff --git a/doc/design/kv/design-crowdb-kv-rpc-client.md b/doc/design/kv/design-crowdb-kv-rpc-client.md deleted file mode 100644 index 21b0ca707..000000000 --- a/doc/design/kv/design-crowdb-kv-rpc-client.md +++ /dev/null @@ -1,294 +0,0 @@ - - - -# CROWDB - Design: Client-Facing KV RPC - -Depends on: [`design-crowdb-kv-rpc.md`](design-crowdb-kv-rpc.md) §3, §5, §10, §11, §12; [`design-crowdb-rpc.md`](../rpc/design-crowdb-rpc.md) §3, §4, §6 -Satisfies: [`design-crowdb-kv.md`](design-crowdb-kv.md) §3, §10.1 - -The client-facing KV service (Put/Get/Delete/BatchWrite/Scan/ -JournalScan/CreateSnapshot/ListSnapshots/SnapshotScan/ReleaseSnapshot + -WatchNotify) uses the **crowdb-rpc flatbuffer transport** — the same -engine as the consensus hot path, but on a separate port and with a -dedicated schema (`kv_client.fbs`). The transport selection is -programmatic (`CrowdbClient::with_rpc_transport`). - -## Table of Contents - -- [1. Design Principles](#1-design-principles) -- [2. Message Surface](#2-message-surface) -- [3. Flatbuffer Schema (kv_client.fbs)](#3-flatbuffer-schema-kv_clientfbs) -- [4. Port Allocation](#4-port-allocation) -- [5. Server-Side Handler (KvRpcService)](#5-server-side-handler-kvrpcservice) -- [6. Transparent Leader-Forwarding](#6-transparent-leader-forwarding) -- [7. WatchNotify Server-Push](#7-watchnotify-server-push) -- [8. Client-Side Transport (KvRpcTransport)](#8-client-side-transport-kvrpctransport) -- [9. CrowdbClient Transport Selection](#9-crowkvclient-transport-selection) -- [10. WatchNotifyClient](#10-watchnotifyclient) -- [11. Zero-Copy Wrapper Classes](#11-zero-copy-wrapper-classes) -- [12. Error Model](#12-error-model) -- [13. Topology cache eviction](#13-topology-cache-eviction) - ---- - -## 1. Design Principles - -- **Same engine, separate port.** The client-facing crowdb-rpc server - binds `rpc_port + 200` (the `KV_CLIENT_RPC_BASE` offset). The - consensus crowdb-rpc server binds `rpc_port + 100`. Both run - simultaneously on the same node. -- **Same retry/topology/NotLeaderHint logic.** The client-side - transport returns the existing crowdb-rpc response types - (`KvResponse`, `KvScanResponse`, `KvJournalScanResponse`), so the - retry loop, topology cache, and `NotLeaderHint` handling in - `CrowdbClient` are unchanged — only the wire send changes. -- **Fire-and-forget WatchNotify.** The server pushes `FBWatchNotify` - frames to subscribed clients via `RpcClient::send` (no response - expected). Dead watchers are lazily removed on the first failed - push. - -## 2. Message Surface - -10 unary request/response pairs + 4 WatchNotify frames: - -- **Put**: `FBKvSetRequest` → `FBKvResponse` -- **Get**: `FBKvGetRequest` → `FBKvResponse` -- **Delete**: `FBKvDeleteRequest` → `FBKvResponse` -- **BatchWrite**: `FBKvBatchWriteRequest` → `FBKvResponse` -- **Scan**: `FBKvScanRequest` → `FBKvScanResponse` -- **JournalScan**: `FBKvJournalScanRequest` → `FBKvJournalScanResponse` -- **CreateSnapshot**: `FBCreateSnapshotRequest` → `FBCreateSnapshotResponse` -- **ListSnapshots**: `FBListSnapshotsRequest` → `FBListSnapshotsResponse` -- **SnapshotScan**: `FBSnapshotScanRequest` → `FBSnapshotScanResponse` -- **ReleaseSnapshot**: `FBReleaseSnapshotRequest` → `FBReleaseSnapshotResponse` -- **WatchSubscribe**: `FBWatchSubscribe` (fire-and-forget, no response) -- **WatchUnsubscribe**: `FBWatchUnsubscribe` (fire-and-forget, no response) -- **WatchNotify**: `FBWatchNotify` (server→client push, fire-and-forget) -- **WatchNotifyError**: `FBWatchNotifyError` (server→client push, fire-and-forget) - -## 3. Flatbuffer Schema (kv_client.fbs) - -The schema lives in `lib/crowdb-protocol/src/fbs/kv_client.fbs`. Codegen -runs via `crowdb-protocol`'s `build.rs`, producing `kv_client_generated.rs` -re-exported as `crowdb_protocol::kv_client_fb`. - -- **`[[ubyte]]` workaround.** Flatbuffers does not support nested - vectors (`[[ubyte]]`). The `FBBytes` wrapper table (`{ data: [ubyte] }`) - is used for byte-vector fields in `FBWatchNotify` (`keys`, `values`). -- **`FBKvClientRetCode` enum.** Maps 1:1 to the `KvErrorCode`: - `Success`, `NotLeader`, `Unavailable`, `Internal`, `JournalScanGcGap`, - `InvalidArgument`. -- **`FBReadMode` enum.** `Linearizable` / `MinSlot`, matching the - `ReadMode`. -- **`forwarded: bool` field.** On `FBKvGetRequest`, `FBKvScanRequest`, - and `FBKvJournalScanRequest` — the loop-guard for transparent - leader-forwarding (the `forwarded` field replaces the - `x-crowdb-kv-forwarded` metadata header). - -## 4. Port Allocation - -The client-facing crowdb-rpc port is derived from the base port: - -``` -client_rpc_port = rpc_port + (KV_CLIENT_RPC_BASE - KV_SERVER_RPC_BASE) - = rpc_port + 200 -``` - -This is parallel to the consensus-side offset (`rpc_port + 100`). The -`KV_CLIENT_RPC_BASE` constant lives in `crowdb-protocol::ports`. The port -mapping is registered as `KvServerClientRpc` in the port-claim registry. - -## 5. Server-Side Handler (KvRpcService) - -`KvRpcService` (in `lib/crowdb-kv/src/rpc/kv_rpc_service.rs`) holds -`Arc` + `Handle` (tokio runtime) + `Arc`. -`register_handlers` wires one handler per request `msg_type` into the -client-facing `RpcServer`. - -Each unary handler: -a. Parse request via `flatbuffers::root::Request>(req.control)`. - On parse failure → `submit_error` with `InvalidArgument`. -b. Dispatch to the existing `KvStore` trait methods - (`kv_put`/`kv_get`/`kv_delete`/`kv_batch_write`/`kv_scan`/ - `kv_journal_scan`/`kv_create_snapshot`/`kv_list_snapshots`/ - `kv_snapshot_scan`/`kv_release_snapshot`) — the same methods the - crowdb-rpc handler calls. The async path spawns a tokio task via - `self.rt.spawn` and submits the response from the task. -c. Build the response flatbuffer via `FlatBufferBuilder`, `finish`, - `submit_response`. - -Server startup: `start_client_rpc_server` (in `kv_server.rs`) creates -the `RpcServer`, binds the client-facing port, constructs the -`KvClientRpcForwarder` + `KvRpcService`, registers handlers, and stores -the server state in `PxKvStore::client_rpc_server_state`. `main.rs` -calls it after `start_rpc_server`. `shutdown_server` stops both -crowdb-rpc servers. - -## 6. Transparent Leader-Forwarding - -`Get`/`Scan`/`JournalScan` preserve the forward step (linearizable -reads only; `MinSlot` served locally). The loop-guard -`x-crowdb-kv-forwarded` metadata header becomes the `forwarded: bool` -field on the request flatbuffer. The handler: - -a. If `read_mode == Linearizable && !req.forwarded` and - `forward_target_for(group_id)` returns `Some(endpoint)` → re-issue - the request to the leader via `KvClientRpcForwarder`. Set - `forwarded = true` on the re-issued request. On success, submit the - leader's response. On forward failure, serve stale local + set - `not_leader_hint = endpoint`. -b. Else serve locally. - -`KvClientRpcForwarder` lives in `crowdb-kv` itself (not `crowdb-kv-client`) -to avoid a crate cycle. It holds an `Arc` + `Arc` -+ connection cache, builds the request flatbuffer with -`forwarded = true`, calls `rpc.call()`, and returns the raw response -control buffer. - -## 7. WatchNotify Server-Push - -`WatchSubscribe` handler: -a. Parse `FBWatchSubscribe` → `group_id`, `prefix`. -b. `store.get_group(group_id)`. If none → push `FBWatchNotifyError` - via `RpcClient::send` on the inbound connection, error="group not - found". No `submit_response` (fire-and-forget). -c. If `!group.local_replica().is_leader()` → push - `FBWatchNotifyError` with `not_leader_hint = group.leader_endpoint()`. -d. Else → build `Connection` via `from_handle`, register it in the - group's `WatchRegistry` with a `CrowdbRpcPushTarget` - (`(Connection, Arc, Arc)`). - -`WatchRegistry` uses a `PushTarget` enum: -- `CrowdbRpc` — `Arc` (crowdb-rpc push path). - -`emit` builds `FBWatchNotify` flatbuffer + `rpc.send` (fire-and- -forget). On `ConnectionClosed`/`ConnectionError` → increment -`closed_watchers` (lazy cleanup; the watcher is removed on the next -emit pass or via the safety-net poller). - -Connection-close cleanup: crowdb-rpc does not deliver a connection-close -callback to the handler. The `WatchRegistry::emit` path detects a dead -push target when `RpcClient::send` returns `ConnectionClosed`. The -safety-net poller covers missed notifications during the gap. - -## 8. Client-Side Transport (KvRpcTransport) - -`KvRpcTransport` mirrors `PxRpcTransport`: it holds `Arc`, -`Arc`, a shared `ConnectionPoolIndex`, and an atomic request-ID -source. `conn_for` derives the client-facing crowdb-rpc port from the base -port via the `KV_CLIENT_RPC_BASE` offset. Pool selection returns its -generation, and retryable errors invalidate only that generation. - -Callers with concurrent bulk scans and latency-bounded point reads may opt -into `with_isolated_point_reads`, or construct `CrowdbKvClient` with -`new_with_isolated_point_reads`. Get replies then use one additional connection -per endpoint, sharing the existing RPC workers, request-ID source, completion -pool, tunables and deadlines. Scans and mutations retain their ordinary pool. -Errors invalidate the selected pool's exact generation; explicit reconnect -clears both pools. The default constructor retains its existing connection -behavior. ChunkDB enables this mode so actual metadata point reads do not wait -behind an incomplete background scan response. Isolation does not cache success -or convert rejection, transport failure or an expired caller deadline to Ready. - -Unary methods (`send_put`, `send_get`, `send_delete`, `send_batch_write`, -`send_scan`, `send_journal_scan`): build request flatbuffer → -`rpc.call` → await → parse via `Ref` wrapper → map to the existing -crowdb-rpc response types (`KvResponse`, `KvScanResponse`, -`KvJournalScanResponse`). `NotLeaderHint` parsed from the -`not_leader_hint` string field → fed into `CrowdbClient`'s existing -retry + topology-cache logic. - -## 9. CrowdbClient Transport Selection - -`CrowdbClient` holds an `Option>` field -(`rpc_transport`). `with_rpc_transport(transport)` sets it. Each public -method (`put`/`get`/`delete`/`batch_write`/`scan`/`scan_count`/ -`journal_scan`) checks `self.rpc_transport` first: when set, delegate -to the transport's `send_*` (with the existing retry/topology/ -`NotLeaderHint`/metrics wrapping — only the wire send changes). - -## 10. WatchNotifyClient - -`WatchNotifyClient` uses the crowdb-rpc transport path: - -- **crowdb-rpc path**: persistent - connection + client-side handler. The reader loop resolves the leader, - opens a connection via `KvRpcTransport::get_conn`, registers - handlers for `FBWatchNotify` + `FBWatchNotifyError` via - `RpcClient::register_handler`, sends `FBWatchSubscribe` as - fire-and-forget `send()`. On `FBWatchNotifyError` with non-empty - `not_leader_hint` → `topology.set_leader` + reconnect signal. A - periodic liveness check (fire-and-forget ping every 5s) detects - dead connections — crowdb-rpc does not deliver a connection-close - callback. - -## 11. Zero-Copy Wrapper Classes - -`FBKvResponseRef`, `FBKvScanResponseRef`, `FBKvJournalScanResponseRef`, -`FBCreateSnapshotResponseRef`, `FBListSnapshotsResponseRef`, -`FBSnapshotScanResponseRef`, `FBReleaseSnapshotResponseRef`, -`FBWatchNotifyRef` — zero-copy views over the response flatbuffer -control buffer. Each provides null-safe accessors (`valid()`, -`ret_code()`, `error_msg()`, `ok()`, etc.). See -[`design-crowdb-rpc.md`](../rpc/design-crowdb-rpc.md) §6 for the wrapper -convention. - -## 12. Error Model - -`FBKvClientRetCode` → `KvErrorCode` mapping: - -- `Success` → `KvErrorNone` -- `NotLeader` → `KvErrorNotLeader` -- `Unavailable` → `KvErrorUnavailable` -- `JournalScanGcGap` → `KvErrorJournalScanGcGap` -- `Internal` / `InvalidArgument` / unknown → `KvErrorInternal` - -`RpcError` → `Error` variants: -- `ConnectionClosed` → retry on next connection (same as crowdb-rpc - `Unavailable`). -- `Timeout` → `Error::Transport` (caller retry budget). -- `SendQueueFull` → retry with backoff. -- `RegistrationFailed`/`AllDown`/`InvalidArg` → non-retryable - `Error::Transport`. -- `NotLeaderHint` is a protocol-level response (`not_leader_hint` - string + `FBKvClientRetCode::NotLeader`), not a transport error — - the client's retry/topology-cache logic is unchanged. -- `JournalScanGcGap` → `Error::JournalScanGcGap` (caller falls back - to a full-scan rebuild). - -## References - -- [`design-crowdb-kv-rpc.md`](design-crowdb-kv-rpc.md) — consensus RPC wire - protocol (same engine, different port + schema). -- [`design-crowdb-rpc.md`](../rpc/design-crowdb-rpc.md) — RPC engine - overview (framing, connection pool, handler pattern, wrapper - convention). -- [`design-crowdb-kv-watch-notify.md`](design-crowdb-kv-watch-notify.md) — - WatchNotify design (trie, prefix matching, safety-net poller). - -## 13. Topology cache eviction - -Within one operation, distinct leader hints are followed immediately. Repeated -or cyclic hints consume the configured retry budget and use the existing -election wait and topology refresh rather than issuing an unbounded RPC loop. -Scan and journal continuations reset this redirect history after a successful -page. Conditional writes retain their original mutation identity on retries. - -After a transport failure or leadership rejection without a usable hint, -discovery searches the target group across management seeds. A reachable seed -that repeats the rejected RPC endpoint cannot hide another member's replacement -route. Concurrent refreshes share the existing refresh gate and interval; a -newer cached route is reused without another fetch. If no replacement is known, -the operation retains its original bounded election wait/backoff and retry -budget. Discovery never establishes authority for a mutation; the destination -still applies its normal leadership and conditional-write checks. - -`TopologyCache` publishes one immutable `TopologySnapshot`. Every group route -contains its leader, replica endpoints, read cursor, endpoint statistics, and -write-slot high-watermark. A refresh builds a complete replacement snapshot; -leader hints and refreshes use compare-and-swap so stale work cannot overwrite -a newer route generation. Eviction retires the complete group route, so a -reused group ID starts with fresh high-watermark and selection state. - -- **I1 — Cache eviction on topology change**: When a group disappears from - topology, its complete route generation is evicted atomically. diff --git a/doc/design/kv/design-crowdb-kv-rpc.md b/doc/design/kv/design-crowdb-kv-rpc.md deleted file mode 100644 index 06212f2c9..000000000 --- a/doc/design/kv/design-crowdb-kv-rpc.md +++ /dev/null @@ -1,549 +0,0 @@ - - - -# CROWDB - Design: RPC Wire Protocol - -Depends on: [`design-crowdb-kv.md`](design-crowdb-kv.md) §2, §3, §9.2, §10 -Satisfies: [`design-crowdb-kv.md`](design-crowdb-kv.md) §3, §9.2, §10.1 - -This document defines the wire-serialization contract for all -node-to-node and client-to-node RPC communication. The consensus hot -path (Prepare/Promise, Accept/Accepted, PreVote/RequestVote, -Heartbeat/StepDown, ChosenNotification, FetchGap) uses the -**crowdb-rpc flatbuffer transport** — a custom epoll/kqueue-based RPC -library with flatbuffer framing and request/response correlation via -oneshot channels. Client-to-node KV RPCs (Get/Set/Delete/Batch/Scan) -use the same crowdb-rpc engine on a separate port with a dedicated -schema — see [`design-crowdb-kv-rpc-client.md`](design-crowdb-kv-rpc-client.md) -for the client-facing transport design. Every message carries -a `version: u32` field for forward/backward compatibility; no -`required` fields; field numbers are append-only. - -## Table of Contents - -- [1. Design Principles](#1-design-principles) -- [2. Message Surface](#2-message-surface) -- [3. Transport Architecture — crowdb-rpc for Consensus](#3-transport-architecture--crowdb-rpc-for-consensus) -- [4. Server-Side Handler (PxRpcService)](#4-server-side-handler-pxrpcservice) -- [5. Client-Side Transport (PxRpcTransport)](#5-client-side-transport-pxrpctransport) -- [6. RPC Routing in PxRemoteReplica](#6-rpc-routing-in-pxremotereplica) -- [7. Flow Control and Parallelism](#7-flow-control-and-parallelism) - - [7.1 Quorum Short-Circuit](#71-quorum-short-circuit) - - [7.2 RPC Deadline](#72-rpc-deadline) -- [8. Paxos Error Model](#8-paxos-error-model) -- [9. Version Compatibility Rules](#9-version-compatibility-rules) -- [10. Flatbuffer Schema (kv_consensus.fbs)](#10-flatbuffer-schema-kv_consensusfbs) -- [11. Zero-Copy Wrapper Classes](#11-zero-copy-wrapper-classes) -- [12. Port Allocation](#12-port-allocation) -- [13. Flatbuffer `bytes` Field Mapping — `bytes::Bytes`](#13-flatbuffer-bytes-field-mapping--bytesbytes) - ---- - -## 1. Design Principles - -1. **All RPCs over crowdb-rpc.** The Paxos hot - path (Prepare/Accept/Heartbeat/Chosen/FetchGap) runs on the - crowdb-rpc flatbuffer transport for zero-copy framing and - epoll/kqueue I/O efficiency. Client-facing KV RPCs (Get/Set/Delete/ - Batch/Scan/Watch) run on the same crowdb-rpc engine on a separate - port with a dedicated schema for client library ergonomics - and `Bytes` mapping. -2. **Pipelined unary calls on persistent connections.** Steady-state - data traffic (`Accept`, `ChosenNotification`) multiplexes over - pipelined unary `call()`s on a persistent crowdb-rpc connection per - `(group_id, peer_id)` pair. Connection setup cost is paid once per - server lifetime. Fire-and-forget frames (`ChosenNotification`, - `BatchChosenNotification`) use `send()` with a no-op completion - callback — no reply is awaited. -3. **Heartbeats on the same connection.** Steady-state `Heartbeat` - traffic flows over the same crowdb-rpc connection as `Accept` and - `Chosen`. The crowdb-rpc transport multiplexes frames at the I/O - layer (epoll/kqueue), so heartbeats are not blocked behind data - frames the way they would in a bidi stream's FIFO send-half. - The term fence handles cross-term reordering; same-term - heartbeat/accept mutate independent state. -4. **No `required` fields.** All flatbuffer fields are optional or - have sensible defaults. A missing `version` field defaults to `0`. -5. **Field numbers are append-only.** Once assigned, a field number is - never reused for a different semantic meaning. This is a hard rule - for rolling upgrades. -6. **Plaintext in P1/P4; TLS hooks reserved.** The transport layer is - plaintext TCP loopback in tests and P1/P4 integration. TLS config - slots exist in the service builder but are unimplemented. - ---- - -## 2. Message Surface - -The wire protocol defines four classic Paxos messages (P1 M2): -`Prepare`, `Promise`, `Accept`, `Accepted`, sufficient to run a full -classic Paxos round across a real network boundary. Election messages -(`Heartbeat`, `RequestVote`, `PreVote`, `StepDown`) and the -`ChosenNotification` / `BatchChosenNotification` / `FetchGap` frames -are added in P1 M3; snapshot and client RPCs land in P4. - -Key reusable sub-message: `AcceptedValue` carries `(slot, round, -leader_id, term, payload)`. Used in `Accept` requests and `Promise` -responses for classic Paxos value-recovery. The `payload` is opaque -bytes in P1 M2; `kind` discrimination (empty = `NoOp`, non-empty = -`Write`) is not a wire field. `ConfigChange` and `DedupCheckpoint` -kinds are designed but not yet implemented. - -The full flatbuffer schema is in `lib/crowdb-protocol/src/fbs/kv_consensus.fbs`; -the crowdb-rpc schema definitions for client KV RPCs are in -`lib/crowdb-kv/src/rpc/fbs/`. This doc covers design decisions only. - ---- - -## 3. Transport Architecture — crowdb-rpc for Consensus - -The consensus RPC path uses the R104 `crowdb-rpc` engine: a C++ -epoll/kqueue transport with flatbuffer framing and request/response -correlation. The Rust facade (`crowdb-rpc-ffi`) exposes `RpcServer` -(listen + register handlers), `RpcClient` (connect + `call()` for -request-response, `send()` for fire-and-forget), and `Connection` -(per-peer persistent connection). - -**Why crowdb-rpc for consensus:** - -- **No per-stream lock.** The crowdb-rpc transport handles - concurrent frames without a per-stream lock, avoiding the ~17% - throughput loss that per-stream serialization would cause at 2T:1C - write workload. The epoll/kqueue I/O loop processes frames - concurrently. -- **Zero-copy framing.** Flatbuffer responses are read in-place via - `FBRef` wrappers — no deserialization into owned types on the - hot path. -- **Fire-and-forget support.** `ChosenNotification` and - `BatchChosenNotification` are sent via `send()` with a no-op - completion callback — no oneshot channel allocation, no reply - correlation overhead. - -**Client-facing KV RPCs:** Client-facing KV RPCs (`Get`, `Set`, `Delete`, -`Batch`, `Scan`, `Watch`) run on the same crowdb-rpc engine on a separate -port with a dedicated schema. The client library -(`crowdb-kv-client`) uses crowdb-rpc for retry, topology cache, and -`NotLeaderHint` handling. The crowdb-rpc server also serves -bounded snapshot-session handlers for unpublished new-member install. - -Follower-forwarded Get, Scan and JournalScan responses restore the caller's -control-table RPC `id` before returning across the original connection. -The forwarding connection uses its own counter; the business `request_id` -remains unchanged. Error responses with a leader hint preserve this same -distinction, including when the business ID is zero. - -**Connection model:** Each `PxKvStore` runs one `RpcServer` on the -crowdb-rpc port (derived from the base port via a fixed offset, see -§12). The shared `PxRpcTransport` holds one `RpcClient` and a lock-free RCU -connection-pool index. Each endpoint entry is an immutable generated pool; -connections are established outside the index and installed with -compare-and-swap. All `PxRemoteReplica` instances in the store share the same -transport. - ---- - -## 4. Server-Side Handler (PxRpcService) - -`PxRpcService` is the crowdb-rpc handler set for consensus RPCs. It -holds `Arc` + a tokio `Handle` (for spawning async -handlers). `register_handlers` wires one handler per `msg_type` into -the `RpcServer`: - -- `EPrepareRequest` → `handle_prepare` — dispatches to - `PxLocalReplica::on_prepare`, builds `FBPromiseResponse`. -- `EAcceptRequest` → `handle_accept` — dispatches to - `PxLocalReplica::on_accept`, builds `FBAcceptedResponse`. -- `EPreVoteRequest` → `handle_pre_vote` — dispatches to - `PxLocalReplica::on_pre_vote`, builds `FBPreVoteResponse`. -- `ERequestVoteRequest` → `handle_request_vote` — dispatches to - `PxLocalReplica::on_request_vote`, builds `FBRequestVoteResponse`. -- `EHeartbeatRequest` → `handle_heartbeat` — dispatches to - `PxLocalReplica::on_heartbeat`, builds `FBHeartbeatResponse`. -- `EStepDownRequest` → `handle_step_down` — dispatches to - `PxLocalReplica::on_step_down`, builds `FBStepDownResponse`. -- `EChosenNotification` → `handle_chosen_notice` — fire-and-forget, - updates the learner's chosen frontier, no response sent. -- `EBatchChosenNotification` → `handle_batch_chosen` — fire-and-forget, - batch frontier update, no response sent. -- `EFetchGapRequest` → `handle_fetch_gap` — looks up the accepted - value at the requested slot, builds `FBFetchGapResponse` (or - `NotFound` error if the slot is not yet chosen). - -Each handler reads the request flatbuffer directly via -`flatbuffers::root::Request>(req.control)`, dispatches to the -existing consensus logic on `PxLocalReplica`, and builds the response -flatbuffer via `FlatBufferBuilder`. The response is submitted via -`server.submit_response(conn_handle, &ctrl, None, msg_type, req_id)`. - -**Malformed-frame rejection:** when `flatbuffers::root` fails (wrong -table type, corrupted bytes) or a required field is missing (e.g. -`FBAcceptRequest.value` is `None`), the handler calls `submit_error` -with `FBKvRetCode::InvalidArgument` before any consensus logic runs. -`submit_error` builds a `FBPromiseResponse` buffer (shared by all error -paths) and submits it with the response `msg_type`. The client's -`send_*` method parses the response as the expected `FBResponseRef`; -since `FBPromiseResponse` has a different vtable layout, `valid()` -returns `false` and the client maps it to -`PxReplicaError::Internal("response malformed")`. Test coverage: -`malformed_accept_request_is_rejected_by_rpc_boundary` in -`paxos_error_test.rs` exercises both rejection paths (wrong table type + -missing `value`) via the `test-util`-gated `send_raw_request` on -`PxRpcTransport`. - ---- - -## 5. Client-Side Transport (PxRpcTransport) - -`PxRpcTransport` is the shared client-side transport for outbound -consensus RPCs. It holds: - -- `server: Arc` — the local server handle (for connection - attachment). -- `rpc: RpcClient` — the client facade for `call()` / `send()`. -- `connections: ConnectionPoolIndex` — per-endpoint immutable connection - pools carrying a generation. `conn_for(endpoint)` lazily connects outside - the index and atomically installs one completed pool. - -**Request-response path (`send_prepare`, `send_accept`, etc.):** -Builds the flatbuffer request via `FlatBufferBuilder`, calls -`rpc.call(server, conn, req_id, control, None, msg_type)`, which -returns a `CallFuture`. The future resolves to a `Response` containing -the control buffer. The response is parsed via the zero-copy -`FBResponseRef` wrapper (§11). - -**Fire-and-forget path (`send_chosen`, `send_batch_chosen`):** Builds -the flatbuffer, calls `rpc.send(server, conn, req_id, control, None, -msg_type, noop_completion(), null_mut())`. The no-op completion -callback satisfies the C++ side's non-null `on_complete` requirement. -No reply is awaited; failures are returned for caller-side -observability but treated as best-effort. - -**Port derivation:** `conn_for(endpoint)` parses the server endpoint -(host:port) and connects to `port + RPC_PORT_OFFSET` (see §12). - -Retryable failures invalidate only the exact generation selected for the -failed request. A late failure from an older connection therefore cannot -discard a replacement pool. - ---- - -## 6. RPC Routing in PxRemoteReplica - -`PxRemoteReplica` routes all RPCs through crowdb-rpc. Each -`ReplicaClient` trait method (`send_prepare`, `send_accept`, -`send_pre_vote`, `send_request_vote`, `send_heartbeat`, -`send_step_down`) delegates to the transport with a -`tokio::time::timeout` wrapper and records metrics. - -`send_chosen_notice` and `send_batch_chosen_notice` use -the transport — fire-and-forget via `transport.send_chosen()` / -`transport.send_batch_chosen()`. - -`send_fetch_gap` delegates to -`transport.send_fetch_gap()` (request-response). The `group_fetchgap` loop -also uses `remote.rpc_transport()` to route FetchGap through the -transport. - ---- - -## 7. Flow Control and Parallelism - -The crowdb-rpc transport enables the parallel-slot pipelining described -in `design-crowdb-kv-slot.md` §5: - -- **Multiple in-flight `Accept` frames per peer.** Each `send_accept` - call creates a new `CallFuture` and returns immediately; the caller - does not block waiting for the peer's reply. This allows slot N+1's - `Accept` to be sent before slot N's `Accepted` response arrives. -- **Connection-level flow control.** The crowdb-rpc engine manages a - send queue per connection; when the queue is full, `send()` / - `call()` returns `SendQueueFull`, which the caller treats as - retryable. -- **Reconnect safety.** On transport failure, the connection is - dropped from the cache; the next `conn_for()` lazily reconnects. - The proposer treats this as retryable and re-sends after reconnect. - -### 7.1 Quorum Short-Circuit - -`run_prepare_phase` and `run_accept_phase` fan out to all peers but -do not wait for all replies. A `FuturesUnordered` (local + each remote, -tagged with `(voting, kind)`) is drained via `StreamExt::next`; the -phase returns as soon as `accepted >= quorum` AND the local reply has -been folded (the W6 invariant: `Chosen`/`Proceed` is never returned -before the local WAL persist / CAS reply is counted). The -still-pending futures are moved into a detached drain task that -continues folding for side effects only: a late `TermStale` triggers -`become_follower`; a late `EpochMismatch` adopts the responder epoch. -The drain captures `self_weak` (upgrades to `Arc`), so a -dropped group lets the upgrade fail and the task exit cleanly. It -honors `tenure_cancel` so a step-down aborts the drain. - -In a 3-node group (quorum = local + 1), the proposal latency is the -quorum-th-fastest reply, not `max(all peers)`. A slow but connected -follower no longer drags every write. Failure detection is preserved. -It just moves off the latency path. - -### 7.2 RPC Deadline - -Each `send_*` method wraps its `CallFuture` await with -`tokio::time::timeout(rpc_timeout)`. On expiry the caller surfaces a -typed retryable error. A connected-yet-unresponsive peer (GC pause, -half-open socket, overloaded server) is surfaced as a retryable -failure within `peer_rpc_timeout_ms` (default 2000 ms, -aligned with the 2 s election max) rather than blocking the fan-out -indefinitely. - ---- - -## 8. Paxos Error Model - -The error categories below determine whether the proposer retries the -same slot, runs classic repair, moves the client operation to a new -slot, redirects the client, or returns a retryable error. - -### 8.1 Error Categories - -- **`NotLeader`** — KV request reaches follower. Client retries with - leader hint. -- **`PrepareRejected`** — Phase 1 blocked by higher promised ballot. - Retry same slot with a higher ballot. -- **`AcceptRejected`** — Phase 2 blocked by higher promised ballot. - Run classic Phase 1 repair on the same slot. -- **`ForeignValueChosen`** — Phase 1 adopts another value or chosen - value differs from client value. Learn the chosen value, then retry - the client operation on a new slot. -- **`QuorumUnavailable`** — Not enough reachable voting peers. Retry - same slot until budget exhausted. -- **`TransportFailure`** — RPC timeout, connect failure, or - unavailable peer. Retry same slot; repeated failures become - `QuorumUnavailable`. -- **`Busy`** — Admission/retry budget exhausted. Client retries with - backoff. -- **`InternalInvariantViolation`** — Missing required value, invalid - state transition. Fail test fast; production maps to internal error. - -### 8.2 Retry Rules - -- **Prepare rejection:** retry Phase 1 for the same slot using a - ballot above the highest observed rejected ballot. Do not move to a - new slot. -- **Accept rejection:** run classic Phase 1 for the same slot. If - Phase 1 discovers a foreign accepted value, repair and learn it - before the client operation moves to a new slot. -- **Transport or quorum failure:** retry the same slot with the same - ballot until the retry budget is exhausted. Increasing the ballot is - only required after a higher-ballot rejection. -- **Foreign value:** adopt it for the current slot. If not the client - value, retry the client operation on a later slot only after the - foreign value is learned. - -Retry backoff (`retry_backoff`) is `base * 2^attempt` with ±50% jitter -(`base * multiplier / 1000` where `multiplier ∈ [500, 1500]`), using a -thread-local `XorShift64` PRNG seeded from monotonic nanos. This -decorrelates retry storms across replicas that collide on the same -slot. The admission permit is held during backoff (releasing it would -let a duplicate `(client_id, seq)` admit concurrently). - -### 8.3 RPC Mapping - -Consensus responses carry rejected metadata directly in the -flatbuffer response fields (`rejected`, `term_stale`, `epoch_mismatch` -booleans + the term/epoch fields). RPC transport failures are mapped -by the caller into `TransportFailure`. Malformed consensus RPCs map -to `InternalInvariantViolation` at the Paxos model level and to -`FBKvRetCode::InvalidArgument` at the crowdb-rpc boundary. - ---- - -## 9. Version Compatibility Rules - -1. **Sender rule:** every message sets `version = 1` (the initial wire version). -2. **Receiver rule:** decode must accept any `version <= max_supported`. Unknown fields are ignored (flatbuffer default behavior). -3. **Upgrade rule:** new fields are added with new field numbers; old fields are never removed or renumbered. -4. **Field-number freeze per message:** field numbers are frozen once - assigned. The frozen ranges are documented in the schema file - comments. Future versions may add new fields starting at the - next free field number. - ---- - -## 10. Flatbuffer Schema (kv_consensus.fbs) - -`lib/crowdb-protocol/src/fbs/kv_consensus.fbs` mirrors the consensus -messages from `pxos.fbs`, following the `diskdb.fbs` conventions -proven by R115: - -- `include "common_type.fbs";` for `FBInt128`. -- `namespace crowdb.kv_consensus.proto;` -- Every request/response table carries `id` (request_id) + - `rpc_create_nano` as its first two fields. -- `FBAcceptedValue` is a table (has a `payload: [ubyte]` vector, - which requires a vtable). -- `FBAcceptRequest` carries consensus value and fence fields only. Client - request identities remain leader-local and are not follower wire metadata. -- `NotLeaderHint` is NOT a separate message — it is fields on the - response tables (`not_leader_hint:string` + `term:uint64` + - `membership_epoch:uint64`). - -Message type IDs registered in `msg_type.fbs` (1000–1099 range): - -``` -EPrepareRequest = 1000, -EPromiseResponse = 1001, -EAcceptRequest = 1002, -EAcceptedResponse = 1003, -EPreVoteRequest = 1004, -EPreVoteResponse = 1005, -ERequestVoteRequest = 1006, -ERequestVoteResponse = 1007, -EHeartbeatRequest = 1008, -EHeartbeatResponse = 1009, -EStepDownRequest = 1010, -EStepDownResponse = 1011, -EChosenNotification = 1012, // fire-and-forget (no response) -EBatchChosenNotification = 1013, // fire-and-forget (no response) -EFetchGapRequest = 1014, -EFetchGapResponse = 1015, -ESnapshotBeginRequest = 1018, -ESnapshotBeginResponse = 1019, -ESnapshotReadRequest = 1020, -ESnapshotReadResponse = 1021, -ESnapshotFinishRequest = 1022, -ESnapshotFinishResponse = 1023, -ESnapshotAbortRequest = 1024, -ESnapshotAbortResponse = 1025, -``` - -Each consensus operation has its own message type. Unary operations use -`RpcClient::call`; chosen notifications use fire-and-forget -`RpcClient::send`. The shared peer connection carries the mixed message types, -and the server dispatches each frame independently by its `msg_type`. - -### 10.1 New-member snapshot lifecycle - -Snapshot bootstrap is a receiver-pulled sequence of bounded unary calls. Begin -pins one immutable engine view and returns a process boot nonce plus monotonic -session number, source membership epoch, engine format, slot/term, total -length, body CRC32C, and a chunk limit no larger than 1 MiB. Read echoes the -identity and requested byte offset; it carries one payload and its CRC32C in -the data buffer. Only the current offset or an exact retry of the immediately -previous offset is accepted. Finish requires the final offset; Abort is -idempotent. - -The source admits at most four sessions by default and expires idle sessions -after 30 seconds. Every Read and Finish rechecks the captured membership epoch. -The receiver keeps one Read in flight, advances only after feeding a verified -payload, and may reconnect without changing identity. An expired identity -causes a fresh Begin and importer. Import is restricted to a fresh unpublished -group; learner seeding and group registration occur only after length, CRC, -source Finish, and engine Finish all succeed. The removed single-frame message -IDs 1016 and 1017 are deliberately unsupported. - -**Build integration:** `lib/crowdb-protocol/build.rs` compiles -`kv_consensus.fbs` via `flatc --rust --gen-all` (inlines -`common_type.fbs` so `FBInt128` resolves). The generated module is -re-exported as `crowdb_protocol::kv_consensus_fb`. - ---- - -## 11. Zero-Copy Wrapper Classes - -`lib/crowdb-protocol/src/fb_wrappers/kv_consensus.rs` defines one -`FBRef` struct per response type: - -```rust -pub struct FBPromiseResponseRef<'a> { - root: flatbuffers::Result<'a, FBPromiseResponse<'a>>, -} -impl<'a> FBPromiseResponseRef<'a> { - pub fn new(buf: &'a [u8]) -> Self { Self { root: flatbuffers::root::(buf) } } - pub fn valid(&self) -> bool { self.root.is_ok() } - pub fn request_id(&self) -> Option { self.root.as_ref().ok()?.id() } - pub fn slot(&self) -> Option { self.root.as_ref().ok()?.slot() } - // ... remaining fields -} -``` - -Same pattern for: `FBAcceptedResponseRef`, `FBHeartbeatResponseRef`, -`FBPreVoteResponseRef`, `FBRequestVoteResponseRef`, -`FBStepDownResponseRef`, `FBFetchGapResponseRef`. - -Request wrappers are NOT needed — the server handler reads the request -flatbuffer directly via `flatbuffers::root::Request>(req.control)`. -The client builds requests with `FlatBufferBuilder` and doesn't need -to read them back. - -Edge cases: -- Malformed flatbuffer → `valid()` returns false; the caller treats it - as a transport error (maps to `PxReplicaError::Internal`). -- Missing optional field → accessor returns `None` / default; the - caller handles the missing-field case. - ---- - -## 12. Port Allocation - -The crowdb-rpc port is derived from the base port via a fixed offset: - -``` -KV_SERVER_RPC_BASE = 28001 (rpc port base for KV server) -KV_RPC_BASE = 28101 (crowdb-rpc port base for KV server) -RPC_PORT_OFFSET = KV_RPC_BASE - KV_SERVER_RPC_BASE = 100 -``` - -When a `PxKvStore` binds on port P, the crowdb-rpc server listens -on port `P + 100`. The client transport's `conn_for(endpoint)` parses -the server endpoint and connects to `rpc_port + 100`. - -For ephemeral ports (port 0 in tests), the server binds first, -then `start_rpc_server` reads the actual bound port from -`server_state.listen_addr` and derives the crowdb-rpc port. - -Port constants are in `lib/crowdb-protocol/src/ports.rs` and the -`KvServerPort` enum. - ---- - -## 13. Flatbuffer `bytes` Field Mapping — `bytes::Bytes` - -By default, `flatc` maps flatbuffer `[ubyte]` fields to `Vec`, -making every clone an O(n) heap allocate + memcpy. For hot-path KV -fields that are cloned across retry loops or fanout, this is -avoidable. - -`lib/crowdb-kv/build.rs` configures selected -flatbuffer `[ubyte]` fields to map to `bytes::Bytes` instead, turning -clones into O(1) atomic ref-count bumps: - -- `AcceptedValue.payload` — Paxos Accept-fanout payload cloned across - N peers. -- `KvSetRequest.key`, `KvSetRequest.value` — single-key write path. -- `KvGetRequest.key` — single-key read path. -- `KvDeleteRequest.key` — single-key delete path. -- `KvBatchItem.key`, `KvBatchItem.value` — batch write path; the - client retry loop clones the entire `items` vec per attempt. -- `KvResponse.value` — read response. -- `KvScanRequest.prefix`, `KvScanRequest.start_after` — scan request. -- `KvScanItem.key`, `KvScanItem.value` — scan response items. - -The client `BatchOp` type (`lib/crowdb-kv-client/src/client.rs`) also uses -`Bytes` for `key` and `value`, so `BatchOp` → `KvBatchItem` conversion -and the per-retry `items.clone()` are both O(1) per item. - -The server-side `encode_kv_batch_items` (`px_kv_store.rs`) still -flattens `KvBatchItem` fields into a `Vec` consensus payload. -That copy remains (eliminating it requires changing the consensus -payload contract, out of scope). - ---- - -## References - -- `design-crowdb-kv-leader-election.md` §6 — heartbeat/lease interaction with - stream ordering -- `design-crowdb-kv-slot.md` §5 — pipelined fanout and per-peer flow control -- `design-crowdb-kv-observability.md` — write-path phase metrics - (`paxos.propose.e2e.l`, `paxos.classic.prepare.l`, - `paxos.accept.quorum_rpc.l`, `paxos.learn.apply.l`) -- `design-crowdb-rpc.md` §6 — Flatbuffer Wrapper Convention -- `design-crowdb-rpc-diskdb-migration.md` — R115's proven migration pattern diff --git a/doc/design/kv/design-crowdb-kv-server.md b/doc/design/kv/design-crowdb-kv-server.md deleted file mode 100644 index 8c50b9504..000000000 --- a/doc/design/kv/design-crowdb-kv-server.md +++ /dev/null @@ -1,230 +0,0 @@ - - - -# CROWDB - Design: KV Server - -Depends on: [`design-crowdb-kv.md`](design-crowdb-kv.md) §15.2 -Satisfies: [`design-crowdb-kv.md`](design-crowdb-kv.md) §15.2 - ---- - -## Table of Contents - -- [1. Overview](#1-overview) -- [2. Design Decisions](#2-design-decisions) - - [2.1 KV engine selection](#21-kv-engine-selection) - - [2.2 Startup ordering](#22-startup-ordering) - - [2.3 Concurrency model](#23-concurrency-model) - - [2.4 HTTP framework: axum](#24-http-framework-axum) - - [2.5 Group lifecycle](#25-group-lifecycle) - - [2.6 Shutdown](#26-shutdown) -- [3. Port Pool](#3-port-pool) -- [4. Group/store cleanup on remove](#4-groupstore-cleanup-on-remove) -- [5. Wipe user data](#5-wipe-user-data) - ---- - -## 1. Overview - -`crowdb-kv-server` is a thin binary that wires the `crowdb-kv` library into a -runnable process. It adds CLI argument parsing, an HTTP management -server for runtime topology control, and graceful startup/shutdown -orchestration. The server does **not** contain business logic — all -consensus, KV operations, and replication are handled by the `crowdb-kv` -library. - -## 2. Design Decisions - -### 2.1 KV engine selection - -`--kv-engine` chooses the `KVEngine` implementation for all groups: - -- **`crowdb-tree` (default)** — durable, file-backed. Each group gets its - own file under `--data-root`, recovered by replaying the WAL on - restart. -- **`memory`** — in-memory, non-durable. Explicit low-durability choice - for tests and dev. - -`--kv-backend` defaults to `block`: the tree uses an aligned block page -store with direct I/O on Linux. `file` explicitly selects buffered file -storage, and `mem-block` explicitly selects an in-memory block device. - -`--wal-backend` defaults to `block-device`: aligned real-file block storage -with direct I/O on Linux and durable synchronization before acknowledgement. -`file`, `uring` and `mem-block` remain explicit alternatives. File-backed -device simulation still traverses the filesystem and can incur variable VFS -and filesystem-log synchronization latency. Backend selection does not disable -durable synchronization. - -Backend selection must match an existing data directory. Defaults select the -backend for a new deployment; they do not convert stored tree formats. Existing -file-backend deployments continue to specify `file` explicitly on restart. - -### 2.2 Startup ordering - -The management API starts **before** stores so the server is observable -even if store creation fails. `--root ` is required on every -start; it derives the four node paths via `CrowDBConfig::apply_root`: -`wal_root = /waldata`, `config_root = /conf`, -`data_root = /ctdata`, `log_dir = /log` (fixed subfolder -names, the only supported layout). `--config ` is optional and -supplies startup tunables in both first-boot and restore modes; when omitted, -tunables come from `CrowDBConfig::default()`. Explicit CLI settings override -the file, as defined by the shared -[`service configuration design`](../config/design-crowdb-config.md) §3. The -config-file watcher runs only when `--config` is passed. Static settings such -as `server.rpc_workers` take effect after restart. - -Boot has two modes, selected by whether group 0 is on disk -(`restore::group0_exists` checks `/store0/group0`): - -- **Restore mode** — group 0 is on disk. `restore::scan_local_groups` - enumerates every `store{S}/group{G}` directory under ``, - and `restore::load_local_groups` creates each `PxKvStore` and loads - every group via `create_group_with_wal` (replays the WAL, opens the - crowdb-tree engine, and applies persisted membership from - `conf/node-config.json` — including remote-replica endpoints). The - local `replica_id` per group is read from `node-config.json`, falling - back to the `--replica` CLI arg. `--stores`/`--groups` are ignored - (warned); local disk is the source of truth for which stores/groups - this node hosts. After load, `reconcile::reconcile_with_group0` - compares local state against group 0's `/kv/replica/` records: - groups that came up with no remotes (`node-config.json` missing or - stale for them) get their remotes seeded from group 0 via a group - rebuild; groups that already have remotes are verified and any - peer present in group 0 but not wired locally is logged (the live - membership is not forcibly overwritten — it may be legitimately - ahead of group 0 during an in-flight reconfiguration). Reconciliation - is best-effort: if group 0 is unreachable or has no `/kv/replica/` - records, the node continues with local state and retries on the next - restart. -- **First-boot mode** — no group 0 on disk. The server boots empty - (or from `--stores`/`--groups` if given) and the operator calls - `POST /system/init` to create store 0 / group 0. - -A scan IO error is treated as empty (first-boot mode); a failed -`create_group_with_wal` for one group is logged and skipped while the -store still starts with its other groups. - -Creating a group persists its local replica identity and initial membership -before opening the WAL directory. A crash during later WAL/tree initialization -therefore restores the requested identity even when creation never returned or -the deployment manifest still marks that step incomplete. A conflicting -persisted local identity fails creation; it is never overwritten by a retry. - -### 2.3 Concurrency model - -`KvStoreRegistry` and each `PxKvStore` publish immutable store/group maps -through `ArcSwap`. Readers clone an `Arc` from one snapshot without taking a -shard lock. Replacement uses compare-and-swap, so a reader sees the old or new -group and never a missing intermediate entry; successful replacement cancels -the retired group's tenure exactly once. `PxGroup` rebuilds and republishes a -group when remote membership changes. - -### 2.4 HTTP framework: axum - -Chosen for lightweight async + tower ecosystem fit with tokio. The full -endpoint list is defined in the Axum route handlers and the OpenAPI spec -at `/openapi.json`; this doc covers design decisions only. - -Key endpoint groups: - -- **Store/group CRUD** — `/stores`, `/stores/:sid/groups` for runtime - topology control. -- **Remote replica wiring** — `/stores/:sid/groups/:gid/remotes` for - adding/removing peer replicas. `batch` endpoint accepts another - server's topology export for bulk wiring. -- **Topology export** — `GET /topology` produces a JSON document that - another server can consume to batch-add remotes. -- **System group** — `POST /system/init` bootstraps store 0 + group 0 - on this node. (`POST /topology/finalize` and `GET /topology/ready` - are removed. Persistent topology-record management moves to - `crowdb-kv-client`'s `KVClusterMetaClient` / `HardwareClient`. See - [`design-crowdb-kv-group0.md`](design-crowdb-kv-group0.md) for the - group-0 sysdata architecture.) The lifecycle endpoints (`add_store`, - `add_group`) are now **internal**; only `crowdb-kv-client`'s - `KVClusterAdmin` calls them. See `../console/design-crowdb-console.md` §4.3 for the - full persistent cluster config design. -- **Admin operations** — `step-down` (force leader step-down), `join` - (new-member snapshot join), `flush` (drain the local replica's L0 - memtable into L1 in memory; used by the bench's - `--flush-after-prepopulate` flag and as an admin drain). - -`POST /system/group0-discovery` accepts one to sixteen HTTP management -origins for a process launched before Group 0 exists. The server atomically -persists these connection hints and its keepalive loop discovers Group 0 -before registering. Explicit startup seeds take precedence. Repeating the -same request is safe; malformed origins and embedded credentials are rejected. -These hints survive process restart and never create stores, groups, replicas, -or membership. Without hints or a local Group 0 replica, registration waits. - -The service instance identity is persisted with the node's fixed-layout -configuration before opening listeners. Restarts reuse it, including when a -previous process could not unregister. A changed explicit instance or node -identity, or a malformed identity file, fails startup. Different durable roots -receive different generated instance identities; duplicate live node identities -remain ambiguous rather than being merged by discovery. - -### 2.5 Group lifecycle - -Local replicas start as `Follower`; no role assignment needed at group -creation. Leader assignment happens via topology wiring or leader -election. Groups can be added/removed at runtime via the management -API, not just at CLI bootstrap. - -### 2.6 Shutdown - -On SIGINT/SIGTERM, Axum stops accepting new HTTP requests, then -`graceful_shutdown` cascades through each store: - -1. **`PxKvStore::shutdown`** — stops the crowdb-rpc server (cuts frontend - load), then cascades into each group. -2. **`PxGroup::shutdown`** — cancels the tenure token and awaits the - election driver + maintenance loop, closes remote crowdb-rpc channels, - then cascades into the local replica. -3. **`PxLocalReplica::shutdown`** — calls `KVEngine::flush` (drain L0 - memtable into L1 B+tree, in-memory) then - `KVEngine::persist_snapshot` (write dirty L1 pages + superblock to - the page store, durable). For `InMemKV` both are no-ops. Acceptor - and learner resource release is deferred to `Drop`. - -This ensures in-memory engine state reaches the block file before the -process exits, so `resume_from_slot` is non-zero on restart and WAL -replay can skip the durable prefix. - -## 3. Port Pool - -The `--ports` argument supports comma-separated single ports and -inclusive ranges (e.g. `28,40..50,59`). Duplicates are silently -deduplicated. If `--stores` exceeds the port pool size, the server -exits with an error. When `--ports` is omitted, ports are OS-assigned -(zeros). - -## 4. Group/store cleanup on remove - -The `remove_group` and `remove_store` mgmt handlers delete the WAL + -engine dirs and update `node-config.json` in addition to the -in-memory `PxKvStore::remove_*`. The `node-config.json` update is the -fence against resurrection on restart — without it, a removed group -reappears from persisted config and collides with a new group at the -same `(store_id, group_id)`. - -- **I1 — ID reuse safety**: After a group/store is removed, its ID - can be safely reused. The `node-config.json` update is the fence. - -## 5. Wipe user data - -`POST /stores/{sid}/groups/{gid}/wipe-user-data` drops and recreates -WAL + engine user data for a group, preserving group0 sysdata + -topology. Explicit test-only measurement fixtures use this endpoint between -write-regression sub-tests while retaining their owned topology. Attached -product benchmarks do not reset groups or manage servers. The handler -steps down if leader, deletes the WAL + engine dirs, recreates the -group via `create_group_with_wal` (replays empty WAL → fresh state), -and does `remove_group` + `add_group` (remove-before-add is critical: -`add_group` calls `inherit_local_state_from(prior)` which would -`Arc::clone` the prior's engine + WAL, undoing the wipe). - -- **I2 — Wipe preserves group0**: `wipe-user-data` drops WAL + engine - user data only; group0 sysdata + `node-config.json` + topology are - untouched. diff --git a/doc/design/kv/design-crowdb-kv-slot.md b/doc/design/kv/design-crowdb-kv-slot.md deleted file mode 100644 index 29b010a89..000000000 --- a/doc/design/kv/design-crowdb-kv-slot.md +++ /dev/null @@ -1,659 +0,0 @@ - - - -# CROWDB - Design: Slots — Parallel Pipelining & Acceptor Retention - -Depends on: [`design-crowdb-kv.md`](design-crowdb-kv.md) §5.1, §6.5, §7.3 -Satisfies: [`design-crowdb-kv.md`](design-crowdb-kv.md) §5.1, §6.5, §7.3 - -This document covers two aspects of CROWDB's slot mechanism: - -- **Part A (§1–§14): Parallel Slot Pipelining** — the consensus protocol that distinguishes CROWDB from Raft-based KV systems. How the leader pipelines proposals, how gaps are detected and repaired, how the safe-slot is maintained, and how the system stays correct under concurrent in-flight slots. -- **Part B (§15–§22): Acceptor Retention** — guarded per-slot state, concurrent trim/reclamation and snapshot-backed retention certificates. - -## Table of Contents - -- [1. Why Parallel Slots](#1-why-parallel-slots) -- [2. Concepts and Invariants](#2-concepts-and-invariants) -- [3. Slot Lifecycle on the Leader](#3-slot-lifecycle-on-the-leader) -- [4. Sliding Window and Backpressure](#4-sliding-window-and-backpressure) -- [5. Pipelined Fanout](#5-pipelined-fanout) -- [6. Per-Key Resolved-Slot](#6-per-key-resolved-slot) -- [7. Safe-Slot Computation and Propagation](#7-safe-slot-computation-and-propagation) -- [8. Gap Detection](#8-gap-detection) -- [9. Gap Repair via Classic Paxos](#9-gap-repair-via-classic-paxos) -- [9A. Follower-Side Apply and Catch-up](#9a-follower-side-apply-and-catch-up) -- [10. Timing Diagrams](#10-timing-diagrams) -- [11. Interaction with Snapshot and WAL GC](#11-interaction-with-snapshot-and-wal-gc) -- [12. Tunables and Defaults](#12-tunables-and-defaults) -- [13. Correctness Analysis for Parallel Slot Writes](#13-correctness-analysis-for-parallel-slot-writes) -- [14. Parallel-Slot Linearizability Analysis](#14-parallel-slot-linearizability-analysis) -- [15. Acceptor Storage Requirements](#15-acceptor-storage-requirements) -- [16. Retained Slot Entries](#16-retained-slot-entries) -- [17. Per-Slot State](#17-per-slot-state) -- [18. Lookup and Reclamation](#18-lookup-and-reclamation) -- [19. Acceptor API](#19-acceptor-api) -- [20. Retention Invariants](#20-retention-invariants) -- [21. Storage Cost](#21-storage-cost) -- [22. Retention Limits](#22-retention-limits) -- [23. Server-side Proposal Coalescing](#23-server-side-proposal-coalescing) - ---- - -# Part A: Parallel Slot Pipelining - -## 1. Why Parallel Slots - -A Raft leader cannot acknowledge slot N+1 until slot N has been committed; the log is contiguous, so a single slow follower stalls the entire commit pipeline (head-of-line blocking). Multi-Paxos has no such constraint: each slot is a separate Paxos instance, and a quorum that decides slot N+1 need not include the same nodes that decide slot N. - -CROWDB exploits this by running many slots in parallel on the leader. Throughput is bounded by network and disk bandwidth, not by per-slot serialized round-trips. The price is two-fold: - -- **Gaps.** A slot may remain undecided long after later slots are decided. We need a mechanism to resolve gaps without stalling the hot path. -- **Conservative cross-key reads.** A `Scan` must wait for a no-gap prefix; point reads do not. - -The blind-ops premise from design-crowdb-kv.md §5.2](design-crowdb-kv.md) makes the trade-off cheap: out-of-order *apply* is safe because no operation reads before writing. - ---- - -## 2. Concepts and Invariants - -- **I1 — Single slot counter.** On a leader, slot assignment is performed by exactly one logical worker. Two writes never receive the same slot, and no two slots are assigned out of arrival order. -- **I2 — Slot determines linearization.** The slot number assigned to an op is the op's position in the global linearization order (design-crowdb-kv.md §6.1](design-crowdb-kv.md)). -- **I3 — Quorum-fsync before ack.** A client write is acked only after a quorum of acceptors (including the leader) have fsynced their `Accepted(slot, ballot, value)` records. This is the durability hook that makes I2 robust to failures. -- **I4 — Apply-order independence for blind ops.** For any key *k*, the engine's final value is `value(max{ slot | slot writes k })`. Apply order between non-overlapping keys is irrelevant; for the same key, the higher slot wins regardless of arrival order ([§13](#13-correctness-analysis-for-parallel-slot-writes)). -- **I5 — Per-key resolved-slot is monotone.** A learner's per-key tracker only ever advances. This is the basis for read-your-writes from followers. -- **I6 — Safe-slot is contiguous.** The cluster-wide safe-slot is the maximum N such that *every* slot ≤ N is chosen and applied on every learner. It is by definition gap-free. - ---- - -## 3. Slot Lifecycle on the Leader - -| State | Trigger to enter | Trigger to leave | Notes | -| --- | --- | --- | --- | -| `Assigned` | client request admitted; counter incremented | leader's WAL fsync of the `PxLogEntry` completes | Visible to no one yet | -| `Proposing` | local fsync done; `Accept` fanned out to followers | quorum of `Accepted` responses received | At this point the value is *chosen* | -| `Chosen` | quorum reached | learner has applied to engine | Ack to client may be sent here (see below) | -| `Applied` | learner.apply() returned | terminal | Used by metrics; safe to evict | -| `OrphanRepair` | leader changed before reaching `Chosen`; repair task takes over | `Chosen` (via repair) | New leader's responsibility | - -**When does the ack happen?** When the slot reaches `Chosen` *and* the leader's own learner has applied it. The leader's own learner application is required so that an immediately-following `Get` on the leader sees the new value (see [§6.1 of design-crowdb-kv.md](design-crowdb-kv.md#61-linearizable-leader-read)). - -**Eviction.** Acceptor entries and WAL segments become eligible below the snapshot-backed reclamation watermark in §11. Local application alone does not authorize eviction. - ---- - -## 4. Sliding Window and Backpressure - -The number of in-flight slots is capped at the **window size** (`max_inflight_proposals`, default 32). The proposer uses an `InflightAdmission` gate backed by one or more `Semaphore`s. If a permit is available, the request is admitted, slot is assigned, and proposing begins. The permit is held for the entire proposal duration. - -**Admission policy** (`reject_on_inflight_window_full: bool`, internal config, not exposed via CLI): - -- **`false` (default)** — If no permit is available, the caller blocks on `acquire().await` until a permit is freed. This eliminates client-side reject-retry storms under contention (e.g., window=1 with 16 writers). Wait time and queue depth are tracked as metrics (`inflight_total_enqueued`, `inflight_total_wait_us`, `inflight_queue_depth`). -- **`true`** — If no permit is available (`try_acquire` fails), the leader immediately returns `Busy` — a retryable error. No queuing. Used only in tests that need to verify fail-fast behavior. - -**Multi-queue routing** (`--inflight-queues`, default 1): The window is split across N semaphores, each sized `ceil(max_inflight / N)`. Proposals are routed round-robin. Multiple queues reduce semaphore contention under high concurrency without affecting correctness (each slot is an independent Paxos instance; admission ordering does not influence consensus safety). - ---- - -## 5. Pipelined Fanout - -As soon as the leader has fsynced its own copy of slot N, it sends `Accept(N, ...)` to all followers. It does **not** wait for slot N-1, N-2 to reach quorum first. - -**Transport: shared per-peer `crowdb-rpc` connection.** Each Accept is a unary -request correlated by `RpcClient`, so concurrent slots may remain in flight -without a dedicated application-level stream. - -**Per-follower flow control.** `crowdb-rpc` uses the configured bounded send -queue plus a bounded overflow queue drained on writable events. Terminal -queue pressure surfaces as `PxReplicaError::Backpressure`; the proposal retry -budget applies backoff without invalidating the healthy connection. - -**Quorum bookkeeping.** For each in-flight slot, the leader keeps a small bitmap of which peers have `Accepted` it. As soon as a majority is reached (counting itself), the slot transitions to `Chosen`. - -**Out-of-order `Accepted` and `Chosen` notifications are fine.** A follower may respond `Accepted(N+2)` before `Accepted(N)`. Each is processed independently. A follower may receive `Chosen(N+2)` before `Chosen(N)` and apply only the parts safe to apply (per-key tracking handles this). - -**Zero-copy accept path.** `Acceptor::accept` and `ReplicaHandler::on_accept` take `&PxLogEntry` instead of owned `PxLogEntry`. The caller (proposer's `run_accept_phase`, crowdb-rpc `handle_accept_inner`, WAL replay) passes a reference; no clone for the acceptor call. The only clone is inside `inner_accept` for `cas_accepted`, where the slot node must own its copy. `WALRecord::from_accepted` already borrows, so the WAL encode after accept is also zero-copy. With `Bytes` payloads these clones were O(1) ref-count bumps; the signature change makes the zero-copy intent explicit and avoids redundant bumps. - ---- - -## 6. Per-Key Resolved-Slot - -Each learner maintains, per key it has applied, the highest slot that has touched that key. This enables: - -- **Read-your-writes** ([§6.2 of design-crowdb-kv.md](design-crowdb-kv.md#62-read-your-writes-follower-read)): a follower can serve `Get(k, slot=N)` as soon as `resolved_slot[k] ≥ N`. -- **Linearizability** ([§14](#14-parallel-slot-linearizability-analysis)): per-key tracking makes "highest slot wins" correct under out-of-order apply. - -**Update rule.** On `apply(slot, batch)`: for each `(k, op, v?)` in the batch, if `slot > resolved_slot[k]`, update `resolved_slot[k] = slot` and write/tombstone `v`. Otherwise drop. Then advance `max_applied` and the contiguous-applied frontier. - ---- - -## 7. Safe-Slot Computation and Propagation - -The safe-slot is the cluster-wide **contiguous applied frontier**: the maximum N such that every slot ≤ N has been chosen and applied on every learner. - -**Aggregation.** The leader collects per-learner `contiguous_applied` reports piggy-backed on heartbeat responses. The safe-slot is recomputed at the end of each quorum heartbeat round: - -``` - safe_slot = min(contiguous_applied[learner]) for learner in voting members -``` - -A peer that has never reported is treated as `0`, so the safe-slot only rises once *all* voting members are heard from. The safe-slot is monotonic within a tenure (`fetch_max`); a new leader resets it to `0` and re-establishes it from fresh heartbeats. - -**Propagation.** The leader includes `safe_slot` in heartbeats, write responses, read responses, and the describe-cluster RPC. - -`Scan(Linearizable)` uses the leader's *own* contiguous frontier rather than the safe-slot. It is strictly ≥ safe-slot at all times ([§6.4 of design-crowdb-kv.md](design-crowdb-kv.md#64-scan-modes)). - ---- - -## 8. Gap Detection - -A "gap" is a slot N < max-chosen-slot for which the leader has no `Chosen` decision yet. The leader's learner maintains `contiguous_chosen` (highest gap-free chosen slot) and `last_chosen_slot` (highest slot ever seen chosen, gaps allowed). A gap exists when `contiguous_chosen < last_chosen_slot`. - -**Repair mechanism:** - -1. **Opportunistic repair.** After each heartbeat round in the leader-state loop, `repair_once()` is called. It finds the lowest gap (`contiguous_chosen + 1`) and runs classic Paxos (Phase 1 + Phase 2) to close it. A no-gap leader returns immediately without any RPCs. -2. **On leader change:** the new leader runs bulk Phase 1 over `[contiguous_chosen+1, ceiling]` as a one-shot sweep (see [`design-crowdb-kv-leader-election.md`](design-crowdb-kv-leader-election.md) §4). - -There is no separate repair task with its own tick or concurrency cap; repair is interleaved with heartbeats at the heartbeat cadence. - ---- - -## 9. Gap Repair via Classic Paxos - -Once a gap is selected, repair runs full classic-Paxos for that slot: - -1. **Pick a fresh ballot.** `(round, leader_id)` with `round = max_seen_round + 1`. -2. **Phase 1 — Prepare.** Send `Prepare(slot, ballot)` to all acceptors. -3. **Wait for quorum of `Promise`s.** Each carries the highest `(ballot', value')` previously accepted, if any. -4. **Choose a value.** If any `Promise` returned an accepted value, re-propose the one with the highest `ballot'`. If none, propose `NoOp`. -5. **Phase 2 — Accept.** Send `Accept(slot, ballot, value)`. Wait for quorum. -6. **Chosen.** Broadcast `Chosen(slot)` and apply locally. - -The repair never invents a fresh user value — if any acceptor had a half-baked accept, that value is re-chosen. Repair touches one slot at a time; the hot path on other slots is unaffected. A repair on slot N does not block new writes at slot M > N. - ---- - -## 9A. Follower-Side Apply and Catch-up - -The leader-side repair in §9 closes gaps in the leader's *chosen* frontier. Followers face a symmetric but distinct problem: they learn about chosen slots via `ChosenNotice` (a fire-and-forget notification the leader broadcasts after quorum), and they may miss or receive stale versions of those slots. This section covers the follower-side apply path and catch-up mechanism. - -### 9A.1 Accept Path Stores Only - -A follower's `handle_accept` stores the accepted value in its acceptor and persists it to the WAL, but does **not** advance `known_commit_slot` or wake the apply loop. Accept is a storage event, not a commit event. The leader may crash before reaching quorum, in which case the accepted value may never be chosen (or may be overwritten by a higher-ballot value during new-leader recovery). - -Advancing the apply frontier on Accept would let a follower apply an un-chosen value, violating Paxos safety, the same class of bug as applying before `leaderCommit` in Raft. - -The one exception is the new-leader transition: after a follower wins election and completes bulk Phase 1 over `[contiguous_chosen+1, ceiling]`, it advances `known_commit_slot` to its new `contiguous_chosen` and wakes the apply loop. This is safe because bulk Phase 1 has resolved (chosen or NoOp-filled) every slot in that range under the new leader's ballot. - -### 9A.2 ChosenNotice — Ballot-Verified Out-of-Order Apply - -When the leader chooses a slot, it broadcasts a `ChosenNotice` carrying `(slot, term, leader_id, ballot_round)` to all followers. The follower's ChosenNotice handler: - -- If the follower's acceptor has an accepted value at `slot` with `ballot == chosen_ballot` → the value matches what was chosen. The follower calls `update_chosen_frontier` and wakes the apply loop. Apply can proceed out-of-order; the per-key slot tracking in §3 makes this safe (highest slot wins per key). -- If the follower's accepted ballot at `slot` is **lower** than the chosen ballot (stale) → the follower has an outdated value. It records a gap for `slot` (driven by FetchGap, §9A.3) and does not apply. -- If the follower has no accepted value at `slot` → it records a gap and does not apply. - -The `ballot_round` field in the ChosenNotice flatbuffer enables this verification. Without it, the follower cannot distinguish a fresh chosen value from a stale re-delivery. - -### 9A.3 Follower-Driven FetchGap Catch-up - -When a follower detects a gap (missing or stale slot in the chosen range), it sends a unary `FetchGap(slot)` request to the leader. The leader: - -- Has completed leader recovery and retains the value within its contiguous - chosen prefix → replies with the full entry (payload + ballot + term). -- Otherwise → returns pending/not found. Normal leader recovery or independent - gap repair must establish the chosen value before a later request can fetch it. - -On receipt, a durable follower first writes the original Accepted entry to its -WAL, then installs it, advances the chosen frontier and wakes the apply loop. -Historical entries retain their original election term; a newer local term -does not reject committed history. Failed persistence leaves the gap pending. -Learning a proven chosen value does not answer a new Accept request and must -not discard a later promise. It retains the original entry alongside that -higher promise; an already higher accepted ballot must carry the same value. -Replay rebuilds highest accepted values before restoring promise fences, so -the physical order of historical fetch and later promise records cannot lose -either durable fact. Accepted history alone is never a FetchGap chosen proof. -`MAX_INFLIGHT_FETCHGAP` (default 16) bounds the entire fetch/persist/install -operation, including error and cancellation paths. - -This replaces the previous leader-driven catch-up that ran inline in `run_heartbeat_round` (up to 64 synchronous `send_accept().await` calls per lagging follower). That inline catch-up could exceed `heartbeat_interval` and trigger spurious elections. The heartbeat round is now pure liveness + lease: one RPC round-trip, no catch-up work. - -### 9A.4 Snapshot Fallback - -If a follower's gap count exceeds `catchup_snapshot_threshold` (default 1024), -it records the large backlog and continues FetchGap under the same fixed -16-request limit. A missing automatic snapshot path must not disable catch-up. - -The bounded snapshot protocol is used only while bootstrapping a fresh, -unpublished member. It does not replace the engine of a running follower and -is not invoked by this diagnostic threshold. A live replacement requires a separate -atomic engine-generation publication design. - -A recovered durable engine's gap-free resume watermark remains authoritative -when WAL reclamation removed the boundary record or all preceding segments. -Restore retains that complete chosen/applied prefix without capping it to the -remaining WAL tip. If the exact boundary term is unavailable, its historical -term is reported as zero (unknown); voting freshness still uses the durable -acceptor log tip. Higher retained entries replay after the snapshot prefix. - -### 9A.5 Apply Loop - -The follower's apply loop targets `max(known_commit_slot, last_chosen_slot)`. For each slot in the apply range: - -- If `slot ≤ known_commit_slot` → apply (committed by definition). -- Else if `learner.is_chosen(slot)` → apply (out-of-order chosen slot). -- Else (accepted but not chosen) → skip and record a gap for FetchGap. - -This ensures accepted-but-not-chosen slots are never applied, while allowing out-of-order chosen slots to apply immediately without waiting for lower slots to be resolved. - ---- - -## 10. Timing Diagrams - -### 10.1 Best case — fully pipelined window - -``` - time → - - slot N : assign─ fsync─ Accept ──► quorum ──► Chosen ─ apply ─ ack - slot N+1 : assign─ fsync─ Accept ──► quorum ──► Chosen ─ apply ─ ack - slot N+2 : assign─ fsync─ Accept ──► quorum ──► Chosen ─ apply ─ ack - - end-to-end latency for slot N = fsync + RTT + apply - per-slot incremental latency ≈ batched-fsync amortization - steady-state throughput = window × (1 / RTT) when network-bound - = disk_bw / record_size when disk-bound -``` - -### 10.2 Gap repair (slow follower) - -``` - time → - - slot N : assign─ fsync─ Accept ──► follower-A: Accepted - │ follower-B: ...........(slow) - slot N+1 : assign─ fsync─ Accept ──► quorum ──► Chosen ─ apply ─ ack(N+1) - slot N+2 : assign─ fsync─ Accept ──► quorum ──► Chosen ─ apply ─ ack(N+2) - - ... after heartbeat round ... - - repair : Prepare(N, ballot') ─► quorum of Promises - │ Promise from leader: Accepted(ballot, v) - Accept(N, ballot', v) ──► quorum ──► Chosen(N) - apply(N) on leader, advance contiguous_applied to N+2 -``` - -### 10.3 Leader change - -``` - time → - - old leader: ...slot N-1: Chosen, applied - slot N: Accepted on 1 follower only (no quorum) - slot N+1: Accepted on 0 followers - - - new leader (term T+1): - Prepare((T+1, me)) over [N, N+1, ..., max_chosen] - Promises: - slot N : (older_ballot, v_N) from one follower - slot N+1 : empty - re-Accept((T+1,me), v_N) at slot N → Chosen - Accept((T+1,me), NoOp) at slot N+1 → Chosen - steady state resumes -``` - ---- - -## 11. Interaction with Snapshot and WAL GC - -Acceptor and WAL reclamation share a certificate distinct from the commit -frontier and voting-member `group_safe_slot`: - -- The leader takes the minimum contiguous-applied frontier of itself and every - configured remote learner, including non-voting learners. Unknown progress - is zero; replies arriving after quorum still update progress. -- It caps that frontier by its local durable snapshot and the greatest durable - snapshot of a voting peer. A singleton voting group needs only its own durable - snapshot, while any attached non-voting learners still constrain progress. -- Heartbeats carry `gc_before_slot` and `gc_membership_epoch`. A follower accepts - the certificate only from its known voting leader in the current term and - membership epoch. The retained certificate binds these identities and the - watermark atomically; a later epoch or term cannot relabel an old certificate. -- Each replica caps its certificate by its own contiguous-applied and durable - snapshot frontiers before reclaiming. A missing WAL/snapshot yields zero. -- Maintenance trims acceptor entries with `slot < gc_slot` and unlinks only WAL - segments wholly below that boundary. It retains the boundary slot. Outstanding - readers retain node/payload owners until they finish. - -A new group or leader tenure clears progress evidence. Membership replacement -also clears evidence, including when it adds a non-voting learner. A joining -replica that needs a prefix already reclaimed uses snapshot catch-up. - -See [WAL garbage collection](design-crowdb-kv-wal.md#7-garbage-collection). - ---- - -## 12. Tunables and Defaults - -| Parameter | Default | Where it lives | -| -------------------------------- | ------- | --------------------------------------------------------------------- | -| `max_inflight_proposals` | 32 | `PaxosConfig` (total semaphore permits) | -| `reject_on_inflight_window_full` | `false` | `PaxosConfig` (internal, not CLI-exposed) | -| `coalesce_max_keys` | 32 | `PaxosConfig` (max ops per coalesced batch; 0 = off) | -| `coalesce_drain_threshold` | 1 | `PaxosConfig` (skip drain at this inflight count) | -| `max_paxos_retries` | 3 | `PaxosConfig` (per-slot Phase-2 retries) | -| `max_slot_retries` | 3 | `PaxosConfig` (new-slot retries before giving up) | -| `retry_base_backoff_ms` | 5 | `PaxosConfig` (exponential backoff base) | -| `bulk_prepare_window` | 1024 | `PxElectionConfig` (bulk Phase-1 batch size) | -| `catchup_snapshot_threshold` | 1024 | `PxElectionConfig` (diagnostic threshold; bounded FetchGap continues) | - -Bulk Phase 1 recovers independent slots concurrently, with at most -`bulk_prepare_window` complete Prepare/Accept operations in flight. It starts -at the new leader's own contiguous frontier, adopts quorum values and opens -reads only after the entire observed prefix has resolved and applied. Tenure -cancellation prevents prepared slots from proceeding to Accept. - -Steady-state gap repair starts only after initial recovery has completed; -the two paths never compete for the same slot. Each background pass drains -its observed backlog independently of heartbeat cadence. Follower-side gaps -use at most `MAX_INFLIGHT_FETCHGAP` (default 16) concurrent requests, including -backlogs above the snapshot diagnostic threshold. - ---- - -## 13. Correctness Analysis for Parallel Slot Writes - -**Key insight:** Parallel slot writes are safe because we only support **blind operations** (`Put`, `Delete`). No operation reads current state before writing. The final value of each key is determined solely by the highest slot that touched it. - -- **Consensus phase (parallel):** Each `PxSlot` is an independent Paxos instance. They can be decided in any order. -- **Apply phase (per-key slot tracking):** `PxLearner`s store `(slot, value)` per key and only accept writes where `slot > current_slot`. Regardless of apply order, the highest slot wins. -- **Gaps don't block point reads:** An undecided slot 3 does not block `Get(k)` if slot 5 for key `k` is already applied. Gaps only block cross-key scans and WAL truncation. -- **Batches:** Intra-batch order is **as written by the client**. A batch uses the batch's slot for each key. - -This would NOT be safe for `CAS` or `Increment`, which read current state. Not supported. - ---- - -## 14. Parallel-Slot Linearizability Analysis - -**Premises:** (1) Only blind ops. (2) Leader serializes slot assignment. (3) Quorum durable-flush before ack. (4) Per-key `(slot, value)` tracking, highest slot wins. (5) Leader reads fenced by lease or ReadIndex. - -**Claim:** The assigned slot number is a valid linearization point for every op. - -**Sketch:** - -- **Real-time order → slot order.** If `ack(A)` completes before `invoke(B)`, the counter has already advanced past `slot(A)`, so `slot(A) < slot(B)`. -- **Blind apply-order independence.** An undecided earlier slot writing *k* will be ordered earlier and immediately overwritten by the later slot's value. -- **Durability before visibility.** Quorum durable-flush before ack ensures a client-observed write cannot be lost by leader change. Gap repair re-chooses the same value. -- **Leader read correctness.** The leader's learner has applied slot N before acking. A subsequent `Get(k)` reflects the highest chosen slot writing *k*. -- **Follower read correctness.** Read-your-writes uses per-key resolved-slot; bounded-stale uses the gap-free `safe-slot`. - -**Single remaining cost — linearizable `Scan`.** Must wait for the leader's **contiguous applied frontier** to cover the target. Latency is bounded by (window size) × (slot-resolution time). Mitigation: `Scan(mode = SafeSlot)` or `Scan(mode = AtSlot(N))`. Point `Get`s bypass gaps entirely. - -**Why `CAS`/`Increment` would break this:** Read-modify-write ops must read current value, which is unknown until all earlier slots for that key are resolved. Would require sequential consensus or per-key dependency tracking. Neither is in scope. - -**Implementation invariants:** Slot assignment is single-point. `Accepted` not sent before durable flush. Classic-Paxos recovery never discards accepted values. Lease/ReadIndex fencing on every linearizable read. Per-key slot comparison atomic with write. - ---- - -# Part B: Acceptor Retention - -## 15. Acceptor Storage Requirements - -Per-slot promised ballots and accepted entries must support sparse, concurrent -Prepare/Accept operations while maintenance reclaims a proven-safe prefix. -Lookups must retain the node before accessing its fields. Reclamation must -support overlapping maintenance callers during group replacement. - -## 16. Retained Slot Entries - -`PxAcceptor` owns a `crossbeam_skiplist::SkipMap`, a -monotonic `trim_slot`, and a monotonic `highest_seen_slot`. Each operation keeps -the returned map `Entry` alive throughout field access. Removing an entry does -not invalidate an outstanding retained entry. Sparse slots allocate individual -nodes; absent slot ranges allocate nothing. - -## 17. Per-Slot State - -`PxSlotNode` contains atomic promised and accepted pointers. Replaced pointers -enter per-node retired lists and are freed only when the node is dropped. This -keeps a field read valid for the entire retained node lifetime. Returned log -entries clone their `Bytes` owner, so a caller may keep a payload after trim. - -## 18. Lookup and Reclamation - -- `open_slot` rejects a slot below the trim boundary, obtains or inserts a - retained node, then checks the boundary again. An insertion racing trim is - removed and rejected instead of resurrecting old history. -- Prepare/Accept keep the entry guard while reading or changing node fields. - Queries and ordered range scans clone values while retaining their entries. -- `trim` advances the boundary with `fetch_max`. `reclaim` removes entries below - that boundary; concurrent removers can safely encounter the same entry. -- A lookup already in progress may complete with its retained state. A newly - admitted request below the boundary is rejected. The boundary is exclusive. - -## 19. Acceptor API - -- `prepare`, `accept`, `accepted_at`, `promised_at`: per-slot protocol access. -- `accepted_iter_range`: ordered retained-history view; missing/trimmed slots - are absent. `JournalScan` reports a GC gap for a start below `trim_slot`. -- `accepted_log_tip`: highest retained accepted slot and its term. -- `highest_seen_slot`: highest slot ever opened, independent of reclamation. -- `trim`, `reclaim`, `trim_slot`, `retained_slots`: reclamation and observability. - -`ReplicaStatus.acceptor` exposes the exclusive trim boundary and retained slot -count. The count describes map entries, not payload bytes or process RSS. - -## 20. Retention Invariants - -- **R1 — Retained access:** every node field access holds a live map entry. -- **R2 — No resurrection:** a request admitted below the current trim boundary - cannot create visible accepted history. -- **R3 — Durable local recovery:** reclamation cannot exceed the local durable - snapshot or contiguous-applied frontier. -- **R4 — Learner coverage:** the leader certificate cannot exceed the least - applied frontier of any configured learner, including non-voting learners. -- **R5 — Certificate identity:** followers use only the certificate bound to - their current membership epoch, term and known voting leader. -- **R6 — Independent payload lifetime:** reclaiming a node does not invalidate - a log entry already cloned by a caller. - -## 21. Storage Cost - -The ordered map adds per-node allocation and lookup overhead. Reclamation walks -only the eligible prefix and releases historical payload owners as readers -finish. No throughput improvement follows from this representation alone; -measure request throughput, tails, retained slots and RSS under sustained load. - -## 22. Retention Limits - -A lagging configured learner or unavailable durable witness can hold retention -at zero. An operator must restore/catch up that learner or change membership; -maintenance cannot discard history merely to satisfy a memory target. A slot -with repeated ballot/value changes retains its old field versions until that -slot is reclaimed. Long-lived readers may delay physical release after logical -removal. The generic `PxSlotList` remains a separate utility and is not the -acceptor's reclamation mechanism. - -## 23. Server-side Proposal Coalescing - -### 23.1 Problem - -Each client `PUT`/`DELETE` is its own Paxos proposal: one slot, one -quorum RPC round, one WAL record, one learner apply. WAL batch -aggregation already amortizes `fsync` across concurrent proposals, and -the leader's local `fsync` is off the critical path, so the -remaining per-proposal fixed cost is the **quorum RPC round**. The -write-flow sweep shows throughput plateaus at ~29K (Intel) / ~48K -(M5 Pro) once the consensus pipeline saturates, independent of the -inflight window above MI=16. The bottleneck is the per-proposal quorum -RPC rate, not `fsync`. - -The `Batch` payload format already supports multiple ops per slot and -`kv_batch_write` exposes it. Without server-side coalescing, concurrent -single-key proposes each take a distinct slot and pay the full quorum -round. The coalescer (enabled by default, `coalesce_max_keys = 32`) -event-batches these into one multi-key Paxos proposal per batch. - -### 23.2 Drain Threshold - -The first op starts a 1-op slot-task **immediately** (no timer). Ops -arriving during the round join a pending batch. On round completion, -`coalesce_drain_after_round` checks the in-flight slot-task count -(`occupied`, permits held) before taking the pending batch. If -`occupied >= coalesce_drain_threshold`, the drain is skipped; the -`max_keys` overflow path handles high load with full batches. If -`occupied < coalesce_drain_threshold`, the pending batch is taken and -the next slot-task is started (if non-empty) or the coalescer goes -idle (if empty). - -At high load, drains almost never fire (many slot-tasks in flight); -the `max_keys` overflow path produces full batches. At low load, -drains always fire (few slot-tasks, threshold not exceeded), so there -is no latency floor; a lone op flushes immediately. Default -`coalesce_drain_threshold = 1` (see §23.5). - -### 23.3 Coalescer Design - -Per-group state on `PxGroup`: - -- `coalescer: parking_lot::Mutex>` — the - accumulating batch, `None` when idle. -- `self_weak: OnceLock>` — set once the group is wrapped - in `Arc`, so spawned slot-tasks can upgrade without holding a strong - self-reference. -- `coalesce_last_activity_us: AtomicU64` — last coalescer activity - timestamp (updated on every enqueue and drain). Used by the - activity-based watchdog. - -The coalescer mutex is retained because it protects one small batch mutation -and is released before slot proposal, RPC, or engine work. An MPSC owner task -would remove producer contention but add scheduling and queue latency to every -write; profiling must show this critical section is material before making that -trade. Peer applied/durable maps and follower gap tracking also retain their -short locks: replica sets are normally 3–7 members and gaps are exceptional, -so their bounded critical sections do not justify per-peer atomics or another -concurrent map. - -`PendingBatch` holds: -- `op_bodies: Vec` — concatenated op bodies (each single-op - payload's leading count bytes dropped; op bodies are self-delimited). -- `op_count: u16` — number of ops accumulated. -- `identities: Vec` — one `(client_id, seq)` request - identity per client operation, all mapping to the shared result slot. -- `waiters: Vec>` — one per coalesced - caller; each receives the shared `ProposeResult` on flush. -- `timer: Option>` — reserved (unused in event mode; - the watchdog is activity-based, not per-batch). - -`propose` flow (refactored into `propose` + `propose_inner`): - -1. Leadership gate (as before). -2. Request-result lookup — a hit returns the cached slot - immediately, never enters a batch. -3. If `coalesce_max_keys == 0` (coalescing disabled) or `self_weak` is - unset: call `propose_inner(payload, &[tag])` directly; bit-identical - to the fallback path. -4. Else (coalescing on, default `coalesce_max_keys = 32`): - `coalesce_enqueue(payload, tag)`: - - Coalescer is `None` (idle) → create a batch with this op, start a - 1-op slot-task **immediately**. Open a fresh empty batch for ops - arriving during this round. - - Coalescer is `Some` (batch exists) → append op body + tag + - waiter. If `op_count >= max_keys` → overflow: take the batch, - spawn a concurrent slot-task, open a new empty batch. -5. `await` the waiter's `ProposeResult`. - -`coalesce_flush_batch` (called by overflow and drain): - -1. Build merged payload: `[op_count as u16 LE][op_bodies]`. -2. Open a fresh empty batch in `coalescer` (for ops during the next - round). -3. `tokio::spawn` `propose_inner(payload, tags)`. On completion, fan - the `ProposeResult` (now `Clone`) to all waiters, then call - `coalesce_drain_after_round`. - -`coalesce_drain_after_round` (called after each slot-task finishes): - -1. Touch activity timestamp. -2. If `occupied >= coalesce_drain_threshold` → return (skip drain; - overflow handles it). -3. Take the pending batch from `coalescer`. -4. If `op_count == 0` → go idle (return). -5. If `op_count > 0` → `coalesce_flush_batch(batch)` (start next round). - -**Watchdog**: A single background task per group sleeps 1000ms, then -checks if there's been no coalescer activity for 1000ms. If so, it -flushes any stuck non-empty batch. Safety net for edge cases (drain -panic, spawn failure). Zero overhead during normal operation. - -### 23.4 Request Identity Threading - -A coalesced batch carries K client request identities but one result slot. -The identities stay with the leader's coalescer waiters and are published to -the leader-local request-result cache only after that value is chosen. They -are not part of `PxLogEntry`, the WAL, or `FBAcceptRequest` because an -Accepted value is not necessarily chosen and followers cannot safely publish -a result from Accept alone. - -`Learner::learn` accepts a slice of `RequestIdentity` values. Normal leader -proposals pass their identities; repair, election, follower notification, and -restore paths pass an empty slice. The cache is lost on leader change or -restart, at which point ordinary data-idempotent operations may be proposed -again. - -### 23.5 Config - -`PaxosConfig` gains: - -- `coalesce_max_keys: usize` — max ops per batch (cap 65535, the - payload count field is `u16`). Default `32`; `0` disables coalescing. -- `coalesce_drain_threshold: usize` — skip drain when in-flight - slot-task count >= this. Default `1` (the last finisher always - drains). `0` = always drain (pure event mode); higher values skip - the drain at high load so the `max_keys` overflow path produces full - batches. - -CLI: `--coalesce-max-keys` on `crowdb-kv-server`, applied in `main.rs` -into `config.paxos`. Wired into the group via `set_from_config` (the -coalescer reads `self.config.paxos.*`). - -### 23.6 Correctness - -- **Request replay**: every coalesced identity is recorded against the shared - slot on the leader after choice. Same-leader retries return that slot; - retries after leader change or restart may re-propose ordinary mutations. -- **Per-key ordering**: unchanged; all ops in a batch share one slot; - across batches, per-key highest-slot-wins applies as before. -- **`ProposeResult::Chosen { slot }` contract**: every coalesced - waiter receives the same slot. `ProposeResult` gains `Clone`. -- **`coalesce_max_keys = 0`**: `propose` calls `propose_inner` with a - one-identity slice and records the same leader-local result after choice. -- **Leadership**: re-checked inside `propose_inner`; a step-down - between batch collection and flush surfaces as `NotLeader` to all - waiters. -- **Shutdown**: slot-tasks hold a `Weak`; on shutdown the - pending batch is dropped (waiters get a closed oneshot → mapped - to `Err`). -- **Drain threshold liveness**: the inflight permit is released (on - permit drop in `propose_inner`) **before** `coalesce_drain_after_round` - is called. So the last finisher always sees `occupied = 0 < threshold` - and takes the batch. The batch is never stuck. The 1000ms watchdog - is a backstop for edge cases (drain panic, spawn failure). - -### 23.7 Benchmark Results (10s mem mode, 3-node cluster, max_keys=32) - -| Threads | Baseline TPS | Timer TPS | Event TPS | Drain TPS | Timer WAL | Drain WAL | -|---|---|---|---|---|---|---| -| 32 | 27,787 | 33,029 | 48,346 | 47,485 | 31,090 | 139,404 | -| 64 | 28,062 | 64,145 | 68,201 | 68,741 | 60,498 | 106,926 | -| 128 | 28,260 | 97,554 | 86,759 | 101,537 | 92,752 | 101,350 | -| 256 | 27,804 | 113,671 | 97,865 | 118,377 | 110,034 | 111,944 | - -Drain beats Timer at high load (128: 102K vs 98K, 256: 118K vs 114K) with -WAL counts close to Timer. At 64 threads it matches Event mode and beats -Timer. At 32 threads it matches Event mode (no regression). The default -threshold of 1 still allows drains at low load, preserving the zero- -latency-floor behavior. diff --git a/doc/design/kv/design-crowdb-kv-state-machine.md b/doc/design/kv/design-crowdb-kv-state-machine.md deleted file mode 100644 index 84e1d3c39..000000000 --- a/doc/design/kv/design-crowdb-kv-state-machine.md +++ /dev/null @@ -1,448 +0,0 @@ - - - -# CROWDB - Design: State Machine - -Depends on: [`design-crowdb-kv.md`](design-crowdb-kv.md) §5.2, §8.3, §8.4 -Satisfies: [`design-crowdb-kv.md`](design-crowdb-kv.md) §8.3, §8.4, §14.1 - -This document specifies the storage engine abstraction used by CROWDB learners. The engine is the **only** consumer of consensus output; it owns the materialized key-value state and serves all reads. The WAL is the durable log; the engine is the materialized projection. - -> **P3 redesign note.** The production engine `crowdb-tree` and the redefined -> (async, snapshot/GC-aware) `KVEngine` abstraction are specified in the crowdb-tree -> sub-design set: [`../tree/design-crowdb-tree.md`](../tree/design-crowdb-tree.md) (overview + engine -> abstraction + language/FFI decisions), [`../tree/design-crowdb-tree-engine.md`](../tree/design-crowdb-tree-engine.md) -> (in-memory engine + memory model + async FFI), [`../tree/design-crowdb-tree-storage.md`](../tree/design-crowdb-tree-storage.md) -> (durable storage + mapping table + snapshot/GC flow). This document remains the -> source of truth for the *semantics* (per-key slot, apply, compare, compaction); -> crowdb-tree docs own the *implementation*. - -## Table of Contents - -- [1. Goals and Non-Goals](#1-goals-and-non-goals) -- [2. Engine Responsibilities](#2-engine-responsibilities) -- [3. Per-Key Slot Tracking and Single Versioning](#3-per-key-slot-tracking-and-single-versioning) -- [4. Apply Semantics](#4-apply-semantics) -- [5. Read Surface](#5-read-surface) -- [6. Snapshot Export and Import](#6-snapshot-export-and-import) -- [7. Compaction Policy](#7-compaction-policy) -- [8. Compare for Cross-Learner Validation](#8-compare-for-cross-learner-validation) -- [9. Engine Implementations](#9-engine-implementations) -- [10. Tunables and Defaults](#10-tunables-and-defaults) - ---- - -## 1. Goals and Non-Goals - -**Goals:** - -- Define one engine surface so the same learner code drives any backend (in-memory, crowdb-tree). -- Encode the per-key resolved-slot semantics that make parallel-slot consensus correct. -- Provide a deterministic state-equality check (`compare`) for cross-learner test verification. -- Provide a streamable snapshot import/export usable by snapshot install. - -**Non-goals:** - -- Multi-version (MVCC) reads. CROWDB is single-version per key; time-travel is not supported. -- Cross-engine queries. The engine is per-group. -- Transactions or snapshot isolation across keys. Batch completion certifies all - key publications; ordinary reads may observe partial in-flight batches. -- Pluggable compression/encryption. Each engine handles its own internals. - ---- - -## 2. Engine Responsibilities - -The engine encapsulates everything below the consensus / learner layer: - -| Responsibility | Notes | -| -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Persist (or hold in memory) the current `(slot, value)` for every live key | Single version, tombstones for deletions | -| Apply a `(slot, batch)` to completion | Per-key publication; replay is idempotent | -| Serve point reads with their resolved-slot | Used by leader linearizable reads, follower RYW | -| Serve ordered range scans | Required for `Scan` API | -| Export and import snapshots | Streamable; resumable on the consumer side | -| Provide a deterministic state digest or comparison | For test-time validation | -| Track watermarks for compaction | `snapshot_slot`, `safe_slot` (provided by learner) | -| Persist snapshot with slot index | The engine persists its KV state and `last_applied_slot` to a snapshot file; on restart, it loads this snapshot to skip re-applying already-applied slots | - -The engine does **not** know about Paxos, terms, ballots, leaders, or the network. It receives `(slot, batch)` and applies; that is the entire write contract. - -### 2.1 State Machine Restart: Engine Durability + WAL Replay - -The state machine's persistence boundary is separate from the WAL — but it is -not a dedicated "state-machine snapshot file" either. It is whatever the -*engine itself* durably persists (crowdb-tree's own snapshot pipeline, -[`../tree/design-crowdb-tree-storage.md §6`](../tree/design-crowdb-tree-storage.md#6-snapshot-recovery-and-exportimport)), -queried through `KVEngine::resume_from_slot()`. This replaces the former -`DurableCommitWatermark` WAL record design. - -**Restart behavior** (`PxLocalReplica::restore_from_replay_with_engine`, -`lib/crowdb-kv/src/cluster/local_replica.rs`): - -1. WAL replay rebuilds acceptor state (`Promised`, `Accepted`, `VoteGranted`), Pass 1, unconditional. -2. The engine reports `resume_from = resume_from_slot()`: the highest slot it - already durably reflects (always `0` for `InMemKV`; crowdb-tree's - `last_applied_slot`, restored from its on-disk commit anchor, for a - durable engine that was cleanly snapshotted before the crash/restart). -3. Pass 2 `learn()`s every WAL-accepted slot in `(resume_from, highest_local]` - into the engine, in order, and fast-forwards the learner's frontier - (`contiguous_chosen`/`contiguous_applied`/`last_chosen_term`) directly to - `resume_from` via `seed_resume_frontier`, skipping the now-redundant - re-`learn()` of the already-durable prefix. -4. Slots above the local WAL's own highest accepted slot are re-learned via - new-leader recovery (bulk Phase 1) or steady-state heartbeat catch-up (§6 - of this doc's parent flows, `design-crowdb-kv-slot.md`). - -Step 3's skip is a pure **optimization**, not a correctness requirement: -`KVEngine::apply` is idempotent (highest-slot-wins per key) and -`update_frontier` tolerates out-of-order slots, so replaying the *entire* -local WAL through `learn()` (`resume_from = 0`, `InMemKV`'s permanent case) -always converges to the same correct state. It's just extra, safe, no-op -work. `persist_snapshot()` (called periodically by the group's maintenance -loop, `lib/crowdb-kv/src/cluster/group_maintenance.rs::run_pass`) is what advances -the durable floor `resume_from_slot()` will report on the *next* restart. - ---- - -## 3. Per-Key Slot Tracking and Single Versioning - -### 3.1 What we store - -For every live key `k`, the engine stores exactly one tuple: - -``` - (k → resolved_slot(k), value_or_tombstone(k)) -``` - -- `resolved_slot(k)` is the highest slot that has touched key `k` and been applied. -- `value_or_tombstone(k)` is the value from that slot, or a tombstone marker if that slot was a `Delete`. - -Tombstones occupy space until compacted away (§7). - -### 3.2 Why single-version - -CROWDB does not provide repeatable reads or time-travel queries. `AtSlot(N)` waits -for the engine's contiguous-applied frontier to reach `N`, then reads current data. - -`Scan(AtSlot(N))` has a freshness precondition, without a common snapshot time. -Each key returns its highest observed slot, which may exceed `N`; different keys -can reflect different times and partial later batches. This does not establish -one linearization point for the whole scan. Stable exports use the separate -covered-prefix snapshot interface. - -If true historical snapshots are ever required, MVCC is a future extension. The single-version restriction comes from design-crowdb-kv.md §1 / §5.2](design-crowdb-kv.md). - -### 3.3 Resolved-slot is monotone per key - -The engine never accepts a write at slot `s` for key `k` if `s ≤ resolved_slot(k)`. This is the runtime expression of [Invariant I5 in `design-crowdb-kv-slot.md`](design-crowdb-kv-slot.md#2-concepts-and-invariants). - -Implication: replays and out-of-order applies are naturally idempotent. If WAL replay tries to apply slot 7 for key `k` and `resolved_slot(k)` is already 9, the apply is a no-op for `k`, consistent with the parallel-slot semantics. - ---- - -## 4. Apply Semantics - -The engine's `apply(slot, batch)` is the only mutator. It must be atomic and idempotent. - -### 4.1 Inputs - -- `slot` — the slot of this batch. All operations in `batch` carry this same slot. -- `batch` — an ordered list of `(key, op, value?)` tuples, where `op ∈ { Put, Delete }`. - -### 4.2 Procedure - -For each `(k, op, v?)` in batch order: - -1. Read current `(resolved_slot(k), current)` if any. -2. If `slot ≤ resolved_slot(k)` → skip (idempotent no-op). -3. Else write `(slot, value)` for `Put`, or `(slot, tombstone)` for `Delete`. -4. Update `resolved_slot(k) = slot`. - -After all tuples are processed, advance the engine's `max_applied = max(max_applied, slot)` and update the contiguous-applied frontier if `slot == contiguous_applied + 1`, possibly cascading. - -### 4.3 Atomicity - -Batch completion is recorded only after every accepted key mutation has been published. Ordinary tree reads and scans may observe part of an in-flight batch; they do not promise a common snapshot version across keys. Equal-slot replay and intra-batch last-occurrence-wins remain required. Durable snapshot export is a separate, covered-prefix operation. - -In-memory engines can hold a write lock for the duration of the batch. File-based engines can use a transactional write group. crowdb-tree's btree operations on a single batch can ride on its own concurrency control. - -### 4.4 Intra-batch order - -For a key `k` appearing multiple times in a batch (rare but legal; see [`design-crowdb-kv-slot.md` §13](design-crowdb-kv-slot.md#13-correctness-analysis-for-parallel-slot-writes)), the *last* occurrence in batch order wins. The earlier ones are folded into the apply procedure naturally (each tuple in turn updates `current`; the loop's final state is what persists). - -### 4.5 Failure during apply - -`apply` either completes or returns an error. On error, the engine must leave its state unchanged (no partial apply). The learner treats engine apply errors as fatal (they indicate disk corruption or out-of-space) and fails the node out of the group. - -### 4.6 Zero-copy batch decode - -`Batch::decode` extracts keys and values from the Paxos payload using -`Bytes::slice(range)`, which creates zero-copy views into the same -allocation as `PxLogEntry.payload` (an O(1) ref-count bump). This -eliminates the per-key and per-value `to_vec()` heap allocations that -would otherwise occur on every `learn_chosen` call. - -`Op::Put(Bytes)` and `BatchOp.key: Bytes` use `Bytes` (ref-counted, -owned, `Send`) rather than `Vec`. The `KVEngine::apply` trait -signature is unchanged; it still takes `&Batch`. Engine -implementations use `Bytes::as_ref()` to obtain `&[u8]` for FFI or -internal storage. `Cell` and `EngineDiff` remain `Vec` since they -represent engine-internal storage, not the decode path. - -Truncated payloads (where a key or value length exceeds the remaining -bytes) yield empty `Bytes` for that field, matching the previous -`unwrap_or(&[])` behavior. - -### 4.7 Async engine apply and the apply fence - -`learn` is the learner's apply entry point: `apply_entry` (the FFI + -memtable insert) → advance the chosen frontier → advance the applied -frontier → cache the leader's request result. Two frontiers are tracked: - -- `contiguous_chosen` — highest slot `S` such that every slot in `[1, S]` - is chosen. -- `contiguous_applied` — highest slot `S` such that every slot in `[1, S]` - is applied to the engine. - -Under V1 (apply == learn, the default), both advance together -in `learn` right after the synchronous `apply_entry`, so -`contiguous_applied` tracks `contiguous_chosen` exactly. - -**`async_engine_apply`** defers the engine apply off the write critical -path.** The leader's propose path splits `learn`: the chosen-frontier -advance and request-result record run **synchronously** (cheap atomics, before -`propose` returns `Chosen`), and only `apply_entry` + the applied-frontier -advance are `tokio::spawn`'d. This keeps `contiguous_chosen` current: a -subsequent read's `read_slot = contiguous_chosen` reflects the -just-chosen slot, while `contiguous_applied` lags by the spawned apply. -Spawned applies can complete out of order, so each frontier stores gaps in a -sharded concurrent map. A compare-exchange drain owner advances the contiguous -cursor and removes consecutive gaps; producers that lose ownership only insert -their slot. The highest-chosen `(slot, term)` pair uses an atomic sequence guard -so election readers cannot observe a slot paired with another update's term. - -**Apply fence (Linearizable read-your-writes).** With `async_engine_apply` on, a -linearizable read that lands between "chosen" and "applied" would miss a -just-written value: the barrier captures `read_slot = contiguous_chosen` -(which is current), but the engine get returns the latest *applied* value. -The fence closes this gap. After the leadership barrier resolves -`read_slot`, the read awaits `contiguous_applied >= read_slot` before -serving the engine get. A `Notify` on the learner is woken -(`notify_waiters`) whenever `contiguous_applied` advances; the fence uses -register-before-load (`notified()` created before the `Acquire` load) so a -wake racing the load is not missed; the load observes the -`Release`-stored new frontier and returns without waiting. - -- **Fast path** (`async_engine_apply` off, or the slot already applied): one - `AtomicU64::load(Acquire)` + compare; no wait, no wake. With `async_engine_apply` off the - fence is a no-op; `contiguous_applied == contiguous_chosen` at the - instant the barrier resolves. -- **Slow path** (`async_engine_apply` on AND a read races a just-chosen-but-not-applied - write): the fence waits for the spawned apply. The wait is bounded by - apply throughput (memtable insert, µs), exactly the latency `async_engine_apply` removed - from the write path; the fence redistributes that µs to an occasional - read, it does not add new latency. -- **MinSlot** is untouched; it already gates on `contiguous_applied` via - the client-supplied `min_slot`. -- **ReadIndex batching** composes: the fence runs per-read after the - (possibly shared) barrier outcome; batched reads share the barrier's - `read_slot` floor but each awaits its own `contiguous_applied >= - read_slot` check, and `notify_waiters` wakes all parked reads together. - -`async_engine_apply` defaults to `true`; test profiles (`for_tests`) and -the `PxGroup::new` test path opt out (`false`) for deterministic -synchronous apply. The setter `set_async_engine_apply` remains for tests -that exercise `async_engine_apply`. The flag is carried across group rebuild via -`set_from_config(group.config())`. - ---- - -## 5. Read Surface - -The engine exposes three read primitives. All are non-mutating and may be served concurrently with writes. - -### 5.1 `get(k) → Option<(slot, value)>` - -Returns `None` if `k` is unset or tombstoned (callers may distinguish if they care, e.g. for range scans excluding tombstones). Returns `Some((resolved_slot(k), value))` if a live value exists. - -The slot is returned because callers (the learner) need it to assemble responses to `Get` with read-mode semantics. - -### 5.2 `scan(range, limit) → iterator<(key, slot, value)>` - -Returns an iterator of live entries (no tombstones) within `range`, in key order, up to `limit` items. The iterator may be backed by a btree cursor (in crowdb-tree) or a sorted-tree iterator (in-memory). - -The ordinary iterator returns the highest version it observes for each key, in key order, without duplicates. Different keys may reflect different times or part of a concurrent batch. `AtSlot(N)` remains a freshness precondition at the serving layer; it does not pin all returned keys to a common version. Flush must preserve continuous visibility of stable keys and the lifetime of borrowed tuples. Stable export uses the engine's separate snapshot interface. - -### 5.3 `multi_get(keys) → map` - -A batched point read. Returned as a map; missing keys are simply absent. - -### 5.4 What the engine does **not** expose - -- No range-write API. All writes go through `apply(slot, batch)`. -- No "delete-range" API. Range deletes are decomposed by the consensus layer into per-key deletes inside a batch. -- No "wait-for-slot" primitive. Wait conditions (RYW, SafeSlot, AtSlot) are implemented in the learner using the engine's `contiguous_applied` watermark, not inside the engine. The Linearizable RYW wait is the apply fence (§4.7): `PxLearner::await_applied` parks on a `Notify` until `contiguous_applied >= read_slot`. - -This minimal surface keeps multi-engine compatibility easy. - ---- - -## 6. Snapshot Export and Import - -### 6.1 Purpose - -Snapshot install ([§8.4 of design-crowdb-kv.md](design-crowdb-kv.md#84-snapshot-and-install)) is the bootstrap path for new or far-lagging members. The engine provides export/import primitives that the snapshot module wraps in a chunked, resumable, throttled transfer. - -In addition to peer transfer, the engine uses snapshot export to persist its own state locally (see §2.1). The local snapshot file stores the KV state and `last_applied_slot`, enabling fast restart without re-applying the entire WAL. - -### 6.2 Export - -`snapshot_export() → stream`: - -- Returns a stream of opaque chunks that, when imported by a peer engine, produce the same logical state as the exporter's current `last_applied_slot`. -- The export reflects the engine state *after* applying every slot ≤ `last_applied_slot` (always the latest durable state). -- Chunks have stable ordering and stable boundaries: the same state always produces the same byte sequence, modulo any internal pagination. This determinism is what makes resumption possible. -- Chunk size is engine-defined; 1–4 MiB is a reasonable default. - -The chunk format is **engine-specific**. The in-memory tree might serialize key/value pairs in btree order; crowdb-tree might dump native pages directly. The snapshot module treats chunks as opaque. - -### 6.3 Import - -`snapshot_import(stream) → ()`: - -- Consumes a stream of chunks produced by the same engine type. -- On success, the engine's state is exactly what `snapshot_export()` produced, with `contiguous_applied` and `max_applied` set to the exported slot. -- Resumable: an import that fails part-way may be restarted from a known chunk offset (the snapshot module records the last successful offset). -- Verified: the snapshot module checks an end-to-end CRC after the last chunk before activating the import. - -### 6.4 Cross-engine snapshots - -The default rule: **snapshot exported by engine X can only be imported by engine X.** A 3-node group all running crowdb-tree can snapshot-install crowdb-tree-formatted snapshots. A test cluster mixing in-memory and file engines must export/import via a portable format. - -A portable interchange format (e.g. sorted key-value pairs with slots) is provided as an optional path for testing and operations, at the cost of being slower than the native path. Production deployments should use a single engine type per group. - -### 6.5 Atomicity of import - -The import is atomic to readers: until the last chunk is processed and the import is activated, the engine continues serving its previous state. Only at the activation step does the engine swap to the imported state. This avoids serving a partial snapshot. - ---- - -## 7. Compaction Policy - -Tombstones and replaced values can be removed once they are no longer needed. The two watermarks that gate compaction: - -- `snapshot_slot` — the engine state at this slot is durable on at least leader + one peer. -- `safe_slot` — every learner has applied every slot up to here. - -### 7.1 When can a tombstone be GC'd? - -A tombstone for key `k` placed at slot `t` may be GC'd when `t < min(snapshot_slot, safe_slot)`. Both must have advanced past `t` because: - -- Until `safe_slot` passes `t`, some learner might still be applying earlier slots that could resurrect `k`. (Cannot happen for blind ops, but could complicate snapshot install if the tombstone is GC'd too early.) -- Until `snapshot_slot` passes `t`, a snapshot install must reproduce the tombstone. Otherwise the receiving peer would think `k` was never deleted. - -The conservative "both must pass" rule prevents observability holes. - -### 7.2 When can an old value be GC'd? - -When a key `k` is overwritten with a higher slot value, the old value is immediately overwritten in single-version storage; there is nothing to GC. The complication is only with tombstones (representing deletes) and with engine-internal versioning (e.g. crowdb-tree's MVCC of pages, which it manages internally). - -### 7.3 Compaction trigger - -- **Periodic.** A background sweeper runs every `compaction_tick` (default 5 min) and removes eligible tombstones in batches. -- **Pressure.** If the engine reports memory or disk pressure, a focused sweep runs. -- **Snapshot-side-effect.** When a snapshot completes, eligible tombstones for slots ≤ that snapshot may be swept as part of the post-snapshot bookkeeping. - -### 7.4 Engine-specific compaction - -- In-memory: tombstones are simply removed from the map. -- crowdb-tree: uses crowdb-tree's internal compaction. The sweeper hands crowdb-tree a "tombstones below slot S are safe to drop" hint and crowdb-tree merges that into its compaction policy. - ---- - -## 8. Compare for Cross-Learner Validation - -A `compare(other) -> diff` operation is required so that `crowdbbench` can verify state equality across learners after a test run (design-crowdb-kv.md §14.1](design-crowdb-kv.md)). - -### 8.1 Semantics - -`compare(other)` returns a diff describing the keys that differ in `(slot, value)` between `self` and `other`. If the engines are in the same logical state (same set of live keys, same `(slot, value)` per key), the diff is empty. - -The compare is **logical**, not byte-level. Two engines may have different physical layouts (e.g. one in-memory and one crowdb-tree) yet be logically equal. Compare must succeed across engine types. - -### 8.2 Required behavior - -- Order: results sorted by key. -- Tombstones: included if they differ. Two engines that both have `k` tombstoned at the same slot are equal. -- Resolved-slot: compared exactly. Two engines with the same value for `k` but different `resolved_slot(k)` are *not* equal; this would indicate that one of them missed an apply. - -### 8.3 Implementation strategy - -For testing, a simple two-cursor merge over both engines' ordered key streams is sufficient. The engines provide a `iter_all() → iterator<(key, slot, value_or_tombstone)>` interface to support this. The cursors advance the smaller key first; when both are at the same key, compare `(slot, value)`; emit a diff if they disagree. - -For very large states, a Merkle-tree-style digest could be added later; not required for the initial implementation. - -### 8.4 Compare under in-flight writes - -`compare` is a snapshot-time operation. It assumes the engines are quiescent (no apply in flight) or that the iterator semantics provide a consistent point-in-time view. The test harness ensures quiescence by stopping client traffic and waiting for `safe_slot == max_chosen` before invoking compare. - ---- - -## 9. Engine Implementations - -Two engine implementations satisfy the surface above. Each is appropriate for a specific phase of development or workload. - -### 9.1 In-Memory Tree - -- Backing store: a sorted in-memory map (e.g. a btree or skiplist). -- Persistence: none. -- Use cases: unit tests, integration tests where startup and teardown are cheap, behavior-validation runs of `crowdbbench`. -- Snapshot format: serialized sorted (key, slot, value-or-tombstone) tuples. -- Concurrency: read-write lock over the whole map for `apply`; concurrent reads via a snapshot pointer. - -### 9.2 crowdb-tree - -- Backing store: the production btree library `crowdb-tree`, a C++ `libcrowdb-tree` - (single-writer COW B+tree with per-leaf delta chains, epoch GC, versioned root) - consumed from Rust over a coarse C ABI. Full design: - [`../tree/design-crowdb-tree.md`](../tree/design-crowdb-tree.md) and its sub-docs. -- Persistence: yes, via a pluggable `PageStore` (local file, raw block device, - remote/RDMA); snapshot + consensus replay for recovery (no second op-log). -- Use cases: production. -- Snapshot format: portable `(key, slot, cell)` tuples first; native page dump as - an optimization. -- Concurrency: single writer (learner apply) + lock-free immutable-page reads, - epoch-based reclamation. - -The trait surface is engine-agnostic; switching engines is a configuration choice. Both pass the same `compare`-based equivalence tests. - -### 9.3 New-member snapshot sessions - -Snapshot transfer is exposed as unique, non-clone export and import session -owners. An exporter pins one stable view and reports its format, covered slot, -encoded length, body CRC32C, and chunk limit. Reads are sequential by byte -offset. An importer accepts arbitrary bounded feeds and changes the fresh -target engine only after `finish` validates the complete portable stream. -Dropping either owner aborts it. Production transfer supports only the -crowdb-tree portable format; cross-engine transfer is not supported. - ---- - -## 10. Tunables and Defaults - -| Parameter | Default | Range | Notes | -| --- | --- | --- | --- | -| `compaction_tick` | 5 min | 10 s – 24 h | Background tombstone sweep cadence | -| `snapshot_chunk_bytes` | 1 MiB | 1 B – 1 MiB | New-member snapshot Read payload limit; kept below the 4 MiB RPC frame ceiling | -| `engine_apply_concurrency` | 1 | 1 – ∞ | Apply is serialized by slot anyway; >1 makes sense only if apply is non-overlapping per-key | -| `engine_read_concurrency` | unbounded | — | Reads do not contend with apply at the engine level | -| `tombstone_grace_slots` | 0 | 0 – ∞ | Optional minimum number of slots to keep a tombstone past the GC watermark, for forensics | - -**Engine choice guidance:** - -- Unit and integration tests → in-memory. -- Manual debugging or operations exercises → ordered file. -- Production → crowdb-tree. - -**Per-key memory cost considerations:** the per-key resolved-slot adds 8 bytes per live key. For 10⁹ live keys this is 8 GiB on every learner, accepted in the requirement ([§7.3.1](design-crowdb-kv.md)). If memory pressure dictates, a future optimization could compress recently-applied resolved-slots into a "below safe-slot" bit (a single bit replacing the 8 bytes when the per-key slot is no longer needed for read-your-writes). diff --git a/doc/design/kv/design-crowdb-kv-test.md b/doc/design/kv/design-crowdb-kv-test.md deleted file mode 100644 index 7a93ef557..000000000 --- a/doc/design/kv/design-crowdb-kv-test.md +++ /dev/null @@ -1,272 +0,0 @@ - - - -# CROWDB - Design: Test Strategy - -Depends on: [`design-crowdb-kv.md`](design-crowdb-kv.md) §17 -Satisfies: [`design-crowdb-kv.md`](design-crowdb-kv.md) §17 - -## Table of Contents - -- [1. Overview](#1-overview) -- [2. Architecture Stack](#2-architecture-stack) -- [3. Test Binary Map](#3-test-binary-map) -- [4. Cross-Cutting Coverage Rules](#4-cross-cutting-coverage-rules) - - [4.1 Runtime resource ownership](#41-runtime-resource-ownership) -- [5. Layer Scope](#5-layer-scope) -- [6. crowdb-tree C++ Test Layers](#6-crowdb-tree-c-test-layers) -- [7. Sequencing](#7-sequencing) -- [8. Test Pairing Rule](#8-test-pairing-rule) - -## 1. Overview - -CROWDB is a distributed key-value store built on Paxos consensus. The test -suite is organized in layers that mirror the system architecture, so a -failure points at the lowest broken layer. Each layer gets its own test -binary and tests only the logic that belongs to that layer. - -This document is the **layer guide**: it defines the test strategy, the -scope of each layer, and the cross-cutting coverage rules that apply across -layers. Consult it to determine which layer a new test belongs to and what -rules apply. Per-layer coverage checklists and the live task backlog live in -[`plan-test.md`](../working/plan-test.md); benchmark design and baseline -results live in [`kv-write-flow-analysis.md`](kv-write-flow-analysis.md). - -## 2. Architecture Stack - -``` -store (PxKvStore: many groups, one node identity, routing) <- lib/crowdb-kv/tests/store.rs - group (PxGroup: 1 local + N remote replicas, full Paxos) <- lib/crowdb-kv/tests/group.rs - replica (PxLocalReplica: acceptor + learner + WAL + slots) <- lib/crowdb-kv/tests/replica.rs - election (PxLocalReplica election state machine, pure logic) <- lib/crowdb-kv/tests/election.rs - wal (WalEngine: durable log) slot (PxSlotList) <- lib/crowdb-kv/tests/wal.rs, slot.rs - unit (pure modules: codec, classifier, kv engine, roles) <- lib/crowdb-kv/tests/{paxos,kv}.rs + wal/slot codec tests -deployment (release containers + host HTTP/RPC clients) <- e2e-test/tests/{kv,deployment}/* -console (mgmt API server + CLI: Axum REST, CLI commands) <- app/crowdb-{web,cli}/tests/* -ui e2e (host Playwright browser + release node containers) <- e2e-test/clients/console-ui/e2e/* -``` - -## 3. Test Binary Map - -| Binary | Layer | Drives | -| ------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | -| `lib/crowdb-kv/tests/paxos.rs` | unit | `acceptor`, `learner`, `error` classifier, `roles` in isolation | -| `lib/crowdb-kv/tests/kv.rs` | unit | `InMemKV` / `KVEngine` / `CrowdbTreeEngine` apply semantics | -| `lib/crowdb-kv/tests/election.rs` | unit | `PxLocalReplica` election state machine (role transitions, vote granting, heartbeat, lease, term fencing) | -| `lib/crowdb-kv/tests/wal.rs` | subsystem | `WalEngine` append/replay/gc/segment + record codec + backends | -| `lib/crowdb-kv/tests/slot.rs` | subsystem | `PxSlotList` / `PxSlotNode` | -| `lib/crowdb-kv/tests/replica.rs` | replica | single `PxLocalReplica` (no peers) | -| `lib/crowdb-kv/tests/group.rs` | group | `PxGroup` multi-node clusters (real loopback crowdb-rpc, no mocks) | -| `lib/crowdb-kv/tests/store.rs` | store | `PxKvStore` routing / lifecycle / status / health | -| `app/crowdb-kv-server/tests/*` | component | management handlers, validation and deterministic local state | -| `e2e-test/tests/kv/*`, `e2e-test/tests/deployment/*` | container E2E | packaged server startup, public management/RPC, persistence and recovery | -| `lib/crowdb-kv-client/tests/*` | client component | `CrowdbClient` retry, topology cache, `NotLeaderHint` follow, `AnyReplica` read distribution + fallback against embedded crowdb-rpc servers | -| `app/crowdb-web/tests/*` | console mgmt API | Axum REST API server: node management, OpenAPI proxy, API forwarding | -| `lib/crowdb-console-shared/tests/*`, `app/crowdb-cli/tests/*` | console mgmt API | shared core (config, API client, health aggregation) + CLI commands | -| `app/crowdb-cli/tests/*` | CLI component | argument validation, protocol stubs and comparisons; real cluster lifecycle has a container owner | -| `e2e-test/clients/console-ui/e2e/*` | UI container E2E | host browser interactions against owned release nodes and fixed endpoints | -| `app/crowdb-web/ui/src/**/*.test.*` | UI component | deterministic hooks, rendering and state transitions | - -## 4. Cross-Cutting Coverage Rules - -**Placement rule:** deterministic algorithms, pinned roles, injected failures and -client/supervisor protocol stubs remain component-local, including embedded RPC -used to verify these internal contracts. Scenarios that start production server -binaries or exercise externally observable assembled service flows belong to the -[container E2E component](../../../e2e-test/README.md). Each mutating owner uses -its own release containers, network and data roots. Rust, SDK and browser clients -run on the host using inspected IPs and fixed ports; there is one canonical owner -per boundary flow. Component tests require neither Docker nor server binaries. - -**KV operation correctness rule:** every layer that applies KV mutations -(replica, group, store) must test all operation types and orderings: -- Put (single key) -- Overwrite (same key, new value) -- Delete (produces tombstone) -- Delete on non-existent key (no-op) -- Batch with multiple puts -- Batch with intra-batch last-wins (put → delete → put on same key) -- Batch with put-then-delete same key (delete wins) -- Batch with delete-then-put same key (put wins) -- Empty batch / NoOp (advances frontier, no keys) -- Mixed put/delete across multiple slots and batches -- Persistence round-trip: all above operations survive WAL replay + restart - -**Cluster verification rule:** every tier that creates a real `PxKvStore`, -`PxGroup`, or `crowdb-kv-server` process must verify two things before -proceeding to tier-specific assertions: - -1. **Leader election succeeded within a bounded timeout.** If leader - election does not complete within the timeout (typically 10 s for - multi-node, 3 s for single-node), the test fails — slow election is a - bug, not a flaky test. The test must assert that exactly one leader - exists, not just that "some leader appeared." - -2. **Basic KV CRUD correctness via the client library.** After leader - election, perform a minimal CRUD cycle through `CrowdbClient` (or - direct `PxKvStore` API for lower layers) and verify correctness: - - Put a key → Get returns the same value - - Overwrite → Get returns new value - - Delete → Get returns not-found - - Scan returns expected keys - - **Edge-case coverage** (at least one test per layer must cover these): - - Empty key (`""`) - - Large key (≥1 KB) - - Key with special bytes (null bytes, high-UTF8, whitespace-only) - - Large value (≥1 MB) - - Small value (1 byte) - - Empty value (`""`) - - Performance is not asserted here; correctness only. A put that takes - 5 s but returns `ok=true` and is readable via get is a pass. Performance - tuning is tracked separately (see Benchmark row in the binary map above - and [`kv-write-flow-analysis.md`](kv-write-flow-analysis.md)). - - KV operations are crowdb-rpc only (no REST API for KV). Tests use - `CrowdbClient` (deployment/UI layers) or `PxKvStore` public API - directly (group/store layers). The console mgmt API layer verifies - topology management via REST but does not perform KV operations; KV - correctness is delegated to the deployment or UI E2E layer. - -**Leader change & reconfig rule:** every tiered layer (Group, Store, -Deployment, UI E2E) that creates a multi-replica cluster must cover: - -1. **Leader change:** trigger step-down on the current leader, verify a new - leader is elected within bounded timeout (10 s multi-node, 3 s - single-node), verify KV operations continue to work after the new leader - is established, verify the old leader rejoins as follower after restart - (if applicable). - -2. **Reconfig — add replica:** add a new replica to a running group with - existing data, verify the new replica catches up (data visible via scan - or get on the new node) within bounded timeout. Since tests write few - keys, catch-up should complete in seconds. Slow catch-up is a bug. - -3. **Reconfig — remove replica:** remove a non-leader replica from a running - group, verify the group continues to accept KV operations (quorum - intact), verify health status reflects the reduced membership. - -4. **Reconfig — remove leader:** remove or stop the leader's replica, verify - a new leader is elected within bounded timeout, verify KV operations - resume. This is the most operationally sensitive scenario; the test must - not block indefinitely waiting for election. - -### 4.1 Runtime resource ownership - -Every test environment that starts subprocesses owns one runtime namespace. -The default root is the workspace-local `.crowdb-runtime/`; tests do not use -the system temporary directory. Its lifecycle classes are: - -- `ephemeral/` for disposable test and development environments; -- `persistent/` for restartable local clusters; -- `artifacts/` for exported logs, profiles, sanitizer reports, and benchmark - output; -- `ports/` for the cross-process assignment registry. - -A namespace manifest binds one logical service identity to its `data/`, -`config/`, `log/`, and `artifacts/` directories and to stable `ServicePort` -assignments. Starting two environments produces disjoint roots and ports. -Restarting a logical service reuses its existing assignment; replacement uses -a new identity. Child-process port handoff never uses port zero or a released -probe socket. An in-process listener may use port zero only while retaining -the bound socket itself. - -Ephemeral claims include the owning PID and process-start identity so stale -claims can be reclaimed without confusing PID reuse. Persistent assignments -remain claimed across stop/restart and a conflicting port is reported rather -than silently renumbered. Successful tests release their namespace; a panic -preserves its tree for diagnosis. `pixi run clean --runtime-only` removes only recorded -ephemeral processes and state. `pixi run clean` additionally removes -rebuildable artifacts, build output, and the complete `.crowdb-runtime/` tree. - -## 5. Layer Scope - -Each layer tests only the logic that belongs to it; a failure points at the -lowest broken layer. Detailed per-layer coverage checklists live in -[`plan-test.md`](../working/plan-test.md). - -- **Unit** — Pure modules with no I/O, async runtime, or network; - deterministic, microsecond tests. Covers `paxos/{acceptor,learner,error, - roles}`, `kv/{mem_kv,op}`, and the `PxLocalReplica` election state - machine (role transitions, vote granting, heartbeat, lease, term fencing). -- **WAL subsystem** — `WalEngine` and WAL internals: durable log, segment - management, replay, GC, I/O backends, pipeline writer. Uses namespaced - workspace-local filesystems or in-memory simulated disks. -- **Slot subsystem** — `PxSlotList` lock-free chunked sparse array: - single-threaded ops, concurrent stress, reclamation watermark - interactions with long-lived read guards. -- **Replica** — Single `PxLocalReplica` with no peers: acceptor + learner + - WAL + slot integration: prepare/accept with WAL persistence, dedup, - snapshot install, WAL replay ordering. -- **Group** — `PxGroup` with 1 local + N remote replicas over real loopback - crowdb-rpc (no mocks): full Paxos rounds, leader election, KV through the - group, durability under crash/restart. Includes RPC boundary rejection - tests: malformed `EAcceptRequest` frames (wrong table type, missing - required `value` field) are rejected with `InvalidArgument` via the - `test-util`-gated `send_raw_request` on `PxRpcTransport`. -- **Store** — `PxKvStore`: multi-group routing, node identity, lifecycle, - topology status. Uses embedded crowdb-rpc server via `PxKvStore::start`. -- **Client E2E** — `CrowdbClient` against embedded `PxKvStore` crowdb-rpc - servers: retry, topology cache refresh, `NotLeaderHint` follow, - `AnyReplica` read distribution, `MinSlot` lagging-follower fallback. No - `crowdb-kv-server` binary is booted. -- **Deployment** — `crowdb-kv-server` binary + HTTP management API + - multi-process clusters: boots the actual binary, exercises the HTTP API, - verifies multi-process cluster formation and lifecycle. -- **Console mgmt API** — Console management API server (Axum) + CLI - frontend: REST endpoints, OpenAPI proxy, API forwarding to - crowdb-kv-server nodes, shared core (config, API client, health - aggregation), CLI commands. KV correctness is delegated to the - deployment or UI E2E layer. -- **UI E2E** — Playwright browser tests against a real `crowdb-web` + - `crowdb-kv-server` backend: drives the SPA as an operator would (clicks, - context menus, dialogs, KV panel), verifies both UI feedback and backend - state. No mocks; the only network interception is route aborting for the - backend-unreachable test. -- **Benchmark** — Self-contained `bench benchmark` lifecycle: deploy a - 3-node cluster, drive write/read load, collect server-side metrics + - logs, produce a report, then clean up. Exercises the full production - path end-to-end. Design, storage modes, and baseline results are in - [`kv-write-flow-analysis.md`](kv-write-flow-analysis.md). - -## 6. crowdb-tree C++ Test Layers - -The C++ crowdb-tree library (`libcrowdb-tree`) has its own test layers, separate -from the Rust test binaries above. They run as `test-tree-ct` in CI. - -| Layer | Where | What it proves | -| --- | --- | --- | -| C++ unit | `liblib/crowdb-tree/tests/unit` | Single component correctness: cell encoding, leaf page build/read, delta replay, consolidation triggers, mapping table, epoch manager, split point. | -| C++ integration | `liblib/crowdb-tree/tests/integration` | Multi-component flows over `InMemoryPageStore` and `FilePageStore`: basic CRUD, batch apply, scan, split/merge, consistent view, snapshot roundtrip, GC. | -| Crash/recovery | `liblib/crowdb-tree/tests/recovery` | Durability: snapshot + recover, torn-page, superblock A/B, double-apply idempotency. Uses a `FaultyPageStore` with fault points for the FI matrix. | -| Rust FFI | `crowdb-kv` `tests/` | `CrowdbTreeEngine` trait conformance (shared parametrized tests with `InMemKV`), async bridge, buffer ownership, error mapping. | -| Cross-engine parity | `crowdb-kv` `tests/` | `InMemKV` and `CrowdbTreeEngine` produce identical state via `compare()` after random op streams, snapshot export/import round-trips, and mid-stream restart. | -| Concurrency | sanitizer CI | No data races / UAF under reader+writer load (TSan/ASan/UBSan); epoch reclamation under load; version pin + GC. | - -The authoritative correctness oracle is **`compare()` against `InMemKV`**: for -any op sequence, the two engines' `EngineView::iter_all` must be byte-for-byte -equal (same key set, same `(slot, cell)`). - -## 7. Sequencing - -Fill gaps bottom-up so a new failure is always attributable to the lowest -layer: -1. Unit + WAL/slot + Election — cheap, deterministic. -2. Replica layer — highest-value gap; unblocks confident group debugging. -3. Group reconfiguration + peer RPC transport. -4. Multi-node store and deployment re-enables, after repair-correctness - fixes tracked in [`plan-test.md`](../working/plan-test.md). - -## 8. Test Pairing Rule - -Every feature or component milestone includes: - -1. **Unit invariants** from the matching test layer (property-based or - deterministic). -2. **Failure-injection** matching [`design-crowdb-kv.md`](design-crowdb-kv.md) §9 - scenarios. -3. **Benchmark integration test** (end-to-end correctness + performance) - once the RPC/client layer is reached. - -See [`plan-test.md`](../working/plan-test.md) for pending test task tracking. diff --git a/doc/design/kv/design-crowdb-kv-wal.md b/doc/design/kv/design-crowdb-kv-wal.md deleted file mode 100644 index b49b67819..000000000 --- a/doc/design/kv/design-crowdb-kv-wal.md +++ /dev/null @@ -1,626 +0,0 @@ - - - -# CROWDB - Design: Write-Ahead Log - -Depends on: [`design-crowdb-kv.md`](design-crowdb-kv.md) §8.1, §8.2 -Satisfies: [`design-crowdb-kv.md`](design-crowdb-kv.md) §8.1, §8.2 - -This document specifies CROWDB's write-ahead log. **There is exactly one durable -log per group: the replica's consensus log (the per-slot acceptor log).** -Everything else (the KV state machine (`KVEngine`), future snapshots, the dedup -cache, the chosen/applied watermarks) is a *derived projection* of that log. -The `Accepted` record carries the actual operation, so this single log doubles -as the replicated command log (Raft-style) and the state-machine input; there is -no separate "state-machine WAL" and no second write of the value. - -Durability rule of thumb: **if an acknowledged write is in the WAL on a quorum, -it cannot be lost** — a value is *chosen* exactly when a quorum has durably -`Accepted` it, and replay + recovery reconstruct the committed state from those -durable accepts. - -## Table of Contents - -- [1. Goals and Non-Goals](#1-goals-and-non-goals) -- [2. Logical Record Shape](#2-logical-record-shape) -- [3. Multi-Disk Segment Layout](#3-multi-disk-segment-layout) -- [4. Write Path and Batched Durable Flush](#4-write-path-and-batched-durable-flush) - - [4.6 I/O Backend Abstraction](#46-io-backend-abstraction) -- [5. Ack Contract and Failure Modes](#5-ack-contract-and-failure-modes) -- [6. Replay, Restore, and Recovery](#6-replay-restore-and-recovery) -- [7. Garbage Collection](#7-garbage-collection) -- [8. Disk Loss and Recovery](#8-disk-loss-and-recovery) -- [9. Tunables and Defaults](#9-tunables-and-defaults) -- [10. Follow-up Implementation Scope](#10-follow-up-implementation-scope) - ---- - -## 1. Goals and Non-Goals - -**Goals:** - -- Bound write latency by the slower of (one durable flush) and (one quorum RTT). -- Saturate aggregate disk bandwidth by parallelizing across multiple physical disks. -- Survive arbitrary process crash with bounded recovery time. -- Survive single-disk loss without data loss (peer recovery via snapshot install). -- Be replayable into a deterministic state. - -**Non-goals:** - -- Cross-group log ordering. Each group has its own WAL; there is no cluster-wide log. -- Compression / deduplication at the WAL layer. (Possible future work.) -- Multi-version time-travel reads. The WAL is not queried by the read path. -- Erasure coding of records. Replication is at the consensus layer, not the disk layer. - ---- - -## 2. Logical Record Shape - -A WAL record persists one of the messages an acceptor must remember durably. Conceptually: - -| Field | Purpose | -| --- | --- | -| `magic`, `version` | Format identifier, for upgrade compatibility | -| `record_type` | One of `{ Promised, Accepted, VoteGranted }` | -| `group_id` | Which group this record belongs to | -| `term` | Acceptor's `current_term` at the time of record (for fence verification on replay) | -| `slot` | Slot number this record concerns (n/a for `VoteGranted`) | -| `ballot` | `(round, leader_id)` for `Promised` / `Accepted` | -| `payload` | Type-specific body — for `Accepted`, the `PxLogEntry` (kind + batch); for `Promised`, empty; for `VoteGranted`, the voted-for node id | -| `payload_len` | Length, for forward-skipping malformed records | -| `crc32c` | CRC of header + payload | - -Records are self-describing and length-prefixed so a corruption in one record does not destroy the rest of the segment unless followed by a CRC failure (see §6). - -**Design note:** the `Accepted` record's payload IS the `PxLogEntry`; there is no separate "raft log" or "state machine log" structure. This keeps disk I/O minimal: one durable flush per accepted batch, no further logging. - -`Promised` records exist for safety: classical Paxos requires acceptors to remember promises across crashes. They are short (no payload). - -### 2.4 When each record is written - -The WAL is written **only on the acceptor/election critical path**, before the -handler replies (the ack contract, §5). The mapping is exhaustive: - -| Handler (`PxLocalReplica`) | Trigger | Record | Payload | -| --- | --- | --- | --- | -| `on_prepare` | acceptor grants a promise | `Promised` | none (slot + ballot + term in header) | -| `on_accept` | acceptor accepts a value | `Accepted` | the `PxLogEntry` (kind + KV batch) | -| `handle_request_vote` | acceptor grants a vote | `VoteGranted` | `voted_for` node id (term in header) | - -**`learn()` writes nothing.** Applying a chosen value to the `KVEngine` is a pure -in-memory projection; its durability comes entirely from the `Accepted` records -that a quorum already persisted. The state machine's KV state is **not** rebuilt -from the WAL on restart. Instead, the state machine will persist its own -snapshot (with a slot index) to disk. On restart, the state machine loads its -snapshot and the WAL replay only rebuilds acceptor state (promises + accepted -values). The learner / `KVEngine` is repopulated by new-leader recovery (§6.4) -or steady-state heartbeat catch-up (§6.5). - -This design avoids a separate `DurableCommitWatermark` WAL record type. The -WAL carries only consensus state (`Promised`, `Accepted`, `VoteGranted`); the -state machine owns its own persistence boundary via snapshots. - ---- - -## 3. Multi-Disk Segment Layout - -### 3.1 Why multiple disks - -A single disk's fsync latency caps single-group write throughput at `1 / fsync_latency` IOPS, which on commodity NVMe is ~10–30k. Multiple disks give us linear scaling up to the network/CPU limit. - -### 3.2 Layout - -Each acceptor configures `wal_disks = [path_1, path_2, ...]`. Each disk holds a sequence of **segments**. A segment is a single append-only byte stream containing records in append order for that disk. It is **not** required to contain a contiguous slot interval. - -With multiple disks, global slot order and physical record order intentionally diverge: - -``` - /wal_disk_1/groupN/seg-0000001.ck slots {1,4,7,...} - /wal_disk_1/groupN/seg-0000004.ck slots {361,364,367,...} - /wal_disk_2/groupN/seg-0000002.ck slots {2,5,8,...} - /wal_disk_2/groupN/seg-0000005.ck slots {362,365,368,...} - /wal_disk_3/groupN/seg-0000003.ck slots {3,6,9,...} -``` - -- Each segment is named with a monotonic `segment_id` within the group. -- Segments are pre-allocated to a target size (`wal_segment_size`, default 64 MiB) and sealed when full. -- Each sealed segment records `(min_slot, max_slot, record_count)` as scan metadata; `min_slot..max_slot` may contain holes because slot ownership is distributed across disks and because non-slot records exist. -- Replay correctness comes from sorting / indexing records by `(group_id, slot, ballot, record_type)` after reading them, not from physical segment order. - -**Slot-to-segment mapping** is recorded in an in-memory **segment index** that -points each slot to its latest `(disk, segment_id, file_offset)` record. The -live index has one mutex-protected shard per disk pipeline. Each writer mutates -only its own shard, and slot lookup uses the same deterministic affinity as the -write path. Replay and GC snapshot shards one at a time; file scanning and -unlink I/O never run while an index shard is locked. A persisted index may be -used as a startup accelerator, but it is only a cache: replay must be able to -rebuild the authoritative index by scanning segment records. - -### 3.3 Slot-to-disk assignment - -When the acceptor receives a slot-addressed record (`Promised` or `Accepted`) for slot N, it chooses the target disk by deterministic slot affinity: - -``` - disk = hash(group_id, slot) % wal_disks.len() -``` - -This is the default and correctness-oriented mapping. All records for the same `(group_id, slot)` land on the same disk, so replay can recover that slot without merging same-slot histories from multiple disks. The mapping is stable across restart as long as `wal_disks` order is stable. - -Each disk owns an independent append queue, active segment, flush worker, and durable-flush state. Concurrent slot writes still parallelize because adjacent slots hash across disks, but same-slot retries and higher-ballot re-accepts preserve disk locality. - -Non-slot records use fixed lanes: - -- `VoteGranted` and other election metadata go to the group metadata lane (`disk = hash(group_id, 0) % wal_disks.len()`). -- Group membership configuration is persisted in a dedicated config file (`group_config.bin`) in the group's WAL directory, not as a WAL record. - -A future load-aware mode is only legal if it preserves slot affinity, for example by moving whole slot shards through an explicit re-sharding protocol. Per-record load-aware placement is not allowed for slot-addressed records. - -Because apply order is determined by the slot index in memory, the *physical* order of records on disk does not affect correctness (Invariant I4). This is what allows multiple disks to flush in parallel. - -### 3.4 Segment rotation - -A segment is rotated (sealed and a new one opened) when it reaches -`wal_segment_size`. (Future: slot-range boundary rotation for GC granularity, -admin force-rotate, and leader-change rotation are design options not yet -implemented.) - -Sealed segments are immutable. - ---- - -## 4. Write Path and Batched Durable Flush - -### 4.1 Goal of batching - -Each individual `Accept` is small (tens of bytes overhead + payload). Per-record durable flush is wasteful: an SSD's flush amortizes over a batch nearly as well as over a single record. CROWDB therefore batches records per WAL disk and completes the record futures only after the selected backend has made the bytes durable. - -The durable operation is backend-specific: - -- **Filesystem file backend:** append with normal file writes, then `fdatasync` / equivalent durable flush. -- **Block-device backend:** issue aligned positional writes, using direct I/O - when selected. If the device requires 4 KiB sectors, the WAL writer pads or - read-modify-writes through the alignment planner. The real file-backed - implementation then calls `sync_data` before releasing durable acknowledgments. -- **RAM / SCM / simulated backends:** use the backend's own durability contract. A volatile RAM backend is for tests only and may complete immediately; persistent-memory backends may require cache-line flush / fence semantics instead of filesystem sync. - -The acceptor code talks to `WalEngine::append_*` and does not branch on these backend details. - -### 4.2 Batch coalescing - -For each disk, the WAL maintains a **pending queue**: - -``` - pending_queue[disk] = [ record_1, record_2, ..., record_k ] -``` - -Records arrive concurrently from many `Accept` handlers. They are appended to the in-memory segment buffer for that disk and pushed onto the disk's pending queue. A single **flush worker** per disk owns the active segment, drains queued records into one backend write batch, calls the backend-specific durable flush, and then **completes all pending records' futures together**. - -The worker is event-driven, not timer-first: - -1. If the queue is empty, the worker awaits a notification from the next enqueue. -2. When notified, it drains all records currently available without waiting for a fixed 1 ms interval. -3. If more records arrive while the write is being assembled, they are included until the batch reaches `wal_flush_batch_bytes` or the drain loop observes no immediately-ready records. -4. The worker issues one backend write / durable-flush operation for the drained batch. -5. On success, it resolves every drained record future; on error, it fails them and marks the affected disk / group unhealthy. - -This gives low latency for a single record while still aggregating bursts naturally. - -### 4.3 Triggers for durable flush - -The flush worker flushes when any of: - -- It is woken by a transition from empty queue to non-empty queue (`rx.recv().await`). -- Pending bytes ≥ `wal_flush_batch_bytes` (default 64 KiB): breaks the drain loop early. -- The watchdog (`wal_flush_watchdog_ms`, default 100 ms) is a safety-net timer that wakes the idle writer every `watchdog` ms to drain any queued records in case of a missed wake (a defensive measure "just in case for bugs"). The idle wakeup does a `try_recv` drain and re-parks if nothing is queued; no I/O, no allocation, ~10 wakeups/s at the default 100 ms. - -Default behavior is **wake-drain-flush**: a lone record flushes immediately, while a concurrent burst gets batched because the worker drains all immediately-ready records before issuing I/O. A coalescing budget (`wal_flush_coalesce_us`) was previously available but was removed after T3 benchmarking showed no gain over the wake-drain-flush baseline. - -### 4.4 Async write completion - -Each `Accept` handler returns a future. The flush worker resolves the future with `Ok` after the backend durable flush completes; on disk error it resolves with `Err` (which the acceptor escalates per §8). The acceptor only emits the network `Accepted` response after the future resolves; this is the **ack contract** ([§5](#5-ack-contract-and-failure-modes)). - -The future-based interface decouples the acceptor's handler from disk timing and allows multiple in-flight `Accept`s to coalesce into one durable flush naturally. - -### 4.5 Why this maps onto Multi-Paxos parallelism - -Different slots may be in flight on different disks at the same time. Slot N may be batching on disk 1 while slot N+1 batches on disk 2. Both durable flushes proceed in parallel. Aggregate flush throughput is `sum over disks of (1 / flush_latency)`, exactly the parallel benefit we want. - -Crucially, the leader's **own** durable flush is not on the critical path of remote acceptors' durable flushes. The leader persists locally, then broadcasts; remote acceptors flush in parallel. Two durable flushes total in serial: leader's, and the slowest of the quorum. With multiple disks each durable flush can itself be parallelized internally, but the consensus-level parallelism is what dominates throughput. - -### 4.6 I/O backend abstraction - -The WAL dispatches all file operations through `IoBackend`, a process-lifetime -enum with four variants: - -- **`File`** — `tokio::fs` with an awaited `sync_data` durability barrier. The - production default; works everywhere. A batch acknowledgement is not released - until this barrier succeeds. -- **`Uring`** — explicitly selected buffered regular files. Positional reads, - vectored writes, `fdatasync`, and `fsync` are submitted to one process-shared - single-pipeline io_uring owner; namespace and metadata operations remain on - the portable filesystem path. Ring setup failure rejects explicit selection - rather than falling back. Borrowed buffers and descriptors remain live until - the CQE is drained, including when a Rust operation future is dropped. -- **`MemBlock(MemBlockDevice)`** — in-memory test harness. Stores segments as - `BTreeMap>` with error injection (`inject_io_error`, - `inject_sync_error`, `set_full`), corruption injection - (`corrupt_at_offset`), and layout tracking. `fdatasync` is a no-op (data is - already "durable" in RAM). Used by all unit/integration tests. -- **`BlockDevice(BlockDevice)`** — real file-backed block device. Always opens - OS paths and does positional I/O (`pwrite`/`pread`) via `FileExt`. - `fdatasync` calls `sync_data`. Whether the path is a regular file, a raw - block device, or a tmpfs file makes no difference; in Unix, a block device - *is* a file. - -On the production multi-thread Tokio runtime, including a runtime with one -worker, synchronous block-device file and layout operations use Tokio's -[`block_in_place`](https://docs.rs/tokio/1.52.3/tokio/task/fn.block_in_place.html). -This allows independently spawned apply, RPC, and timer tasks -to run while the OS call blocks. The closure completes before returning, so -borrowed WAL buffers and the file remain live without an extra payload copy. -Write, durable flush, index publication, and acknowledgment keep their original -ordering and error behavior. This changes scheduling rather than durability, -timeouts, retries, or configured worker/resource limits. Current-thread and -non-runtime callers retain synchronous compatibility; they do not gain -concurrent task progress. The production path uses independently spawned WAL -workers on the ordinary multi-thread runtime, not a `LocalSet` or same-task -`join!` branches. - -`BlockDevice` has two constructors selected by the `O_DIRECT` flag: - -- `BlockDevice::new()` — aligned (4K), buffered (no `O_DIRECT`). Default for - benchmarks and general use. All alignment/RMW logic runs, `fdatasync` - performs a real `sync_data`, but the page cache absorbs writes. -- `BlockDevice::ssd()` — aligned (4K), `O_DIRECT` (Linux only). Models a real - SSD/NVMe with direct I/O bypassing the page cache. - -`MemBlockDevice` has two constructors selected by alignment: - -- `MemBlockDevice::new()` — unaligned (IU=1, byte-addressable). Models RAM / - SCM / PMEM. Default for tests. -- `MemBlockDevice::with_alignment(align)` — aligned in-memory. For testing - RMW logic without real files. - -Both `BlockDevice` and `MemBlockDevice` track write counts, fdatasync counts, -logical/physical bytes written, and RMW counts for observability. The -`WalEngine::backend_label()` method returns a short string (`"file"`, -`"uring"`, `"mem"`, `"block"`) for metric names. `WalEngine::block_device_snapshot()` -reads cumulative counters for the engine collector to compute per-window -deltas. - -The WAL bench (`lib/crowdb-kv/benches/wal.rs`) exercises four backends: `Mem` -(in-memory `MemBlockDevice`), `File` (`tokio::fs`), `Uring` (buffered -CQE-backed file I/O), and `Block` -(`BlockDevice::new()` with `wal_skip_fsync: true`). The `Block` case hits all -block code paths (alignment planning, RMW, amplification tracking, -`pwrite`/`pread` syscalls) at high TPS since `fdatasync` is skipped per -batch. - -The crowdb-tree C++ storage layer has a similar but cleaner separation: -`TextPageStore` (debug text files), `MemPageStore` (in-memory), and -`BlockPageStore` (real block device with `O_DIRECT`). Each is a distinct C++ -class; no flag-based branching. The Rust FFI `PageStoreBackend` enum (`File`, -`Block`, `MemBlock`) maps to these. No structural change was needed on the -crowdb-tree side for this refactor. - ---- - -## 5. Ack Contract and Failure Modes - -### 5.1 The contract - -> An `Accepted` response is sent to the leader only after the corresponding WAL record's backend durable flush has completed. -> A client write is acked only after a quorum of acceptors have sent `Accepted`. - -This is repeated from design-crowdb-kv.md §8.1](design-crowdb-kv.md) because everything else in this section is a consequence of it. This is the **strict** mode. The **early-ack** mode (§5.4) relaxes the leader's own durability ordering but keeps the quorum-durability requirement; both modes are described in §5.4. - -### 5.2 Failure cases - -- **Crash before durable flush.** The record may or may not be on disk. Replay reads what is on disk; the slot may end up with no record on this acceptor. That is fine; the leader either had a quorum from other acceptors (slot is chosen) or not (slot will be repaired). In strict mode no client was acked for this acceptor's slot, so no expectation is violated; in early-ack mode (§5.4) the leader may have acked the client on the remote quorum alone, and the chosen-but-not-locally-durable slot is re-adopted by `repair_once` from the followers on restart. - - On restart, the new leader computes its recovery ceiling from durable state, not from an assumed contiguous log tail: `ceiling = max(local highest_seen_slot, peers' highest_seen_slot, persisted next_slot - 1 if present)`. Because parallel writes may leave holes inside `[floor+1, ceiling]`, the leader must run bulk Phase 1 over the whole open interval and fill empty slots with `NoOp`. It does **not** rely on `latest_slot + window_size` to discover hidden data; any value absent from every acceptor's durable state could not have formed a durable quorum and therefore could not have been acknowledged. - -- **Crash after durable flush, before sending `Accepted`.** Replay finds the record. After the acceptor rejoins the group, the leader observes the existing accept (via heartbeat / slot-status query) and includes this acceptor in the quorum. -- **Crash after sending `Accepted`, before the leader acked the client.** Same as above; the record is on disk; if a quorum was reached, the slot is chosen; the new leader's bulk Phase-1 will see this and re-confirm. -- **Durable-flush failure (returns `Err`).** Treated as a disk fault. The acceptor stops accepting new records on that disk and marks itself failed for the affected group; see §8. - -### 5.3 What we never do - -- Ack a client write before a quorum of acceptors have durably flushed. This holds in both ack modes (§5.4): the early-ack mode defers only the *leader's own* durable flush, never the remote quorum's. -- Declare a slot `Chosen` without a quorum of `Accepted` responses (remote durable + leader CAS in early-ack mode; remote durable + leader durable in strict mode). - -### 5.4 Ack modes (`wal_early_ack`) - -The ack contract in §5.1 is the **strict** mode: every acceptor (including the leader) completes its WAL durable flush before its `Accepted` counts toward `Chosen`, and the client is acked only after the leader's own flush joins the quorum. `wal_early_ack` (default `true`, gated on `quorum > 1`) is a **relaxed** mode that drops the leader's local fsync off the write critical path: - -- The leader runs `on_accept_inner` (the term-fence CAS + in-memory `acceptor.accept`) concurrently with the remote `send_accept` fan-out via `tokio::join!`. -- `Chosen` is declared as soon as the **remote quorum** has durably flushed *and* the leader's local CAS has landed. The leader's own WAL `Accepted` record is appended by a fire-and-forget `tokio::spawn` (`spawn_accept_persist`) that runs off the critical path; its failure is logged and does not retract `Chosen`. -- The client is acked at that point. The per-proposal critical path becomes the quorum RPC round-trip only (the local fsync, ~10–100 µs on NVMe, runs concurrently and is no longer on the path). - -**Crash-recovery guarantee.** A crash in the window between the leader's CAS landing and the deferred local persist is Paxos-safe: the value is chosen (remote quorum accepted durably), so the client's observed `Chosen` is not violated. The leader's acceptor state for that slot is lost on crash (the CAS was in memory, the WAL record never landed), but on restart `replay_group` rebuilds from durable state only and `repair_once` re-adopts the chosen value for the gap slot from the followers (classic prepare + accept re-runs and re-chooses the highest-ballot accepted value). A value the client observed as `Chosen` therefore remains readable after crash + restart: either the local WAL raced the kill and has it, or repair re-adopts it from the quorum. - -**Single-node gate.** `wal_early_ack` defaults to `true` only when `quorum > 1`. A single-node group (quorum = 1) has no survivors to re-drive a chosen-but-not-locally-durable slot after a crash, so the leader's local fsync must stay on the critical path and the strict contract (§5.1) applies. - -**What is not relaxed.** The remote quorum's durable flush is still on the critical path in both modes; `Chosen` is never declared on in-memory-only remote accepts. The relaxation is strictly the leader's *own* durability ordering, recovered by `repair_once` rather than awaited inline. - ---- - -## 6. Replay, Restore, and Recovery - -Turning the on-disk WAL back into a live, serving replica has three stages, then -a steady state: - -- **6.1 Replay** — rebuild in-memory *acceptor* state from the records. -- **6.2 Restore** — rebuild a live replica shell from acceptor state (Pass 1), then rebuild the learner / `KVEngine` from that same acceptor state (Pass 2), skipping only the prefix the engine already durably reflects. -- **6.3 Wiring** — attach a fresh WAL engine and seed the proposer conservatively. -- **6.4 Recovery** — a new leader re-confirms *chosen* values from a quorum and fills holes. -- **6.5 Steady-state apply** — followers keep their state machine current from the leader's commit watermark. - -The split matters: replay/restore are purely *local* (this node's WAL), but `Accepted` ≠ `chosen` (a value is chosen only on a quorum). Restore therefore re-derives the acceptor's local view exactly as it was (Pass 1), then re-derives the learner's committed KV state from that *same locally-accepted* data (Pass 2), which is safe precisely because `Accepted` records already carry a chosen-or-not-yet-known value idempotently (re-`learn()`ing a value that was never actually chosen by a quorum is harmless: it just sits in the engine until either superseded by a higher slot for the same key, or never observed as chosen by anyone else, in which case it was never visible to a client anyway). Slots this node never locally accepted are, separately, left to: - -- New-leader recovery / steady-state heartbeat catch-up re-learning the value from quorum-confirmed consensus (§6.4 / §6.5). - -**No `DurableCommitWatermark` WAL record.** The state machine owns its own persistence boundary, but that boundary is the *engine's* durability (crowdb-tree's on-disk snapshot), not a separate state-machine-level snapshot file; see §6.2 below and `design-crowdb-kv-state-machine.md §2.1`. The WAL does not duplicate this information. - -### 6.1 Replay — rebuild acceptor state (`replay_group`) - -1. Discover all segments under `/group/seg-*.ck`, order by `segment_id`. - A zero-byte final segment on each WAL disk can result from a crash between - file creation and header writing. Remove that empty tail; it contains no - acknowledged records. A nonempty unreadable segment, invalid header, or - empty intermediate segment prevents local recovery; never skip it and serve - a partial acceptor history. The direct block backend - aligns physical reads and copies the requested byte range, including short - header reads and record ranges spanning block boundaries. -2. Walk each segment's records in order; verify `magic`, `version`, `crc32c`. - On the first failure, **truncate that segment at the offset and stop** (a torn - tail from a crash mid-write); later segments are still processed. -3. Rebuild per-`(group, slot)` acceptor state keeping the **highest-ballot** - `Promised` / `Accepted` per slot (later/higher-ballot records win, Paxos rule). - Retain promises and accepted values independently while scanning; release - superseded payloads immediately. Segment locations, counts and election - state still account for every verified record. -4. `current_term` = max `term` across all records. -5. `voted_for` = the node from the latest `VoteGranted` whose `term == current_term`. This is election safety state, not just debug metadata: after crash, the node must not grant a second vote in the same term. -Output: `ReplayResult { records, max_segment_id, current_term, voted_for }`. - -### 6.2 Restore — rebuild live acceptor state (`restore_from_replay`) - -Pass 1 rebuilds acceptor state without assuming any local accept was ever chosen by a quorum: - -- Feed `Promised` through the acceptor restore path so the highest promise per slot is preserved. -- Feed `Accepted` through the acceptor restore path so the highest accepted value per slot is available to future Phase 1. -- Seed `current_term`, `voted_for`, role from `ReplayResult`. - -The learner / `KVEngine` is rebuilt in a second pass over the *same* replayed -WAL records (`PxLocalReplica::restore_from_replay_with_engine`), not from a -separate state-machine snapshot file; see -[`design-crowdb-kv-state-machine.md §2.1`](design-crowdb-kv-state-machine.md#21-state-machine-restart-engine-durability--wal-replay) -for the full mechanics. In short: - -- Pass 2 walks every slot with an accepted entry (rebuilt in Pass 1 above) - and `learn()`s it into the engine, in order. This alone is always - sufficient and correct: `KVEngine::apply` is idempotent - (highest-slot-wins per key), so it works unconditionally, including for - `InMemKV` which has no durable floor at all. -- **Optimization for a durable engine:** before Pass 2, the engine reports - its own durable floor via `KVEngine::resume_from_slot()` (crowdb-tree's - `last_applied_slot`, restored from its on-disk snapshot). Pass 2 then - starts just above that floor instead of at slot 1, and the learner's - frontier is fast-forwarded directly to it, skipping the redundant - re-`learn()` of an already-durable prefix. -- Slots above the local WAL's own highest accepted slot are left to §6.4 / - §6.5 (heartbeat catch-up / new-leader bulk Phase 1) to re-learn from peers. - -This avoids resurrecting a value that was accepted locally but never chosen, -and, via the resume-floor check, never re-applies a write below a point -the engine has already durably advanced past (crowdb-tree rejects such writes -outright; see `../tree/design-crowdb-tree-engine.md`). - -### 6.3 Wiring a live group (`create_group_with_wal`) - -`replay_group` → create `WalEngine` and resume `next_segment_id = max_segment_id + 1` → `restore_from_replay` → `set_wal` → wrap in `PxGroup`. - -- Seed the live pipeline index with the verified replay index before starting - writers. Its slot locations and segment metadata cover retained history as - well as future appends, so GC can reclaim pre-restart segments. -- Initialize the WAL's local snapshot watermark from the freshly opened - engine's durable resume floor. Peer progress and the reclamation certificate - remain independent gates; local recovery does not authorize prefix removal. - -The group also records a conservative local slot tip: - -``` - local_tip = max(highest_seen_slot, snapshot_slot) -``` - -Where `snapshot_slot` is the state machine's persisted snapshot slot (0 if no snapshot exists yet). - -A restarted node must not serve as leader until election and §6.4 recovery establish the quorum-derived ceiling. After recovery is issued, the leader seeds new client proposals at `next_slot = ceiling + 1`. - -### 6.4 New-leader recovery (bulk Phase 1) - -On winning election a leader sweeps slots `(floor, ceiling]`: - -- `floor = local contiguous_chosen`; `ceiling = max(local highest_seen, - next_slot − 1, peers' highest_seen)` (peers' values arrive in the vote replies). -- For each slot it runs Phase-1 `Prepare` across a quorum and **adopts the - highest-ballot accepted value** observed (an empty slot becomes a `NoOp`), then - re-`Accept`s under its term and `learn()`s it. -- Quorum intersection guarantees any chosen value is seen, so a committed value - is **re-confirmed, not overwritten**. This repairs a leader that restored an - incomplete or merely-accepted prefix. - -(Steady-state gap repair below the frontier uses the same adopt-from-quorum logic, one slot at a time; see [`design-crowdb-kv-slot.md`](design-crowdb-kv-slot.md).) - -Parallel slot assignment means holes are normal after crash / restart. Recovery resolves every slot in `(floor, ceiling]` independently: - -- If Phase 1 observes an accepted value from any quorum member, the leader adopts the highest-ballot value and re-accepts it. -- If Phase 1 observes no accepted value for that slot, the leader writes a `NoOp` for the slot. -- `NoOp` has no KV payload, but it is a real chosen slot and advances contiguous watermarks; future replay sees it and does not need to repair the hole again once it is covered by the state machine snapshot. - -Slot indexes are generated only by the active leader's proposer: - -1. A leader tenure is fenced by election `term`; stale leaders are rejected by term / ballot checks. -2. During recovery, the leader computes `ceiling` from local and quorum-reported highest seen slots. -3. It resolves all holes up to `ceiling` with adopted values or `NoOp`. -4. It initializes `next_slot = ceiling + 1`. -5. Each admitted client proposal atomically fetches and increments `next_slot` exactly once. - -This gives unique slot numbers within a leader tenure and a continuous chosen prefix after recovery, even though physical WAL placement is non-contiguous and multiple slots are processed in parallel. - -### 6.5 Steady-state apply - -- **Leader:** applies on commit (`propose` → quorum `Accept` → `learn`). -- **Follower:** does **not** apply on `Accept` (which only persists) nor on the - payload-less `ChosenNotice` (which carries no value). Instead each heartbeat - carries the leader's `committed_safe_slot`, and the follower applies its - *contiguous accepted prefix* up to it (`apply_committed_up_to`). A follower - missing a committed `Accept` is re-sent it by the leader's heartbeat catch-up, - so its prefix converges over successive heartbeats. -- **Reads:** linearizable reads run on the leader behind a lease / ReadIndex - barrier (non-leaders redirect); bounded-stale / best-effort reads serve local - applied state. - -### 6.6 Truncation safety - -A truncated record is one whose CRC fails or whose payload is shorter than -`payload_len` (crash mid-write). Truncate-on-failure is safe because appends are -append-only and the bad record is the last durable one on that disk; the missing -slot was either chosen on a quorum without this node (recovered by §6.4) or never -chosen (no client was acked). A CRC failure inside a **sealed** segment is fatal: -the node fails out of the group and rebuilds from peers via snapshot install. - -### 6.7 Replay performance - -Binary replay uses a 1 MiB read-ahead buffer for frame markers, format probes -and complete frames. A larger frame receives enough space for its own length. -Every frame still passes its normal CRC and format checks. Retained payloads -own their bytes so replacing an accepted record can release it independently -of neighboring records in the input buffer. Memory scales with the retained -highest-ballot state and index, rather than the number of replaced versions. - ---- - -## 7. Garbage Collection - -WAL GC is the mechanism that keeps disk usage bounded. It interacts with snapshots and the safe-slot. - -### 7.1 GC watermark - -The local watermark is `min(reclamation_certificate, local_contiguous_applied, -local_durable_snapshot)`. The leader derives the certificate from every -configured learner's applied progress and durable snapshots on itself plus a -voting peer (local durability alone for a singleton voting group). Followers -receive the certificate in a heartbeat bound to the current membership epoch, -term and known voting leader. A missing certificate or local snapshot gives zero. - -Only records with `slot < gc_slot` are eligible. The same boundary trims -in-memory acceptor history; retained readers keep their owners alive. The full -contract is in [slot retention](design-crowdb-kv-slot.md#11-interaction-with-snapshot-and-wal-gc). - -### 7.2 GC granularity - -GC runs at **segment granularity**, not record granularity. A whole segment is unlinked from disk when *every* record in it has `slot < gc_slot`. This is cheap (one `unlink()` per segment) and avoids any in-place rewriting. - -- Slot-based eligibility does not retire election state. Retain a segment - proving the highest term and the segment containing the latest vote at the - highest recorded vote term, even when their slot records are below the GC - watermark. A mixed vote/data segment follows the same rule. -- Derive these election fields from every verified record during replay and - update them as live segments are written. They are in-memory index metadata; - WAL record and segment formats remain unchanged. - -### 7.3 GC trigger - -GC is triggered by: - -- Periodic tick (default 30 s): the GC worker checks each disk and unlinks eligible segments. -- Disk-pressure signal (not yet implemented): if any WAL disk's usage exceeds `wal_disk_high_watermark` (default 80%), GC would run immediately and may also force a snapshot to advance `snapshot_slot`. - -### 7.4 Interaction with leader change - -After a leader or membership change, fresh progress and a matching certificate must be established. Until then reclamation pauses. Snapshot creation remains local and can continue independently. - -### 7.5 Force-retain window - -A configurable `wal_min_retention` (default 1 hour, optional) would keep even GC-eligible segments around for a grace period to support post-incident debugging. **Not yet implemented** — the GC worker currently skips the retention check (V1 simplification). - ---- - -## 8. Disk Loss and Recovery - -### 8.1 Single-disk failure - -When the backend durable-flush operation returns an error, the OS reports the disk read-only, or repeated I/O errors exceed a threshold, the acceptor declares the disk failed: - -1. Stop using the disk for new writes (the `failed` AtomicBool is set; all subsequent `append` calls return `Err`). -2. Mark the group affected (a multi-disk WAL with one disk lost has incomplete state for slots that landed on the lost disk). -3. **Fail the node out of the group** (not yet implemented): the node would send a step-out RPC to the leader, the leader records the failed acceptor as not-eligible, and (if necessary) triggers a reconfiguration to maintain quorum. -4. Rebuild from peers via **snapshot install** ([§8.4 of design-crowdb-kv.md](design-crowdb-kv.md#84-snapshot-and-install)). - -This is the **fail-out semantics** decided in design-crowdb-kv.md §8.1](design-crowdb-kv.md). We do not try to keep operating with partial WAL state on the surviving disks; that path requires per-slot replication across disks and adds disproportionate complexity. - -### 8.2 All-disk failure - -The node is non-functional; it cannot serve any group. It must be replaced by an operator. After replacement (with empty disks) the new node bootstraps via Group-0 and snapshot installs all groups it should host. - -### 8.3 Detecting silent corruption - -CRC32C catches all common bit-flips and torn writes. Logical corruption (e.g. a slot record with valid CRC but inconsistent term/ballot relative to other state) is detected by replay sanity checks; if found, the node fails out of the affected group. - -### 8.4 Network-level data loss is not a WAL concern - -The WAL is per-node. Inter-node consistency is the consensus layer's job. If a network drops a record, the leader retransmits; if it drops repeatedly, gap repair handles it. - ---- - -## 9. Tunables and Defaults - -| Parameter | Field in `WalConfig` | Default | Notes | -| --- | --- | --- | --- | -| `wal_disks` | `wal_disks` | required | ≥ 1; more → higher throughput | -| `wal_segment_size` | `wal_segment_size` | 64 MiB | Trade GC granularity vs file count | -| `wal_flush_batch_bytes` | `wal_flush_batch_bytes` | 64 KiB | Batch cap per durable flush | -| `wal_flush_watchdog_ms` | `wal_flush_watchdog_ms` | 100 ms | Safety-net timer: wakes idle writer to drain missed wakes | -| `wal_disk_high_watermark_pct` | `wal_disk_high_watermark_pct` | 80% | Triggers eager GC (not yet implemented) | -| `wal_min_retention_secs` | `wal_min_retention_secs` | 3600 (1 h) | Forensics retention (not yet implemented) | -| `gc_tick_secs` | `gc_tick_secs` | 30 s | GC scan cadence | -| `wal_aligned` | `wal_aligned` | false | Selects block-aligned I/O backend | -| `wal_io_unit_bytes` | `wal_io_unit_bytes` | 4096 | IU size when `wal_aligned` is true | -| `wal_record_format` | `wal_record_format` | Auto | Binary (zero-copy) or text-line | -| `wal_early_ack` | `CROWConfig.wal_early_ack` | `true` (gated on `quorum > 1`) | Early-ack mode (§5.4): declare `Chosen` on remote quorum durable flush + leader CAS, deferring the leader's local WAL persist to a background spawn. Default `false` for single-node groups (quorum = 1); no survivors to re-drive a chosen-but-not-locally-durable slot. | - -**Choosing durable-flush batching:** - -- Latency-critical, low write rate → defaults (wake-drain-flush; no fixed delay). -- Throughput-oriented → `wal_flush_batch_bytes = 64–512 KiB`. -- WAN replication piggybacking → tune batching only after measuring end-to-end RTT and tail latency. - -**Choosing segment size:** - -- Few large segments → fewer files, faster replay scan. -- Many small segments → finer-grained GC, more files. -- 64 MiB is the engineering compromise. - -**Choosing disk count:** - -- 1 disk: single durable-flush limit. Fine for development. -- 2–4 disks: typical production for a write-heavy workload. -- More than 4: usually network or CPU becomes the bottleneck before more disks help. - ---- - -## 10. Follow-up Implementation Scope - -Most of the design above has been implemented. The following items are **not yet -implemented**: disk-pressure-triggered eager GC (§7.3), retention window (§7.5), -and the full fail-out procedure with step-out RPC and reconfiguration (§8.1). -Remaining pending test tasks are tracked in [`plan-test.md`](../working/plan-test.md): - -- Slot-affinity WAL placement instead of per-record round-robin. -- Backend-specific durable flush and alignment semantics. -- Event-driven wake-drain-flush workers. -- Replay-only restore for acceptor state, with learner apply gated by durable commit evidence. -- New-leader recovery ceiling, `NoOp` hole fill, and post-recovery `next_slot` seeding. -- Dedup checkpoint documentation and tests. diff --git a/doc/design/kv/design-crowdb-kv-watch-notify.md b/doc/design/kv/design-crowdb-kv-watch-notify.md deleted file mode 100644 index 0ef9f8fef..000000000 --- a/doc/design/kv/design-crowdb-kv-watch-notify.md +++ /dev/null @@ -1,895 +0,0 @@ - - - -# CROWDB - Design: crowdb-kv Watch/Notify Extension - -Depends on: [`design-crowdb-kv-rpc.md`](design-crowdb-kv-rpc.md) §3, §5, §6; [`design-crowdb-kv.md`](design-crowdb-kv.md) §3; [`design-crowdb-kv-state-machine.md`](design-crowdb-kv-state-machine.md) §2 -Satisfies: [`design-crowdb-diskdb.md`](../diskdb/design-crowdb-diskdb.md) §8 (Follow-up — group-0 notify/watch), §15 - -This document covers the crowdb-kv watch/notify extension, the diskdb -server-side notify handler, and the client endpoint cache proactive -refresh that together replace fixed-interval polling as the primary -change-detection mechanism for group-0 sysdata. The notify is a -**client-pulled `WatchNotify` bidi stream**: diskdb opens the stream to -the group-0 leader and subscribes to prefixes, and the leader pushes -notifies over that stream. No separate notify-endpoint registration is -needed for notify delivery. Polling stays as a safety net at a raised -interval. Architecture decisions and rationale live here; the root -design context is `design-crowdb-diskdb.md` §8 and -`design-crowdb-kv-rpc.md` §3. - -## Table of Contents - -- [1. WatchNotify crowdb-rpc Bidi Stream](#1-watchnotify-crowdb-rpc-bidi-stream) - - [1.1 Why](#11-why) - - [1.2 Schema](#12-schema) - - [1.3 Server-side handler](#13-server-side-handler) -- [2. Watch Registry + Apply-Path Trigger](#2-watch-registry--apply-path-trigger) - - [2.1 Why](#21-why) - - [2.2 WatchRegistry](#22-watchregistry) - - [2.3 PxLearner integration](#23-pxlearner-integration) - - [2.4 PxGroup integration](#24-pxgroup-integration) - - [2.5 Why not the proposal path](#25-why-not-the-proposal-path) -- [3. Coalescing (deferred)](#3-coalescing-deferred) -- [4. WatchNotify Client](#4-watchnotify-client) - - [4.1 Why](#41-why) - - [4.2 WatchNotifyClient](#42-watchnotifyclient) -- [5. diskdb Notify Handler + Polling Safety Net](#5-diskdb-notify-handler--polling-safety-net) - - [5.1 Why](#51-why) - - [5.2 Trigger extension](#52-trigger-extension) - - [5.3 KeepAlive changes](#53-keepalive-changes) - - [5.4 Notify handler task](#54-notify-handler-task) - - [5.5 Polling safety net](#55-polling-safety-net) -- [6. Client Endpoint Cache Proactive Refresh](#6-client-endpoint-cache-proactive-refresh) - - [6.1 Why](#61-why) - - [6.2 Design](#62-design) -- [7. Configuration](#7-configuration) - - [7.1 diskdb config](#71-diskdb-config) - - [7.2 crowdb-diskdb-client config](#72-crowdb-diskdb-client-config) -- [8. Module Structure](#8-module-structure) -- [9. Server Wiring](#9-server-wiring) -- [10. Test Design](#10-test-design) - ---- - -## 1. WatchNotify crowdb-rpc Bidi Stream - -### 1.1 Why - -crowdb-kv has no watch/notify capability without this extension. The only -change-detection mechanism available to external components is polling -(prefix scans of group 0). A push mechanism requires a long-lived -stream from the group-0 leader to each watcher, fired when a watched -prefix is written. `WatchNotify` uses a persistent crowdb-rpc connection: -the client submits a fire-and-forget subscription frame and registers push -handlers for notify and error frames on that connection. - -### 1.2 Schema - -`lib/crowdb-kv/src/rpc/fbs/kv_client.fbs` carries the watch/notify messages -and one bidi RPC on `KvService`: - -```fbs -// Client-to-leader watch subscription. The client opens a WatchNotify -// bidi stream to a node, sends WatchSubscribe frames for each -// (group_id, prefix) it wants to watch, and receives WatchNotify frames -// when watched keys change. If the node is not the leader of the -// subscribed group_id, it returns a WatchNotifyError with -// not_leader_hint and the client reconnects to the leader. -message WatchSubscribe { - uint32 version = 1; - uint64 group_id = 2; - bytes prefix = 3; // watch all keys with this byte prefix -} - -message WatchUnsubscribe { - uint64 group_id = 1; - bytes prefix = 2; -} - -// Pushed from leader to watcher when a watched key changes. Carries -// the changed keys and their latest values so the watcher can act -// without a re-read. slot = the apply slot of the triggering write. -message WatchNotify { - uint64 group_id = 1; - bytes prefix = 2; // which watched prefix matched - repeated bytes keys = 3; // changed keys (deduplicated) - uint64 slot = 4; - repeated bytes values = 5; // values[i] = latest value for keys[i] - // (empty bytes for a Delete tombstone) -} - -message WatchNotifyError { - uint64 group_id = 1; - string not_leader_hint = 2; // empty for non-leader errors - string error = 3; -} - -message WatchNotifyRequest { - oneof frame { - WatchSubscribe subscribe = 1; - WatchUnsubscribe unsubscribe = 2; - } -} - -message WatchNotifyResponse { - oneof frame { - WatchNotify notify = 1; - WatchNotifyError error = 2; - } -} -``` - -Add to the `KvService` service: - -```fbs - rpc WatchNotify(stream WatchNotifyRequest) returns (stream WatchNotifyResponse); -``` - -- **Field numbers are append-only** — matches the RPC design's - compatibility rule (`design-crowdb-kv-rpc.md` §5). -- **`bytes` fields** — `prefix` and `keys` are raw key bytes as stored - in the engine. For group-0 sysdata these are UTF-8 text paths (e.g. - `/hw/disk/1/100/5/abcd`) because `HardwareClient` puts keys via - `key.to_path().as_bytes()` and scans via `prefix.as_bytes()`. For - data groups the keys are binary-encoded (`BinaryKey::to_bytes()`). The - schema is encoding-agnostic; the subscriber and the engine agree on the - encoding because they both use the same key types. No `Bytes` mapping - needed — group-0 keys are text paths < 128 bytes; `Vec` is fine. -- **Fat notify (keys + values)** — `values[i]` is the latest value for - `keys[i]` (empty bytes for a `Delete` tombstone). `WatchRegistry::emit` - extracts the value from each `BatchOp` (`Op::Put(v)` → value bytes, - `Op::Delete` → empty) and includes it in the notify frame. The client - can act on a notify without a re-read, eliminating the - notify-storm → re-read-storm amplification. The `keys`-only contract - is preserved as a subset (clients that only need invalidation ignore - `values`). - -### 1.3 Server-side handler - -`KvRpcService` registers fire-and-forget handlers for `FBWatchSubscribe` and -`FBWatchUnsubscribe`. Subscribe resolves the group, checks leadership, and -registers the connection plus prefix in `WatchRegistry`. A non-leader sends -`FBWatchNotifyError` with the current leader hint. Apply-path notifications -are pushed with `RpcClient::send_to_handle`; disconnect cleanup removes the -connection's registrations. - -- **Leader check** — uses `local_replica.is_leader()`, the same atomic - role check the propose path uses for its leadership gate. A - non-leader node returns `not_leader_hint` per-subscribe rather than - closing the stream, so a client subscribed to multiple groups gets - hints only for the groups this node doesn't lead. -- **Leader change mid-stream** — on step-down, the old leader clears - its `WatchRegistry` (see §2.4). Dropping all `Watcher` tx senders - closes the clients' outbound streams. The client reconnects to the - new leader and re-subscribes. The safety-net polling covers the gap. -- **`not_leader_hint` source** — `PxGroup::leader_endpoint()` returns - the believed leader's endpoint (or empty if unknown). The handler - sends this as the hint, matching the `KvResponse.not_leader_hint` - pattern in the unary RPC path. - -## 2. Watch Registry + Apply-Path Trigger - -### 2.1 Why - -The notify trigger fires on the leader's **apply path** (after a -value is Paxos-chosen AND applied to the engine), not on the proposal -path. This is the etcd model: watchers are fed from the apply stream, -not the proposal path. The proposal path only fires for slots the -leader proposes itself; slots the leader learns via heartbeat catch-up -or repair never trigger a notify on the proposal path. The apply path -fires for **every** chosen slot on **every** replica, covering all -three entry points: -- `Learner::learn` — sync path: leader proposals + follower learn. -- `spawn_learn_chosen`'s spawned task — async path: deferred engine - apply. -- `apply_loop_task` — background catch-up: slots learned via heartbeat - `known_commit_slot` advance or gap-fill. - -The apply path's central function is `PxLearner::apply_entry`, which -decodes `entry.payload` via `Batch::decode` into `Batch { ops: Vec }`. -The changed keys are already extracted there, so the notify trigger -incurs no extra decode. The registry therefore lives on `PxLearner` (set -via a setter at group construction) so the trigger fires from one hook -point with no cross-struct lookup. - -### 2.2 WatchRegistry - -`lib/crowdb-kv/src/cluster/watch_registry.rs`: - -```rust -/// One watcher: a prefix + an outbound channel to push notify frames. -pub(crate) struct Watcher { - pub prefix: Vec, - pub tx: mpsc::Sender>, -} - -/// Per-group watch registry. Wired into `PxLearner` via -/// `set_watch_registry`; the learner's `apply_entry` calls `emit` -/// after each successful engine apply, gated by `has_watchers`. -pub(crate) struct WatchRegistry { - // Byte-level prefix trie (PrefixTrie): subscribe/unsubscribe/ - // remove_all take a parking_lot::RwLock write lock (rare; - // watcher setup/teardown), emit takes a read lock. emit walks - // each changed key through the trie once (O(key_len) per key), - // collecting watchers at every node whose prefix matches. - trie: PrefixTrie, - next_id: AtomicU64, - /// Atomic fast-path flag: true iff at least one watcher is - /// registered. The apply path checks this (one Acquire load) - /// before touching the trie — zero overhead when no watchers. - has_watchers: AtomicBool, - /// Counter incremented when try_send fails with Full; exposed - /// via dropped_notifies() for metrics wiring. A critical log is - /// emitted on each drop. - dropped_notifies: AtomicU64, -} - -impl WatchRegistry { - pub fn new() -> Self; - /// Register a watcher for `prefix`. Returns the `watcher_id` for - /// later removal. Sets `has_watchers = true`. - pub fn subscribe(&self, prefix: Vec, tx: mpsc::Sender<...>) -> u64; - /// Remove a specific watcher by `(prefix, watcher_id)`. Updates - /// `has_watchers` if the registry becomes empty. - pub fn unsubscribe(&self, prefix: &[u8], watcher_id: u64); - /// Remove all watchers whose `watcher_id` is in the list (stream- - /// end cleanup). Updates `has_watchers`. - pub fn remove_all(&self, watcher_ids: &[u64]); - /// Clear all watchers (leader step-down). Drops all tx senders, - /// closing client streams. Sets `has_watchers = false`. - pub fn clear(&self); - /// For a set of changed keys (from `Batch::decode`), find matching - /// prefixes and enqueue notify frames. Non-blocking: uses - /// `try_send`. - pub fn emit(&self, group_id: u64, slot: u64, changed: &[BatchOp]); - /// True if at least one watcher is registered. Atomic load — the - /// apply-path fast path. - pub fn has_watchers(&self) -> bool; - pub fn dropped_notifies(&self) -> u64; -} -``` - -- **`has_watchers` atomic fast path** — the apply-path hook checks - `registry.has_watchers()` (one `Acquire` load of an `AtomicBool`) - before decoding or matching. When no watchers are registered (the - common case in tests and non-diskdb clusters), the cost is one - predicted-not-taken branch, cheaper than a trie-emptiness scan and - the true zero-overhead gate. `has_watchers` is set to `true` on the - first `subscribe` and recomputed on `unsubscribe` / `remove_all` / - `clear`. -- **Trie-based prefix index** — `WatchRegistry` is backed by a - byte-level prefix trie (`PrefixTrie`) instead of a - `DashMap, Vec<(u64, Watcher)>>`. `emit` walks each changed - key through the trie once (`O(key_len)` per key), collecting - watchers at every node whose prefix matches, reducing the per-apply - cost from `O(watchers × keys)` to `O(key_len × keys)`. No mature - crate provided the exact "find all registered prefixes that are a - prefix of this key" API, so a hand-rolled trie was the natural - choice. -- **`emit` matching** — for each `BatchOp` in the decoded batch, walk - the key through the trie and check `op.key.starts_with(&entry.prefix)`. - Group matches by prefix, build one `WatchNotify { prefix, keys, values, slot }` - per prefix per watcher, send via `tx.try_send` (non-blocking — if the - watcher's channel is full, drop the notify; the safety-net polling - covers missed notifies). -- **`try_send` not `send().await`** — the apply path must not block on - a slow watcher. A full channel means the watcher is lagging; dropping - the notify is correct (the safety-net polling will catch up). The - drop is observable: `dropped_notifies` is incremented and a - `tracing::error!` is emitted ("critical: watch notify dropped -- - watcher channel full, client will catch up via safety-net poller"). -- **Delete ops** — `BatchOp { op: Op::Delete, key }` still carries a - key; a delete on a watched prefix notifies the watcher with the key - and an empty value (tombstone). The watcher re-reads and sees the - deletion. - -### 2.3 PxLearner integration - -`PxLearner` (`lib/crowdb-kv/src/paxos/learner.rs`) holds: - -```rust -/// Optional per-group watch registry. Set when the group is -/// constructed; None in tests that don't use watch/notify. The -/// apply-path hook in `apply_entry` checks `has_watchers()` before -/// touching the registry. RwLock> (not OnceLock) so it -/// can be re-wired when a group is rebuilt via -/// `inherit_local_state_from` — the learner is Arc::clone-shared -/// across rebuilds, but the new group has a new WatchRegistry. -watch_registry: RwLock)>>, // (group_id, registry) -``` - -- **`(group_id, registry)` tuple** — `PxLogEntry` has no `group_id` - field, and `apply_entry` takes only `(slot, &payload)`. The - `group_id` is stored alongside the registry so `emit` can populate - `WatchNotify.group_id` without threading `group_id` through every - call site. -- **Set via `set_watch_registry(group_id, Arc)`** — - called once during `PxGroup` construction (after the learner is - created) and again in `inherit_local_state_from` after constructing - the inherited replica. Tests that don't use watch/notify leave it - unset (`has_watchers()` returns false via the `None` fast path). -- **`apply_entry` hook** — after `self.engine.apply(slot, &batch).await` - succeeds, before the method returns: - -```rust -if let Some((group_id, registry)) = self.watch_registry.read().unwrap().as_ref() { - if registry.has_watchers() { - registry.emit(*group_id, slot, &batch.ops); - } -} -``` - - The `batch` is already decoded at the top of `apply_entry`, so the - keys are available with no extra decode. When `has_watchers()` is - false, the cost is one `RwLock` read + one `AtomicBool` load — zero - overhead when unused. -- **Fires on ALL apply paths** — `apply_entry` is called from `learn` - (sync), `spawn_learn_chosen` (async), and `apply_loop_task` - (catch-up). All three paths trigger the hook, covering leader - proposals, follower learn, heartbeat catch-up, and gap-fill repair: - the etcd model. -- **Followers don't emit** — followers also call `apply_entry`, but - their `WatchRegistry` is empty (cleared on step-down, or never - populated if they were never leader). `has_watchers()` returns false - → no emit. Only the leader (which holds the client streams) emits. - The registry's emptiness is the gate, so no explicit leader check is - needed in the hook. -- **Re-wire on group rebuild** — without the `RwLock>` - (vs `OnceLock`), subscribes would go to the new registry while - `emit` fired on the old (orphaned) registry, and notifies never reached - the client. `inherit_local_state_from` calls `set_watch_registry` - after constructing the inherited replica. - -### 2.4 PxGroup integration - -`PxGroup` (`lib/crowdb-kv/src/cluster/group.rs`) holds: - -```rust -pub(crate) watch_registry: Arc, -``` - -- Constructed in `PxGroup::new` as `Arc::new(WatchRegistry::new())`. - Wired into the learner via - `local_replica.learner.set_watch_registry(group_id, Arc::clone(&watch_registry))` - right after construction. -- **Cleared on step-down** — the leader step-down path is - `PxGroup::step_down`, called for `HigherTerm` / `LeaseUnrenewable` / - `Admin` reasons. `self.watch_registry.clear()` runs at the end of - `step_down` (after `become_follower`). This drops all `Watcher` tx - senders, closing the clients' outbound streams (clean reconnect). -- **Also cleared on propose-path direct step-down** — the propose path - has two direct `become_follower` calls that bypass `step_down`: - one on a higher term during prepare and one on a higher term during - accept. Both have `&self` (`PxGroup`), so `self.watch_registry.clear()` - runs before each `replica.become_follower(...)` call. This covers the - case where the leader discovers a higher term mid-proposal and steps - down without going through `step_down`. -- **Not cleared on `shutdown`** — `shutdown` drops the whole `PxGroup`, - which drops the `Arc`, which drops all watchers. - -### 2.5 Why not the proposal path - -The apply-path hook was chosen over a proposal-path hook because the -proposal path has a critical gap: it only fires for slots the leader -**proposes itself**. Three classes of chosen slots never trigger a -notify on the proposal path: - -- **Heartbeat catch-up slots** — when the leader advances - `known_commit_slot` via heartbeat and the background `apply_loop_task` - applies slots the leader accepted as a follower before winning the - election. These slots are real writes to real keys; watchers must be - notified. -- **Repair slots** — gap-fill repair calls `fan_out_chosen_notice` on - the repair path, not the propose path. A repaired slot is a real - write; watchers must be notified. -- **Foreign-value retry** — on the apply path, both the foreign value - and the retried client value fire `apply_entry` → both notify. This - is correct and automatic — no special handling needed. - -The apply-path hook covers all three with one code point. The cost is -that `apply_entry` runs on followers too (where the registry is empty), -but the `has_watchers()` atomic check makes this a single predicted- -not-taken branch, zero overhead. - -## 3. Coalescing (deferred) - -Burst writes to the same prefix (e.g. a batch of disk-status updates -via `batch_write`) generate one notify per write. A debounce window -would coalesce writes to the same prefix into one notify, reducing -watcher wakeup + re-read amplification under burst load. - -The watch/notify extension ships without coalescing. `apply_entry` -calls `registry.emit` directly (one notify per changed key per matching -prefix). Coalescing is deferred to a follow-up requirement (see -`doc/backlog/backlog.md` — watch/notify coalescing / debounce). The -safety-net poller covers any notify-drop scenarios; the missing -coalescing is a load optimization, not a correctness gap. - -## 4. WatchNotify Client - -### 4.1 Why - -diskdb (and future clients) need a reusable client that opens the -`WatchNotify` stream to the group leader, subscribes to prefixes, -and delivers notify frames via a channel: the client-side mirror of -§1's server handler. - -### 4.2 WatchNotifyClient - -`lib/crowdb-kv-client/src/watch_notify.rs`: - -```rust -/// A live watch subscription. Dropping it unsubscribes and closes the -/// stream. -pub struct WatchSubscription { - /// Notify frames for this subscription. Receiver end of an mpsc; - /// the client reads `WatchNotify` frames here. - pub notify_rx: mpsc::Receiver, - /// Internal handle to unsubscribe on drop. - inner: WatchSubscriptionInner, -} - -impl Drop for WatchSubscription { - fn drop(&mut self) { /* send WatchUnsubscribe, or drop the stream */ } -} - -/// Client for the crowdb-kv `WatchNotify` bidi stream. -pub struct WatchNotifyClient { - kv: Arc, -} - -impl WatchNotifyClient { - pub fn from_shared(kv: Arc) -> Self; - - /// Subscribe to `(store_id, group_id, prefix)`. Opens a bidi stream - /// to the group leader (discovered via the topology cache), sends a - /// `WatchSubscribe` frame, and returns a `WatchSubscription` whose - /// `notify_rx` yields `WatchNotify` frames. - /// - /// On leader-change (stream closes), the client automatically - /// reconnects to the new leader and re-subscribes; the caller's - /// `notify_rx` stays open across the reconnect. Missed notifies - /// during the reconnect gap are caught by the caller's safety-net - /// polling. - pub async fn subscribe( - &self, - store_id: u64, - group_id: u64, - prefix: &[u8], - ) -> Result; -} -``` - -a. `subscribe` resolves the group leader endpoint via `CrowdbClient`'s - topology cache (`kv.topology.leader(store_id, group_id)`, falling - back to `kv.topology.refresh()` on cache miss — same mechanism as - `Put`/`Get` via `resolve_leader`). -b. Gets a crowdb-rpc connection from the shared `ConnectionPool` - (`kv.pool.get(&endpoint)`) — reuses the existing connection pool, - no separate channel management. -c. Opens a `KvServiceClient::new(channel).watch_notify(...)` bidi - stream to that endpoint. -d. Sends `WatchSubscribe { version: 1, group_id, prefix }`. -e. Spawns a reader task that loops on the inbound stream: on - `WatchNotify` frame, forwards to the subscriber's `notify_rx`; on - `WatchNotifyError { not_leader_hint }`, refreshes topology - (`kv.topology.set_leader(store_id, group_id, hint)`), reconnects - to the new leader, re-subscribes, and continues. On stream - error/disconnect, same reconnect logic. -f. `WatchSubscription::drop` sends `WatchUnsubscribe` (if the stream - is still open) and aborts the reader task. - -- **`store_id` required** — the topology cache is keyed by - `(store_id, group_id)`. Group 0 is `(0, 0)` (`CrowdbClient::SYSTEM_STORE` - / `SYSTEM_GROUP`). The `subscribe` signature takes `store_id` so the - client can resolve the leader endpoint. -- **One stream per subscription** — simpler than multiplexing multiple - prefixes over one stream (each subscription is independent; a diskdb - instance typically watches 3 prefixes). The schema already supports - multiple `WatchSubscribe` frames on one stream, so a future - optimization can multiplex subscriptions over a shared stream with - no schema change. -- **Reconnect backoff** — capped exponential (50 ms → 2 s). -- **Proactive topology refresh on reconnect** — the reader loop calls - `topology.refresh()` unconditionally at the top of every (re)connect - iteration, before looking up the leader endpoint, instead of only - refreshing when the cached leader was `None`. This avoids connecting - to a stale leader cached before a leader change that happened during - the disconnect gap. The reactive `not_leader_hint` path is kept as a - fallback for leader changes that happen between the refresh and the - stream open. -- **`ConnectionPool` reuse** — `pool` is `pub(crate)` on `CrowdbClient`; - `WatchNotifyClient` is in the same crate, so it accesses the pool - directly. No new connection management code. - -## 5. diskdb Notify Handler + Polling Safety Net - -### 5.1 Why - -diskdb's `KeepAlive` sync loop (`liveness/keepalive.rs`) wakes on a -fixed timer (`Trigger::TimerFn`). To trigger an immediate sync on -notify, the loop must also wake on an external signal. The `Trigger` -enum has an `Event(Notify)` variant but not a hybrid timer+event. The -notify handler (a `WatchNotifyClient` reader) must wake the keepalive -without going through the timer. - -### 5.2 Trigger extension - -`Trigger` in `app/crowdb-diskdb/src/bg_task.rs`: - -```rust -pub enum Trigger { - Timer(Duration), - TimerFn(Box Duration + Send + Sync>), - Event(Arc), - /// Wake on either a timer tick (dynamic interval from config) OR - /// an external notify signal. Used by keepalive when - /// `notify_enabled` is true: the timer is the safety-net polling - /// interval, the notify is woken by the WatchNotify handler. - TimerOrEvent { - interval_fn: Box Duration + Send + Sync>, - notify: Arc, - }, -} -``` - -`wait_trigger`: - -```rust -Trigger::TimerOrEvent { interval_fn, notify } => { - tokio::select! { - () = tokio::time::sleep(interval_fn()) => {} - () = notify.notified() => {} - } -} -``` - -- **Backward-compatible** — existing tasks keep `Timer`/`TimerFn`/ - `Event`; only keepalive uses `TimerOrEvent` when `notify_enabled`. -- **`select!` is race-safe** — both branches can fire; `select!` polls - both and picks one. No panic if both are ready. - -### 5.3 KeepAlive changes - -In `app/crowdb-diskdb/src/liveness/keepalive.rs`: - -a. Field `sync_trigger: Option>` — `Some` when - `notify_enabled`, `None` otherwise (keeps the struct usable in tests - without notify). -b. Builder method `.with_sync_trigger(notify: Arc) -> Self`. -c. In `trigger()` (the `BackgroundTask` impl): if `sync_trigger` is - `Some`, return `Trigger::TimerOrEvent { interval_fn, notify: - sync_trigger }`; else fall back to the existing `TimerFn` path. -d. Method `pub fn trigger_now(&self)` — calls - `self.sync_trigger.notify_one()` if set. Called by the notify - handler on each `WatchNotify` frame. -e. **`rpc_endpoint` already populated** — `main.rs` calls - `.with_rpc_endpoint(config.load().server.listen_addr.clone())`, - and `keepalive.rs` passes `self.rpc_endpoint` to `heartbeat_diskdb` - on every tick. The endpoint registration (item 1 of the original - scope) is already done; no change needed here. - -### 5.4 Notify handler task - -`app/crowdb-diskdb/src/liveness/notify.rs`: - -```rust -/// diskdb-side watch/notify handler. Subscribes to group-0 prefixes -/// and wakes the keepalive sync loop on notify. -pub struct NotifyHandler { - watch: WatchNotifyClient, - keepalive_trigger: Arc, - /// (store_id, group_id, prefix) triples to subscribe to. - subscriptions: Vec<(u64, u64, Vec)>, -} - -impl NotifyHandler { - pub fn new( - watch: WatchNotifyClient, - keepalive_trigger: Arc, - ) -> Self; - - /// Open subscriptions and loop on notify frames. Each frame wakes - /// the keepalive via `keepalive_trigger.notify_one()`. Runs until - /// the stop signal. - pub async fn run(self, stop: tokio::sync::Notify); -} -``` - -a. On `run`, subscribe to the group-0 prefixes diskdb cares about. - The prefixes are **text-encoded** (matching the engine's group-0 - key encoding — `HardwareClient` puts/scans text paths), centralized - as `DISKDB_WATCH_PREFIXES` in `crowdb-protocol` (`key/diskdb.rs`): - - `/hw/dg_owner/` (ownership map — global; `list_owners` scans - globally and filters to this instance) - - `/hw/dg_bind/` (binding map — global; `list_binds` scans globally) - - `/hw/disk/` (disk metadata + status — global; `observe_disks` - scans per owned disk-group, but the notify is a coarse trigger) - All three use `(G0_STORE, G0_GROUP)` = `(0, 0)`. Adding a new - keepalive-relevant group-0 key kind requires updating only the - constant in `crowdb-protocol`, not the diskdb handler. -b. For each subscription, spawn a reader loop: on `WatchNotify` frame, - call `keepalive_trigger.notify_one()` (wakes keepalive → - `KeepAlive::tick()` → re-reads the changed keys from group 0). -c. The notify is a **trigger only** — diskdb re-reads the actual values - via the normal sync path (`observe_ownership`, `observe_disks`). - This keeps the notify payload small and avoids a duplicated read - path. -d. On subscription drop/reconnect, the `WatchNotifyClient` handles - reconnect internally (§4.2); the handler just keeps reading. -e. On `stop.notified()`, drop all subscriptions (which closes the - streams) and exit. - -- **Global prefixes, not per-node** — `list_owners()` / `list_binds()` - scan the global `/hw/dg_owner/` / `/hw/dg_bind/` prefixes and filter - to `instance_id == self.instance_id`. Watching the global prefix - means the keepalive wakes on any ownership/bind change anywhere, then - re-scans and filters to this instance. An irrelevant change (another - instance's ownership) causes a cheap no-op re-scan. Per-node prefixes - would be more selective but require knowing `rack_id`/`node_id` - before the first sync. The diskdb discovers its node identity from - the ownership map, not from config, so global prefixes avoid this - bootstrapping dependency. -- **Not a `BackgroundTask`** — `NotifyHandler` is a long-lived task - with its own `run` method, not a `BgRunner` cycle task. It is spawned - directly in `main.rs` alongside the bg runner, with the same stop - signal. This avoids forcing the notify loop into the - trigger→cycle→trigger model (it is event-driven, not periodic). -- **`StopHandle::notified()` helper** — supports long-lived tasks that - don't follow the trigger→cycle model. - -### 5.5 Polling safety net - -When `notify_enabled` is true: -- The keepalive timer interval (from `sync.sync_interval_secs`) is the - safety-net polling interval. The config default rises (e.g. 60 s) - when notify is on; the operator sets it explicitly. -- The timer catches any missed notifies (reliability fallback): if a - notify is dropped (full channel) or the WatchNotify stream is - disconnected, the next timer tick still syncs. -- When `notify_enabled` is false (default), behavior is unchanged — - fixed-interval polling at `sync_interval_secs` (default 10 s), no - notify handler, no `WatchNotifyClient`. - -## 6. Client Endpoint Cache Proactive Refresh - -### 6.1 Why - -The `DiskdbClient` endpoint cache (`lib/crowdb-diskdb-client/src/client.rs`) -refreshes **on demand**: at startup, on cache miss, and on error-retry -(`refresh_endpoints` reads `read_all_diskdb_instances`). When a diskdb -instance moves (endpoint change in its service-registry record at -`/srv/diskdb/`), a client holding the old endpoint cache -entry fails one `allocate_blocks`/`free_blocks` attempt, triggers -`refresh_endpoints` (on-demand safety net), and retries. This section -extends that to **proactive** refresh: the client subscribes to the -`/srv/diskdb/` prefix via `WatchNotifyClient`; on a notify (instance -register/deregister/move), it refreshes the affected cache entries so -the next call routes to the new endpoint without a failed attempt + -retry. The on-demand refresh remains as the safety net; proactive -refresh is an optimization, not a correctness requirement. - -### 6.2 Design - -Add a `WatchNotifyClient` subscription to `DiskdbClient`: - -a. In `DiskdbClient::new` (or a new `with_watch_notify` builder), if - a `WatchNotifyClient` is provided, subscribe to - `(G0_STORE, G0_GROUP, "/srv/diskdb/")`. -b. Spawn a reader task: on `WatchNotify` frame, call - `self.refresh_endpoints()` (the existing method). The notify is a - coarse trigger — the refresh re-reads ALL diskdb instances from - group 0 and updates the cache. This is the same "trigger, not - transport" model as the diskdb notify handler (§5.4). -c. The reader task runs for the client's lifetime. On stream close - (leader change), `WatchNotifyClient` reconnects automatically - (§4.2); the reader task just keeps reading. -d. If no `WatchNotifyClient` is provided (default), the client uses - on-demand refresh only (unchanged behavior). - -- **`/srv/diskdb/` prefix** — the service-registry instance key path - is `/srv/diskdb/` (`InstanceKey::to_path()`). The global - prefix `/srv/diskdb/` covers all diskdb instances. - `ServiceRegistryClient::read_all_diskdb_instances` scans this prefix. -- **On-demand refresh stays as safety net** — if a notify is missed - (stream disconnected, channel full), the next `allocate_blocks` to a - moved instance fails once, triggers `refresh_endpoints`, and - retries. The cache is eventually consistent either way. - -## 7. Configuration - -### 7.1 diskdb config - -`app/crowdb-diskdb/src/ddb_config.rs`: - -```rust -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct NotifyConfig { - /// static: enable watch/notify (default: false). When true, - /// diskdb subscribes to group-0 prefixes and the keepalive timer - /// serves as a safety-net poller at `sync.sync_interval_secs`. - pub notify_enabled: bool, -} - -impl Default for NotifyConfig { - fn default() -> Self { - Self { notify_enabled: false } - } -} -``` - -- `pub notify: NotifyConfig` on `DdbConfig`. -- `validate()`: if `notify_enabled` and `sync.sync_interval_secs == 0`, - error (same existing check; no new constraint). - -### 7.2 crowdb-diskdb-client config - -An optional `watch_notify` toggle on `DiskdbClient`'s config (or -builder): - -```rust -/// When true, the client subscribes to `/srv/diskdb/` and proactively -/// refreshes the endpoint cache on notify. Default: false (on-demand -/// refresh only). -pub watch_notify_enabled: bool, -``` - -- **Default false** — proactive refresh is an optimization; the default - ships with on-demand refresh. Operators opt in per client. - -## 8. Module Structure - -``` -lib/crowdb-kv/src/ - rpc/ - fbs/kv_client.fbs # WatchNotify messages + RPC - kv_service.rs # watch_notify handler - cluster/ - watch_registry.rs # WatchRegistry, Watcher, PrefixTrie - group.rs # watch_registry field, wire into learner, clear on step_down - group_election.rs # clear registry in step_down - group_propose.rs # clear registry before direct become_follower calls - mod.rs # export watch_registry - paxos/ - learner.rs # watch_registry RwLock