diff --git a/README.md b/README.md
index 6d881e902..2e6cc70e7 100644
--- a/README.md
+++ b/README.md
@@ -16,10 +16,10 @@ SPDX-License-Identifier: Apache-2.0
- 18-operation native CLI ·
+ native CLI and 18-tool Workbench contract ·
direct Python and Rust SDKs ·
commit, snapshot, and restore ·
- conditional publication and exact replay
+ durable append identity and recovery
@@ -49,7 +49,7 @@ SPDX-License-Identifier: Apache-2.0
> [!IMPORTANT]
-> This README describes the current `main` contract. An installed release may
+> This README describes the checked-in source contract. An installed release may
> pin different component revisions; run `nokv version --json` to identify the
> exact NoKV commit, Workbench schema, and
> [Holt](https://github.com/NoKV-Lab/holt) build. Implemented and
@@ -90,7 +90,10 @@ same conformance and fault evidence:
only when their revision, path head, indexes, references, and receipt commit.
- **No silent overwrite:** create, replace, edit, append, rename, and remove
are explicit and generation-fenced. A stale writer gets a conflict.
-- **Exact retry:** request identity and operation records distinguish
+- **Durable append identity:** save one logical operation ID and its inputs before
+ dispatch. A replacement worker can recover the original receipt after a lost
+ reply, or finish a safely cleaned attempt under that same ID.
+- **Explicit outcomes:** request identity and operation records distinguish
definitely-not-applied, applied-with-lost-response, and unknown outcomes.
- **Durable decision points:** commits retain an immutable revision closure;
restore builds a hidden same-root incarnation and publishes it atomically.
@@ -106,7 +109,8 @@ same conformance and fault evidence:
| --- | --- | --- |
| Persist one run | `workbench_create`, `workbench_put_file`, `workspace-path append`, `workbench_edit`, `workbench_commit` | Five-section workspace, explicit write modes, immutable revisions, and a sealed run manifest |
| Reopen work from another process | `workbench_list`, `workbench_stat`, `workbench_read`, `workbench_find` | Path-shaped discovery and verified reads without retaining the original sandbox |
-| Prevent stale or duplicate writes | generations, explicit request identity, exact replay | Typed conflict for stale state; the same outcome for an exact retry after response loss |
+| Prevent stale writes | generations and explicit write modes | Typed conflict when another writer has advanced the path |
+| Redeliver one event after a worker dies | `workspace-path append --operation-id`, `operation status/inspect/recover` | One saved action, one committed append receipt; inspect and resume recovery across processes |
| Search many artifacts or runs | `workbench_grep`, `workbench_search`, `workbench_aggregate`, `workbench_catalog`, `workbench_find` | Literal body search plus typed indexed metadata query and aggregation |
| Freeze a short-lived view | `workbench_snapshot`, `workbench_snapshot_renew`, `workbench_snapshot_list` | Leased point-in-time reads with explicit lifecycle state |
| Create a durable replay point | `workbench_commit` then `workbench_restore` | Retained revision closure and atomic restore into a new destination |
@@ -152,8 +156,9 @@ harness can use its lifecycle primitives through the native CLI or SDK:
snapshots provide shorter-lived point-in-time recovery and inspection.
- **One recoverable workspace:** verified materialize/collect, indexed search,
immutable lineage, and retention-aware GC share one workspace contract. The
- Rust SDK additionally exposes change polling; quarantine and reconciliation
- remain internal lifecycle mechanisms rather than CLI commands.
+ Rust SDK additionally exposes change polling. Stable append exposes native
+ `operation status`, `inspect`, and `recover` commands; other lifecycle
+ quarantine and reconciliation mechanisms remain internal.
The harness still decides which state is authoritative and when execution may
resume. Restoring state does not grant a replacement worker permission to act;
@@ -207,7 +212,7 @@ The names and normalized input schemas below are fixed by the
| --- | --- |
| `workbench_create` | Create one jailed Workbench with `input`, `scripts`, `outputs`, `logs`, and `metadata` sections. |
| `workbench_put_file` | Publish create-only or replace-only bytes; it is never upsert. |
-| `workbench_append` | Append through immutable publication plus generation CAS and bounded conflict retry. |
+| `workbench_append` | Append through immutable publication, generation CAS and bounded conflict retry within one invocation. Use `workspace-path append --operation-id` for durable redelivery across invocations. |
| `workbench_edit` | Replace exact UTF-8 text, revalidate after conflicts, and avoid a new revision for a byte-identical result. |
| `workbench_list` | List direct children with scope-bound cursors at live state or a leased snapshot. |
| `workbench_stat` | Read a compact artifact or implicit-prefix card without loading the body. |
@@ -246,101 +251,72 @@ real-service acceptance gate is still **not qualified**.
| `nokv serve` | Start one explicit metadata owner from create, same-namespace reopen, or recovery-log state. | Shared recovery publication is opt-in and not currently qualified. |
| `nokv mcp` | Deprecated stdio transport retained only because qualification runners still use it. | **Unsupported for integration.** |
-For an append that must survive a lost reply or a harness restart, save one
-operation id before sending the request, then reuse it with the same inputs:
+### Redeliver an append after a worker restart
-```shell
-nokv [route/agent/object options] workspace-path append run-42 logs events.jsonl \
- --operation-id aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa \
- --expected-workspace-incarnation-id bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb \
- --text 'step-7-done' --content-type text/plain
-```
-
-Replace the example ids with the caller's stable logical operation id and the
-intended workspace's incarnation. On the first call, omitting the incarnation
-observes and fences the existing workspace. On recovery, omission resolves the
-original durable operation first, even if the workspace was deleted or recreated.
-Append never creates a missing workspace.
-
-Use exactly one of `--text`, `--base64`, or `--file`. Text defaults to
-`text/plain; charset=utf-8` on creation; binary and file input default to
-`application/octet-stream`. Existing content type is inherited unless
-`--content-type` overrides it. Explicitly use the same type when replaying across
-text, file, and Python inputs. The delta limit is 16 MiB; `--max-artifact-bytes`
-may lower it. The resulting body also defaults to a 16 MiB limit;
-`--max-logical-size` raises that bound and is part of the stable intent.
-`--block-size` defaults to 4 MiB and matches Python's `block_size` argument;
-retain the same value across retries and surface changes.
-
-The receipt identifies both the logical `operation_id` and its successful
-`publication_operation_id`. Its revision, generation, workspace revision, size,
-digest, and incarnation describe the original publication, even if another
-writer has advanced or removed the path. `replayed` and nullable `commit_version`
-describe this SDK call and are not stable receipt fields.
-
-Query a result without resending the delta or contacting the object store:
-
-```shell
-nokv [route/agent options] operation status aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
-```
+A queue consumer may append an event successfully, then die before saving its
+acknowledgement. Give that business action a durable identity: save its root,
+operation ID, destination, exact bytes and append options **before dispatch**.
+The replacement worker uses the saved ID and the same inputs. NoKV returns the
+original committed receipt without appending the event again. A different
+business action uses a new ID, even if its bytes are identical.
-The response supplies a machine-readable recovery action:
+For a provisioned deployment and an existing `run-001`, this command sends a
+previously saved delivery. The environment variables identify that saved job;
+generate neither a new ID nor new bytes when retrying it. Object settings must
+match the provisioned namespace, including its root and region.
-| `state` | `next_action` | Caller action |
-| --- | --- | --- |
-| `committed` | `none` | Consume the original `receipt`. |
-| `pending` | `poll` | Query the same logical identity again after waiting. |
-| `ready_to_retry` | `resubmit_same` | Resubmit the original append inputs under the same logical identity. |
-| `quarantined` | `retry_cleanup` | Inspect the failure, restore provider availability, then ask the owner to retry cleanup. |
+```bash
+nokv \
+ --root-id "$NOKV_ROOT_ID" \
+ --agent-id "$NOKV_AGENT_ID" \
+ --workbench-root /agents/research/wb \
+ --etcd-endpoint "$NOKV_ETCD_ENDPOINT" \
+ --object-bucket "$NOKV_BUCKET" \
+ --object-endpoint "$NOKV_OBJECT_ENDPOINT" \
+ --object-root "$NOKV_OBJECT_ROOT" \
+ --object-region "$NOKV_OBJECT_REGION" \
+ workspace-path append run-001 logs events.jsonl \
+ --operation-id "$APPEND_OPERATION_ID" \
+ --file "$APPEND_DELTA_FILE" --content-type application/x-ndjson
+```
-Status also retains the current physical attempt's `cause_code`,
-`failure_message`, and structured `attempt_failure`, including when the logical
-operation is `ready_to_retry`.
+The CLI binds the workspace incarnation on first admission and resolves that
+saved incarnation on retry; it never creates a missing workspace. Specify
+`--expected-workspace-incarnation-id` when the application already owns that
+identity. Retain the exact content-type choice, block size and logical-size
+limit with the job. In particular, changing an omitted content-type override
+to an explicit one can change the intent even if the displayed type is equal.
-Inspect and recover without object credentials:
+After an uncertain outcome, query by the **same root and logical ID**:
-```shell
-nokv [route/agent options] operation inspect aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa --limit 32
-nokv [route/agent options] operation recover aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa \
- --expected-state-digest
+```bash
+nokv \
+ --root-id "$NOKV_ROOT_ID" \
+ --agent-id "$NOKV_AGENT_ID" \
+ --etcd-endpoint "$NOKV_ETCD_ENDPOINT" \
+ operation status "$APPEND_OPERATION_ID"
```
-Inspection returns `operation_token` and `publication_token`, each with
-`operation_id` and `state_digest`, plus the object namespace, `registered_count`,
-`cleanup_cursor`, `remaining_count`, and `entries`. Entries describe retained
-staged objects, not a historical inventory of retired keys. Each entry includes
-its sequence, identity, expected length, digest, and optional multipart token.
-The page limit defaults to 32 and may range from 1 to 192. Continue with the
-opaque base64 `next_cursor` using `--cursor`; a stale cursor returns
-`Conflict/OperationState`, requiring a fresh inspection instead of mixing pages.
-
-Save the logical token before requesting recovery. Reuse its state digest after
-a lost reply or process restart. The owner performs sealing; this command never
-resends the append delta. `requested=true` means a cleanup request was durably
-accepted, including a replay. Its historical `recovery_receipt` contains the
-logical id, publication id, cleanup retry count, and expected state digest.
-The top-level status is a fresh observation and may be newer than that receipt;
-acceptance is not cleanup completion. Omitting the digest targets the current
-state once and is a no-op for pending, cleaned, or committed operations. A later
-quarantine needs an intentional new request with a newly inspected token.
-`AppendCleanupUnresolved` preserves the saved digest, any known recovery receipt,
-and `next_action=retry_same_cleanup`; generic retry handlers must not choose a
-new token. After `ready_to_retry`, resubmit the original append inputs.
-
-An interrupted publication must be fenced and cleaned before a successor
-attempt can start. Status queries only observe this lifecycle. Persist the delta
-or a way to reproduce the exact bytes until the operation commits. An unknown
-result or `NotFound` observation is not proof that a concurrent request cannot
-commit; errors preserve the logical id and use `next_action=query_same`.
-`cause_code=RequestReplayMismatch` means the identity belongs to different
-inputs or a different lifecycle: recover the original intent before retrying.
-Never generate a replacement id merely to bypass an uncertain outcome.
-
-The older `workbench_append` tool retains per-invocation identities and cannot
-recover one action across separate processes. Use `workspace-path append` for
-new retrying harness integrations. The fixed 18-tool contract remains unchanged.
-The [append product specification](docs/development/append-product-spec.md)
-defines the supported recovery, size, retention, and release boundaries.
+| Observation | Next action |
+| --- | --- |
+| `committed` | Save the original `receipt` durably, then acknowledge the delivery. |
+| `pending` | Poll the same ID. |
+| `ready_to_retry` | Resubmit the saved inputs with the same ID. |
+| `quarantined` | Inspect, save the logical state token, restore provider availability, and request owner cleanup with that token. |
+| Uncertain error or `NotFound` | Retain the job and ID; this observation alone cannot rule out an in-flight commit. |
+
+Status and inspection use metadata only. Recovery also takes no caller payload
+or object credentials, but the owner needs access to the object provider to
+finish cleanup. A recovery admission receipt is not an append commit receipt.
+Historical append receipts do not pin historical artifact bytes.
+
+The [durable append guide](docs/append.md) covers a complete CLI/Python delivery
+record, error handling, `inspect` pagination, exact-token `recover`, size limits
+and deployment prerequisites. The [product specification](docs/development/append-product-spec.md)
+defines the storage and retention guarantees; the [qualification record](docs/development/append-qualification.md)
+identifies the executed crash and redelivery tests. The fixed 18-tool
+`workbench_append` facade has no caller-supplied logical identity and is not the
+entrypoint for this cross-process guarantee.
### Programmatic and non-CLI capabilities
@@ -353,7 +329,7 @@ do not expose identical method shapes.
| Python adapters | Workbench-scoped fsspec, checkpoint helpers, and optional torch Distributed Checkpoint reader/writer | Bounded to an explicit Workbench; not arbitrary-root POSIX or FUSE |
| Rust `WorkspaceClient` | Lower-level typed workspace, publication, query, lifecycle, routing, and batch-range workflows | Recommended when the caller must own typed retry and recovery integration |
| Rust-only extensions | Polling change feed, generic custom-index registration, raw non-append operation status, and phased publish/restore primitives | No native CLI or Python method today; change feed is polling, not push |
-| Metadata and server lifecycle | snapshot reap, commit/tag holds, reference-fenced GC, quarantine reconciliation, Holt reopen, and optional shared recovery records | Internal/operator mechanisms, not independent end-user commands |
+| Metadata and server lifecycle | snapshot reap, commit/tag holds, reference-fenced GC, generic quarantine reconciliation, Holt reopen, and optional shared recovery records | Internal/operator mechanisms; stable append cleanup separately has public status/inspect/recover entrypoints |
Python `append_bytes`, `operation_status`, `operation_inspect`, and
`operation_recover` share the native stable append contract. Python inspection
@@ -473,7 +449,17 @@ The labels below mean exactly:
- **Live-qualified:** the supported entrypoint ran against the required real
dependencies at the pinned revision and produced accepted evidence.
-Evidence snapshot:
+The [stable append qualification record](docs/development/append-qualification.md)
+adds feature-specific evidence at NoKV `13590c1`: 1,158 Rust tests passed
+(11 ignored), 34 Python SDK tests passed, local real-service append/recovery
+matrices completed, and a Linux demo consumer passed the SIGKILL red/green
+redelivery gate. The remote [Rust](https://github.com/NoKV-Lab/NoKV/actions/runs/35992447765),
+[Python wheel](https://github.com/NoKV-Lab/NoKV/actions/runs/35992447625), and
+[amd64/arm64 image](https://github.com/NoKV-Lab/NoKV/actions/runs/35992447760)
+checks also passed. This qualifies the documented append scenarios, not the
+complete 18-tool or installed-Python Gate 0 surface or a new release.
+
+Earlier general-surface evidence snapshot (the counts below are historical):
- NoKV commit [`0f1995ebee96`](https://github.com/NoKV-Lab/NoKV/commit/0f1995ebee96048e5d4f9d4745d84c3518c64351);
- pinned Holt `0.8.6` dependency.
@@ -488,7 +474,7 @@ Evidence snapshot:
| Holt 0.8.6 | 705 listed entries; 10 marked ignored; all executed entries passed; the NoKV adapter adds 32 unit and one reopen/conformance integration test | Layer-tested for the embedded local boundary |
| Holt third-party A/B | the current v0.8.6 Holt/RocksDB/SQLite/sled comparator target was attempted; local `librocksdb-sys`/`libclang` build failed before measurement | No result; no performance claim |
-The 10 ignored NoKV workspace tests are six real-etcd tests, two direct object
+The 10 ignored tests in that earlier snapshot are six real-etcd tests, two direct object
provider tests, and two Python S3-admission tests. They are gaps, not passes.
See [Workspace acceptance](docs/development/workspace-acceptance.md) for the
normative gate and [Benchmarks and evidence](docs/benchmarks.md) for evidence
@@ -591,7 +577,7 @@ Credentials use the provider's normal chain or
agent skill / harness embedded Python native control plane
| | |
v v v
- 18-tool CLI facade direct Python API lower-level Rust SDK
+ native CLI direct Python API lower-level Rust SDK
+------------------------+-------------------------+
|
typed NoKV protocol
@@ -616,6 +602,9 @@ immutable revisions are visible and retained by a workspace.
## Documentation
- [Documentation index](docs/index.md)
+- [Durable append guide](docs/append.md)
+- [Append product specification](docs/development/append-product-spec.md)
+- [Append qualification record](docs/development/append-qualification.md)
- [Product design](docs/product-design.md)
- [Architecture](docs/architecture.md)
- [Workbench contract](docs/workbench-contract.md)
diff --git a/crates/nokv-client/src/lib.rs b/crates/nokv-client/src/lib.rs
index 873cdd543..088224aca 100644
--- a/crates/nokv-client/src/lib.rs
+++ b/crates/nokv-client/src/lib.rs
@@ -8,6 +8,13 @@
//! The SDK owns route refresh, exact request replay, retry policy, and typed
//! result extraction. Metadata layout and shard-owner implementation remain
//! outside this crate.
+//!
+//! For an action that must survive caller replacement, persist the root, logical ID,
+//! full `IdempotentAppendOptions` and exact delta before calling
+//! `WorkspaceClient::append_artifact_idempotent`. Its status, inspection and
+//! exact-token recovery APIs share the durable append lifecycle. The lower-level
+//! `append_artifact` API and the fixed Workbench facade are not substitutes for
+//! saving that business identity. A replayed receipt does not retain old bytes.
mod append_operations;
mod artifact;
diff --git a/crates/nokv-python/README.md b/crates/nokv-python/README.md
index 97beb3132..e5dcd62d8 100644
--- a/crates/nokv-python/README.md
+++ b/crates/nokv-python/README.md
@@ -62,74 +62,13 @@ not for installing the SDK.
`expected`) with nothing written. Callers that omit the argument keep the
0.11.0 behaviour; a server older than 0.11.1 rejects a fenced request as an
invalid argument instead of ignoring the fence.
-- `Client.append_bytes(workbench, path, data, operation_id, ...)` accepts Python
- `bytes` under a caller-owned stable logical identity. The first admission
- requires an existing workspace. An omitted `expected_workspace_incarnation_id`
- first resolves a recorded operation's original incarnation, then observes the
- live workspace only when no operation was found. Explicit incarnation fences
- are always checked. Save the identity before the first call and reuse the
- exact inputs after a lost reply.
- The result matches native `workspace-path append`: `status`, `operation`,
- `state=committed`, `next_action=none`, logical `operation_id`, successful
- `publication_operation_id`, `workbench_id`, `path`, `artifact_revision_id`,
- `generation`, `workspace_revision`, `logical_size`, `body_digest`, and
- `workspace_incarnation_id`. These describe the original publication.
- `replayed` and nullable `commit_version` are call metadata.
- `content_type=None` inherits an existing artifact's type and uses
- `application/octet-stream` on creation; an explicit type applies to both.
- Use an explicit matching type when replaying between Python and CLI text.
- The delta limit is 16 MiB, checked before copying the Python bytes.
- `max_logical_size=None` means a 16 MiB resulting-body limit; an explicit larger
- bound is part of the intent and must be retained across retries. The default
- block size is 4 MiB and maps to CLI `--block-size`. A retry may start a successor publication only after the
- predecessor is fenced and cleaned; the logical identity never changes.
-- `Client.operation_status(operation_id)` queries an append without resending
- its payload or depending on a current workspace. `Client(root_id, routing)`
- needs no object-store configuration; configured object stores are initialized
- lazily on object operations, so a fresh status client works during an S3
- outage. The result reports `state` and `next_action` from the shared Rust SDK:
- `committed/none` with the original `receipt`, `pending/poll`,
- `ready_to_retry/resubmit_same`, or `quarantined/retry_cleanup`. It also
- contains the logical and publication identities, attempt number and phase,
- activity deadline, original target and incarnation, progress, `cause_code`,
- `failure_message`, and `attempt_failure` for a failed physical attempt. The
- logical operation may still be `ready_to_retry` while retaining that failure.
- A query never advances an attempt. Retain or reproduce the original delta
- until commitment; follow `resubmit_same` using `append_bytes`.
-- `Client.operation_inspect(operation_id, *, cursor=None, limit=32)` returns the
- same status plus logical `operation_token`, child `publication_token`, object
- namespace, `registered_count`, `cleanup_cursor`, `remaining_count`, and a page
- of retained staged `entries`. Each entry has `sequence`, `object_identity`,
- `expected_length`, `expected_digest`, and optional `multipart_token`. Retired
- keys are not a historical inventory. The limit is 1 to 192; `next_cursor` is
- opaque `bytes` or `None`. Continue with the unchanged cursor. A changed state
- returns `Conflict`; restart inspection rather than combining different pages.
-- `Client.operation_recover(operation_id, expected_state_digest=None)` asks the
- owner to retry quarantined cleanup. Both inspection and recovery are metadata
- only and work with `Client(root_id, routing)` without S3 credentials. Persist
- the inspected `operation_token["state_digest"]` (a lowercase hex string) and
- pass it on every retry of one recovery request. `requested=True` means durable
- acceptance, including replay; `recovery_receipt` preserves that round's
- logical id, publication id, cleanup retry count, and expected state digest.
- Top-level status is a fresh observation and may be newer than the receipt.
- Acceptance is not cleanup completion. Without a digest, the method targets
- the current state once and is a no-op for pending, cleaned, or committed
- operations. A later quarantine requires a new intentional recovery request.
- When status reaches `ready_to_retry`, redeliver the original append inputs;
- recovery does not reconstruct or publish the delta.
-- Append and status failures carry `operation_id`, observed `state`, `code`,
- expected incarnation in `expected`, `cause_code`, and `next_action=query_same`.
- `publication_operation_id` is `None` until status supplies an observed
- publication. `AppendError` is a `RuntimeError`; incarnation conflicts retain
- the `WorkspaceIncarnationMismatch` subtype with the same recovery attributes.
- `cause_code=RequestReplayMismatch` identifies changed inputs or another
- lifecycle, so recover the original intent instead of blindly resubmitting.
- `retryable` is false for generic retry handlers. An unknown result or a
- `NotFound` observation never authorizes a replacement logical identity.
- Cleanup errors use `code=AppendCleanupUnresolved` and
- `next_action=retry_same_cleanup`, retain `expected_state_digest`, and expose
- any known `recovery_receipt` and its publication id. Retry using that exact
- digest; do not automatically choose a fresh token after a lost reply.
+- `Client.append_bytes(workbench, path, data, operation_id, ...)` appends Python
+ `bytes` under a caller-persisted logical identity. `operation_status`,
+ `operation_inspect`, and `operation_recover` query its outcome and request
+ owner-executed cleanup without payloads or object credentials. These methods
+ share the native CLI's durable append state machine; see
+ [Durable append](../../docs/append.md) for the CLI-first guide, a shared
+ persisted-input example, all defaults and result fields, and recovery steps.
- `WorkbenchFileSystem` is an fsspec compatibility adapter bound to one explicit
Workbench. Paths must be one of `input`, `scripts`, `outputs`, `logs`, or
`metadata`, optionally followed by an artifact-relative path. Sections and
@@ -151,8 +90,86 @@ Snapshots require a committed Workbench. Commit and restore must be driven by
the canonical Workbench lifecycle facade; clients must not synthesize a
run-manifest or duplicate the durable workflow locally.
-Stable append's size, retention, owner recovery, and release contract is specified
-in [the append product specification](../../docs/development/append-product-spec.md).
-The fsspec adapter's append mode remains outside this contract; retrying harnesses
-should use native `nokv workspace-path append` first, or `Client.append_bytes`
-when embedded Python execution is required.
+## Durable append in Python
+
+These methods describe the checked-in API. Do not assume the example `0.11.0`
+release wheel above includes them. A main-branch merge does not publish a new
+wheel. Use a matching qualified source build, or a release that explicitly
+includes durable append; record the exact commit/asset identity rather than
+only `nokv.__version__`. Source builds and the owner must satisfy the same
+[format and release boundary](../../docs/development/append-product-spec.md).
+
+| Method | Input and result |
+| --- | --- |
+| `append_bytes(workbench, path, data, operation_id, content_type=None, block_size=4194304, max_logical_size=None, expected_workspace_incarnation_id=None)` | `data` is `bytes`; `path` includes the section, such as `logs/events.jsonl`. Returns the original committed receipt. |
+| `operation_status(operation_id)` | Metadata-only observation with `state`, `next_action`, and nullable historical `receipt`. |
+| `operation_inspect(operation_id, *, cursor=None, limit=32)` | Metadata-only retained staging page. `next_cursor` is opaque `bytes`; limits are 1–192. |
+| `operation_recover(operation_id, expected_state_digest=None)` | Ask the owner to retry quarantined cleanup. The digest is a saved 64-character lowercase hex string. Acceptance is not completion. |
+
+The following reads a previously persisted application intent. It needs a
+reachable, provisioned root and etcd endpoint, but no `ObjectStoreConfig`:
+
+```python
+import json
+import os
+from pathlib import Path
+
+from nokv import Client, RoutingConfig
+
+job = json.loads(Path("append-intent.json").read_text())
+routing = RoutingConfig.etcd(
+ [os.environ["ETCD_ENDPOINT"]],
+ key_prefix=os.environ.get("ETCD_KEY_PREFIX", "/nokv/control"),
+)
+client = Client(job["root_id"], routing)
+status = client.operation_status(job["operation_id"])
+print(status["state"], status["next_action"], status["receipt"])
+```
+
+See the [complete input record and CLI/Python submission example](../../docs/append.md)
+for a new append or redelivery. Save the root, logical ID, exact bytes, target,
+content-type policy, block size, and resulting-size limit before dispatch.
+The first admission needs an existing workspace. If incarnation is omitted,
+the SDK resolves an admitted operation's original incarnation before consulting
+the live workspace name. Explicit fences are never replaced silently.
+
+The delta limit and default total-body limit are each 16 MiB; increasing
+`max_logical_size` does not increase the delta limit. `content_type=None`
+inherits an existing type and defaults to `application/octet-stream` on
+creation. An explicit type applies to both and is a different intent policy
+from omission, even when the strings match the current file. For CLI/Python
+redelivery, use the same explicit type from the first call. Keep block size and
+all other options identical. Existing producer, manifest identity, and index
+fields are inherited.
+
+Follow `committed/none`, `pending/poll`, `ready_to_retry/resubmit_same`, or
+`quarantined/retry_cleanup`. Queries do not advance an attempt. Inspection's
+cursor continues one exact observation; its logical
+`operation_token["state_digest"]` identifies a cleanup recovery request.
+Persist that digest before `operation_recover` and reuse it after an uncertain
+reply. A later quarantine needs an intentional new recovery round. Once cleanup
+finishes, redeliver the original append input; cleanup cannot reconstruct it.
+
+Runtime append errors expose `operation_id`, optional raw `state`, `code`,
+`cause_code`, `next_action`, and `retryable=False`. `AppendError` and
+`WorkspaceIncarnationMismatch` are separate `RuntimeError` subtypes; catch both
+when handling fenced append errors. Local argument type/hex errors may instead
+be `TypeError` or `ValueError`. `RequestReplayMismatch` as a cause means the
+original intent or lifecycle differs, not a transient retry. A timeout or
+`NotFound` observation does not authorize a replacement ID.
+`AppendCleanupUnresolved` additionally retains `expected_state_digest`, any
+known `recovery_receipt` and publication ID, and
+`next_action="retry_same_cleanup"`. Reuse that digest instead of selecting a
+new token automatically.
+
+Persist the stable receipt before acknowledging an external queue. Its ten
+fields identify the original logical action, successful publication/revision,
+target/incarnation, versions, complete body size, and digest. `replayed` and
+nullable `commit_version` are call metadata, not stable receipt fields. A
+retained receipt does not permanently retain historical body bytes.
+
+The [product specification](../../docs/development/append-product-spec.md)
+defines owner leases, provider sealing, retention, and qualification.
+`WorkbenchFileSystem` has no append mode, and the legacy 18-tool
+`workbench_append` operation is not the durable identity API. Use native
+`nokv workspace-path append` first, or `Client.append_bytes` for embedded work.
diff --git a/crates/nokv-python/src/lib.rs b/crates/nokv-python/src/lib.rs
index c2a3a37d0..f675f3fff 100644
--- a/crates/nokv-python/src/lib.rs
+++ b/crates/nokv-python/src/lib.rs
@@ -8,6 +8,11 @@
//! Metadata and lifecycle semantics remain in `nokv-client`. Durable bytes
//! remain behind `nokv-object`. The only local-filesystem behavior in this
//! crate is the explicit materialize/collect adapter.
+//!
+//! `Client.append_bytes` accepts a caller-persisted logical operation ID and
+//! preserves the shared SDK append contract across process restarts. Metadata-only
+//! status and inspection, and owner-executed recovery, do not require a caller
+//! object store. The caller still owns its durable delivery record and queue ACK.
mod client;
mod local_adapter;
diff --git a/docs/append.md b/docs/append.md
new file mode 100644
index 000000000..4bd61f90a
--- /dev/null
+++ b/docs/append.md
@@ -0,0 +1,384 @@
+
+
+# Durable append and recovery
+
+A queue worker appends an event, NoKV commits it, and the worker dies before
+acknowledging its queue message. With a saved logical operation ID, the next
+worker can recover the original receipt instead of appending the event twice.
+NoKV also lets that same logical action make progress after an unsuccessful
+publication attempt has been safely cleaned up.
+
+Use the native `nokv workspace-path append` command for this contract. Embedded
+Python callers use `Client.append_bytes`; both delegate recovery to the same
+Rust SDK. The 18-tool `workbench_append` operation retains its existing
+per-invocation identity and generation-CAS behavior. Repeating that tool in a
+new process does not recover one durable logical action. The deprecated MCP
+sidecar and fsspec append mode are not integration paths for this feature.
+
+This guide describes the checked-in implementation. A merged change is not a
+published release: use a binary and, when applicable, a Python wheel built
+from a matching qualified commit or a release that explicitly includes this
+feature. `nokv version --json` reports the binary's source identity. A package
+version string alone does not distinguish builds from unreleased commits.
+
+## Before sending an append
+
+Use an existing, provisioned deployment as described in
+[deployment preflight](workbench-preflight.md). The CLI requires its real
+`RootId`, bound `AgentId`, and etcd routing; a static owner address alone is
+insufficient for these Agent-facing commands. The first append requires an
+existing workspace. It may create a missing file, but never the workspace.
+New publications need the deployment's matching S3-compatible object namespace
+and a provider admitted for append sealing. Status, inspection, and cleanup
+recovery need metadata connectivity but no caller-side object configuration.
+
+Persist an outbox or queue record **before dispatch**, containing:
+
+- The root and a 128-bit logical `operation_id`, encoded as 32 lowercase hex
+ characters. Keep it unique across append, publish, commit-build, and restore
+ operations in that root.
+- The workbench, normalized section/path, and intended workspace incarnation
+ when known. Keep the root with the ID: the same ID in a different root does
+ not identify the original action.
+- The exact delta bytes, or durable storage from which those exact bytes can
+ be recovered. A content digest alone is insufficient.
+- The content-type policy, block size, and effective maximum resulting size.
+ Preserve all these options across retries.
+
+Generate an ID once when accepting a business action, then commit the record
+using the application's durable queue/database transaction. In-memory state,
+a freshly generated UUID on each delivery, or an unflushed JSON file is not a
+durable outbox. Separate actions receive separate IDs even if their bytes match.
+Retargeting an admitted action to a fork or another path is an intent mismatch.
+
+The incarnation identifies a particular lifetime of a workspace name. If it
+is omitted, the SDK first looks for an admitted operation and uses its original
+incarnation; only an unknown operation resolves the current workspace name.
+Thus a committed action remains recoverable after deletion/recreation, while
+an unfinished old action cannot write into the replacement workspace. An
+explicit `expected_workspace_incarnation_id` must match; it is never silently
+updated. Save a known incarnation when accepting work for a particular instance.
+
+## CLI: submit the saved action
+
+The following Bash examples require `jq`, an existing workspace, and the
+deployment values in `AGENT_ID`, `ETCD_ENDPOINT`, `OBJECT_BUCKET`, `OBJECT_ROOT`,
+`OBJECT_REGION`, and `OBJECT_ENDPOINT`. Configure the same provider credentials
+as your other NoKV object operations. An optional `ETCD_KEY_PREFIX` defaults to
+`/nokv/control`. Connection settings belong to deployment configuration, not
+to the immutable append intent.
+
+For illustration, a retained application record named `append-intent.json`
+has this shape. Replace the example root/ID/workbench with values from your
+durable queue; this JSON shape is an application example, not a NoKV wire API.
+The base64 value encodes `{"step":7}` followed by a newline.
+
+```json
+{
+ "root_id": "11111111111111111111111111111111",
+ "operation_id": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+ "workbench": "run-42",
+ "section": "logs",
+ "path": "events.jsonl",
+ "expected_workspace_incarnation_id": null,
+ "content_type": "application/x-ndjson",
+ "block_size": 4194304,
+ "max_logical_size": 16777216,
+ "data_base64": "eyJzdGVwIjo3fQo="
+}
+```
+
+Read the saved values on every delivery:
+
+```bash
+set -euo pipefail
+job=append-intent.json
+operation_id=$(jq -er '.operation_id' "$job")
+route=(--root-id "$(jq -er '.root_id' "$job")"
+ --agent-id "${AGENT_ID:?}"
+ --etcd-endpoint "${ETCD_ENDPOINT:?}"
+ --etcd-key-prefix "${ETCD_KEY_PREFIX:-/nokv/control}")
+objects=(--object-bucket "${OBJECT_BUCKET:?}"
+ --object-root "${OBJECT_ROOT:?}"
+ --object-region "${OBJECT_REGION:?}"
+ --object-endpoint "${OBJECT_ENDPOINT:?}")
+incarnation=$(jq -r '.expected_workspace_incarnation_id // empty' "$job")
+fence=()
+if [[ -n "$incarnation" ]]; then
+ fence=(--expected-workspace-incarnation-id "$incarnation")
+fi
+
+nokv "${route[@]}" "${objects[@]}" workspace-path append \
+ "$(jq -er '.workbench' "$job")" "$(jq -er '.section' "$job")" \
+ "$(jq -er '.path' "$job")" \
+ --operation-id "$operation_id" "${fence[@]}" \
+ --base64 "$(jq -er '.data_base64' "$job")" \
+ --content-type "$(jq -er '.content_type' "$job")" \
+ --block-size "$(jq -er '.block_size' "$job")" \
+ --max-logical-size "$(jq -er '.max_logical_size' "$job")"
+```
+
+CLI paths are relative to the selected section: use `logs events.jsonl`, not
+`logs logs/events.jsonl`. Valid sections are `input`, `scripts`, `outputs`,
+`logs`, and `metadata`. Paths reject absolute paths, empty components, `.` and
+`..`, backslashes, and NULs. Append accepts exactly one of `--text`, `--base64`,
+or `--file`; it does not add a newline. For larger deltas, use `--file` with a
+retained regular file to avoid OS command-line length limits. The reader rejects
+symlinks and non-regular inputs and caps bytes read even if the file grows.
+
+All options that select routing or objects precede `workspace-path`; append
+options follow the target. `--operation-id` is required. The incarnation,
+content type, block size, and resulting-size bound are optional with the
+defaults below. There is no caller-supplied generation for this command: the
+SDK handles publication CAS and fenced successor attempts.
+
+On success, stdout contains JSON with `status=success`, `operation=append`,
+`state=committed`, and `next_action=none`. The stable receipt fields are:
+
+| Fields | Meaning |
+| --- | --- |
+| `operation_id` | The caller's logical action. |
+| `publication_operation_id`, `artifact_revision_id` | Its successful publication attempt and immutable revision. |
+| `workbench_id`, `path`, `workspace_incarnation_id` | The original target; `path` includes the section. |
+| `workspace_revision`, `generation` | Versions at that publication. |
+| `logical_size`, `body_digest` | Complete resulting body length and SHA-256 digest URI. |
+
+`replayed` and nullable `commit_version` describe this SDK call; exclude them
+when comparing or retaining the stable receipt. A replay can return generation
+1 even if the current path is at generation 3 or has been removed.
+
+Durably save the stable receipt before acknowledging the business queue
+message. If the consumer crashes between those steps, redelivery reads the
+saved action and either its saved receipt or NoKV's original result. Keep
+completed IDs deduplicated according to the application's delivery contract.
+NoKV cannot atomically acknowledge an external queue for the caller.
+
+## Query first, then follow the recovery action
+
+The following commands reuse the `route` array and `operation_id` above and
+do not use the `objects` array:
+
+```bash
+nokv "${route[@]}" operation status "$operation_id"
+```
+
+| `state` | `next_action` | What to do |
+| --- | --- | --- |
+| `committed` | `none` | Persist the original `receipt`, then acknowledge the action. |
+| `pending` | `poll` | Wait and query this ID again. An active/abandoned child has not reached a safe retry point. |
+| `ready_to_retry` | `resubmit_same` | Redeliver the same bytes and options under the same logical ID. |
+| `quarantined` | `retry_cleanup` | Inspect the failure, correct the dependency problem, then request owner cleanup with the saved token. |
+
+Status queries do not advance an attempt. An unsuccessful physical attempt
+does not mean the logical action has failed permanently. Only completed
+cleanup permits a successor; repeated queries do not reconstruct the delta.
+Schedule polling with the caller's backoff and deadline. Owner activity leases
+and cleanup scheduling can make recovery take longer than one client call.
+The owner setting `--append-activity-lease-ms` defaults to 30 minutes, accepts
+1 second to 24 hours, and has an additional clock-skew allowance. There is no
+independent client upload heartbeat; operators must budget for upload progress
+before shortening the lease.
+
+Status includes the current `publication_operation_id`, zero-based `attempt`,
+`attempt_phase`, `cleanup_retry_count`, `activity_deadline_ms`, original target,
+`observed_state`, `progress`, `cause_code`, `failure_message`, `attempt_failure`,
+and nullable `receipt`. `progress` contains completed/total rows and bytes;
+totals may be null. `attempt_failure` retains the physical attempt's code,
+message, retryability, conflict kind, and current generation when present.
+Use the public `state`/`next_action` pair for decisions; an exception's raw
+`state` may instead be `Running`, `Quarantined`, or unknown.
+
+### Inspect pages and save a recovery token
+
+```bash
+nokv "${route[@]}" operation inspect "$operation_id" --limit 32 > inspection.json
+cursor=$(jq -r '.next_cursor // empty' inspection.json)
+if [[ -n "$cursor" ]]; then
+ nokv "${route[@]}" operation inspect "$operation_id" \
+ --limit 32 --cursor "$cursor" > next-page.json
+fi
+```
+
+Continue until `next_cursor` is null. Each page includes the status fields,
+`action=inspect`, `operation_token`, `publication_token`, `object_namespace_id`,
+`registered_count`, `cleanup_cursor`, `remaining_count`, and `entries`.
+Each entry has `sequence`, `object_identity`, `expected_length`,
+`expected_digest`, and nullable `multipart_token`.
+
+The page contains retained staging rows for the current child, not every key
+ever used by the logical action. `remaining_count` is the retained interval
+`registered_count - cleanup_cursor`, not a count of unpublished bytes or proof
+that a published object should be cleaned. Do not delete inspected keys.
+
+The base64 continuation cursor binds the root, logical ID, exact observation,
+and last sequence. A changed phase, child, or cleanup state invalidates it with
+`Conflict/OperationState`; discard the partial inspection and start again.
+Malformed cursors and cursors used with another root/ID are `InvalidArgument`.
+The default page limit is 32, with an allowed range of 1–192.
+
+`operation_token.state_digest` is different: it is a 64-character lowercase
+hex token for one cleanup recovery request. Persist that token in the durable
+operator task before sending `recover`; a plain `inspection.json` redirection
+in this example is not the durable handoff. After correcting the reported
+dependency failure, use the token saved while the operation was quarantined:
+
+```bash
+if [[ "$(jq -er '.state' inspection.json)" == quarantined ]]; then
+ recovery_digest=$(jq -er '.operation_token.state_digest' inspection.json)
+ nokv "${route[@]}" operation recover "$operation_id" \
+ --expected-state-digest "$recovery_digest"
+fi
+nokv "${route[@]}" operation status "$operation_id"
+```
+
+The owner seals the failed child's staged keys; the caller needs neither
+payload nor object credentials for this request. `requested=true` means the
+cleanup request was durably accepted, including a replay. Its
+`recovery_receipt` contains `operation_id`, `publication_operation_id`,
+`cleanup_retry_count`, and `expected_state_digest`. The top-level status and
+`operation_token` are a fresh observation and can be newer than that historical
+recovery receipt. Neither acceptance nor `replayed=true` proves cleanup or
+append completion.
+
+Repeat an uncertain recovery request with **the same saved digest**, even if
+the operation has since progressed or become quarantined again. If that request
+was durably accepted, it replays that round's receipt. Otherwise, a now-stale
+token can be rejected without starting a new round. A new round requires a
+deliberate new inspection/token. Without `--expected-state-digest`, each
+call targets the current observation once: pending, cleaned, and committed
+operations return `requested=false` with no recovery receipt. Once status is
+`ready_to_retry`, submit the original append again; `recover` never publishes
+the delta or starts its successor on the caller's behalf.
+
+## Python: use the same retained intent
+
+The Python path combines section and relative path into `logs/events.jsonl`.
+This example reads the same durable record as the CLI example, using the same
+explicit content type and bounds. The installed wheel must include the feature
+and match the deployment; see the [SDK installation guide](../crates/nokv-python/README.md).
+
+```python
+import base64
+import json
+import os
+from pathlib import Path
+
+from nokv import Client, ObjectStoreConfig, RoutingConfig
+
+job = json.loads(Path("append-intent.json").read_text())
+routing = RoutingConfig.etcd(
+ [os.environ["ETCD_ENDPOINT"]],
+ key_prefix=os.environ.get("ETCD_KEY_PREFIX", "/nokv/control"),
+)
+objects = ObjectStoreConfig.s3(
+ bucket=os.environ["OBJECT_BUCKET"],
+ root=os.environ["OBJECT_ROOT"],
+ region=os.environ["OBJECT_REGION"],
+ endpoint=os.environ["OBJECT_ENDPOINT"],
+)
+client = Client(job["root_id"], routing, object_store=objects)
+result = client.append_bytes(
+ job["workbench"],
+ f'{job["section"]}/{job["path"]}',
+ base64.b64decode(job["data_base64"], validate=True),
+ job["operation_id"],
+ content_type=job["content_type"],
+ block_size=job["block_size"],
+ max_logical_size=job["max_logical_size"],
+ expected_workspace_incarnation_id=job["expected_workspace_incarnation_id"],
+)
+print(result) # Persist the stable receipt before acknowledging the queue.
+
+# A separate metadata-only client needs no ObjectStoreConfig.
+metadata = Client(job["root_id"], routing)
+status = metadata.operation_status(job["operation_id"])
+page = metadata.operation_inspect(job["operation_id"], limit=32)
+```
+
+`append_bytes` accepts `bytes`; there is no separate Python `append_file`
+method. Preserve input bounds when reading files in application code. Its
+success/status fields match the CLI. `operation_inspect(operation_id, *,
+cursor=None, limit=32)` returns `next_cursor` as opaque **bytes**, not a base64
+string; pass it unchanged to the next Python call. For a cross-language
+handoff, base64-encode/decode it exactly once.
+
+`operation_recover(operation_id, expected_state_digest=None)` takes the saved
+lowercase hex **string** from `page["operation_token"]["state_digest"]`.
+The same quarantine check, durable token persistence, and status decision
+table apply. A completed append can also be replayed through a metadata-only
+client when the original payload/options are available: receipt validation
+happens before object-store initialization. Querying status needs no payload.
+
+## Defaults, errors, and boundaries
+
+| Option | CLI | Python | Default / rule |
+| --- | --- | --- | --- |
+| Input | Exactly one of `--text`, `--base64`, `--file` | `data: bytes` | At most 16 MiB per delta; CLI `--max-artifact-bytes` can lower this. |
+| Result size | `--max-logical-size` | `max_logical_size` | 16 MiB including existing body; explicit larger bounds do not raise the delta limit. |
+| Block size | `--block-size` | `block_size` | 4 MiB; positive and within provider admission limits. |
+| Content type | `--content-type` | `content_type` | Omission inherits an existing type; creation defaults differ by input form. |
+| Incarnation | `--expected-workspace-incarnation-id` | `expected_workspace_incarnation_id` | Omission resolves original admission first, otherwise the live workspace. |
+
+CLI text creation defaults to `text/plain; charset=utf-8`; CLI file/base64 and
+Python default to `application/octet-stream`. An explicit type applies on both
+creation and append. The full intent binds the creation default **and whether
+an override was supplied**, even when the current file already has that type.
+Do not switch from omitted type to explicit type on retry. For interchangeable
+CLI/Python delivery, choose one explicit type on the first submission and keep
+it. A default/explicit 16 MiB result limit normalizes to the same intent; a
+different bound or block size does not. Existing producer, manifest identity,
+and typed index fields are inherited; these append interfaces do not accept
+replacement metadata fields.
+
+CLI success JSON goes to stdout. Runtime errors exit nonzero and are written
+to stderr as `nokv: { ... }`, with top-level `status`, `code`, `message`, and
+`retryable`; recovery attributes are under `details`. Early argument-parsing
+errors can be plain text. Python exposes runtime append failures as
+`nokv.AppendError`; workspace-incarnation conflicts use
+`nokv.WorkspaceIncarnationMismatch`, a separate `RuntimeError` subtype with
+append recovery attributes. Invalid Python argument types or identity encoding
+may raise `TypeError`/`ValueError` before a request is constructed.
+
+| Observation | Response |
+| --- | --- |
+| `AppendUnresolved`, timeout, or unknown outcome | Keep the root, ID, and intent; query the same operation. Unknown state is not proof of no effect. |
+| `NotFound` | Verify root/ID and query or redeliver the saved action as appropriate; a concurrent admission may still occur. Never infer permission for a replacement ID. |
+| `cause_code=RequestReplayMismatch` | Restore the original intent or correct the caller's identity assignment; changing inputs cannot recover this action. |
+| `Conflict` with workspace-incarnation conflict | Investigate the original workspace lifetime. Do not silently target its replacement. |
+| `InvalidArgument` | Fix invalid input before a new admission; preserve an already admitted action's original intent. |
+| `AppendFailed` with `ProviderAdmissionRejected`, `ProviderAdmissionUnavailable`, or `ProviderAdmissionInconclusive` | Inspect capability/configuration or provider availability. Admission failures are not cached; a repaired provider can be checked again using the same client. |
+| `AppendCleanupUnresolved` | Retry `recover` with its saved `expected_state_digest`; retain any known `recovery_receipt`. |
+
+Append errors preserve `operation_id`, optional raw `state`, `cause_code`, and
+`next_action=query_same`. Cleanup uncertainty instead uses
+`next_action=retry_same_cleanup`; it preserves the expected digest and any
+already observed historical cleanup receipt/publication ID. Generic retry
+handlers must not interpret `retryable=false` as permission to discard the ID
+or allocate a fresh one. Follow the explicit action rather than parsing text.
+
+A zero-byte delta is still an action: a new ID can create an empty file or
+advance an existing generation, while replaying that ID cannot do so again.
+Concurrent successful actions have a publication order; their caller start
+times do not establish FIFO order. This is bounded byte append, not a
+transaction across a queue, an external API, and NoKV, nor a guarantee of
+unlimited log size or sustained throughput.
+
+Logical receipts and child records currently have no automatic expiration.
+Failed attempts retain small permanent revision reservations and zero-byte
+seals so delayed PUTs cannot reopen abandoned keys. A historical receipt proves
+the original result; it does **not** retain its body forever. Use the documented
+commit/tag/retention lifecycle when historical bytes must remain readable.
+Capacity-plan this retained metadata and provider-key overhead. Owner lease,
+schema upgrade, provider qualification, and acceptance details are in the
+[append product specification](development/append-product-spec.md). The
+[qualification record](development/append-qualification.md) identifies the
+executed local and remote tests and their remaining boundaries.
+
+Rust integrations use `IdempotentAppendOptions` and
+`WorkspaceClient::append_artifact_idempotent`, with the shared
+`get_append_operation`, `inspect_append_operation`, and
+`recover_append_operation` methods. Reuse those compositions instead of
+implementing another append or cleanup state machine.
diff --git a/docs/architecture.md b/docs/architecture.md
index f999e1e80..9223acd31 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -7,6 +7,10 @@ SPDX-License-Identifier: Apache-2.0
Status: normative workspace architecture.
+For the supported append workflow, start with the [durable append guide](append.md).
+The [append product contract](development/append-product-spec.md) defines its
+identity, recovery, and qualification boundaries.
+
## System Shape
```mermaid
@@ -236,6 +240,30 @@ Failed uploads never become visible. Response loss after commit returns the
same result on retry. Generic random writes are absent; append is immutable
segment publication plus stream-head CAS.
+Stable append adds a root-scoped logical operation above that publication
+pipeline. The caller retains one action ID and its complete intent, including
+the delta. The logical parent binds the full intent digest and original
+workspace incarnation; each numbered child owns one immutable publication
+plan. Parent advancement and child admission are one metadata command, and a
+successor is permitted only after the current child is durably `Cleaned`.
+An unknown result or active child is not permission to allocate another action.
+
+Successful publication installs the logical receipt in the same transaction as
+the child result, path, revision, indexes, and references. Replaying that action
+returns the stored result independently of a later live head or workspace-name
+reuse. A receipt retains the historical result metadata, not a permanent hold
+on the old body. The caller-facing `operation_id` remains the logical ID;
+`publication_operation_id` identifies the physical attempt that produced it.
+
+Public `operation status`, `operation inspect`, and `operation recover` use
+metadata only on the caller. Inspection pages bind the exact current parent
+and child. Recovery re-enqueues a quarantined child for the fenced owner; it
+does not append data or create a successor. A saved logical state token names
+one cleanup retry, so response-loss redelivery returns its original admission
+receipt even after another quarantine or a successor. The cleanup retry count
+belongs to the child and starts at zero for each new child. Full intent is
+still required when the caller resubmits the append after cleanup.
+
Commit replay resolves its deterministic build-operation identity before any
live workspace lookup. A terminal retry authenticates the complete stored
request and commit-owned run-manifest binding, then verifies the corresponding
@@ -272,6 +300,15 @@ GC claims only the current zero-count epoch and atomically moves the revision
from `Available` to `Deleting`. New references require `Available`, closing the
restore/commit-versus-delete race.
+An abandoned stable-append revision follows a different lifetime rule. Its
+registered keys are conditionally replaced or created as permanent zero-byte
+seals; its revision claim remains bound to the failed child. This closes the
+window in which a delayed immutable PUT could recreate an orphan after DELETE.
+Ordinary published-revision GC does not delete these unpublished reservations.
+Their metadata and provider-key cost remains even after the logical append
+succeeds. Generic publication cleanup and published-revision GC retain their
+existing deletion behavior; see [Object Layout](object-layout.md).
+
## Snapshot And Commit Reads
NoKV's logical history and hold records support two different products:
@@ -347,9 +384,19 @@ durable distributed
ACK after the configured shared logical-log boundary
```
-The two modes have separate SLOs and benchmark rows. Recovery uses checkpoint
-images plus the logical command log. Owner epoch prevents an old process from
-committing or deleting objects after failover.
+The two modes have separate acknowledgement and qualification boundaries.
+The serving local adapter is the locked Holt 0.8.6 dependency. Recovery uses
+the same exclusive Holt namespace or the admitted shared logical-log path
+described below. Owner epochs fence metadata commands; they cannot cancel a
+provider request already in flight. Stable append additionally uses monotonic
+seals to make late failed-child uploads harmless.
+
+The current workspace RPC schema is v12, system format is 13, and publication
+value format is 7. Existing stores are opened read-only and checked before
+writable Holt recovery. Incompatible stores are rejected without rewriting
+their files; this feature supplies no system-format migration or mixed-version
+write mode. Same-directory process recovery is not cross-host failover or
+machine power-loss qualification.
Current implementation status: the local Holt WAL boundary and the
object-backed shared-log boundary are executable. Every acknowledged metadata
diff --git a/docs/benchmarks.md b/docs/benchmarks.md
index ab2fac0fa..afadc3a24 100644
--- a/docs/benchmarks.md
+++ b/docs/benchmarks.md
@@ -210,6 +210,8 @@ Matrix:
- create-only success and exists conflict;
- replace-only success, missing path, and stale generation;
- append head CAS;
+- stable logical-ID replay, same-ID contention and distinct-ID contention;
+- changed-intent rejection, including content-type override presence and limits;
- byte-identical edit;
- exact request replay;
- request-id mismatch;
@@ -217,7 +219,12 @@ Matrix:
- object upload and verification failure.
Report object time and metadata time separately, plus the complete user-visible
-latency.
+latency. Separate the first publication from a committed receipt replay and
+from resumption after cleaned attempts; they perform different object and
+metadata work. Report operation-history size, retained seal/reservation counts
+and client/owner retry budgets for stable append. The
+[append qualification record](./development/append-qualification.md) proves
+listed functional outcomes, not a throughput, p99 or long-term capacity SLO.
### Query
@@ -241,6 +248,8 @@ Required workloads include:
- commit-head/tag replacement and commit retirement;
- restore across entry-count and shared-revision distributions;
- publication abort and staged-object cleanup;
+- stable append failed-key sealing, bounded ledger inspection and exact-token
+ owner cleanup retry, including receipt replay after another attempt;
- revision GC under create/replace/remove churn;
- ambiguous provider deletion and quarantine reconciliation.
diff --git a/docs/development/append-idempotency.md b/docs/development/append-idempotency.md
index acbd01233..184ae132c 100644
--- a/docs/development/append-idempotency.md
+++ b/docs/development/append-idempotency.md
@@ -18,11 +18,22 @@ response loss, or later changes to the live file. An unfinished operation expose
Status queries need neither the original payload nor an available object store.
The native CLI is the primary interface, followed by the Python and Rust SDKs.
+Start with the [durable append guide](../append.md) for current submission and
+operator commands. Public inspection reads the current child's retained ledger;
+public recovery asks the owner to retry quarantined cleanup using a saved
+logical state token. It neither supplies the missing delta nor creates the
+next publication attempt. Failed child keys are permanently sealed, not
+deleted, before the same logical action can admit a successor.
+
The earlier single-attempt implementation prevented duplicate appends after a
lost success response but could not finish an action after an abandoned upload
was cleaned. Its executable
[identity recovery gate](../../scripts/workbench/append_identity_recovery_gate.py)
-is retained for reproducible historical evidence. Its safety-only pending cases
+is retained for reproducible historical evidence and shared harness fixtures.
+Its safety-only pending cases
do not establish completion qualification for the current product contract.
-Use the [product acceptance gate](../../scripts/workbench/append_product_acceptance_gate.py)
+Use the [product acceptance gate](../../scripts/workbench/append_product_acceptance_gate.py),
+the [public operations gate](../../scripts/workbench/append_operations_acceptance_gate.py),
and the normative contract's acceptance matrix for the current implementation.
+The [qualification record](append-qualification.md) states which versions and
+fault boundaries were executed; the presence of a harness is not a passing run.
diff --git a/docs/development/append-product-spec.md b/docs/development/append-product-spec.md
index bf3f86f3d..d36f61e3e 100644
--- a/docs/development/append-product-spec.md
+++ b/docs/development/append-product-spec.md
@@ -5,6 +5,11 @@ SPDX-License-Identifier: Apache-2.0
# Durable append product contract
+This is the normative implementation contract. The
+[durable append guide](../append.md) provides CLI/Python examples; the
+[qualification record](append-qualification.md) separates executed evidence
+from requirements below.
+
A queue worker saves an action ID and its event before appending the event to a
workspace file. If its process dies, its replacement submits the same action
and receives one durable result. Concurrent writers may change the file while
@@ -28,6 +33,9 @@ retained staged ledger. `nokv operation recover` / `Client.operation_recover`
request owner-executed cleanup through `artifact_append_recovery_v1`. Neither
operation requires the delta, a current workspace binding, object credentials
on the caller, or a transcript captured before the failure.
+Metadata routing and caller configuration are still required; metadata-only
+does not mean an offline lookup or that the serving owner needs no object
+provider for cleanup.
The frozen Workbench tool schemas remain unchanged.
The caller must durably retain the following before its first submission:
@@ -78,6 +86,16 @@ attempt identity, and eventual compact receipt. Each child publication carries
its parent/attempt binding and its immutable generation-dependent plan. An
active child cannot be replaced by a newly planned child.
+The identities and counters have different lifetimes:
+
+| Field | Scope and meaning |
+| --- | --- |
+| `operation_id` | Caller-owned logical action; unchanged across all redeliveries and safe successors. |
+| `attempt` | Current child number, starting at zero; advances only with atomic predecessor-cleaned admission. |
+| `publication_operation_id` / `artifact_revision_id` | Derived immutable identities of that numbered child. |
+| `cleanup_retry_count` | Number of accepted operator cleanup retries for this child; a new child starts at zero. |
+| `operation_token.state_digest` | Exact observed logical parent/current-child state; retaining it identifies one operator recovery request. |
+
## Atomicity and recovery invariants
1. Creating or advancing the parent and admitting its child happen in one
@@ -134,6 +152,14 @@ cleanup retry count, and receipt. An attempt failure describes its child, not a
the logical action. Applications must use
structured fields, not parse human-readable messages.
+The compact successful append receipt binds the logical and publication IDs,
+target/incarnation, artifact revision, workspace revision, path generation,
+whole-body size and digest. These historical fields are immutable.
+`commit_version` and `replayed` are call-envelope metadata, not fields of the
+append receipt: a later metadata-only lookup may have no commit version, and a
+replay may report different call metadata. Do not compare entire response
+envelopes to decide whether an append happened twice.
+
A missing logical operation is a point-in-time observation. Another caller may
admit it immediately afterward. Resubmitting the same ID is safe because atomic
admission handles that race; changing to a fresh ID is not justified.
@@ -234,6 +260,15 @@ means the request has a durable admission receipt, including replay;
`replayed` distinguishes replay from fresh admission. Neither field means the
append committed or cleanup finished.
+For example, cleanup token T can admit round 1 for child A. If that cleanup
+quarantines again, resending T returns the same round-1 receipt; it does not
+start round 2. A deliberate request using a new observation can admit round 2.
+After A reaches `Cleaned` and the same logical append publishes child B,
+replaying T still returns A's original recovery receipt while the response's
+current operation describes B with `cleanup_retry_count=0`. This is not a
+counter regression. The recovery call's commit version belongs to the original
+cleanup-admission command and is preserved on exact-token replay.
+
Omitting the expected digest explicitly requests recovery against the currently
observed state. Active, cleaned and committed attempts return an observation
with `requested=false` and no recovery receipt. A quarantined attempt can start
@@ -301,8 +336,9 @@ a promise for a deployment using the default lease.
Published and cleaned child records and logical receipts are currently retained
without automatic expiration. This is an explicit identity-retention cost. Failed attempts also retain their
-revision reservations and zero-byte object seals; seals retain no former payload
-bytes but still incur provider key/metadata overhead.
+revision reservations and zero-byte object seals; the live sealed object has
+no former payload bytes but still incurs provider key/metadata overhead.
+Historical provider versions, if enabled, are a separate retention concern.
Removing these records without an equivalent tombstone/replay protocol would
allow forgotten IDs to execute again. Receipts preserve metadata, not permanent
access to the old revision's object bytes. Ordinary snapshot/hold/GC policies
@@ -355,13 +391,15 @@ The workspace RPC schema is `nokv.workspace.rpc.v12`, the system format is 13,
and the publication value format is 7. A parent row is a new operation kind;
the child includes the monotonic cleanup retry counter. These version gates
also reject the preceding unreleased append candidate's v11/system-12 layout.
-Older stores and protocol clients are rejected explicitly. Existing Holt stores
-are inspected through the locked backend's read-only open path before writable
-recovery, so a format rejection does not rewrite their files. There is no silent
-migration, compatibility decoder, marker-only upgrade, or mixed-version write
-mode. Operators must preserve existing data and use the repository's qualified
-upgrade/export path when one exists; this feature does not manufacture an
-upgrade procedure for an incompatible store.
+Older stores and protocol clients are rejected explicitly. The serving backend
+is the published Holt 0.8.6 dependency pinned in Cargo. Existing Holt stores
+are inspected through its read-only open path before writable recovery, so a
+format rejection does not rewrite their files. There is no automatic
+system-format migration, marker-only upgrade, or mixed-version write mode.
+This feature provides no qualified migration/export-import procedure for an
+incompatible store. Preserve that namespace intact; do not edit its marker or
+reuse it as a fresh current-format store. Object-namespace adoption is a
+separate binding operation and cannot upgrade the system format.
Red and green fault tests initialize separate version-appropriate stores. A
green restart test must reopen the exact same green Holt directory. NoKV's
diff --git a/docs/development/append-qualification.md b/docs/development/append-qualification.md
new file mode 100644
index 000000000..c6acd1c71
--- /dev/null
+++ b/docs/development/append-qualification.md
@@ -0,0 +1,139 @@
+
+
+# Durable Append Qualification Record
+
+Status: feature-scoped safety and completion evidence. This record covers the
+local qualification finalized on 2026-09-23 (Australia/Sydney) and remote CI
+on 2026-09-24. It is not a whole-platform production or performance approval.
+The [user guide](../append.md) explains the API; the
+[product specification](./append-product-spec.md) defines its contract.
+
+## Source And Profile
+
+The tested NoKV source is
+[`13590c1cc12300d79396ea902a06b2dbb010a75b`](https://github.com/NoKV-Lab/NoKV/commit/13590c1cc12300d79396ea902a06b2dbb010a75b).
+The feature-branch commit that integrates main,
+[`3b83f8a216edb8ae09f79b77a22f746a386dd79f`](https://github.com/NoKV-Lab/NoKV/commit/3b83f8a216edb8ae09f79b77a22f746a386dd79f),
+has the identical Git tree, `2ce0d3728d50a767da314528daac53cbfac10d04`.
+This is source equivalence, not a claim that the frozen binary was built from
+the later commit.
+
+The local gates used a clean source tree, matching CLI and installed Python
+extension, macOS arm64, one local Holt authority, isolated etcd 3.7.1, and real
+RustFS. The linked dependency was registry **Holt 0.8.6**, not a local Holt
+checkout. The RustFS image was pinned to
+`rustfs/rustfs@sha256:e620d37756fff072b10bf648c7bb9d370d7e91a928b7e6a5e1ac85bdfb4e4dab`.
+The tested schema is RPC v12, system format 13, publication value format 7.
+
+Fault tests explicitly used the supported 1,000 ms append activity lease plus
+the normal 30-second clock grace, 90-second completion and 120-second command
+budgets, and 100 independent CLI replays. These budgets and measured durations
+are not the default 30-minute lease or a recovery SLA.
+
+## Local Fault Qualification
+
+| Execution | Result | Evidence scope |
+| --- | --- | --- |
+| [Core product gate](../../scripts/workbench/append_product_acceptance_gate.py), 21 scenarios | Safety and completion `PASS`; all listed scenarios executed. | Same/different logical identities, complete intent conflicts, cross-root isolation, deterministic admission races, caller death at five observed publication boundaries, owner death/reopen, provider PUT response loss and read corruption, late PUT blocked by a zero-byte seal, size/block/batch boundaries, 100 replays, metadata-only history, CLI/Python parity, and real incompatible wire/store rejection without mutation. |
+| [Public operations gate](../../scripts/workbench/append_operations_acceptance_gate.py), four scenarios | Safety and completion `PASS`; all listed scenarios executed. | Public inspection and owner recovery; 257 registered keys with a 32-row cleaned prefix; CLI/Python pages and concurrent recover; recovery ACK loss, owner reopen, repeated quarantine, successor attempts and stale former-child cursors; owner death after a real seal PUT succeeded but before its response returned. |
+
+The core gate's public quarantine scenario also runs in the operations gate.
+These are **21 + 4 executions, not 25 distinct requirements**. Required
+completion is checked alongside at-most-once effect, exact ordered bytes,
+generation changes, stable receipt fields, and relevant provider inventory.
+After a successor is published, an old inspection cursor is rejected while
+the old saved recovery digest still replays its original admission receipt.
+
+Workspace deletion or incarnation replacement during old-child cleanup is
+covered by metadata integration tests. It is not claimed as public lifecycle
+E2E: there is no corresponding public deletion/rebinding endpoint in this
+qualification. The historical
+[identity gate](../../scripts/workbench/append_identity_recovery_gate.py)
+reproduces the earlier duplicate-effect defect and preserves the narrower
+single-attempt design's evidence; it cannot replace the current completion
+gates. Commands and prerequisites are in the
+[runner guide](../../scripts/workbench/README.md#stable-append-gates).
+
+The local raw logs, databases, frozen artifacts, and JSON receipts are retained
+by the project but are **not published in this repository or as public CI
+artifacts**. Their recorded identities are:
+
+| Local receipt or artifact | SHA-256 |
+| --- | --- |
+| `product-final-13590c1/product-results.json` | `b458994291406dc001f63d9a1fff39edac81d3c2f9663dd981273201d9a2958c` |
+| `operations-final-13590c1/operations-results.json` | `44aa7777ed156473f216236ccf23de1d850b98b8eee38082d675643ead5695da` |
+| Frozen CLI | `19b6c785ad9fe3a3c78d4409bc76f56dbf4a69f76b6bc2a9e5346bc10eb7b774` |
+| Installed Python native extension | `43778cbc080ef099ac545dec4a804f7c2a8e53db1f61858a46d17378ce45ac21` |
+
+Each gate's result identifies its executed script and source hash separately
+from `environment.json`, which identifies the shared identity-gate stack
+helper. These hashes locate retained evidence; they do not substitute for
+access to the raw records or an independent run.
+
+## Remote NoKV CI
+
+The workflows below tested the same `13590c1` source tree. Their test merge
+checkout and branch tree were verified equal.
+
+| Run | Result | Scope |
+| --- | --- | --- |
+| [Rust](https://github.com/NoKV-Lab/NoKV/actions/runs/35992447765) | `PASS` | Default workspace: 1,158 passed, 11 ignored. Diagnostics feature: 429 test executions passed; restore-crash feature: 188 passed. Formatting, clippy, contract checks, and existing local-WAL, live Workbench, object namespace, restore composition, and fork/restore gates passed. |
+| [Python SDK](https://github.com/NoKV-Lab/NoKV/actions/runs/35992447625) | `PASS` | Release `manylinux_2_28_x86_64` wheel built and installed into a clean Python 3.12 environment; 34 SDK tests passed. |
+| [Docker](https://github.com/NoKV-Lab/NoKV/actions/runs/35992447760) | `PASS` | amd64 and arm64 image builds; this is not arm64 runtime fault qualification. |
+
+Feature test counts overlap and must not be added as unique tests. The 11
+ignored tests are not counted as executed successes. These NoKV workflows
+**did not execute the new complete append 21 + 4 fault matrix**. Existing
+Workbench runs through the deprecated sidecar do not qualify the corresponding
+native CLI/Python 18-tool path. The pre-#423 aggregator's deliberate
+`NOT QUALIFIED` result is also not erased by a successful CI job.
+
+## Real Downstream Queue Redelivery
+
+The demo's
+[push CI](https://github.com/NoKV-Lab/nokv-demo/actions/runs/35993046954) and
+[PR CI](https://github.com/NoKV-Lab/nokv-demo/actions/runs/35993047249)
+tested demo commit `13cc33cd733b024591620f8e69a35c153bb97aba`, pinned to NoKV
+`13590c1`. **The demo repository is private; these links and its retained CI
+evidence require repository access.** They are not public download links.
+
+Each run passed 434 Python tests plus three subtests, 75 Node tests, TypeScript
+checks, and a real Linux consumer redelivery gate. The gate killed the actual
+consumer after NoKV committed but before its local queue acknowledgement:
+
+- The legacy invocation was redelivered and grew the body from **13 to 26
+ bytes**, reproducing the duplicate effect.
+- A new process read the stable event from SQLite outbox and redelivered the
+ same complete intent and id. The body remained **13 bytes**, with all ten
+ immutable append receipt fields equal. Separate ids produced independent
+ effects, and the same id with changed intent was rejected.
+
+The test exercised the real Runtime/Storage/native CLI integration and
+independent byte verification, without paid model calls or a cloud sandbox.
+Its source and runner are documented in the access-restricted
+[demo redelivery guide](https://github.com/NoKV-Lab/nokv-demo/blob/13cc33cd733b024591620f8e69a35c153bb97aba/docs/APPEND_REDELIVERY.md).
+This demonstrates the tested consumer's durable queue boundary, not every
+agent framework or execution environment.
+
+## Remaining Qualification Boundaries
+
+- Full [Workspace Acceptance](./workspace-acceptance.md), including Gate 0's
+ complete native/Python 18-tool and snapshot-expiry evidence, remains separate.
+- Process SIGKILL and same-directory Holt reopen do not qualify cross-host HA,
+ disk loss, physical power failure, or every internal WAL/fsync boundary.
+- Rejection of previous wire/store formats is qualified; migration, mixed
+ writers, and cross-format rollback are not.
+- Receipt, failed-revision reservation, zero-byte guard, and outbox retention
+ have no long-term capacity or high-cardinality GC qualification. Guards must
+ survive external object deletion/overwrite/expiration; versioned-bucket
+ historical payload retention needs its own evidence.
+- The tested RustFS endpoint does not qualify all S3-compatible providers.
+ Throughput, long-running load, tail latency, and workload SLOs need separate
+ measurements; both local append gates report performance `NOT QUALIFIED`.
+
+Original red evidence and intermediate harness/setup failures remain retained.
+This dated record does not rewrite them, change the frozen qualification
+ledger, or assert a current PR approval or merge status.
diff --git a/docs/development/metadata-store-interface.md b/docs/development/metadata-store-interface.md
index 104843a13..f59520ce2 100644
--- a/docs/development/metadata-store-interface.md
+++ b/docs/development/metadata-store-interface.md
@@ -6,7 +6,8 @@ SPDX-License-Identifier: Apache-2.0
# Metadata Store Interface
Implemented: the storage-neutral interface, local Holt adapter, and `MetaShard`
-cutover. The serving local profile uses Holt through `TxnStore`. Pending:
+cutover. The serving local profile uses the pinned Holt 0.8.6 dependency through
+`TxnStore`. Pending:
FoundationDB, replicated Holt, provider-neutral runtime admission, and
owner-safe response delivery.
@@ -398,9 +399,15 @@ create/reopen/replace/remove for both a short path with a 61,203-byte typed
projection and a maximum-length path with 64 dependencies and a 57,243-byte
projection. A separate successful replacement changes all 60 index fields at
once: the before/after event is 61,323 bytes and the fully derived transaction
-is 9,859,091 bytes. The short-path replace-to-empty and remove transactions are
-also pinned at 11,797,794 and 11,791,459 bytes. These are characterized storage
-and metadata-engine compatibility results, not a universal domain-size proof.
+is 9,861,120 bytes. The short-path replace-to-empty and remove transactions are
+also pinned at 11,799,847 and 11,793,075 bytes in the current source fixtures.
+Append's parent/child receipt, active-publication marker, and cleanup retry
+state participate in this same budget, including history and recovery-outbox
+amplification. The metadata command item bound is 260 and its recovery codec
+uses the same bound; the complete maximum-path/dependency/index append fixture
+checks the derived transaction against the unchanged serving limits. These are
+characterized storage and metadata-engine compatibility results, not a
+universal domain-size proof.
The local-WAL server qualifies restart of the same exclusive namespace with an
empty shared recovery frontier. It does not qualify rolling upgrade onto
another store, copied-directory recovery, or checkpoint/log failover.
@@ -458,6 +465,13 @@ fresh location. The adapter does not complete a partial tree catalog.
Existing mode requires the exact configured catalog. It must reject any
unmarked namespace that contains domain records.
+For the current system format 13 and publication value format 7, the server
+first opens an existing Holt namespace read-only and asks `MetaShard` to verify
+its schema, identity, and recovery chain. Only a successful inspection permits
+writable recovery. Rejection during this read-only inspection does not trigger
+writable recovery or checkpoint an incompatible store on disk. This is an
+admission check, not a migration or a new runtime adapter profile.
+
The target server configuration is tagged. It does not encode cluster
files, credentials, or namespace settings into a provider URI. A missing build
feature causes a startup error and never falls back to Holt.
@@ -477,8 +491,10 @@ attach_root
1. Validate the open mode, root placements, control record, and shared recovery
frontier.
2. Before any new acquisition, initialize or exclusively reopen the explicit
- local namespace. Holt replays its WAL; `MetaShard` validates the catalog,
- schema, logical-shard identity, system rows, and full recovery-outbox chain.
+ local namespace. An existing namespace first receives the read-only
+ inspection above; writable Holt recovery then replays its WAL. The Holt
+ adapter validates the catalog; `MetaShard` validates the schema,
+ logical-shard identity, system rows, and full recovery-outbox chain.
3. Compare the local owner fence with the durable control state. A normal
successor requires the exact previous epoch. An interrupted `Recovering`
attempt accepts only that recovery epoch or its immediate predecessor.
diff --git a/docs/development/nokv-agent.md b/docs/development/nokv-agent.md
index 997ce3b8f..5e214a884 100644
--- a/docs/development/nokv-agent.md
+++ b/docs/development/nokv-agent.md
@@ -14,12 +14,16 @@ The normative behavior is [Workbench Contract](../workbench-contract.md).
## Package Position
```text
-native full CLI (primary)
+native CLI: nokv workbench (primary Workbench surface)
-> nokv-agent
-> nokv-client traits
-> versioned protocol and direct object data path
```
+The native `workspace-path append` and `operation status/inspect/recover`
+commands call the shared client append lifecycle directly. They are not extra
+Workbench tools and do not change this crate's frozen schemas.
+
The direct Python SDK is the secondary embedded product surface and calls the
client boundary directly. Downstream Agent frameworks should normally implement
skills over the CLI. The `nokv mcp` sidecar is deprecated transport wiring
@@ -66,7 +70,10 @@ workbench_restore
The normalized input schemas are frozen in
`crates/nokv-agent/workbench_contract_schema.json`. Tool registration
fails closed when a name or normalized schema differs. The native CLI exposes
-the same operations directly.
+the same operations directly. `workbench_append` allocates per-invocation
+identities and has no caller-supplied durable operation ID. Retrying a business
+action from another process requires the separate
+[stable append entrypoints](../append.md), not an added field in that tool.
## Adapter Responsibilities
@@ -93,7 +100,9 @@ The crate must delegate:
- `workbench_put_file(replace=false)` is create-only.
- `workbench_put_file(replace=true)` is replace-only.
- Append, edit, and replace preserve generation CAS behavior.
-- An exact retry returns the same result.
+- An exact retry that preserves the operation identity returns the same result;
+ a new `workbench_append` invocation does not preserve a previous invocation's
+ identity.
- Reusing a request identity with different inputs fails.
- Snapshot reads stay at their fixed read version.
- Snapshot renewal extends only and never revives a reaped snapshot.
diff --git a/docs/development/pr_review_checklist.md b/docs/development/pr_review_checklist.md
index 01bc9d356..259f0330f 100644
--- a/docs/development/pr_review_checklist.md
+++ b/docs/development/pr_review_checklist.md
@@ -101,6 +101,18 @@ boundaries, or Workbench behavior.
index rather than the child manifest's ordered row position?
- Do publish finalization and staged-object cleanup race through one durable
operation CAS before either metadata visibility or external deletion?
+- Does stable append bind a caller-persisted logical id to the complete intent,
+ with the parent receipt and path publication atomic and a successor fenced by
+ its cleaned predecessor?
+- Do historical append receipts survive later live changes without another
+ effect, while different logical ids with identical payloads remain distinct?
+- Does failed-append cleanup preserve conditional zero-byte seals and revision
+ reservations against late PUTs, quarantining an uncertain outcome?
+- Do inspection cursors bind the exact parent/child state and retained ledger,
+ rejecting missing rows and stale pages rather than mixing observations?
+- Does recovery replay the receipt for the saved state digest after response
+ loss, concurrent calls, repeated quarantine, and owner restart, without
+ silently choosing a newer recovery round or accepting a provider verdict?
## Snapshot, Commit, Restore, And GC
@@ -156,6 +168,9 @@ boundaries, or Workbench behavior.
second, and the Rust SDK third, without presenting the deprecated `nokv mcp`
sidecar as a supported integration surface?
- Do all 18 tool names and normalized input schemas remain stable?
+- Is caller-owned stable append documented as a separate native CLI/SDK API,
+ without adding an operation id to `workbench_append` or changing its delta
+ digest into a whole-body digest?
- Does golden-transcript validation cover observable result and error behavior,
rather than treating input-schema validation as equivalent?
- Is put still create-only or replace-only, never upsert?
@@ -177,6 +192,9 @@ boundaries, or Workbench behavior.
- Does every applicable
[acceptance gate](./workspace-acceptance.md) report `PASS`, `FAIL`, or
`NOT QUALIFIED`?
+- Do append claims distinguish safety from completion, local fault matrices
+ from remotely executed jobs, overlapping cases from distinct requirements,
+ and consumer redelivery from whole-platform or performance qualification?
## Required Validation
diff --git a/docs/development/workspace-acceptance.md b/docs/development/workspace-acceptance.md
index 0d7acc8bd..924527ac6 100644
--- a/docs/development/workspace-acceptance.md
+++ b/docs/development/workspace-acceptance.md
@@ -34,7 +34,8 @@ Every qualification record includes:
- NoKV commit and dirty-worktree state;
- Rust toolchain, operating system, CPU, memory, and storage devices;
-- Holt commit and durability profile;
+- linked Holt version, source and checksum (or commit for a Git dependency),
+ plus durability profile;
- logical-shard count, physical owners, root placement, and owner epochs;
- object provider, endpoint class, bucket policy, and consistency assumptions;
- client/adapter versions and the Workbench schema digest;
@@ -147,6 +148,50 @@ Required evidence:
expose a partial artifact;
- abort and cleanup race publication through one durable operation state.
+### Stable Append
+
+The native `workspace-path append` and direct SDK append APIs require separate
+evidence from the frozen `workbench_append` tool, whose schema has no
+caller-owned stable operation id. The
+[append product specification](./append-product-spec.md) defines this bounded
+contract. Required evidence includes:
+
+- persist a root-scoped logical id and complete intent before dispatch; replay
+ returns the original append receipt after process loss and later live writes,
+ while different ids with identical bytes remain independent;
+- reject changed intent, incarnation, and cross-kind identity reuse with exact
+ typed errors, without mutation;
+- atomically bind the logical operation to its publication attempt and publish
+ its receipt with the path; admit a successor only after its predecessor is
+ durably cleaned;
+- kill callers at observed publication boundaries, lose successful responses,
+ and reopen the same Holt directory after owner death; verify eventual
+ completion as well as at-most-once effect and exact ordered bytes;
+- delay a real PUT until after cleanup; prove a permanent conditional zero-byte
+ seal rejects its late arrival, and quarantine an unprovable seal outcome;
+- query status and inspect the retained ledger through public CLI/Python APIs
+ without delta or provider credentials; paginate past a cleaned prefix and
+ reject stale, foreign-root, and former-child cursors;
+- persist the inspected state digest for one cleanup request; concurrent
+ calls, lost recovery acknowledgements, owner restart, and repeated quarantine
+ replay its original receipt without admitting another round;
+- recover through the owner, not a caller-supplied provider verdict; distinguish
+ cleanup admission from completion and from the committed append receipt;
+- check zero bytes, block and batch boundaries, result/delta limits, repeated
+ replay without live revision changes, CLI/Python parity, and actual old-format
+ rejection without changing the old files;
+- redeliver a durable downstream queue through its real consumer after killing
+ it between NoKV commit and the queue's local acknowledgement.
+
+Run the [core product gate](../../scripts/workbench/append_product_acceptance_gate.py)
+and [public operations gate](../../scripts/workbench/append_operations_acceptance_gate.py)
+with matching frozen CLI/Python artifacts; invocation and historical baseline
+instructions are in the [runner guide](../../scripts/workbench/README.md#stable-append-gates).
+The [dated append qualification record](./append-qualification.md) separates the
+local 21 + 4 executions (one overlapping scenario), remote NoKV CI, and the
+separate Linux demo consumer test. Passing these feature gates does not resolve
+Gate 0's full-profile gaps or qualify every gate in this document.
+
## Gate 4: Snapshot, Commit, And Restore
The restart/composition runner is
diff --git a/docs/index.md b/docs/index.md
index 72dd75a57..c6b52d7c6 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -4,12 +4,15 @@ layout: home
hero:
name: NoKV
text: Agent-native distributed workspace and artifact storage.
- tagline: A native full CLI and direct Python SDK over path-primary ordered metadata.
+ tagline: Durable workspaces and recoverable append identities through a native CLI and direct SDKs.
image:
src: /img/logo.png
alt: NoKV
actions:
- theme: brand
+ text: Durable Append
+ link: /append
+ - theme: alt
text: Architecture
link: /architecture
- theme: alt
@@ -23,7 +26,9 @@ hero:
link: /development/workspace-acceptance
features:
- title: Stable Agent surface
- details: Use the native full CLI first, the direct Python SDK for embedded callers, and the lower-level Rust SDK for native integrations. One 18-tool Workbench contract fixes behavior across all three.
+ details: Use the native CLI first, Python for embedded callers and Rust for native integrations. The frozen 18-tool Workbench facade and the explicit append recovery APIs share one metadata lifecycle.
+ - title: Durable append identity
+ details: Save an operation ID and its inputs before dispatch. Replacement workers can retrieve the original receipt or resume safely cleaned work under that same ID.
- title: Path-primary metadata
details: One normalized full path is namespace truth. Exact artifacts use point reads; child listing uses component-safe delimiter scans.
- title: Immutable revisions
@@ -50,8 +55,9 @@ SPDX-License-Identifier: Apache-2.0
Application surface
CLI · Python SDK · Rust SDK
Downstream skills use the native CLI by default; embedded callers use
- the Python SDK, and native integrations use the Rust SDK. All three share
- the same 18-tool semantics.
+ the Python SDK, and native integrations use the Rust SDK. The full CLI
+ includes the stable 18-tool Workbench facade plus explicit append
+ submission and recovery commands.
Metadata layer
@@ -82,11 +88,17 @@ SPDX-License-Identifier: Apache-2.0
Recovery is explicit.
Leased snapshots pin MVCC history. Durable commits retain exact revisions.
Restore stages a new Workbench incarnation and reveals it only after a
- verified member seal.
+ verified member seal. For an interrupted append, query its saved logical ID;
+ inspect the staged ledger and request owner cleanup when needed.
+
Follow the append recovery guide.
## Documentation Map
+- Append a durable event: [Caller guide](./append.md),
+ [Product Contract](./development/append-product-spec.md), and
+ [Qualification Record](./development/append-qualification.md).
+
- Product and interface: [Product Design](./product-design.md),
[Architecture](./architecture.md), and
[Workbench Contract](./workbench-contract.md).
@@ -125,7 +137,8 @@ SPDX-License-Identifier: Apache-2.0
evidence required for release.
-
Workbench contract →
+
Durable append guide →
+
Workbench contract
Metadata schema
Acceptance plan
Path model comparison
diff --git a/docs/metadata-schema.md b/docs/metadata-schema.md
index e909673ee..f8e078dda 100644
--- a/docs/metadata-schema.md
+++ b/docs/metadata-schema.md
@@ -363,13 +363,15 @@ ArtifactRevisionClaim (reserved key inside the artifact_revision tree)
begin with a different operation id fails while it exists. Staged rows
derive permanent object keys from the revision id alone, so without the
claim two operations could own identical provider keys and an aborted
- loser's cleanup could delete the winner's published objects. The claim is
- deleted in the same command that publishes the revision or finishes the
- owning operation's cleanup. A quarantined operation keeps its claim
- fail-closed: its provider-side object state is unresolved, so the revision
- identity stays unclaimable until `ReconcileQuarantinedArtifactPublish`
- resolves the operation under an operator verdict and releases the claim in
- the same command that transitions it to `Cleaned`.
+ loser's cleanup could delete the winner's published objects. Successful
+ publication deletes the claim atomically with the new ArtifactRevision.
+ Generic publication cleanup also releases its own claim when it reaches
+ Cleaned, including an authorized ReconcileQuarantinedArtifactPublish.
+ Stable append cleanup instead retains the failed child's claim permanently:
+ its unpublished revision owns zero-byte object seals that must never become
+ reusable keys. Quarantine never releases that ownership. Append recovery
+ uses RetryAppendCleanup and owner-executed sealing, not a caller's generic
+ reconciliation verdict.
Exact revision keys are 32 bytes, so the 33-byte discriminated key can
never collide with one.
@@ -477,19 +479,30 @@ operation's later lookup by admitting the same identity. Internal manifest
publications retain distinct operation IDs. Terminal operation retention keeps
this exclusion after success or cleanup.
+An `Operation(Append)` parent records the full 32-byte intent digest, original
+target and workspace incarnation, current attempt number, child publication
+ID, artifact revision ID, and optional compact success result. The child's
+`Operation(Publish)` record carries the matching logical ID/attempt binding
+and immutable plan. Its `cleanup_retry_count` counts owner cleanup requests for
+that child; it is not the parent attempt number and is zero on each successor.
+
A logical append parent and its child publication are admitted together. Only a
current child proven `Cleaned` permits atomically advancing the parent and
admitting a successor. Final publication writes the parent receipt together
with the child `Published` result and visible path/revision/index/reference
changes. Parent and child records are retained for replay; they do not create
an extra body-retention reference. See the
-[append product contract](development/append-product-spec.md).
+[append product contract](development/append-product-spec.md) and
+[caller/operator guide](append.md).
`Operation(ActivePublish)` is an internal recovery index keyed by the physical
publication ID. It contains only Uploading, Finalizing, Aborting, and Cleaning
children. Publication transitions add or remove it in the same command that
changes the child. The lifecycle scans this index instead of retained terminal
-history. A malformed marker or missing referenced child fails closed.
+history after its initial read-only validation of all retained parents,
+publications, and markers. Missing or orphan markers, an inconsistent binding,
+or a missing child fail closed; the scan does not silently repair the index.
+Startup validation therefore still grows with retained history.
`ReadChanges` treats `(commit_version, event_sequence)` as an append-only log
position. Its opaque cursor is bound to the root, query scope, and optional
@@ -718,11 +731,13 @@ Finalizing -> Aborting # fenced proof of no path/dedupe publication
Finalization first CASes `Uploading -> Finalizing`; cleanup first CASes
`Uploading -> Aborting`, so only one can win. The metadata publication command
changes `Finalizing -> Published` atomically with the new path/revision. A
-crash in `Finalizing` is resumed from the ledger; cleanup may take it over only
-through the shown `Finalizing -> Aborting` CAS after proving that no
-path/dedupe publication exists. Publication and takeover both change the same
-operation row, so one wins. Cleanup may mutate the ledger or issue external
-DELETE only while it owns `Aborting`/`Cleaning`.
+crash in `Finalizing` is resolved against the durable ledger and publication
+evidence; it is not a promise to reconstruct and resume the caller's frozen
+upload. Cleanup may take it over only through the shown
+`Finalizing -> Aborting` CAS after proving that no path/dedupe publication
+exists. Publication and takeover both change the same operation row, so one
+wins. Cleanup advances only under the authoritative abort/cleaning state;
+provider deletion or sealing is performed by the fenced lifecycle worker.
For a stable append child, successful cleanup is `Aborted/Sealed`, not
`Aborted/Deleted`. Its exact staged keys become permanent zero-byte objects
@@ -740,10 +755,21 @@ Metadata-only `InspectAppendCleanup` returns bounded contiguous pages of actual
retained staged rows, bound to the exact current logical/child state. Generic publications and published
revision GC retain their existing deletion semantics.
-A late upload completion must observe the operation state; after abort it joins
-cleanup instead of publishing. Ambiguous multipart completion, late PUT, or
-DELETE remains ledger-owned and `Quarantined` until reconciled. Object listing
-is never used to discover staged ownership.
+Inspection reports `registered_count = staged_object_cursor` and the remaining
+interval `[cleanup_cursor, registered_count)`, not all keys ever planned or
+previously sealed. Published children may still retain staged rows, but reading
+those rows does not authorize cleanup. Missing rows cause an error, not a
+partial successful page. A changed state or successor invalidates continuation.
+`RetryAppendCleanup` binds the full expected logical state token in its durable
+command receipt. Replaying that token returns the original cleanup admission
+and commit version even after a later failure or successor; omitting a token
+at the SDK/CLI layer deliberately observes the current state for a new request.
+
+A client reporting upload completion must observe the operation state; after
+abort it cannot revive publication. The durable ledger retains cleanup
+ownership. For stable append, an unproved seal outcome quarantines the child;
+ordinary publication retains its existing ambiguous-upload/delete handling.
+Object listing is never used to discover staged ownership.
For generic publications, reconciliation is operator-driven, never scanner-driven. The operator verifies
provider-side object state for the operation's staged keys out-of-band and
@@ -761,8 +787,10 @@ owns it.
The final metadata command creates the `ArtifactRevision` as `Available`, its
manifest, the first path reference, `PathCurrent`, workspace revision, indexes,
-event, and dedupe result. A failed upload is invisible. A response-loss retry
-returns the stored result without allocating another revision.
+event, and dedupe result. A failed upload is invisible. Once publication has
+committed, an exact response-loss retry returns its stored result without
+allocating another revision. A stable append whose earlier child failed may
+admit a new revision only after predecessor cleanup is proven.
Append stores immutable segments in the new revision manifest and atomically
advances the path generation. A manifest row names the revision that physically
diff --git a/docs/object-layout.md b/docs/object-layout.md
index 4be9cee34..f4240d7ba 100644
--- a/docs/object-layout.md
+++ b/docs/object-layout.md
@@ -5,8 +5,9 @@ SPDX-License-Identifier: Apache-2.0
# Object Layout
-NoKV keeps artifact bytes outside Holt. Metadata stores compact immutable
-revision and manifest records; an S3-compatible provider stores the blocks.
+NoKV keeps artifact bytes outside the serving Holt 0.8.6 adapter. Metadata
+stores compact immutable revision and manifest records; an S3-compatible
+provider stores the blocks.
## Permanent Block Identity
@@ -18,7 +19,9 @@ nokv/artifacts/{logical_shard_id}/{root_id}/{artifact_revision_id}/blocks/{objec
- `object_index` is a fixed-width hexadecimal counter;
- physical process addresses, owner epochs, bucket endpoints, and Workbench
names are absent;
-- an object key is never reused for different bytes.
+- a published block key is never reused for a different payload;
+- an abandoned stable-append key may only move to its permanent zero-byte seal,
+ as described below; it cannot be reused for another artifact.
The logical shard, root, and revision remain stable when physical ownership
moves.
@@ -84,6 +87,12 @@ Create, replace, append, and edit retain distinct metadata predicates. An exact
request retry returns the same deterministic result and never allocates a
second published revision.
+For [durable append](append.md), the logical action ID survives across physical
+publication attempts. The current child must be durably cleaned before a new
+child/revision can be admitted. A failed child's keys are not reused by that
+successor. A successful logical receipt is installed atomically with the
+publication and remains queryable independently of the current live path.
+
## Reused Blocks
A new revision may reference blocks physically owned by older revisions. The
@@ -118,6 +127,34 @@ A claimed revision rejects new references. Provider deletion then advances
through durable states. Timeout or uncertain completion is quarantined and
reconciled; object listing is never used as reachability truth.
+## Failed Append Keys
+
+Stable append cleanup permanently occupies every registered failed-child key
+with an empty object. It conditionally creates an absent key with
+`If-None-Match: *`, or replaces an existing payload with an empty seal using
+`If-Match` on its observed ETag. This path never DELETEs a key: a delayed DELETE
+could otherwise remove a newer seal, allowing a late upload to recreate an
+unreachable payload. A nonempty immutable create cannot replace a seal.
+
+Only the fenced owner performs this operation after verifying the current
+parent/child binding, revision reservation, and absence of a published
+revision. Metadata cleanup records `Sealed`, retires the processed staging
+rows, and eventually marks the child `Cleaned`. The failed child's revision
+claim is retained permanently, including after a successor succeeds. A missing
+key alone is not sealing proof; an uncertain result quarantines the child.
+
+Public `operation recover` requests owner cleanup using a saved logical state
+token. It neither supplies a trusted provider verdict nor writes the objects
+from the caller. Generic publication reconciliation and ordinary
+published-revision GC continue to use their existing deletion semantics.
+
+These seals and revision reservations have no automatic expiration. They retain
+key/metadata overhead even though the live object has zero payload bytes.
+Logical receipts also remain, but do not pin historical body contents against
+normal reference/hold/GC rules. With bucket versioning, replacing the current
+payload does not prove removal of older versions. External expiration or
+overwrite policies must not remove live seals or other NoKV-owned objects.
+
## Provider Boundary
AWS S3, RustFS, MinIO, and Ceph RGW can use the same provider-neutral
@@ -138,9 +175,17 @@ block larger than the admitted maximum fails before object or metadata
publication. Admission v1 deliberately does not qualify multipart creation,
completion, or abort; those paths cannot inherit a single-PUT receipt.
+Stable append and serving owners advertising it additionally require the
+append-sealing admission profile: conditional creation of an empty seal,
+ETag-conditional replacement, rejection of late immutable writes, and stable
+empty-key observations must be exercised on that handle. A generic single-PUT
+receipt is insufficient. A tiered store seals the durable provider and attempts
+to evict its cached copy; cache eviction is best effort. Only the durable seal
+is authoritative; an evictable cache cannot provide the proof.
+
The provider interface also retains idempotent deletion with explicit
ambiguous-outcome handling. Public provider-admission errors and ambiguous
-create/delete errors do not render endpoint, bucket, or physical object-key
+create/delete/seal errors do not render endpoint, bucket, or physical object-key
details.
Provider credentials and endpoints are deployment configuration, not durable
diff --git a/docs/product-design.md b/docs/product-design.md
index 259a5fdb1..b3973292d 100644
--- a/docs/product-design.md
+++ b/docs/product-design.md
@@ -19,8 +19,10 @@ The supported front doors, in delivery order, are:
2. the direct Python SDK for embedded programmatic callers;
3. the Rust SDK for lower-level native integrations.
-The complete 18-tool [Workbench contract](./workbench-contract.md) fixes shared
-behavior across these surfaces. Downstream Agent systems normally provide
+The frozen 18-tool [Workbench contract](./workbench-contract.md) defines the
+Workbench facade. The full CLI also exposes durable append submission and
+operation recovery; the direct Python and Rust SDKs expose corresponding
+methods over the same lifecycle. Downstream Agent systems normally provide
skills that invoke the CLI, or call the Python SDK when an in-process boundary
is preferable.
@@ -75,7 +77,8 @@ Workbench-specific result shaping stays above the storage core:
- exact-string edit behavior;
- grep matching;
- section projection;
-- the delta digest returned by append;
+- the delta digest returned by `workbench_append` (the stable append receipt
+ instead identifies the resulting artifact and its whole-body digest);
- friendly errors, and the JSON-RPC result envelope the qualification harness
consumes;
- stable `run_manifest.json` and `restore_manifest.json` projections.
@@ -159,14 +162,45 @@ revision, manifest, path, workspace revision, indexes, event, reference
changes, and deterministic replay result.
Readers therefore see the previous complete revision or the new complete
-revision, never a partial body. A response lost after commit is safely replayed
-with the same request id. Failed or abandoned uploads remain invisible and are
+revision, never a partial body. An exact metadata-command replay uses the same request
+id. Across process restarts, stable append instead binds the complete intent
+to a caller-persisted logical operation ID. Failed or abandoned uploads remain invisible and are
recovered from a durable staged-object ledger.
Whole-artifact replacement is the generic write primitive. Logs use immutable
append segments plus a conditional stream-head advance. Range reads and
multipart uploads remain first-class SDK operations.
+## Durable Append Delivery
+
+Consider a queue event that is appended remotely before its consumer dies. A
+new consumer must know whether it is redelivering that action or appending a
+new one. The caller saves the root, logical operation ID, exact delta and full
+intent before the first request; NoKV durably binds that identity to the
+workspace incarnation, target, content-type options, block size and size limit.
+Different business actions use different IDs even when their bytes match.
+
+The logical operation owns predecessor-fenced publication attempts. An attempt
+publishes at most one receipt; a successor is admitted only after its predecessor
+cannot publish and its registered private objects are safely cleaned. After
+commitment, a lost reply is resolved by the original receipt; earlier uncertainty
+is queried under the same logical ID and follows its recovery state. Reusing an
+ID with different intent is rejected. The native `workspace-path append`
+command and the Python SDK provide this contract; `workbench_append` retains
+its fixed per-invocation contract and has no caller-supplied durable identity.
+
+The public operator path is `operation status`, bounded `operation inspect`,
+and exact-token `operation recover`. A recovery receipt records acceptance of
+one owner cleanup request, not completion of the append. The caller polls and
+resubmits the same saved append only when `ready_to_retry` allows it; it durably
+saves the committed append receipt before acknowledging its own queue event.
+A receipt is evidence of a past effect, not an artifact-retention hold or a
+transaction with the queue, model call, or other external side effect.
+
+The [append guide](./append.md) defines the caller workflow; the
+[product contract](./development/append-product-spec.md) defines its complete
+intent, retention and failure boundaries.
+
## Discovery And Provenance
Typed secondary indexes are updated atomically with the path entry. They serve:
@@ -188,7 +222,7 @@ and evidence an Agent needs to find and cite.
## Recovery Products
-NoKV exposes three different promises:
+Snapshots, commits and tags retain readable state in different ways:
| Mechanism | Purpose | Retention |
| --- | --- | --- |
@@ -201,6 +235,10 @@ shares immutable revisions inside one root/shard, and reveals the destination
with one final marker transition. Normal reads do not pay for a lazy overlay
chain.
+An append operation receipt serves a different purpose: it answers whether one
+logical action committed. Its retained identity does not pin the old body. Use
+a commit or a live snapshot when those bytes must remain readable.
+
## Safe Object Lifetime
Every visible path and durable commit owns an exact strong revision reference.
@@ -214,6 +252,13 @@ require `Available`, so restore/commit/publication cannot race with deletion.
Snapshots and in-progress scans protect older metadata with read-version
history holds. Ambiguous provider deletes are quarantined rather than guessed.
+Failed stable append attempts also retain revision reservations and conditional
+zero-byte object seals. These permanently prevent delayed immutable PUTs from
+recreating private payloads after cleanup. They are separate from published
+revision references and must not be removed by bucket expiration or ordinary
+GC. Quarantined append cleanup can be retried through the owner; generic
+publication/GC reconciliation remains an internal lifecycle operation.
+
This reference model intentionally stays root/shard-local.
## Distribution
@@ -259,4 +304,7 @@ unknown, malformed, incompatible, or mixed store before serving requests.
Source presence and unit tests do not prove durability, recovery, failover, GC,
or product behavior. Required boundary-level evidence is defined in
-[Workspace Acceptance](./development/workspace-acceptance.md).
+[Workspace Acceptance](./development/workspace-acceptance.md). The
+[append qualification record](./development/append-qualification.md) names the
+executed local and remote scenarios without extending them to full-platform
+release, old-store migration, cross-host failover or physical power loss.
diff --git a/docs/rustfs.md b/docs/rustfs.md
index d5d5d569a..5ada22461 100644
--- a/docs/rustfs.md
+++ b/docs/rustfs.md
@@ -21,10 +21,12 @@ native full CLI / Python SDK / lower-level Rust SDK
-> RustFS
```
-NoKV sends immutable block puts, ranged gets, integrity checks, and fenced
-deletes through the common object package. The NoKV metadata schema remains the
-authority for paths, manifests, references, operations, and deletion
-eligibility. Holt is the current serving local metadata adapter.
+NoKV sends immutable block puts, ranged gets, integrity checks, fenced deletes,
+and failed-append key sealing through the common object package. The NoKV
+metadata schema remains the authority for paths, manifests, references,
+operations, and deletion
+eligibility. The locked Holt 0.8.6 dependency is the serving local metadata
+adapter. See the [durable append guide](append.md) for caller and operator use.
## Deployment Identity
@@ -91,10 +93,19 @@ above that profile is rejected. Multipart upload, completion, and abort are not
qualified by admission v1 and must remain fail-closed until a separate endpoint
probe covers them.
+Stable append needs a stronger admission receipt that also exercises monotonic
+empty-object sealing: create-if-absent, ETag-conditional replacement, and
+rejection of a delayed nonempty immutable create. The native serving owner
+advertises append support and requires this qualified capability before owner
+acquisition. Generic-only admission does not satisfy that requirement. A
+provider outage during admission is not cached as a permanent failure; the
+same client handle can retry admission after the dependency recovers.
+
The deployment must additionally ensure:
- timeout and retry settings preserve ambiguous outcomes for reconciliation;
-- lifecycle policies cannot delete reachable NoKV objects independently;
+- lifecycle policies cannot independently delete reachable NoKV objects or
+ permanent seals from failed appends;
- bucket listing is not required for metadata recovery or reachability.
NoKV should use a dedicated bucket or an exclusive prefix with a policy that
@@ -139,20 +150,51 @@ Publication follows object-first, metadata-last ordering:
3. publish metadata;
4. acknowledge the deterministic result.
-If metadata publication fails, staged-object records drive cleanup. If delete
-completion is uncertain, the operation enters quarantine and reconciliation.
-The system does not infer success from a later bucket listing.
+If metadata publication fails, staged-object records drive cleanup. Ordinary
+publication and published-revision GC retain their deletion/reconciliation
+contract. Stable append instead turns the failed child's registered keys into
+permanent zero-byte objects. It uses `If-None-Match: *` for an absent key and
+`If-Match` against an observed ETag for an existing payload. It never DELETEs
+those keys. The retained revision claim prevents their reuse by another
+publication, and a late immutable PUT cannot replace the seal.
+
+An unproved seal result quarantines the child. Use metadata-only
+`operation inspect` and `operation recover` with a saved logical state digest
+after correcting the provider failure. The fenced owner performs the seals and
+records progress; the operator client does not need provider credentials or a
+pre-failure transport transcript. The generic publication reconciliation RPC's
+caller-supplied verdicts do not authorize stable append cleanup. The system
+does not infer ownership or safe cleanup from bucket listing.
+
+Zero-byte seals and failed revision reservations currently have no expiration.
+Budget for their key/metadata cost. Bucket versioning may retain old payload
+versions after the current object is sealed; the append contract does not
+qualify removal of those historical versions. Do not use bucket lifecycle
+expiration to remove live seals. These constraints are detailed in
+[Object Layout](object-layout.md#failed-append-keys).
Temporary provider failures remain retryable across the object, client, and
Workbench error boundaries. After bounded attempts, callers receive
`ObjectUnavailable` with `retryable: true` and an attempt count. Public
-provider-admission errors and ambiguous create/delete errors do not include
+provider-admission errors and ambiguous create/delete/seal errors do not include
endpoint, bucket, prefix, or physical object keys. Immutable create and delete
operations with ambiguous completion remain reconciliation cases rather than
blind retries.
+Stable append additionally preserves the logical action ID in
+`AppendUnresolved`; operator recovery preserves the expected state digest and
+any known admission receipt in `AppendCleanupUnresolved`. Follow their
+structured next action. A transient failure is not permission to create a new
+logical action or silently select a fresh cleanup token.
+
## Existing Roots
+The namespace-binding procedure below is not a system-format upgrade. The
+current owner accepts RPC v12, system format 13, and publication value format
+7. An older incompatible Holt store is rejected without mutation before
+writable recovery. The adoption flag cannot migrate it or justify editing its
+schema marker. This feature provides no in-place migration procedure.
+
Roots created before object-namespace binding have no durable evidence from
which NoKV could infer the historical bucket/prefix. Automatic adoption would
therefore make a typo authoritative. Upgrade them only while all owners are
@@ -199,7 +241,8 @@ Full RustFS qualification still requires:
Retain raw evidence using [Benchmarks](./benchmarks.md) and
[Workspace Acceptance](./development/workspace-acceptance.md).
-An endpoint admission receipt qualifies only the exercised single-PUT and
-range-read contract. It does not by itself qualify multipart behavior, exact
-crash recovery, process restart, owner replacement, or ambiguous-completion
+An endpoint admission receipt qualifies only its exercised single-PUT,
+range-read, and, when explicitly requested, append-sealing profile. It does
+not by itself qualify multipart behavior, exact crash recovery, process
+restart, owner replacement, or ambiguous-completion
reconciliation.
diff --git a/docs/workbench-contract.md b/docs/workbench-contract.md
index 00efefad8..9b8cc115d 100644
--- a/docs/workbench-contract.md
+++ b/docs/workbench-contract.md
@@ -25,9 +25,10 @@ The contract sources are:
- `scripts/workbench/workbench_contract.py` for checking tool names and
normalized input schemas.
-CLI and SDK wiring are consumers of this contract, not schema authorities.
-Every surface delegates to the transport-free facade and is neither a required
-deployment component nor a separate state machine.
+CLI and SDK wiring for these tools are consumers of this contract, not schema
+authorities. They delegate to the transport-free facade. The full native CLI
+and direct SDKs also expose operations beyond this frozen tool profile; those
+operations use the client state machines and their own documented contracts.
The native CLI accepts exactly these 18 names under `nokv workbench`. It fails
closed unless every possible destination owner supports the durable restore
@@ -155,6 +156,30 @@ Workbench responses do not contain `inode`, `source_root`, or
`destination_root`. Those names are not Workbench identities, routing inputs,
conditional-write tokens, provenance fields, or result projections.
+### Append Across Process Retries
+
+`workbench_append` has no caller-supplied operation id. Its identity is allocated
+per invocation: retrying the whole tool call in a new process can append the
+same bytes again after a lost success response. Its generation CAS protects
+concurrent writes, but cannot identify a redelivered business event.
+
+For a queue consumer or durable workflow, use native
+`nokv workspace-path append ... --operation-id HEX32`, or Python
+`Client.append_bytes(..., operation_id=...)`. Persist the root, logical id,
+complete intent, and delta before dispatch. Reuse them on redelivery; an
+independent event needs a different id even when its bytes are identical.
+Successful replay returns the original receipt without advancing the live
+generation. Its `body_digest` describes the complete resulting body, unlike
+the Workbench tool's delta `digest`.
+
+The native `operation status`, `operation inspect`, and `operation recover`
+commands provide metadata-only observation and owner-executed cleanup recovery.
+Recovery acceptance is distinct from append commitment; only the committed
+append receipt acknowledges the business event. These commands do not add
+tools or fields to the frozen 18-tool schema. See the
+[durable append guide](./append.md) and
+[product specification](./development/append-product-spec.md).
+
## Commit Identity
`workbench_commit` continues to publish
diff --git a/docs/workbench-preflight.md b/docs/workbench-preflight.md
index f6ed4cb8a..b7a21f5ad 100644
--- a/docs/workbench-preflight.md
+++ b/docs/workbench-preflight.md
@@ -43,6 +43,43 @@ git diff --check
The contract check proves only the exact 18 names and normalized input schemas.
It does not qualify persistence, object I/O, failover, restore, or latency.
+## Stable Append Deployment
+
+For cross-process append retries, use native `workspace-path append` and
+`operation status|inspect|recover`, or their direct Python SDK equivalents.
+The frozen `workbench_append` tool has no caller-supplied stable id and is not
+a durable queue's redelivery boundary. See the [append guide](./append.md).
+
+Stable append requires the current RPC v12, system format 13, and publication
+value format 7. Its public cleanup APIs require `artifact_append_recovery_v1`.
+Use matching owner, CLI, and Python builds; old formats are rejected before
+modification, not automatically migrated. Record the linked backend from
+`nokv version --json`: the qualified build uses registry Holt 0.8.6, which is
+independent of a developer's local Holt checkout.
+
+In addition to ordinary write admission, the concrete object-provider handle
+must pass append seal admission: conditional creation and replacement of a
+zero-byte guard must prevent a delayed create-if-absent PUT from resurrecting
+an aborted revision. Preserve those guards and failed revision reservations.
+External deletion, overwrite, or bucket expiration of guards breaks this
+contract. A provider's brand alone does not establish conformance.
+
+Persist the logical id, root, complete intent, and delta before dispatch. For
+quarantined cleanup, inspect publicly, save `operation_token.state_digest`,
+then call `operation recover` with that digest on every retry of the same
+recovery request. Status, inspection, and recovery need metadata routing but
+no caller-side S3 credentials or delta. The owner still needs a healthy,
+admitted provider to finish cleanup. `requested=true` acknowledges recovery
+admission; the queue may acknowledge its event only after append commitment.
+
+The server's `--append-activity-lease-ms` defaults to 1,800,000 ms and accepts
+1,000 through 86,400,000 ms, with an additional 30-second clock grace. The local
+fault qualification uses 1,000 ms explicitly; its recovery timings do not
+measure the default lease. Run the
+[stable append gates](../scripts/workbench/README.md#stable-append-gates) in fresh
+isolated evidence directories. Their [qualification record](./development/append-qualification.md)
+states the tested profile and limits separately from the full deployment gates.
+
## Default Deployment Shape
A serving shard is one `nokv serve` process over one exclusive Holt store. That
@@ -96,13 +133,13 @@ tool advertisement. This is a fail-closed deployment identity check, not
authentication. A legacy root without a binding requires a one-time,
operator-verified provision with `--adopt-legacy-agent-binding`; NoKV never
infers identity from the presentation path.
-Before serving any Agent-facing command, the CLI performs the typed workspace
+Before serving the Workbench facade, the CLI performs the typed workspace
RPC preflight for every capability required by the 18-tool profile; a missing
capability or route mismatch stops startup.
Bring-up must stop if:
-- the tool set is not exactly 18 tools;
+- the Workbench facade tool set is not exactly 18 tools;
- any normalized input schema differs;
- the root route is stale or belongs to another logical shard;
- the root has no durable Agent binding or is bound to another AgentId;
@@ -125,6 +162,10 @@ evidence for:
- revision retention and GC fencing;
- golden Workbench results and errors, not only input schemas.
+Stable append has separate native CLI/Python fault evidence and a real demo
+consumer redelivery test. Those results do not replace the complete 18-tool
+workflow or qualify unrelated commit, restore, and GC lifecycles.
+
Keep raw commands, environment profile, logs, and result artifacts with the
qualification report.
@@ -136,7 +177,9 @@ prove or implement all of the following:
forgeable client assertion alone;
- production adoption or bounded abort/cleanup for interrupted commit and
restore operations, including release of their history/revision holds;
-- a tracked resolution for late direct PUT completion after publication abort;
+- late direct PUT completion after generic publication abort; stable append's
+ conditional-seal closure is separately tested, not evidence for every
+ publication lifecycle;
- reconciliation that drives ambiguous object deletion out of quarantine;
- destructive provider operations fenced against control-plane lease transfer,
not only a preceding shard-local owner check.
diff --git a/scripts/release/README.md b/scripts/release/README.md
index aefe4ebb5..ce2ee0cf2 100644
--- a/scripts/release/README.md
+++ b/scripts/release/README.md
@@ -57,6 +57,30 @@ a NoKV deployment: callers must still supply the selected control-plane
endpoint, object-store configuration, root identity, and stable Agent
presentation root.
+## Durable append availability and upgrade boundary
+
+The [durable append guide](../../docs/append.md) describes the checked-in source
+contract. A merged feature PR, a passing Linux wheel check, or an unchanged
+18-tool schema does not establish that an older Formula or wheel contains
+`workspace-path append` and `operation status/inspect/recover`. Verify the
+release's source identity with `nokv version --json`, its command help and its
+release-specific qualification before deploying it. For Python, use a wheel
+built from the matching source revision and follow the
+[SDK installation instructions](../../crates/nokv-python/README.md).
+
+The append contract requires workspace RPC v12, metadata system format 13 and
+publication value format 7. An older store is rejected without mutation; this
+feature does not include an in-place migration, a mixed-version serving mode or
+a cross-format rollback procedure. Plan a separately qualified migration before
+upgrading an existing deployment. Do not delete an old store or change its
+format marker to make startup succeed.
+
+An upgraded owner must also qualify conditional failed-key sealing on its
+object provider. Keep the retained zero-byte seals, failed-revision reservations
+and operation receipts; a blanket bucket expiration rule can invalidate the
+retry guarantee. See the [provider contract](../../docs/rustfs.md) and the
+[append qualification record](../../docs/development/append-qualification.md).
+
## Release invariants
`.github/workflows/release-homebrew.yml` accepts only a canonical stable tag of
diff --git a/scripts/workbench/README.md b/scripts/workbench/README.md
index b4194ee2e..7e223eccf 100644
--- a/scripts/workbench/README.md
+++ b/scripts/workbench/README.md
@@ -5,9 +5,11 @@ SPDX-License-Identifier: Apache-2.0
# Workbench Validation
-These assets validate the same 18-tool Workbench semantics exported by
-`crates/nokv-agent`. The native full CLI is the primary integration surface,
-and the direct Python SDK is second for embedded callers.
+These assets validate the 18-tool Workbench semantics exported by
+`crates/nokv-agent` and separate native workspace APIs, including stable append.
+The native full CLI is the primary integration surface, and the direct Python
+SDK is second for embedded callers. The append gates do not extend the frozen
+tool schema or the deprecated sidecar.
Several runners here still reach the 18 tools through a `nokv mcp` child
process. That sidecar is deprecated and is not a supported NoKV integration
@@ -117,6 +119,89 @@ The deployment must provide one persisted `RootId` placement, its
metadata owner, and S3-compatible artifact credentials. Unknown or mixed
metadata schemas are rejected; the sole marker is `nokv_workspace`.
+## Stable Append Gates
+
+These runners exercise native CLI append and the installed Python SDK. Setup
+uses native Workbench commands; no append scenario is driven through MCP.
+Their contract is the [append specification](../../docs/development/append-product-spec.md),
+with a [dated qualification record](../../docs/development/append-qualification.md).
+
+| Runner | Purpose | Qualification boundary |
+| --- | --- | --- |
+| [append_identity_recovery_gate.py](./append_identity_recovery_gate.py) | Reproduce duplicate append after lost success, owner death, and a new invocation; also supplies shared stack helpers. | Its `qualified` mode records the earlier single-attempt design. It is historical safety evidence, not the current completion or operator gate. |
+| [append_product_acceptance_gate.py](./append_product_acceptance_gate.py) | 21 scenarios covering identity, completion, real provider faults, limits, CLI/Python parity, and old-format rejection. | Bounded append on one local Holt authority and real RustFS; full execution requires the matching Python SDK and a frozen previous-format binary. |
+| [append_operations_acceptance_gate.py](./append_operations_acceptance_gate.py) | Four public inspection/recovery scenarios, including pagination, cleaned prefixes, concurrent recovery, ACK loss, ABA, stale former-child cursors, and owner death during seal. | Metadata-only public CLI/Python recovery with owner-executed sealing; one quarantine scenario overlaps the core gate. |
+
+Install `etcd`, `etcdctl`, AWS CLI, and Docker on `PATH`, with Docker running.
+The runners create isolated etcd, a digest-pinned RustFS container and volume,
+fresh buckets, and a Holt owner. Use a fresh evidence directory for every run;
+keep original failures and frozen binaries. Run Python without `-O`, `-OO`, or
+`PYTHONOPTIMIZE`; the historical identity runner uses assertions.
+
+Build and freeze the CLI and Python wheel from the same clean source revision,
+then install that wheel into an isolated Python environment. The runners do
+not build or install them. Set these paths for your checkout and artifacts:
+
+```bash
+NOKV_APPEND_SOURCE="$PWD"
+NOKV_APPEND_BIN=/absolute/path/to/frozen-nokv
+NOKV_APPEND_PYTHON=/absolute/path/to/isolated-env/bin/python
+NOKV_PREVIOUS_FORMAT_BIN=/absolute/path/to/frozen-e5f81daf68-nokv
+NOKV_APPEND_EVIDENCE="$PWD/target/append-acceptance/new-run"
+
+python3 scripts/workbench/append_product_acceptance_gate.py \
+ --source-dir "$NOKV_APPEND_SOURCE" \
+ --nokv-bin "$NOKV_APPEND_BIN" \
+ --python-executable "$NOKV_APPEND_PYTHON" \
+ --legacy-nokv-bin "$NOKV_PREVIOUS_FORMAT_BIN" \
+ --evidence-dir "$NOKV_APPEND_EVIDENCE/product" \
+ --scenario all --replay-count 100 \
+ --append-activity-lease-ms 1000 \
+ --completion-timeout-seconds 90 --timeout-seconds 120
+
+python3 scripts/workbench/append_operations_acceptance_gate.py \
+ --source-dir "$NOKV_APPEND_SOURCE" \
+ --nokv-bin "$NOKV_APPEND_BIN" \
+ --python-executable "$NOKV_APPEND_PYTHON" \
+ --evidence-dir "$NOKV_APPEND_EVIDENCE/operations" \
+ --scenario all --append-activity-lease-ms 1000 \
+ --completion-timeout-seconds 90 --timeout-seconds 120
+```
+
+The previous-format fixture is RPC v11/system format 12 (`e5f81daf68` in the
+recorded run); it is distinct from the original duplicate-append baseline.
+To reproduce that earlier defect with a preserved pre-feature binary:
+
+```bash
+python3 scripts/workbench/append_identity_recovery_gate.py \
+ --source-dir "$NOKV_APPEND_SOURCE" \
+ --nokv-bin /absolute/path/to/frozen-pre-feature-nokv \
+ --mode baseline \
+ --evidence-dir "$NOKV_APPEND_EVIDENCE/historical-red"
+```
+
+The baseline records the binary's embedded build head separately from the
+current helper source head; do not relabel them as equal. The product gate
+creates the old-format fixture with its old binary, checks rejection without
+file changes, and reads it back with that old binary. It does not migrate it.
+
+Inspect `product-results.json` or `operations-results.json`: both safety and
+completion must pass, `all_listed_scenarios_executed` must be true, and
+`qualified_scopes` must contain the requested matrix. A selected subset, missing
+Python, or missing required baseline is not full qualification. The per-gate
+result identifies the executed harness; `environment.json` identifies the
+shared identity-gate stack helper separately. Retain both hashes, binary and
+extension identities, source manifests, fault transcripts, exact readbacks,
+and `cleanup.json`.
+
+The 1,000 ms activity lease is a supported test configuration, followed by the
+normal 30-second grace; observed duration is not the default recovery latency
+or an SLA. Same-directory Holt reopen is not cross-host HA. The new full 21 + 4
+matrix was run locally; the dated remote NoKV workflows run their existing
+suite, while a separate demo workflow tests actual Linux queue redelivery.
+These results do not alter the frozen pre-#423 qualification ledger or resolve
+the complete Workbench Gate 0.
+
## Live Workbench evidence
Dry-run validates the redacted command graph and exact 18-tool coverage without