diff --git a/docs/specifications/README.md b/docs/specifications/README.md index 80006df..81938f5 100644 --- a/docs/specifications/README.md +++ b/docs/specifications/README.md @@ -3,7 +3,7 @@ Canonical specification for each implemented subsystem, translating [`prd.md`](../prd.md) into concrete engineering decisions. Split by feature rather than kept as one document, so a change to one subsystem doesn't require editing (or risk going stale in) an unrelated one. - [`architecture.md`](./architecture.md) — process model (one Node process, `/ws` + `/mcp`) and SvelteKit app structure. -- [`workspace-sharding.md`](./workspace-sharding.md) — proposed Workspace catalog, content-shard, and committed-catalog-write/SSE design for #112; pending #31 measurement before approval. +- [`workspace-sharding.md`](./workspace-sharding.md) — approved Workspace catalog, content-shard, and committed-catalog-write/SSE design (#112); #113 must still prove it with real shard-aware transport measurements before it's treated as implemented. - [`data-model.md`](./data-model.md) — the Record/Document/Collection types and their Yjs mapping. - [`collaboration.md`](./collaboration.md) — presence and holds, built on Yjs Awareness. - [`mcp-tools.md`](./mcp-tools.md) — the MCP tool surface and permission scoping. diff --git a/docs/specifications/architecture.md b/docs/specifications/architecture.md index 6be1fd0..7d6ddf6 100644 --- a/docs/specifications/architecture.md +++ b/docs/specifications/architecture.md @@ -52,9 +52,10 @@ architecture boundary is therefore document- and Collection-level Yjs shards, with workspace navigation/catalog updates kept outside those content shards. That is a direction validated by the baseline, not a claim that shard-aware routing is shipped: Phase 0 still resolves every live connection to the one -default shard. #112 defines the proposed catalog/shard design, and #113 must -prove it with real per-shard transport measurements before this architecture -can describe it as implemented. +default shard. [`workspace-sharding.md`](./workspace-sharding.md) (#112) +records the **approved** catalog/shard design, and #113 must prove it with +real per-shard transport measurements before this architecture can describe +it as implemented. **Why HTTP MCP transport, not stdio:** Claude Desktop, Claude Code, and ChatGPT all support pointing at a remote MCP server via URL + bearer token in their own config (per the docs linked in the PRD). That matches the "simple local access token" decision better than stdio, which would mean each client spawns and manages its own subprocess — more moving parts for no benefit when everything's already running as one long-lived local service. diff --git a/docs/specifications/collaboration.md b/docs/specifications/collaboration.md index 0db93cb..f319f80 100644 --- a/docs/specifications/collaboration.md +++ b/docs/specifications/collaboration.md @@ -8,7 +8,7 @@ Built on Yjs Awareness, not a custom channel. Yjs ships an **Awareness** protoco - Each connected client (browser tab or MCP session) publishes an Awareness state: `{ actor: ActorId, heldRecordIds: string[] }`. - **Human cursor presence → implicit hold:** the browser UI updates its own Awareness state's `heldRecordIds` to `[currentBlockId]` whenever the cursor moves, debounced ~1.5s on move-away (per PRD). -- **Agent hold request:** the MCP server's `hold_records` tool handler checks the _aggregate_ Awareness state across all connected clients for each requested record ID. A record already present in another client's `heldRecordIds` is denied for this request; everything else is granted. This directly implements the PRD's per-record (not all-or-nothing) acceptance criterion. +- **Agent hold request:** the MCP server's `hold_records` tool handler checks the _aggregate_ Awareness state across all connected clients for each requested record ID. A record already present in another client's `heldRecordIds` is denied for this request; everything else is granted. This directly implements the PRD's per-record (not all-or-nothing) acceptance criterion. Accurate for Phase 0's single global `Y.Doc`/Awareness pair; [`workspace-sharding.md`](./workspace-sharding.md) (#112, approved) moves Awareness to be shard-local once #113 lands, with a separate workspace-scoped hold coordinator (§3.3) aggregating across shards for a cross-document agent batch — the per-record semantics above don't change, only where the aggregation happens. - **TTL:** two distinct timeouts are involved, not one: - y-protocols' own Awareness implementation clears any client's state after **30s** of no heartbeat (`outdatedTimeout`) — this is what catches an abruptly-dropped connection (e.g. a browser tab closing without a clean disconnect). - Agent holds additionally carry a dedicated **100s** `AGENT_HOLD_TTL_MS` timer (`src/lib/server/holds.ts`), scheduled server-side whenever a hold is granted or renewed, sitting inside the PRD's 90–120s auto-release target. This is what auto-releases a hold an agent forgot to explicitly release even while its underlying connection/token is still otherwise alive — the 30s Awareness timeout alone wouldn't cover that case, since a stateless MCP agent has no persistent heartbeat to time out in the first place. diff --git a/docs/specifications/data-model.md b/docs/specifications/data-model.md index ca56396..96c9156 100644 --- a/docs/specifications/data-model.md +++ b/docs/specifications/data-model.md @@ -150,7 +150,7 @@ Blocks are deliberately a small, documentation-oriented set rather than one type ## 4. Yjs mapping -One `Y.Doc` for the whole workspace (single-tenant, single workspace — no need for multiple docs yet). +One `Y.Doc` for the whole workspace (single-tenant, single workspace — no need for multiple docs yet). This is accurate Phase-0 behavior, not a permanent architectural stance: [`workspace-sharding.md`](./workspace-sharding.md) (#112) has approved a future one-`Y.Doc`-per-Document/Collection direction, backed by a server-owned SQLite catalog for title/hierarchy/scope; #113 implements and measures it before this table is rewritten to match. | Concept | Yjs type | Notes | | ----------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | diff --git a/docs/specifications/mcp-tools.md b/docs/specifications/mcp-tools.md index 6005b17..b53d253 100644 --- a/docs/specifications/mcp-tools.md +++ b/docs/specifications/mcp-tools.md @@ -22,4 +22,4 @@ All tools operate on `WorkspaceRecord` uniformly — no separate "block tools" v | `create_record` | `parentId, afterRecordId?, blockType? \| properties?, referencedRecordId?` | `{recordId}` | no hold needed — no prior content to protect; `referencedRecordId` sets a `page_link` block's target in the same call — only accepted when `blockType` is `page_link` and `parentId` names a Document; the target must be an existing, caller-accessible Document, else the call is rejected with one generic "not an accessible Document" error that never distinguishes "doesn't exist" from "exists but out of scope" (see [`internal-links.md`](./internal-links.md) §4, [`audit-coverage.md`](./audit-coverage.md) §3); `blockType: 'collection_view'` is accepted the same generic way every `BlockType` is, but unlike `page_link` its `referencedRecordId`/`viewConfig` aren't yet validated on this path (no accessible-Collection check, no `viewConfig` field at all in the current MCP schema) — see [`collection-views.md`](./collection-views.md) §6 | | `delete_record` | `recordId` | `{success: boolean}` | no hold needed | -**Permission scoping:** each access token (see [`persistence.md`](./persistence.md) §1) carries an allowlist of Document/Collection IDs. Every tool call resolves the target record and checks its `parentId` against that allowlist before granting a hold or applying any write — a denied record's Awareness/`Y.Doc` state is only ever read (to resolve it and to run the check itself), never mutated on its behalf. This is what "per-agent permission scoping" means concretely, even single-tenant: a token scoped to one Document can't read or write anything else, which is directly testable per the PRD's permission-denied acceptance criterion. A denied call is itself audited (as `_denied`, carrying only the target ID the caller already supplied) — see [`audit-coverage.md`](./audit-coverage.md) §3 — so a trust review can see repeated denied attempts, not just successful ones. +**Permission scoping:** each access token (see [`persistence.md`](./persistence.md) §1) carries an allowlist of Document/Collection IDs, checked with operation-specific rules rather than one uniform record-parent check: `get_document`, `query_collection`, and document mutations (`create_document`, `move_document`, `delete_document`) authorize the target Document/Collection ID itself; record operations (`hold_records`, `write_record`, `create_record`, `delete_record`, `release_records`) authorize the target record's `parentId`; `list_documents`, `list_collections`, and `search_workspace` filter their results down to allowed Document/Collection IDs rather than authorizing a single target; and `move_document` validates both the moved `documentId` and the destination `parentDocumentId`. [`workspace-sharding.md`](./workspace-sharding.md) (#112, approved) §5 extends this same scoping into a trusted-routing rule at the MCP boundary once #113 lands, resolving a token's Workspace/Space scope and target shard before any hold or write — a denied record's Awareness/`Y.Doc` state is only ever read (to resolve it and to run the check itself), never mutated on its behalf. This is what "per-agent permission scoping" means concretely, even single-tenant: a token scoped to one Document can't read or write anything else, which is directly testable per the PRD's permission-denied acceptance criterion. A denied call is itself audited (as `_denied`, carrying only the target ID the caller already supplied) — see [`audit-coverage.md`](./audit-coverage.md) §3 — so a trust review can see repeated denied attempts, not just successful ones. diff --git a/docs/specifications/persistence.md b/docs/specifications/persistence.md index 4823168..07cfc7d 100644 --- a/docs/specifications/persistence.md +++ b/docs/specifications/persistence.md @@ -8,7 +8,7 @@ [Drizzle](https://orm.drizzle.team/) (Apache 2.0 — same license as this project's own goal, not just compatible with it) manages every SQLite table, including the read model in §2 — one schema/migration story (`drizzle-kit`) instead of hand-written SQL for some tables and a separate library for others. It's also the SvelteKit-idiomatic choice: the Svelte CLI ships a built-in `drizzle` setup command. -- `snapshots` — periodic binary snapshots of the `Y.Doc` state (via `Y.encodeStateAsUpdate`). Simpler than an update-log replay for now; revisit if snapshot size becomes a problem. Keyed by `(workspace_id, shard_id)` (`src/lib/server/workspace-store.ts`'s `DEFAULT_WORKSPACE_ID`/`DEFAULT_SHARD_ID` today — see `architecture.md` §1) — every read/write/retention-prune query in `store.ts`'s `getSnapshotStore()` scopes by that key, so a second workspace's snapshots, once #13/#6 land, can never collide with or prune the first's. A given key's snapshot is loaded lazily, the first time `resolveWorkspaceContext()` resolves that workspace/shard — not eagerly for every context on process start, since Phase 0 only ever resolves the one default key anyway. The [2026-08-30 capacity baseline](../benchmarks/crdt-capacity-baseline-2026-08-30.md) confirms this snapshot path for a small daily workspace, but also establishes a global-snapshot budget: at 2 MiB or more, or when the associated event-loop p99 reaches 100 ms, prioritize document/Collection sharding or compaction rather than extending the Phase 0 global-state envelope. Those are engineering escalation signals, not a claim that either future design is already implemented. +- `snapshots` — periodic binary snapshots of the `Y.Doc` state (via `Y.encodeStateAsUpdate`). Simpler than an update-log replay for now; revisit if snapshot size becomes a problem. Keyed by `(workspace_id, shard_id)` (`src/lib/server/workspace-store.ts`'s `DEFAULT_WORKSPACE_ID`/`DEFAULT_SHARD_ID` today — see `architecture.md` §1) — every read/write/retention-prune query in `store.ts`'s `getSnapshotStore()` scopes by that key, so a second workspace's snapshots, once #13/#6 land, can never collide with or prune the first's. A given key's snapshot is loaded lazily, the first time `resolveWorkspaceContext()` resolves that workspace/shard — not eagerly for every context on process start, since Phase 0 only ever resolves the one default key anyway. The [2026-08-30 capacity baseline](../benchmarks/crdt-capacity-baseline-2026-08-30.md) confirms this snapshot path for a small daily workspace, but also establishes a global-snapshot budget: at 2 MiB or more, or when the associated event-loop p99 reaches 100 ms, prioritize document/Collection sharding or compaction rather than extending the Phase 0 global-state envelope. Those are engineering escalation signals, not a claim that either future design is already implemented. [`workspace-sharding.md`](./workspace-sharding.md) (#112) is the approved sharding design these signals point toward, including its own snapshot/recovery contract (§6) that supersedes the single `(workspace_id, shard_id)` key above once #113 implements it. - `audit_log` — append-only: `id, actor_json, action, target_record_id, timestamp, diff_json`. Populated on every write/delete tool call, denied MCP attempt, and UI-originated write — including UI writes that bypass the service layer entirely, via a generic `Y.Doc`-level observer — satisfying the PRD's audit-log requirement; see [`audit-coverage.md`](./audit-coverage.md) for exactly what's covered, what's deliberately excluded, and how attribution and volume are handled. - `access_tokens` — `token_hash, client_label, allowed_document_ids, allowed_collection_ids, created_at`. `client_label` is a free-text field set when the token is created (e.g., "Claude Desktop") — this is what powers the "Brylie · via Claude Desktop" attribution tag, since there's no per-vendor OAuth to source it from otherwise. diff --git a/docs/specifications/workspace-sharding.md b/docs/specifications/workspace-sharding.md index a3447a7..3e6adea 100644 --- a/docs/specifications/workspace-sharding.md +++ b/docs/specifications/workspace-sharding.md @@ -1,7 +1,13 @@ # Workspace catalog and CRDT sharding -**Status:** Proposed — #31's capacity baseline is complete; design approval is -pending review of the contracts in this document. +**Status:** Approved (2026-08-30) — the catalog/content-shard boundary, +ownership model, trusted-routing rules, committed-catalog-write/SSE contract, +and migration design in this document are the approved direction for #113 +(implementation) and #114 (migration and isolation). Approval is conditional +per §8/§10: #113 must reproduce the required measurements with real +shard-aware transport before this boundary is treated as load-bearing, and +its results may still adjust the Collection partition threshold, event- +retention window, or snapshot cadence. **Depends on:** [`prd.md`](../prd.md), [`architecture.md`](./architecture.md), [`persistence.md`](./persistence.md), [`collaboration.md`](./collaboration.md), @@ -29,13 +35,13 @@ not as the universal transport for every reactive interface element. - Yjs WebSocket and Awareness connections are scoped to the explicitly opened Document or Collection shard. -This is intentionally a proposed decision, not an implementation commitment. -The completed [#31 capacity baseline](../benchmarks/crdt-capacity-baseline-2026-08-30.md) +This is the approved architecture direction, not yet an implementation +commitment. The completed [#31 capacity baseline](../benchmarks/crdt-capacity-baseline-2026-08-30.md) supports a small daily Phase-0 workspace and establishes a global-state escalation boundary. Its document-size projection supports this shard direction, -but does not substitute for real shard-aware transport measurements. The -approval record for #112 must state which assumptions survived that measurement -and which must be proved by #113. +but does not substitute for real shard-aware transport measurements. §10 +records which assumptions survived that measurement and which #113 must still +prove. ## 2. Terms and boundaries @@ -323,3 +329,46 @@ require further partitioning or compaction. the same Compendium version concurrently; a later scale-out implementation proves lease, outbox-claim, and event-backplane coordination within one Deployment instance. + +## 10. Approval decision (recorded 2026-08-30) + +Approved for #113/#114 to build against, on the completed #31 baseline and +the contracts specified above: + +- The catalog/content-shard split in §1–§3: a durable SQLite workspace + catalog owning Space, Document, and Collection identity/title/hierarchy/ + scope, separate from one Y.Doc per Document and one Y.Doc per Collection. +- The trusted-routing rule in §5: every boundary (page load, WebSocket + upgrade, SSE, MCP, persistence) resolves a server-authorized Workspace/ + Space/shard context; a client-supplied selector is never authority. +- The committed-catalog-write/durable-operation/SSE contract in §4, including + the `pending_content → content_durable → publishable → published` state + machine, the public-revision/outbox-cursor ordering guarantees, and the + `catalog-resync` behavior for scope changes and cursor gaps. +- The migration design in §7: versioned, idempotent, checksum-verified, + ID-preserving migration of the current single-space workspace into one + default Space, with the original snapshot retained read-only until + verification passes. +- The deployment-ownership and coordination rules in §2: one process per + Deployment instance today; a future scale-out deployment requires durable + shard-ownership leases, transactional outbox claims, and a shared event + backplane — independent mutable in-memory replicas are invalid. + +Explicitly deferred, not settled by this approval: + +- The exact Collection row-partition threshold, event-retention window, and + snapshot cadence — §8 requires #113 to supply these from real shard-aware + measurements, not from the #31 global-workspace projection. +- Whether the Document/Collection shard granularity itself holds at the + concurrency and payload sizes #113 measures; a materially different result + reopens this document rather than being silently absorbed into #113. +- Where a **saved, shareable view configuration** (PRD P1) is addressed once + it exists as its own catalog-navigable artifact rather than an embedded + `collection_view` block's `viewConfig` (data-model.md §4) — out of scope + here because the feature itself isn't built yet; #113 or a follow-up must + extend §3.1's catalog fields to cover it before that feature lands. +- Storage-engine changes to `data-model.md` §4 ("One `Y.Doc` for the whole + workspace") and `collaboration.md`'s aggregate-Awareness description — both + remain accurate descriptions of current Phase-0 code and are superseded + only once #113 actually implements per-shard `Y.Doc`s, not by this + approval alone.