docs: propose workspace catalog and CRDT sharding design

docs/specifications/README.md

@@ -3,6 +3,7 @@
 Canonical specification for each implemented subsystem, translating [`prd.md`](../prd.md) into concrete engineering decisions. Split by feature rather than kept as one document, so a change to one subsystem doesn't require editing (or risk going stale in) an unrelated one.
 
 - [`architecture.md`](./architecture.md) — process model (one Node process, `/ws` + `/mcp`) and SvelteKit app structure.
+- [`workspace-sharding.md`](./workspace-sharding.md) — proposed Workspace catalog, content-shard, and committed-catalog-write/SSE design for #112; pending #31 measurement before approval.
 - [`data-model.md`](./data-model.md) — the Record/Document/Collection types and their Yjs mapping.
 - [`collaboration.md`](./collaboration.md) — presence and holds, built on Yjs Awareness.
 - [`mcp-tools.md`](./mcp-tools.md) — the MCP tool surface and permission scoping.

docs/specifications/workspace-sharding.md

@@ -0,0 +1,325 @@
+# Workspace catalog and CRDT sharding
+
+**Status:** Proposed — #31's capacity baseline is complete; design approval is
+pending review of the contracts in this document.
+
+**Depends on:** [`prd.md`](../prd.md), [`architecture.md`](./architecture.md),
+[`persistence.md`](./persistence.md), [`collaboration.md`](./collaboration.md),
+and [`mcp-tools.md`](./mcp-tools.md)
+
+**Tracked by:** #112 (design), #113 (implementation), #114 (migration and
+isolation), and #6 (multi-space experience)
+
+## 1. Decision to validate
+
+Compendium uses CRDTs where several clients concurrently edit rich content,
+not as the universal transport for every reactive interface element.
+
+- A server-owned **workspace catalog** is durable SQLite state. It owns Spaces,
+  document and Collection titles, hierarchy, membership, access scopes, and the
+  current catalog revision.
+- A **Document shard** is one Y.Doc containing one Document's ordered blocks
+  and rich text.
+- A **Collection shard** is one Y.Doc containing one Collection's schema and
+  rows. It is the initial unit; a later row partition requires a measured
+  threshold and a separate design decision.
+- The browser uses a small server-sent-events feed for catalog changes. SSE
+  delivers invalidations; it never becomes a second source of truth or a
+  replacement for collaborative content sync.
+- Yjs WebSocket and Awareness connections are scoped to the explicitly opened
+  Document or Collection shard.
+
+This is intentionally a proposed decision, not an implementation commitment.
+The completed [#31 capacity baseline](../benchmarks/crdt-capacity-baseline-2026-08-30.md)
+supports a small daily Phase-0 workspace and establishes a global-state
+escalation boundary. Its document-size projection supports this shard direction,
+but does not substitute for real shard-aware transport measurements. The
+approval record for #112 must state which assumptions survived that measurement
+and which must be proved by #113.
+
+## 2. Terms and boundaries
+
+| Term                | Meaning                                                                                                                                                   | Authority                                            |
+| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
+| Deployment instance | Server-owned instance identity, persistence target, and event scope. One or more coordinated processes with all three values belong to the same instance. | Server configuration; see #111.                      |
+| Workspace           | The top-level durable knowledge environment within a deployment instance.                                                                                 | Server-authorized request context.                   |
+| Space               | An organizational and permission boundary within one Workspace. A person switches Spaces to work on different projects.                                   | Workspace catalog.                                   |
+| Catalog             | The SQLite metadata projection needed to route, list, search, and authorize content without opening every content shard.                                  | Catalog service and committed database transactions. |
+| Content shard       | The Y.Doc for one Document or Collection.                                                                                                                 | Authorized Yjs and service-layer mutations.          |
+
+A deployment instance is not a Workspace, and a Space is not a tenant. This
+keeps local operational isolation (#111), single-user multi-space organization
+(#6), and eventual multi-user authentication (#3) independently evolvable.
+
+Phase 0 and the first #113 implementation run one application process per
+Deployment instance. Starting independent processes against the same instance
+identity and SQLite target is unsupported until the coordination contract below
+is implemented; separate personal, development, and test instances must use
+distinct server-owned identities and persistence targets. A future scale-out
+deployment still represents one Deployment instance only when every process
+shares the same identity, catalog/outbox database, authorization policy, and
+event backplane. It must use durable shard ownership leases, transactional
+outbox claims, and a deployment-scoped event backplane; independent mutable
+in-memory replicas are invalid. #111 owns the startup diagnostics and isolation
+tests, while #113 must not silently introduce a second writer.
+
+## 3. Ownership model
+
+### 3.1 Workspace catalog
+
+The catalog is the authoritative location for the metadata required to navigate
+and authorize a Workspace without subscribing to its content shards:
+
+- Workspace and Space identities, titles, ordering, and membership.
+- Document and Collection identities, titles, owning Space, hierarchy/order,
+  and shard locator.
+- Space-level access grants and the derived scopes assigned to access tokens.
+- A workspace-wide record locator with a unique `(workspace_id, record_id)`
+  key, owning-shard locator, and record kind.
+- Catalog revision and committed event records.
+
+Document titles and hierarchy are catalog fields. They are not duplicated into a
+Document Y.Doc and derived later, because the sidebar needs them before opening
+the Document. Record IDs remain stable global identities; a title is never an
+identity or authorization key. Creating a record reserves its supplied or
+generated ID in the workspace-wide locator transaction before inserting it into
+the target shard. A duplicate `(workspace_id, record_id)` is rejected; a retry
+with the same operation ID returns its original result rather than creating a
+second record. The hold coordinator resolves this locator before using its
+`(workspaceId, recordId)` key, so a hold cannot conflate records in different
+shards.
+
+The existing `snapshots`, `access_tokens`, and `audit_log` storage must become
+explicitly scoped by Workspace and, where applicable, Space and shard. A
+catalog migration must preserve existing IDs and create one default Space for
+the current single-space workspace.
+
+### 3.2 Document and Collection shards
+
+A Document shard owns its block records, their order, Y.Text content, and the
+rich-content collaboration state for that Document. A Collection shard owns its
+schema, rows, and Collection-view source data. A content shard does not own its
+title, parent hierarchy, or Space membership.
+
+Relations, page links, and collection-view references remain ID-backed. A
+cross-Space reference is valid only when the caller's permission scope permits
+both sides; rendering and search must not use a catalog lookup to reveal an
+inaccessible target's title.
+
+### 3.3 Ephemeral coordination
+
+Awareness belongs to the active content shard: cursors, active editing state,
+and visual agent placeholders must not broadcast to clients outside that shard.
+
+The server retains a workspace-scoped, ephemeral hold coordinator keyed by
+`workspaceId` and record ID. It aggregates the active shard Awareness state
+needed to decide an MCP `hold_records` request, while projects only the result
+into shards whose authorized clients need to render it. This preserves a
+cross-document agent batch without leaking cursor or presence state to an
+unrelated Space.
+
+## 4. Committed catalog writes and SSE
+
+Catalog mutations use one service-layer transaction:
+
+```text
+authorize caller and target scope
+  → update catalog rows
+  → increment workspace catalog revision
+  → append catalog outbox event and audit entry
+  → commit SQLite transaction
+  → publish the committed event to active SSE subscribers
+```
+
+Operations that also move, create, or delete content use a durable operation
+row rather than publishing directly from the initial catalog transaction:
+
+```text
+pending_content → content_durable → publishable → published
+```
+
+The initial catalog transaction records the operation ID, affected shards,
+hidden desired catalog state, and a non-publishable outbox row. It does not
+allocate a public catalog revision or alter the published catalog projection.
+The shard transition writes the operation ID into its durable
+snapshot/recovery marker; only then does one transaction promote the desired
+state into the published catalog, allocate the next public Workspace revision,
+and mark its outbox row `publishable`. An outbox consumer may claim and emit
+only `publishable` rows.
+
+Page loads, catalog APIs, MCP routing reads, and SSE all use the published
+catalog projection only. While an operation is `pending_content`, a create and
+its reserved record IDs are unrouteable; a move continues to serve its previous
+published locator/hierarchy; and a delete continues to serve its previous
+published state. A failed or recovered operation either promotes atomically or
+leaves that prior public state intact. Restart recovery resumes a pending
+operation idempotently by inspecting durable markers, never by assuming an
+internal catalog intent proves the content transition completed. This prevents a
+client from being routed to absent or stale content.
+
+Public revisions are allocated only at that promotion transaction and are
+strictly ordered per Workspace. An outbox consumer delivers a contiguous prefix
+of publishable public revisions; it cannot advance a stream past an earlier
+unresolved operation. If durable recovery cannot provide that prefix, the
+consumer emits `catalog-resync` rather than advancing a cursor across the gap.
+
+SSE is driven by the committed catalog write, never by the periodic persistence
+of a Y.Doc snapshot. A snapshot callback would be delayed, might batch unrelated
+content changes, and cannot safely identify which navigation metadata changed.
+
+The SSE endpoint emits compact invalidations, for example:
+
+```text
+id: opaque-authorized-stream-cursor
+event: catalog-changed
+data: {"documents":["…"],"spaces":["…"]}
+```
+
+The client treats this event as a hint to fetch authoritative catalog state. It
+does not mutate its catalog cache solely from event payload. On reconnect, the
+client supplies an opaque `Last-Event-ID` cursor. The cursor is scoped to the
+Deployment instance, Workspace, and an authorization-scope fingerprint; it
+maps server-side to the highest catalog revision delivered to that authorized
+stream, but never exposes a raw workspace revision or event ID. If the scope
+has changed, the cursor fails validation, retained source events no longer
+cover its hidden revision, or filtering makes replay coverage ambiguous, the
+server sends `catalog-resync` and the client reloads the authorized catalog
+snapshot. Inaccessible Space changes therefore create neither an observable
+cursor gap nor an existence signal.
+
+An SSE subscriber is authorized for a Workspace and a set of Spaces before it
+is registered. Filtering happens before an event is emitted, not after it
+reaches the browser. A permission revocation closes or resynchronizes affected
+subscriptions immediately. A move or permission change that crosses an
+authorization boundary sends a generic `catalog-resync` to subscribers whose
+authorized catalog may lose or gain the entry. That signal contains no Document
+ID, destination Space, title, hierarchy, or inaccessible metadata; the
+subscriber reloads its authorized catalog snapshot to remove or add entries.
+
+For the initial single-process deployment, an in-process publisher may drain
+only `publishable` outbox rows immediately. Multiple application processes
+require the deployment coordination rules in §2; independent mutable in-memory
+replicas are invalid.
+
+## 5. Trusted routing
+
+Every boundary resolves a server-authorized context before reading or mutating
+state:
+
+| Boundary                  | Required context                                                 | Result                                         |
+| ------------------------- | ---------------------------------------------------------------- | ---------------------------------------------- |
+| Page load and catalog API | deployment instance, Workspace, authorized Spaces                | Published catalog snapshot or a scoped subset. |
+| Yjs WebSocket upgrade     | deployment instance, Workspace, content shard, caller scope      | One authorized Document or Collection Y.Doc.   |
+| SSE connection            | deployment instance, Workspace, authorized Spaces, last revision | Scoped catalog invalidations.                  |
+| MCP tool                  | token-derived Workspace and Space scope, target shard            | Authorized catalog or content operation.       |
+| Persistence and audit     | Workspace, Space where applicable, shard                         | Independently keyed durable state.             |
+
+Path segments, query values, and Yjs room names are selectors only. They may
+choose among contexts the server has already authorized for that caller, but
+they never grant authority. The browser receives its bootstrap configuration
+from the server; it does not invent a Workspace identity.
+
+## 6. Persistence and recovery
+
+Catalog writes are immediately transactional SQLite writes. Yjs content retains
+its CRDT-first write path and is persisted as shard-scoped snapshots, with the
+update-log versus periodic-snapshot decision revisited from #31's recovery and
+write-churn measurements.
+
+There is no false atomicity claim between a catalog transaction and an arbitrary
+later Yjs snapshot. Operations that change both catalog and content use the
+durable operation state in §4: record an operation ID, make retries idempotent,
+persist a shard completion marker, and recover on restart from the operation,
+outbox, and relevant shard snapshot. Deleting or moving a Document must not
+publish its catalog event until its required durable transition is complete.
+
+Each shard supports lazy loading and flushing. It may unload only when it has no
+authorized live connections, no active holds requiring it, and no unflushed
+state. The catalog remains small and resident enough to route requests and
+serve the sidebar without opening content shards.
+
+## 7. Migration
+
+The Phase-0 workspace migrates into one Workspace with one default Space. A
+versioned migration first loads the legacy snapshot into an isolated Y.Doc and
+records a manifest keyed by `(legacy_snapshot_id, migration_version)`. It then
+deterministically creates:
+
+- one catalog row and Document shard for each legacy Document, with its ordered
+  block records and Y.Text content;
+- one catalog row and Collection shard for each legacy Collection, with its
+  schema and rows; and
+- workspace-wide record-locator entries for every migrated block and row.
+
+The migration copies IDs exactly: `parentId`, record order, relation targets,
+page-link targets, and collection-view references retain their original IDs and
+are resolved only after the corresponding target locator exists. Each target
+shard is snapshotted with the migration operation ID and content checksum before
+the manifest records it durable. The catalog exposes the new workspace only
+after every planned target has a durable marker and the locator manifest is
+complete. Re-running a migration reuses verified targets from that manifest and
+creates no duplicate rows, records, or snapshots.
+
+The original mixed Y.Doc snapshot and its audit/recovery metadata remain a
+read-only legacy recovery unit until all target checksums, IDs, references,
+token grants, and restart recovery checks pass and an explicit retention policy
+permits removal. A failed migration leaves that unit intact and does not expose
+a partial catalog. The migration preserves:
+
+- Document, Collection, record, and relation IDs.
+- Document hierarchy and order.
+- Collection schema, rows, and embedded-view references.
+- Token grants, actor attribution, audit history, and snapshot recovery data.
+
+Migration is versioned, idempotent, observable, and tested against a copied
+representative database.
+
+## 8. Measurements that decide approval
+
+The completed #31 baseline records the Phase-0 global-workspace envelope. #113
+must rerun a workspace seeded with representative data and a realistic
+concurrent browser/MCP workload with real shards, recording:
+
+- Per-Document and per-Collection encoded state size.
+- Catalog size, catalog refresh payload, and SSE reconnect/resync behavior.
+- Cold load, initial Yjs sync, update-apply latency, and fan-out bytes per
+  active shard.
+- Server/client memory, CPU, event-loop delay, snapshot/restart time, and
+  recovery behavior.
+- The cost of an all-Yjs navigation catalog compared with committed catalog
+  writes plus SSE invalidation.
+
+The approval decision must state whether the proposed Document/Collection
+granularity is accepted, whether Collections need an initial partition rule,
+the retained event window, snapshot cadence, and any measured thresholds that
+require further partitioning or compaction.
+
+## 9. Required verification
+
+- A client connected to one content shard never receives another shard's Yjs
+  updates or Awareness state.
+- A Space-scoped SSE subscriber never receives an event that reveals an
+  inaccessible title, hierarchy, or existence signal.
+- A missed SSE event causes a revision-aware catalog resync, not a stale UI.
+- Pending content operations are absent from routing reads; a move/delete
+  retains its previous published state until its successor is durable.
+- Public outbox delivery cannot advance an SSE cursor past an unresolved earlier
+  Workspace operation; recovery produces a contiguous prefix or resync.
+- An opaque SSE cursor cannot reveal an inaccessible Space change; a changed or
+  ambiguous authorized scope causes a catalog resync.
+- A move out of scope, move into scope, and permission change each trigger a
+  generic scoped resync without naming an inaccessible target or destination.
+- UI and MCP operations enforce the same catalog, Space, and shard scope.
+- A cross-document agent hold works only for records the caller can access and
+  does not expose active editing state outside the relevant shard.
+- A duplicate caller-supplied record ID in two shards of one Workspace is
+  rejected, while independent Workspaces remain isolated.
+- Crash/restart recovery never emits a move/delete catalog event before its
+  target shard transition is durable, and retries remain idempotent.
+- Existing data migrates losslessly into the default Space and survives restart:
+  fixtures cover mixed Documents/Collections, IDs, embedded views, links,
+  recovery metadata, and a re-run after partial completion.
+- #111 proves two local deployment instances remain isolated even when they run
+  the same Compendium version concurrently; a later scale-out implementation
+  proves lease, outbox-claim, and event-backplane coordination within one
+  Deployment instance.