Skip to content

Phase B: split content into per-Document/per-Collection Y.Doc shards; shard-aware WS/MCP/holds/audit routing #120

Description

@brylie

Implements the actual shard split from the approved #112 design, building on
#113 Phase A's catalog (PR #119). Original scope below, refined by two
detailed scoping/implementation passes this session — see PR #124 (holds
eviction-wiring fix) and PR #125 (shard-aware service-layer resolution).

Done

  • holds.ts's initHoldEviction process-global one-shot guard bug
    fixed in PR fix: wire hold eviction per-Awareness-instance, not process-wide #124 with a per-Awareness-instance WeakSet guard.
  • The record-locator gap for individual rows — PR Make every service function shard-aware, without cutting over shard assignment yet #125 adds
    reserveRecordLocator/releaseRecordLocator/resolveShardForParent/
    resolveShardForRecord to the catalog, closing the gap where
    write_record/delete_record/hold_records/release_records only ever
    receive a bare recordId with no parent hint.
  • Every service-layer function's resolveWorkspaceContext() call derives
    a real selector
    records.ts, collections.ts, holds.ts, search.ts,
    and permissions.ts's requireAccessibleRecord all resolve their actual
    target shard via the catalog now, instead of defaulting. Proven correct
    for a genuinely separate shard via tests that manually construct one
    (PR Make every service function shard-aware, without cutting over shard assignment yet #125), without cutting over shard assignment yet — see below.
  • The cross-shard holds design forkhold_records/release_records
    now group recordIds by resolved shard and operate against each shard's
    own Awareness, merging results (a cross-document agent batch is a stated
    acceptance criterion — collaboration.md).
  • src/lib/data/records.ts and src/lib/server/audit-observer.ts turned
    out to need no changes for Collection shards specifically: a
    per-collection shard doc with the same internal map shape (a collections
    map with one entry, a records map with just that collection's rows)
    works correctly unchanged, and topLevelMaps stays correct since the map
    names don't change.

What's left

  1. The actual shard-assignment cutover. createCollection still
    deliberately assigns shardId: 'default' — PR Make every service function shard-aware, without cutting over shard assignment yet #125's resolution
    mechanism is proven correct but not yet exercised in production. Flipping
    this over requires, together, in one PR (a genuine flag-day cutover, no
    safe partial version — the browser's live Table view would silently lose
    sync, or the SSR page would show nothing, the moment a Collection's
    content moves to a shard the client doesn't know to connect to):
    • createCollection assigns a real, distinct shardId per Collection.
    • src/lib/server/attach-ws.ts: parse the WS room-name path segment into
      a real, catalog-validated shard selector instead of discarding it (the
      known gap from issue Workspace-scoped CRDT registry and authenticated sync routing #30). In Phase 0 (no auth at all, by design — see
      README.md's "Out of scope" section), the reasonable trust model is:
      validate a client-supplied room name against the catalog's real shard
      locator rather than trust it verbatim — equivalent trust to what
      already exists everywhere else in Phase 0, just made explicit.
    • src/lib/client/yjs-client.ts (currently a bare module singleton, one
      hardcoded 'workspace' room) + the 8+ Svelte components/routes that
      call it directly (TableCollectionView.svelte,
      CalendarCollectionView.svelte, BoardCollectionView.svelte,
      FieldMenu.svelte, FieldManagerDialog.svelte, Sidebar.svelte,
      CollectionViewBlock.svelte, table/[id]/+page.svelte,
      doc/[id]/+page.svelte) need real per-shard room connections.
    • src/routes/table/[id]/+page.server.ts (bypasses the service layer
      already, pre-existing) needs to resolve and hand the client the real
      shard selector.
  2. Document sharding. A separate, larger piece than Collections —
    Documents have hierarchy and recursive delete, unlike Collections, so
    src/lib/data/records.ts's parentKindOf, deleteDocument's recursive
    descendant delete, and repairEmbeddedViewsAfter*Removal (currently
    whole-doc scans) need to become catalog-driven cross-shard operations.
  3. src/lib/mcp/tokens.ts: add workspace/shard scoping to access_tokens'
    allowlists (currently flat, workspace-unaware document/collection id
    arrays) — needed once more than one workspace is real, not strictly
    blocking the Collection-shard cutover above.

Not in scope: the SSE catalog feed (Phase C, #121), lazy load/unload
(Phase D, #122), and the capacity benchmark re-run (Phase E, #123).

Done when: no single global content Y.Doc path remains — every boundary
(routes, services, MCP, WebSocket, holds, audit) resolves and operates on
the correct per-Document/per-Collection shard.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions