From d16dbbe1d9c222eac6f465deab5c00c30cd7e4c5 Mon Sep 17 00:00:00 2001 From: Brylie Christopher Oxley Date: Sun, 30 Aug 2026 16:13:56 +0300 Subject: [PATCH 1/5] test: benchmark CRDT workspace capacity --- .../crdt-capacity-baseline-2026-08-30.md | 84 ++++++ package.json | 2 + tests/benchmark/workspace-capacity.test.ts | 273 ++++++++++++++++++ tests/e2e/harness.ts | 45 ++- vite.config.ts | 11 + 5 files changed, 413 insertions(+), 2 deletions(-) create mode 100644 docs/benchmarks/crdt-capacity-baseline-2026-08-30.md create mode 100644 tests/benchmark/workspace-capacity.test.ts diff --git a/docs/benchmarks/crdt-capacity-baseline-2026-08-30.md b/docs/benchmarks/crdt-capacity-baseline-2026-08-30.md new file mode 100644 index 0000000..8f15184 --- /dev/null +++ b/docs/benchmarks/crdt-capacity-baseline-2026-08-30.md @@ -0,0 +1,84 @@ +# CRDT capacity baseline — 2026-08-30 + +Issue [#31](https://github.com/brylie/compendium/issues/31) establishes a +measured baseline before the workspace catalog and shard design in [#112](https://github.com/brylie/compendium/issues/112) +is approved. This is deliberately a real transport benchmark: it uses the +application's temporary SQLite database, Yjs WebSocket endpoint, and MCP HTTP +server rather than measuring isolated Yjs objects only. + +## Method + +The benchmark seeds deterministic documents, blocks, collections, and rows; +then connects Yjs clients, performs a human-originated WebSocket mutation, and +runs sequential MCP `hold_records` and `write_record` calls. It captures: + +- encoded global Yjs state and persisted snapshot sizes; +- initial client-sync bytes and elapsed time; +- receiver-side WebSocket bytes for one fan-out mutation; +- MCP write p50/p95 elapsed time; +- process heap delta, CPU time, and event-loop-delay p99; +- snapshot-backed context restart time and restored state size. + +The bounded `daily` profile is suitable for CI. The `large` profile is a +manual regression check. Run them with: + +```sh +npm run benchmark:workspace +npm run benchmark:workspace:large +``` + +The harness uses a fresh temporary database and random localhost port, so it +cannot read or modify a running daily workspace database. + +## Results + +Measurements were taken on 2026-08-30 in the local Node test environment. +They are a baseline and trend signal, not universal production SLOs. + +| Metric | Daily: 12 docs, 192 blocks, 3 collections / 120 rows, 3 clients, 12 MCP writes | Large: 120 docs, 2,880 blocks, 8 collections / 3,200 rows, 8 clients, 80 MCP writes | +| ----------------------------------- | -----------------------------------------------------------------------------: | ----------------------------------------------------------------------------------: | +| Encoded global state | 142,694 B | 2,848,008 B | +| Aggregate initial-sync bytes | 142,776 B | 8,544,250 B | +| All-clients initial-sync elapsed | 93.5 ms | 973.2 ms | +| Receiver fan-out bytes (one edit) | 116 B | 406 B | +| Fan-out convergence elapsed | 27.2 ms | 25.9 ms | +| MCP write p50 / p95 | 5.5 / 8.4 ms | 4.9 / 9.0 ms | +| Persisted snapshot | 143,424 B | 2,852,778 B | +| Snapshot-backed restart | 9.4 ms | 130.1 ms | +| Process heap delta | 25.3 MB | 396.1 MB | +| Event-loop p99 | 43.94 ms | 270.01 ms | +| One-document shard state projection | 6,795 B | 10,147 B | + +## Interpretation and decision boundary + +The daily profile fits comfortably inside the initial CI guardrails: less than +2 MiB aggregate initial sync, less than 2 seconds initial sync and restart, +and less than 1.5 seconds MCP write p95. It is therefore safe to start using +Compendium for a small daily Tech with Brylie workspace while the workspace +work proceeds. + +The large profile exposes the existing global-document cost: every new client +receives the whole 2.85 MB workspace state, and the one-process benchmark +showed a 396 MB heap increase with a 270 ms event-loop p99. A same-shape +single-document state is roughly 10 KB, which makes document-level Yjs shards +the appropriate next boundary. The catalog/SSE design in #112 keeps titles and +navigation outside that document state so unrelated document edits need not +grow client synchronization or CRDT fan-out. + +This projection is intentionally not presented as a shipped shard-aware +transport result: current Phase 0 routing still resolves every connection to +the global shard. #113 must rerun these profiles with real document and +collection routes before comparing end-to-end shard transport bytes. + +## Gates for the next implementation phase + +- Keep the `daily` profile in CI with its current conservative limits. +- Treat a global snapshot at or above 2 MiB, event-loop p99 at or above 100 ms, + or a daily CI guardrail failure as a trigger to prioritize sharding or + compaction work rather than increasing the global-state envelope. +- #113 must report per-shard encoded state, sync bytes, apply latency, and + receiver fan-out with two active document shards plus a collection shard. +- Browser heap remains a follow-up measurement: this benchmark's heap value is + the Node host running simulated Yjs clients, not a browser DevTools heap + snapshot. Add browser-memory collection when #24 establishes the user-facing + sync-latency SLO. diff --git a/package.json b/package.json index eab2670..f5fe563 100644 --- a/package.json +++ b/package.json @@ -14,6 +14,8 @@ "test:e2e:tier-a": "vitest --run tests/e2e/tier-a.test.ts", "test:e2e:tier-b": "playwright test", "test:e2e": "npm run test:e2e:tier-a && npm run test:e2e:tier-b", + "benchmark:workspace": "vitest run --project benchmark", + "benchmark:workspace:large": "COMPENDIUM_BENCHMARK_PROFILE=large vitest run --project benchmark", "test": "npm run test:unit -- --run", "test:coverage": "npm run test:unit -- --run --coverage", "lint": "prettier --check . && eslint .", diff --git a/tests/benchmark/workspace-capacity.test.ts b/tests/benchmark/workspace-capacity.test.ts new file mode 100644 index 0000000..c0eb861 --- /dev/null +++ b/tests/benchmark/workspace-capacity.test.ts @@ -0,0 +1,273 @@ +import { monitorEventLoopDelay, performance } from 'node:perf_hooks'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import * as Y from 'yjs'; +import { createTestHarness, type TestHarness } from '../e2e/harness'; +import { + createCollection, + createDocument, + createRecord, + getRecordYText, + listDocuments +} from '$lib/data/records'; +import { plainText, yTextToRichText } from '$lib/data/richtext'; +import { closeDb, getSnapshotStore } from '$lib/server/store'; +import { + flush, + resetWorkspaceStoreForTests, + resolveWorkspaceContext +} from '$lib/server/workspace-store'; +import type { ActorId, PropertyDefinition } from '$lib/data/types'; + +const human: ActorId = { kind: 'human', userId: 'benchmark-seed' }; + +type Profile = { + documents: number; + blocksPerDocument: number; + collections: number; + rowsPerCollection: number; + clients: number; + mcpWrites: number; +}; + +const PROFILES: Record = { + // Bounded enough for CI, while resembling a small working knowledgebase. + daily: { + documents: 12, + blocksPerDocument: 16, + collections: 3, + rowsPerCollection: 40, + clients: 3, + mcpWrites: 12 + }, + // Run manually before changes that affect the global Phase-0 workspace. + large: { + documents: 120, + blocksPerDocument: 24, + collections: 8, + rowsPerCollection: 400, + clients: 8, + mcpWrites: 80 + } +}; + +function percentile(values: number[], fraction: number): number { + const sorted = [...values].sort((a, b) => a - b); + return sorted[Math.min(sorted.length - 1, Math.floor((sorted.length - 1) * fraction))] ?? 0; +} + +function parseMcpText(result: unknown): T { + const response = result as { content?: Array<{ text?: string }>; isError?: boolean }; + if (response.isError) throw new Error(response.content?.[0]?.text ?? 'MCP tool failed'); + return JSON.parse(response.content?.[0]?.text ?? '') as T; +} + +describe('CRDT workspace capacity baseline (issue #31)', () => { + let harness: TestHarness; + + beforeEach(async () => { + harness = await createTestHarness(); + }); + + afterEach(async () => { + await harness.cleanup(); + }); + + it('measures the real global-workspace transport envelope and a document-shard projection', async () => { + const profileName = process.env.COMPENDIUM_BENCHMARK_PROFILE ?? 'daily'; + const profile = PROFILES[profileName]; + if (!profile) throw new Error(`Unknown benchmark profile: ${profileName}`); + + const seedClient = harness.getYjsClient(); + const schema: PropertyDefinition[] = [ + { key: 'title', label: 'Title', type: 'text' }, + { + key: 'status', + label: 'Status', + type: 'select', + options: [ + { id: 'idea', label: 'Idea' }, + { id: 'ready', label: 'Ready' }, + { id: 'done', label: 'Done' } + ] + } + ]; + const documentIds: string[] = []; + const blockIds: string[] = []; + + // Fixture seeding is intentionally direct Yjs data creation. The measured + // workload below uses the public MCP and WebSocket transports; keeping + // setup outside that window makes the profile reproducible and focused. + seedClient.doc.transact(() => { + for (let documentIndex = 0; documentIndex < profile.documents; documentIndex += 1) { + const document = createDocument(seedClient.doc, { + title: `Benchmark document ${documentIndex + 1}` + }); + documentIds.push(document.id); + for (let blockIndex = 0; blockIndex < profile.blocksPerDocument; blockIndex += 1) { + const block = createRecord( + seedClient.doc, + { parentId: document.id, blockType: 'paragraph' }, + human + ); + getRecordYText(seedClient.doc, block.id)?.insert( + 0, + `Seed ${documentIndex + 1}.${blockIndex + 1}: durable workspace context. ` + ); + blockIds.push(block.id); + } + } + for (let collectionIndex = 0; collectionIndex < profile.collections; collectionIndex += 1) { + const collection = createCollection(seedClient.doc, { + title: `Benchmark collection ${collectionIndex + 1}`, + schema + }); + for (let rowIndex = 0; rowIndex < profile.rowsPerCollection; rowIndex += 1) { + createRecord( + seedClient.doc, + { + parentId: collection.id, + properties: { + title: { type: 'text', value: `Row ${collectionIndex + 1}.${rowIndex + 1}` }, + status: { type: 'select', value: rowIndex % 3 === 0 ? 'ready' : 'idea' } + } + }, + human + ); + } + } + }); + + await harness.waitForCondition(() => getRecordYText(seedClient.doc, blockIds[0]) !== undefined); + const globalStateBytes = Y.encodeStateAsUpdate(seedClient.doc).byteLength; + + const eventLoop = monitorEventLoopDelay({ resolution: 10 }); + eventLoop.enable(); + const heapBefore = process.memoryUsage().heapUsed; + const cpuBefore = process.cpuUsage(); + + const peers = Array.from({ length: profile.clients }, () => harness.getYjsClient()); + const initialSyncStart = performance.now(); + await Promise.all( + peers.map((peer) => + harness.waitForCondition( + () => peer.provider.synced && getRecordYText(peer.doc, blockIds[0]) !== undefined, + { + timeoutMs: 20_000 + } + ) + ) + ); + const initialSyncMs = performance.now() - initialSyncStart; + const initialSyncBytes = peers.reduce((total, peer) => total + peer.traffic.receivedBytes, 0); + + for (const peer of peers) peer.traffic.reset(); + const writer = peers[0]; + const fanoutStart = performance.now(); + const writerText = getRecordYText(writer.doc, blockIds[0]); + expect(writerText).toBeDefined(); + const fanoutMarker = `Human fan-out mutation ${Date.now()}.`; + writer.doc.transact(() => writerText?.insert(writerText.length, ` ${fanoutMarker}`)); + await Promise.all( + peers.slice(1).map((peer) => + harness.waitForCondition(() => { + const text = getRecordYText(peer.doc, blockIds[0]); + return text ? plainText(yTextToRichText(text)).includes(fanoutMarker) : false; + }) + ) + ); + // The peer state can converge before ws emits its frame accounting callback + // in the same Node turn. Let that callback drain before recording bytes. + await new Promise((resolve) => setTimeout(resolve, 25)); + const fanoutLatencyMs = performance.now() - fanoutStart; + const fanoutBytes = peers + .slice(1) + .reduce((total, peer) => total + peer.traffic.receivedBytes, 0); + + const { token } = harness.createToken({ + clientLabel: 'Capacity benchmark MCP agent', + allowedDocumentIds: documentIds, + allowedCollectionIds: [] + }); + const mcp = await harness.getMcpClient(token); + const mcpLatencies: number[] = []; + for (let writeIndex = 0; writeIndex < profile.mcpWrites; writeIndex += 1) { + const blockId = blockIds[writeIndex % blockIds.length]; + const start = performance.now(); + const hold = await mcp.callTool({ + name: 'hold_records', + arguments: { recordIds: [blockId] } + }); + expect(parseMcpText<{ granted: string[] }>(hold).granted).toContain(blockId); + await mcp.callTool({ + name: 'write_record', + arguments: { recordId: blockId, markdown: `MCP benchmark revision ${writeIndex + 1}` } + }); + await harness.waitForCondition(() => { + const text = getRecordYText(peers[0].doc, blockId); + return text + ? plainText(yTextToRichText(text)).includes(`MCP benchmark revision ${writeIndex + 1}`) + : false; + }); + mcpLatencies.push(performance.now() - start); + } + + await flush(); + const snapshot = getSnapshotStore('default', 'default').loadLatest(); + expect(snapshot).not.toBeNull(); + const snapshotBytes = snapshot?.byteLength ?? 0; + const restartStart = performance.now(); + closeDb(); + resetWorkspaceStoreForTests(); + const restored = resolveWorkspaceContext(); + const restartMs = performance.now() - restartStart; + const restoredStateBytes = Y.encodeStateAsUpdate(restored.doc).byteLength; + eventLoop.disable(); + + // A controlled state-size projection: one document's records in their own + // Y.Doc. It is not a replacement for #113's shard-aware transport; it + // quantifies the current global blast radius that #112 is intended to fix. + const projectedDocument = new Y.Doc(); + const projectionDocument = createDocument(projectedDocument, { title: 'Projected document' }); + for (let index = 0; index < profile.blocksPerDocument; index += 1) { + const block = createRecord(projectedDocument, { parentId: projectionDocument.id }, human); + getRecordYText(projectedDocument, block.id)?.insert(0, `Projected block ${index + 1}.`); + } + const projectedDocumentStateBytes = Y.encodeStateAsUpdate(projectedDocument).byteLength; + projectedDocument.destroy(); + + const result = { + profile: profileName, + fixture: profile, + globalStateBytes, + initialSync: { totalBytes: initialSyncBytes, durationMs: Number(initialSyncMs.toFixed(1)) }, + fanout: { receiverBytes: fanoutBytes, durationMs: Number(fanoutLatencyMs.toFixed(1)) }, + mcpWriteLatency: { + p50Ms: Number(percentile(mcpLatencies, 0.5).toFixed(1)), + p95Ms: Number(percentile(mcpLatencies, 0.95).toFixed(1)) + }, + snapshotBytes, + restartMs: Number(restartMs.toFixed(1)), + restoredStateBytes, + process: { + heapDeltaBytes: process.memoryUsage().heapUsed - heapBefore, + cpuUserMicros: process.cpuUsage(cpuBefore).user, + eventLoopP99Ms: Number((eventLoop.percentile(99) / 1e6).toFixed(2)) + }, + shardProjection: { oneDocumentStateBytes: projectedDocumentStateBytes } + }; + console.log(`CRDT_CAPACITY_RESULT ${JSON.stringify(result)}`); + + // Conservative daily-workspace guardrails. Larger profiles are reporting + // runs, not CI gates, until #24 defines user-facing latency SLOs. + if (profileName === 'daily') { + expect(initialSyncBytes).toBeLessThan(2 * 1024 * 1024); + expect(initialSyncMs).toBeLessThan(2_000); + expect(percentile(mcpLatencies, 0.95)).toBeLessThan(1_500); + expect(restartMs).toBeLessThan(2_000); + } + expect(globalStateBytes).toBeGreaterThan(projectedDocumentStateBytes); + expect(fanoutBytes).toBeGreaterThan(0); + expect(restoredStateBytes).toBeGreaterThan(0); + expect(listDocuments(restored.doc)).toHaveLength(profile.documents); + }); +}); diff --git a/tests/e2e/harness.ts b/tests/e2e/harness.ts index e2343c0..c99a05b 100644 --- a/tests/e2e/harness.ts +++ b/tests/e2e/harness.ts @@ -2,7 +2,7 @@ import { createServer, type Server, type IncomingMessage, type ServerResponse } import { mkdtempSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import WebSocket from 'ws'; +import WebSocket, { type RawData } from 'ws'; import * as Y from 'yjs'; import { WebsocketProvider } from 'y-websocket'; import { Client } from '@modelcontextprotocol/sdk/client/index.js'; @@ -47,6 +47,7 @@ export interface TestHarness { doc: Y.Doc; provider: WebsocketProvider; awareness: WebsocketProvider['awareness']; + traffic: WebSocketTraffic; disconnect: () => void; }; waitForCondition: ( @@ -56,6 +57,27 @@ export interface TestHarness { cleanup: () => Promise; } +/** + * Bytes received from the real Yjs WebSocket server by one test client. + * + * This deliberately measures the client-visible transport boundary: it covers + * initial state transfer and server fan-out without relying on internals of + * y-websocket's connection handler. It is useful to E2E tests and capacity + * benchmarks, while production telemetry remains a separate concern. + */ +export interface WebSocketTraffic { + receivedBytes: number; + receivedMessages: number; + reset: () => void; +} + +function rawDataByteLength(data: RawData): number { + if (typeof data === 'string') return Buffer.byteLength(data); + if (Buffer.isBuffer(data)) return data.byteLength; + if (data instanceof ArrayBuffer) return data.byteLength; + return data.reduce((total, chunk) => total + chunk.byteLength, 0); +} + async function nodeRequestToWebRequest( req: import('node:http').IncomingMessage, baseUrl: string @@ -199,17 +221,36 @@ export async function createTestHarness(): Promise { doc: Y.Doc; provider: WebsocketProvider; awareness: WebsocketProvider['awareness']; + traffic: WebSocketTraffic; disconnect: () => void; } { const doc = new Y.Doc(); + const traffic: WebSocketTraffic = { + receivedBytes: 0, + receivedMessages: 0, + reset: () => { + traffic.receivedBytes = 0; + traffic.receivedMessages = 0; + } + }; + class InstrumentedWebSocket extends WebSocket { + constructor(address: string | URL, protocols?: string | string[]) { + super(address, protocols); + this.on('message', (data: RawData) => { + traffic.receivedBytes += rawDataByteLength(data); + traffic.receivedMessages += 1; + }); + } + } const provider = new WebsocketProvider(wsUrl, 'workspace', doc, { - WebSocketPolyfill: WebSocket as unknown as typeof globalThis.WebSocket + WebSocketPolyfill: InstrumentedWebSocket as unknown as typeof globalThis.WebSocket }); yjsProviders.push(provider); return { doc, provider, awareness: provider.awareness, + traffic, disconnect: () => { provider.destroy(); doc.destroy(); diff --git a/vite.config.ts b/vite.config.ts index 2da5527..e3615d3 100644 --- a/vite.config.ts +++ b/vite.config.ts @@ -70,6 +70,17 @@ export default defineConfig({ setupFiles: ['./tests/setup/isolate-persistence.ts'] } }, + { + extends: './vite.config.ts', + test: { + name: 'benchmark', + environment: 'node', + include: ['tests/benchmark/**/*.test.ts'], + setupFiles: ['./tests/setup/isolate-persistence.ts'], + fileParallelism: false, + testTimeout: 120_000 + } + }, { extends: './vite.config.ts', test: { From 0ae88205689f6be4316e17ca56cb00b0c188663c Mon Sep 17 00:00:00 2001 From: Brylie Christopher Oxley Date: Sun, 30 Aug 2026 16:30:59 +0300 Subject: [PATCH 2/5] test: stabilize CRDT capacity measurements --- .../crdt-capacity-baseline-2026-08-30.md | 24 ++++++++++--------- tests/benchmark/workspace-capacity.test.ts | 15 ++++++++---- tests/e2e/harness.ts | 7 +++--- vite.config.ts | 1 + 4 files changed, 29 insertions(+), 18 deletions(-) diff --git a/docs/benchmarks/crdt-capacity-baseline-2026-08-30.md b/docs/benchmarks/crdt-capacity-baseline-2026-08-30.md index 8f15184..155c2b0 100644 --- a/docs/benchmarks/crdt-capacity-baseline-2026-08-30.md +++ b/docs/benchmarks/crdt-capacity-baseline-2026-08-30.md @@ -10,7 +10,9 @@ server rather than measuring isolated Yjs objects only. The benchmark seeds deterministic documents, blocks, collections, and rows; then connects Yjs clients, performs a human-originated WebSocket mutation, and -runs sequential MCP `hold_records` and `write_record` calls. It captures: +runs sequential MCP `hold_records` and `write_record` calls. BroadcastChannel +sharing is disabled for benchmark clients, so initial-sync bytes are observed +only at the WebSocket boundary. It captures: - encoded global Yjs state and persisted snapshot sizes; - initial client-sync bytes and elapsed time; @@ -38,15 +40,15 @@ They are a baseline and trend signal, not universal production SLOs. | Metric | Daily: 12 docs, 192 blocks, 3 collections / 120 rows, 3 clients, 12 MCP writes | Large: 120 docs, 2,880 blocks, 8 collections / 3,200 rows, 8 clients, 80 MCP writes | | ----------------------------------- | -----------------------------------------------------------------------------: | ----------------------------------------------------------------------------------: | | Encoded global state | 142,694 B | 2,848,008 B | -| Aggregate initial-sync bytes | 142,776 B | 8,544,250 B | -| All-clients initial-sync elapsed | 93.5 ms | 973.2 ms | -| Receiver fan-out bytes (one edit) | 116 B | 406 B | -| Fan-out convergence elapsed | 27.2 ms | 25.9 ms | -| MCP write p50 / p95 | 5.5 / 8.4 ms | 4.9 / 9.0 ms | -| Persisted snapshot | 143,424 B | 2,852,778 B | -| Snapshot-backed restart | 9.4 ms | 130.1 ms | -| Process heap delta | 25.3 MB | 396.1 MB | -| Event-loop p99 | 43.94 ms | 270.01 ms | +| Aggregate initial-sync bytes | 570,858 B | 28,480,307 B | +| All-clients initial-sync elapsed | 87.9 ms | 1,421.8 ms | +| Receiver fan-out bytes (one edit) | 114 B | 406 B | +| Fan-out convergence elapsed | 26.9 ms | 26.2 ms | +| MCP write p50 / p95 | 6.0 / 8.6 ms | 5.5 / 9.2 ms | +| Persisted snapshot | 143,421 B | 2,852,778 B | +| Snapshot-backed restart | 9.5 ms | 85.7 ms | +| Process heap delta | 50.8 MB | 518.9 MB | +| Event-loop p99 | 43.55 ms | 417.07 ms | | One-document shard state projection | 6,795 B | 10,147 B | ## Interpretation and decision boundary @@ -59,7 +61,7 @@ work proceeds. The large profile exposes the existing global-document cost: every new client receives the whole 2.85 MB workspace state, and the one-process benchmark -showed a 396 MB heap increase with a 270 ms event-loop p99. A same-shape +showed a 518.9 MB heap increase with a 417.07 ms event-loop p99. A same-shape single-document state is roughly 10 KB, which makes document-level Yjs shards the appropriate next boundary. The catalog/SSE design in #112 keeps titles and navigation outside that document state so unrelated document edits need not diff --git a/tests/benchmark/workspace-capacity.test.ts b/tests/benchmark/workspace-capacity.test.ts index c0eb861..9609fe0 100644 --- a/tests/benchmark/workspace-capacity.test.ts +++ b/tests/benchmark/workspace-capacity.test.ts @@ -77,7 +77,9 @@ describe('CRDT workspace capacity baseline (issue #31)', () => { const profile = PROFILES[profileName]; if (!profile) throw new Error(`Unknown benchmark profile: ${profileName}`); - const seedClient = harness.getYjsClient(); + // BroadcastChannel is deliberately disabled: each client must receive its + // initial state through the WebSocket transport being measured. + const seedClient = harness.getYjsClient({ disableBc: true }); const schema: PropertyDefinition[] = [ { key: 'title', label: 'Title', type: 'text' }, { @@ -145,7 +147,9 @@ describe('CRDT workspace capacity baseline (issue #31)', () => { const heapBefore = process.memoryUsage().heapUsed; const cpuBefore = process.cpuUsage(); - const peers = Array.from({ length: profile.clients }, () => harness.getYjsClient()); + const peers = Array.from({ length: profile.clients }, () => + harness.getYjsClient({ disableBc: true }) + ); const initialSyncStart = performance.now(); await Promise.all( peers.map((peer) => @@ -176,8 +180,11 @@ describe('CRDT workspace capacity baseline (issue #31)', () => { ) ); // The peer state can converge before ws emits its frame accounting callback - // in the same Node turn. Let that callback drain before recording bytes. - await new Promise((resolve) => setTimeout(resolve, 25)); + // in the same Node turn. Wait for that bounded transport observation. + await harness.waitForCondition( + () => peers.slice(1).every((peer) => peer.traffic.receivedBytes > 0), + { timeoutMs: 2_000, intervalMs: 5 } + ); const fanoutLatencyMs = performance.now() - fanoutStart; const fanoutBytes = peers .slice(1) diff --git a/tests/e2e/harness.ts b/tests/e2e/harness.ts index c99a05b..46c6336 100644 --- a/tests/e2e/harness.ts +++ b/tests/e2e/harness.ts @@ -43,7 +43,7 @@ export interface TestHarness { allowedCollectionIds: string[]; }) => { token: string; record: AccessToken }; getMcpClient: (token: string) => Promise; - getYjsClient: () => { + getYjsClient: (options?: { disableBc?: boolean }) => { doc: Y.Doc; provider: WebsocketProvider; awareness: WebsocketProvider['awareness']; @@ -217,7 +217,7 @@ export async function createTestHarness(): Promise { return client; } - function getYjsClient(): { + function getYjsClient(options: { disableBc?: boolean } = {}): { doc: Y.Doc; provider: WebsocketProvider; awareness: WebsocketProvider['awareness']; @@ -243,7 +243,8 @@ export async function createTestHarness(): Promise { } } const provider = new WebsocketProvider(wsUrl, 'workspace', doc, { - WebSocketPolyfill: InstrumentedWebSocket as unknown as typeof globalThis.WebSocket + WebSocketPolyfill: InstrumentedWebSocket as unknown as typeof globalThis.WebSocket, + disableBc: options.disableBc }); yjsProviders.push(provider); return { diff --git a/vite.config.ts b/vite.config.ts index e3615d3..34a5fd6 100644 --- a/vite.config.ts +++ b/vite.config.ts @@ -65,6 +65,7 @@ export default defineConfig({ exclude: [ 'src/**/*.svelte.{test,spec}.{js,ts}', 'tests/**/*.spec.{js,ts}', + 'tests/benchmark/**', 'src/lib/client/**/*.{test,spec}.{js,ts}' ], setupFiles: ['./tests/setup/isolate-persistence.ts'] From 598796112269ec62b8cce4edaeafbaccc722e410 Mon Sep 17 00:00:00 2001 From: Brylie Christopher Oxley Date: Sun, 30 Aug 2026 16:34:22 +0300 Subject: [PATCH 3/5] docs: add CRDT capacity benchmark QA guidance --- CLAUDE.md | 21 ++++++++++ README.md | 9 +++++ docs/specifications/README.md | 2 +- docs/specifications/e2e-testing.md | 62 ++++++++++++++++++++++++++++++ 4 files changed, 93 insertions(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index 015bf71..287cc48 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,6 +19,7 @@ Compendium is a shared, real-time knowledge workspace: one SvelteKit app where a - [`persistence.md`](docs/specifications/persistence.md) — SQLite via Drizzle (snapshots, audit, tokens, read model) - [`service-layer.md`](docs/specifications/service-layer.md) / [`service-layer-manifest.md`](docs/specifications/service-layer-manifest.md) — where permission+audit logic must live - [`e2e-testing.md`](docs/specifications/e2e-testing.md) — why/how the Tier A + Tier B suites exist +- [`crdt-capacity-baseline-2026-08-30.md`](docs/benchmarks/crdt-capacity-baseline-2026-08-30.md) — current measured global-workspace envelope and sharding decision gates - [`design-system.md`](docs/specifications/design-system.md) — UI tokens/conventions ## Commands @@ -36,6 +37,9 @@ npm run test:e2e:tier-a # vitest, protocol-level MCP+Yjs parity tests (tes npm run test:e2e:tier-b # playwright, DOM-level (requires `npm run build` first — serves via build/handler.js) npm run test:e2e # both tiers +npm run benchmark:workspace # bounded CRDT capacity profile; isolated temp DB + localhost server +npm run benchmark:workspace:large # manual CRDT capacity profile; run before/after shard or persistence changes + npm run test:coverage # coverage; thresholds are 80% stmts/branches/functions/lines (vite.config.ts) npm run check # svelte-kit sync && svelte-check (typecheck) npm run lint # prettier --check . && eslint . @@ -108,3 +112,20 @@ Unit tests calling `records.ts`/`services/*.ts` directly, and manual/Playwright- - **Tier B** (`tests/e2e/tier-b.spec.ts`, Playwright): real browser, but the triggering action still comes from a real MCP client call in the test's Node context. Reserved for behavior that specifically needs a rendered DOM (held-block shimmer, live sidebar tree updates) — keep this tier small. - Shared harness: `tests/e2e/harness.ts` (the only place that should know how to boot a full server instance for tests). - Vitest is split into three projects (`vite.config.ts`): `server` (node env, most of `src/**` + `tests/**`), `client` (jsdom, `src/lib/client/**`), `component` (jsdom + `browser` resolve condition, `src/**/*.svelte.test.ts` — needed because Vitest's default SSR condition resolves `svelte` to a build without `mount()`). + +## Capacity benchmark: required QA for CRDT and sharding work + +Read [`docs/specifications/e2e-testing.md`](docs/specifications/e2e-testing.md) §6 and the current +[`docs/benchmarks/crdt-capacity-baseline-2026-08-30.md`](docs/benchmarks/crdt-capacity-baseline-2026-08-30.md) +before changing the Yjs schema, WebSocket routing or fan-out, workspace/shard resolution, snapshot persistence, +or the document/collection ownership model. + +- Run `npm run benchmark:workspace` for every PR that changes one of those boundaries. It is deliberately + separate from `npm run test` and coverage so normal checks stay bounded. +- Run `npm run benchmark:workspace:large` manually before and after any sharding, compaction, persistence, + or sync-protocol redesign. Compare like-for-like runs; performance values are trend evidence, not universal SLOs. +- Keep benchmark clients on WebSocket transport (`disableBc: true`) when reporting sync bytes. Do not re-enable + BroadcastChannel sharing to make a profile look smaller. +- If a guardrail fails or the result crosses a documented sharding trigger, stop treating it as a test-only + regression: record the result in a dated benchmark note and link the implementation issue/PR. Update the + baseline only after explaining the fixture or architecture change that makes the comparison valid. diff --git a/README.md b/README.md index a775c11..f1e5456 100644 --- a/README.md +++ b/README.md @@ -184,6 +184,8 @@ for the shared record primitive. ```sh npm run test # unit and component tests npm run test:e2e # real MCP↔Yjs and browser-level flows +npm run benchmark:workspace # bounded CRDT capacity profile +npm run benchmark:workspace:large # manual sharding/persistence profile npm run check # Svelte and TypeScript checks npm run lint # formatting and lint rules npm run build # production build @@ -193,6 +195,13 @@ The E2E suites intentionally cross real transport boundaries. A feature is not considered integrated merely because its UI and service functions pass in isolation. +The capacity benchmark is intentionally separate from routine tests so it can +measure a real temporary SQLite + WebSocket + MCP workspace without making +ordinary checks environment-sensitive. Run the bounded profile for CRDT, +sync, snapshot, or routing changes; run both profiles before and after sharding +or persistence redesign. See the [testing strategy](docs/specifications/e2e-testing.md#6-capacity-benchmark--crdt-and-sharding-regression-gate) +and the [current baseline](docs/benchmarks/crdt-capacity-baseline-2026-08-30.md). + ## Roadmap The [Compendium project board](https://github.com/users/brylie/projects/6) is diff --git a/docs/specifications/README.md b/docs/specifications/README.md index 94e5a61..967b9f4 100644 --- a/docs/specifications/README.md +++ b/docs/specifications/README.md @@ -13,7 +13,7 @@ Canonical specification for each implemented subsystem, translating [`prd.md`](. - [`persistence.md`](./persistence.md) — SQLite via Drizzle: snapshots, audit log, access tokens, and the query read model. - [`audit-coverage.md`](./audit-coverage.md) — how direct UI mutations (which bypass the service layer entirely) and denied MCP attempts get an audit trail, and what's deliberately excluded. - [`service-layer.md`](./service-layer.md) / [`service-layer-manifest.md`](./service-layer-manifest.md) — how permission/audit logic is centralized once and shared by MCP and UI. -- [`e2e-testing.md`](./e2e-testing.md) — the MCP/UI parity testing strategy. +- [`e2e-testing.md`](./e2e-testing.md) — MCP/UI parity and CRDT capacity benchmark strategy. - [`design-system.md`](./design-system.md) — UI tokens and conventions. ## Out of scope for the current architecture diff --git a/docs/specifications/e2e-testing.md b/docs/specifications/e2e-testing.md index 1688669..baba470 100644 --- a/docs/specifications/e2e-testing.md +++ b/docs/specifications/e2e-testing.md @@ -87,3 +87,65 @@ This harness is the only thing that should know how to boot a full server instan ## 5. Relationship to existing and future unit tests This spec doesn't replace `records.ts`'s existing unit tests, or the equivalent tests the service layer (`service-layer.md`) will need — those stay valuable for fast, fine-grained coverage of CRDT and business-rule logic. Tier A is specifically for the narrower, higher-value class of bug that only exists at the transport boundary between two independent real clients — write a Tier A test whenever a change touches anything permission-, grant-, hold-, or attribution-related, since those are exactly the categories where "worked in the unit test, broke for a real second agent call" has already happened once. + +## 6. Capacity benchmark — CRDT and sharding regression gate + +Tier A proves that a small number of real clients converge correctly. It does +not establish that the shared state stays within a workable resource envelope +as documents, Collections, concurrent clients, and MCP activity grow. The +capacity benchmark fills that gap while preserving the same real server, Yjs +WebSocket, MCP HTTP, and temporary SQLite boundaries. + +### Profiles and commands + +```sh +npm run benchmark:workspace # `daily`: bounded profile, suitable for CI +npm run benchmark:workspace:large # `large`: manual pre/post-change comparison +``` + +The benchmark lives in `tests/benchmark/workspace-capacity.test.ts` and runs +in its own Vitest project. It is intentionally excluded from `npm run test` +and coverage: performance work must stay discoverable and repeatable without +making ordinary correctness checks slow or environment-sensitive. Every run +creates a temporary SQLite database and random local port; it must never point +at a developer's running workspace database. + +`daily` uses a small knowledgebase fixture and carries conservative CI +guardrails. `large` is deliberately not a CI gate: use it before and after a +change where state topology or transport cost could change, then publish the +two results with the environment and fixture in a dated note under +`docs/benchmarks/`. + +### When an engineer or agent must run it + +Run the bounded profile for a PR that changes any of the following: + +- the Yjs record schema, encoded representation, or document/Collection + ownership model; +- `attach-ws.ts`, WebSocket connection lifecycle, routing, or update fan-out; +- `workspace-store.ts`, shard selection, context lifecycle, or snapshot load/ + flush behavior; +- MCP write paths that change edit churn, cross-client update application, or + persistence behavior. + +Run both profiles for shard-aware routing, catalog/SSE integration, compaction, +snapshot-format, or persistence redesign. For the workspace-catalog work, +this is a required before-and-after acceptance check, not an optional +optimization exercise. + +### How to interpret and maintain results + +The suite records encoded state/snapshot size, client-visible initial-sync and +fan-out bytes, convergence and MCP-write timing, restart time, and Node host +resource signals. Initial-sync measurements must have `disableBc: true` for +every benchmark provider, including the seed client: otherwise same-process +BroadcastChannel sharing bypasses WebSocket traffic and makes the transport +envelope appear smaller than it is. + +Use the current [CRDT capacity baseline](../benchmarks/crdt-capacity-baseline-2026-08-30.md) +as the decision record. Compare like-for-like results rather than treating +machine-specific timings as universal SLOs. A daily-profile guardrail failure, +a global snapshot of 2 MiB or more, or event-loop p99 of 100 ms or more is a +sharding/compaction escalation: document it and link the related issue or PR. +When an intentional fixture or architecture change makes a new baseline valid, +write a new dated note; do not silently overwrite an old decision record. From c1eddc924b1ceadd1ceca974120c925ce94a9e6b Mon Sep 17 00:00:00 2001 From: Brylie Christopher Oxley Date: Sun, 30 Aug 2026 16:40:14 +0300 Subject: [PATCH 4/5] docs: align benchmark profile guidance --- CLAUDE.md | 6 ++++-- README.md | 7 ++++--- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 287cc48..309f916 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -122,8 +122,10 @@ or the document/collection ownership model. - Run `npm run benchmark:workspace` for every PR that changes one of those boundaries. It is deliberately separate from `npm run test` and coverage so normal checks stay bounded. -- Run `npm run benchmark:workspace:large` manually before and after any sharding, compaction, persistence, - or sync-protocol redesign. Compare like-for-like runs; performance values are trend evidence, not universal SLOs. +- Run `npm run benchmark:workspace:large` manually before and after shard-aware routing, catalog/SSE integration, + compaction, snapshot-format, persistence, or sync-protocol redesign. Compare like-for-like runs; performance + values are trend evidence, not universal SLOs. [`e2e-testing.md`](docs/specifications/e2e-testing.md) §6 is + canonical if this short list needs interpretation. - Keep benchmark clients on WebSocket transport (`disableBc: true`) when reporting sync bytes. Do not re-enable BroadcastChannel sharing to make a profile look smaller. - If a guardrail fails or the result crosses a documented sharding trigger, stop treating it as a test-only diff --git a/README.md b/README.md index f1e5456..79734dc 100644 --- a/README.md +++ b/README.md @@ -198,9 +198,10 @@ isolation. The capacity benchmark is intentionally separate from routine tests so it can measure a real temporary SQLite + WebSocket + MCP workspace without making ordinary checks environment-sensitive. Run the bounded profile for CRDT, -sync, snapshot, or routing changes; run both profiles before and after sharding -or persistence redesign. See the [testing strategy](docs/specifications/e2e-testing.md#6-capacity-benchmark--crdt-and-sharding-regression-gate) -and the [current baseline](docs/benchmarks/crdt-capacity-baseline-2026-08-30.md). +sync, snapshot, or routing changes; run both profiles before and after +shard-aware routing, catalog/SSE integration, compaction, snapshot-format, +persistence, or sync-protocol redesign. See the [testing strategy](docs/specifications/e2e-testing.md#6-capacity-benchmark--crdt-and-sharding-regression-gate) +for the canonical selection rules and the [current baseline](docs/benchmarks/crdt-capacity-baseline-2026-08-30.md). ## Roadmap From 335017aecca4801d49fea0845c26467729c759ed Mon Sep 17 00:00:00 2001 From: Brylie Christopher Oxley Date: Sun, 30 Aug 2026 16:45:07 +0300 Subject: [PATCH 5/5] docs: require capacity checks for sync changes --- CLAUDE.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 309f916..b4d883d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -120,8 +120,8 @@ Read [`docs/specifications/e2e-testing.md`](docs/specifications/e2e-testing.md) before changing the Yjs schema, WebSocket routing or fan-out, workspace/shard resolution, snapshot persistence, or the document/collection ownership model. -- Run `npm run benchmark:workspace` for every PR that changes one of those boundaries. It is deliberately - separate from `npm run test` and coverage so normal checks stay bounded. +- Run `npm run benchmark:workspace` for every PR that changes one of those boundaries or any other Yjs sync + behavior. It is deliberately separate from `npm run test` and coverage so normal checks stay bounded. - Run `npm run benchmark:workspace:large` manually before and after shard-aware routing, catalog/SSE integration, compaction, snapshot-format, persistence, or sync-protocol redesign. Compare like-for-like runs; performance values are trend evidence, not universal SLOs. [`e2e-testing.md`](docs/specifications/e2e-testing.md) §6 is