diff --git a/.github/workflows/docs-deploy.yml b/.github/workflows/docs-deploy.yml index 2f94e747..ca172826 100644 --- a/.github/workflows/docs-deploy.yml +++ b/.github/workflows/docs-deploy.yml @@ -25,6 +25,10 @@ jobs: with: ref: ${{ github.sha }} persist-credentials: false + - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7 + with: + go-version-file: go.mod + cache: true - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7 with: node-version: '24' @@ -32,6 +36,10 @@ jobs: - run: npm ci env: ELECTRON_SKIP_BINARY_DOWNLOAD: '1' + - name: Initialize protocol artifacts and SDK for example tests + run: | + npm run generate + npm run build - run: npm install --prefix "$RUNNER_TEMP/wrangler" --no-save wrangler@4.140.0 - run: echo "$RUNNER_TEMP/wrangler/node_modules/.bin" >> "$GITHUB_PATH" - run: npm run check:docs diff --git a/apps/docs/README.md b/apps/docs/README.md index 7671613f..7e546b56 100644 --- a/apps/docs/README.md +++ b/apps/docs/README.md @@ -20,6 +20,11 @@ npm run storybook:docs # http://localhost:6007 npm run build:storybook:docs ``` +On a clean checkout, run `npm run generate` with the Go version in `go.mod` +before `test:docs`. The published SDK examples are typechecked against the actual +SDK source and its generated protocol declarations. This test prerequisite does +not add an SDK or daemon dependency to the deployed site. + Install Chromium once with `npx playwright install chromium` (CI uses `--with-deps`). Browser tests require a completed `build:docs`, build Storybook automatically, and start static servers on ports 3101 and 6008. Content-independent diff --git a/docs/release-readiness.md b/docs/release-readiness.md new file mode 100644 index 00000000..57197616 --- /dev/null +++ b/docs/release-readiness.md @@ -0,0 +1,48 @@ +# Development release after backend reorganization + +The reorganization froze development at `271c0f8d2a35648d1b45056d57432590b783483c` +(protocol 6.9). The release PR targets older main +`88764670f1c6e2a57e22463e96cfb0d6021aea33` (protocol 6.8). Strict equality with +that older main also rejects intentional features already present before the +reorganization. The frozen baseline, SDK, generated protocol, database schema, +and runtime behavior are unchanged by this release-readiness repair. + +## Historical differences checked for this release + +| Origin | Observable change in the deterministic compatibility fixture | +| --- | --- | +| [f62b5eec6](https://github.com/context-labs/whip/commit/f62b5eec628b2d583793aa788dc2eb677941e0f8) — concrete effort vocabulary | Protocol minor 8 → 9; new root session and agent effort resolves from empty to `low`. | +| [54fca9c1a](https://github.com/context-labs/whip/commit/54fca9c1af25fb85bbf581f05ed5a401bcc5fb2f) — title generation | Removes `session.autotitle`; advertises `session_title_notifications`; admission emits one `session.title.updated` event before `turn.started`. This advances subsequent event/trace sequences and cursors by one. The fallback title grows from 64 to 128 characters, and user messages retain their originating turn. | +| [12f0ea076](https://github.com/context-labs/whip/commit/12f0ea076) — native folder creation | Adds `host.directory.create` beside `host.directory.pick`. | +| [b8cc56a48](https://github.com/context-labs/whip/commit/b8cc56a48d781998784fc7ce1cc541bea2cd59fb) — conversation cleanup | An empty compaction model selects the conversation's model/provider rather than a built-in DeepSeek route. Configuration/provider revision hashes change accordingly. Assistant messages retain recording timestamps. | + +These differences were reproduced in both Linux and macOS release CI artifacts; +their normalized parent and frozen-reference transcripts match across platforms. +They are expectations for this exact historical release pair, not permission +for further changes to these fields. + +`scripts/backend-compat/release-transition.mjs` checks the old values before +constructing the expected release transcript. It requires that transcript to +match both the independently built frozen reference and the candidate, including +all untouched fields, event order, payloads, counts, identities and cursors. +The ordinary immediate-base comparison remains strict for every other base SHA. +Existing fixed-SDK, generated-artifact, lifecycle, schema, checkpoint, trace and +base → candidate → base rollback checks remain enabled. + +## Docs deployment prerequisite + +The deployment workflow omitted protocol generation after generated files +stopped being tracked. The same prerequisite was already restored in regular +docs CI by [567bdebba](https://github.com/context-labs/whip/commit/567bdebbae9aa21a7a09a7686a979824ea998a51). +Set up Go from `go.mod`, generate the protocol and build the SDK before checking +the published examples. The static site still has no SDK or daemon runtime +dependency. + +## Release acceptance + +Require full CI, the docs preview deployment, and isolated desktop acceptance +before merging development to main. Exercise REPL/child state and traces through +the compatibility lifecycle/rollback scenarios; verify the shared renderer's +REPL, agent, provider and effort controls; and run the staged desktop lifecycle +smoke against temporary homes. These checks use local fixture providers and +must not consume personal provider credentials or existing sessions. diff --git a/scripts/backend-compat/README.md b/scripts/backend-compat/README.md index b9bb7412..b15d88fe 100644 --- a/scripts/backend-compat/README.md +++ b/scripts/backend-compat/README.md @@ -62,6 +62,15 @@ forces two separate fixture fragments; smaller fragments could validly coalesce and would be a poor exact-event regression oracle. Comparator tests prove that response, omission, error, cursor, event order/count and identity mutations fail. +The release from main `88764670` to the pre-reorganization development baseline +has an explicit, commit-scoped expectation for previously reviewed protocol 6.8 +→ 6.9 features. See [release readiness](../../docs/release-readiness.md) for the +source commits and every expected difference. The gate checks those exact old +values and requires the result to match both the frozen reference and candidate. +It does not skip the immediate-base comparison or normalize away changed fields. +All other base revisions retain strict equality, and all rollback checks remain +enabled. The frozen baseline and source/artifact freeze are unchanged. + This is a cross-cutting regression gate, not exhaustive behavioral equivalence. It does not replace existing provider encoding/retry tests, detailed trace body and pagination tests, REPL/mail/budget tests, or product acceptance. Add focused diff --git a/scripts/backend-compat/release-transition.mjs b/scripts/backend-compat/release-transition.mjs new file mode 100644 index 00000000..e4ba54d7 --- /dev/null +++ b/scripts/backend-compat/release-transition.mjs @@ -0,0 +1,100 @@ +import assert from 'node:assert/strict'; +import { compare } from './compare.mjs'; + +// The already reviewed features between main and the frozen pre-reorganization +// development baseline. This expectation applies to this exact pair only. +// Provenance and scope: docs/release-readiness.md. +export const releaseTransition = { + base: '88764670f1c6e2a57e22463e96cfb0d6021aea33', + baseline: '271c0f8d2a35648d1b45056d57432590b783483c', +}; + +function replace(object, key, before, after) { + assert.ok(Object.hasOwn(object, key), `missing release source field ${key}`); + assert.deepEqual(object[key], before, `unexpected release source field ${key}`); + object[key] = after; +} + +function add(object, key, value) { + assert.ok(!Object.hasOwn(object, key), `release source already has ${key}`); + object[key] = value; +} + +export function expectedReleaseTranscript(parent) { + const expected = structuredClone(parent); + const { initialization, configuration, updated, provider, initial, snapshot, history, live, replay } = expected; + + // f62b5eec6: protocol 6.9 and concrete effort resolved at session creation. + replace(initialization, 'protocol_minor', 8, 9); + for (const state of [initial, snapshot]) { + replace(state.meta, 'effort', '', 'low'); + replace(state.agents[0], 'effort', '', 'low'); + } + + // b8cc56a48: compaction defaults to the conversation model/provider. Revision + // hashes are exact fixture values, not wildcarded configuration differences. + for (const state of [configuration, updated]) replace(state, 'compact_model', 'deepseek-v4-flash-0731', ''); + replace(configuration, 'revision', 'f82bc34858a0838f80a01e6e9b9c319905987aa2721f374d499a3b918f555f85', '4f2ca9fffdfa2e329309d36d357715e0abdaca3765524f3d651a0dbaef8de834'); + for (const state of [updated, provider]) replace(state, 'revision', 'a5719c0971e0b596bcc0ae551e8d8f85b84c0f28a3337b8252b416a3b271e44f', 'aa5d3e546d8cbfd6980e2452c78200a2419b5d88ea4d1b35fd61ad745d2a3b68'); + + // 12f0ea076: create a folder from the native directory picker. + const operations = initialization.operations; + assert.ok(!operations.some(operation => operation.name === 'host.directory.create')); + const picker = operations.findIndex(operation => operation.name === 'host.directory.pick'); + assert.ok(picker >= 0, 'release source lost directory picker'); + operations.splice(picker, 0, { name: 'host.directory.create', surface: 'rpc', execution: 'ephemeral', permission: 'host-runtime' }); + + // 54fca9c1a: title ownership moved to admission, replacing the explicit + // autotitle command with a notification and an ordered durable title event. + const autotitle = operations.findIndex(operation => operation.name === 'session.autotitle'); + assert.ok(autotitle >= 0, 'release source lost autotitle'); + assert.deepEqual(operations[autotitle], { name: 'session.autotitle', surface: 'runtime', execution: 'command', permission: 'root-association' }); + operations.splice(autotitle, 1); + for (const capabilities of [initialization.capabilities, initialization.negotiated_capabilities]) { + assert.ok(!capabilities.includes('session_title_notifications')); + } + initialization.capabilities.push('session_title_notifications'); + const engine = initialization.negotiated_capabilities.indexOf('execution_engines'); + assert.ok(engine >= 0, 'release source lost execution engine capability'); + initialization.negotiated_capabilities.splice(engine, 0, 'session_title_notifications'); + const title = 'A'.repeat(127) + '…'; + replace(snapshot.meta, 'title', 'A'.repeat(63) + '…', title); + for (const events of [live, replay.events]) { + assert.deepEqual(events.map(event => event.kind), ['command.queued', 'turn.started', 'span.started', 'stream.text', 'stream.text', 'turn.succeeded', 'span.ended']); + for (let i = 0; i < events.length; i++) { + assert.equal(events[i].seq, String(i + 1)); + if (i === 0) continue; + if (events[i].kind.startsWith('span.')) replace(events[i].payload, 'updated_seq', String(i + 1), String(i + 2)); + events[i].seq = String(i + 2); + } + const event = { root_id: events[0].root_id, seq: '2', kind: 'session.title.updated', payload: { title } }; + if (Object.hasOwn(events[0], 'subscription_id')) event.subscription_id = events[0].subscription_id; + events.splice(1, 0, event); + } + replace(replay, 'latest', '7', '8'); + replace(snapshot, 'cursor', '7', '8'); + replace(snapshot.accounting, 'revision', '7', '8'); + replace(snapshot.agents[0].last_turn, 'event_seq', '6', '7'); + for (const messages of [snapshot.messages, history.messages.map(entry => entry.message)]) { + assert.equal(messages[0].role, 'user'); + add(messages[0], 'presentation', { version: 1, turn_id: snapshot.agents[0].last_turn.turn_id }); + // b8cc56a48: preserve assistant recording time in the durable transcript. + assert.equal(messages[1].role, 'assistant'); + add(messages[1], 'sent_at', ''); + } + return expected; +} + +export function comparePRBase(parent, candidate, { baseRevision, baselineRevision, baseline, label }) { + if (baseRevision !== releaseTransition.base) { + compare(parent, candidate, label); + return 'strict'; + } + assert.equal(baselineRevision, releaseTransition.baseline, 'release expectation belongs to a different frozen baseline'); + const expected = expectedReleaseTranscript(parent); + // Prove the exception describes the pre-reorganization reference itself. + // A new candidate response can never become its own compatibility oracle. + compare(expected, baseline, `${label}: reviewed pre-reorganization transition`); + compare(expected, candidate, label); + return 'reviewed 6.8 to 6.9 transition'; +} diff --git a/scripts/backend-compat/release-transition.test.mjs b/scripts/backend-compat/release-transition.test.mjs new file mode 100644 index 00000000..f37b210d --- /dev/null +++ b/scripts/backend-compat/release-transition.test.mjs @@ -0,0 +1,81 @@ +import assert from 'node:assert/strict'; +import { test } from 'node:test'; +import { comparePRBase, releaseTransition } from './release-transition.mjs'; + +function transcript(upgraded) { + const revision = upgraded ? 'aa5d3e546d8cbfd6980e2452c78200a2419b5d88ea4d1b35fd61ad745d2a3b68' : 'a5719c0971e0b596bcc0ae551e8d8f85b84c0f28a3337b8252b416a3b271e44f'; + const title = 'A'.repeat(upgraded ? 127 : 63) + '…'; + const kinds = ['command.queued', ...(upgraded ? ['session.title.updated'] : []), 'turn.started', 'span.started', 'stream.text', 'stream.text', 'turn.succeeded', 'span.ended']; + const events = kinds.map((kind, i) => ({ root_id: 'root', seq: String(i + 1), kind, + payload: kind === 'session.title.updated' ? { title } : kind.startsWith('span.') ? { updated_seq: String(i + 1), trace_id: 'trace' } : { text: 'preserved payload' }, + })); + const messages = () => [ + { role: 'user', content: 'input', ...(upgraded ? { presentation: { version: 1, turn_id: 'turn' } } : {}) }, + { role: 'assistant', content: 'output', ...(upgraded ? { sent_at: '' } : {}) }, + ]; + const effort = upgraded ? 'low' : ''; + return { + initialization: { + protocol_minor: upgraded ? 9 : 8, + capabilities: ['commands', 'execution_engines', ...(upgraded ? ['session_title_notifications'] : [])], + negotiated_capabilities: ['commands', ...(upgraded ? ['session_title_notifications'] : []), 'execution_engines'], + operations: [ + ...(upgraded ? [{ name: 'host.directory.create', surface: 'rpc', execution: 'ephemeral', permission: 'host-runtime' }] : []), + { name: 'host.directory.pick', surface: 'rpc', execution: 'query', permission: 'host-runtime' }, + ...(!upgraded ? [{ name: 'session.autotitle', surface: 'runtime', execution: 'command', permission: 'root-association' }] : []), + ], + }, + configuration: { compact_model: upgraded ? '' : 'deepseek-v4-flash-0731', revision: upgraded ? '4f2ca9fffdfa2e329309d36d357715e0abdaca3765524f3d651a0dbaef8de834' : 'f82bc34858a0838f80a01e6e9b9c319905987aa2721f374d499a3b918f555f85' }, + updated: { compact_model: upgraded ? '' : 'deepseek-v4-flash-0731', revision }, + provider: { revision, name: 'unchanged provider' }, + initial: { meta: { effort }, agents: [{ effort }] }, + snapshot: { meta: { effort, title }, agents: [{ effort, last_turn: { turn_id: 'turn', event_seq: upgraded ? '7' : '6' } }], + cursor: upgraded ? '8' : '7', accounting: { revision: upgraded ? '8' : '7' }, messages: messages() }, + history: { messages: messages().map(message => ({ message })) }, + live: events.map(event => ({ ...structuredClone(event), subscription_id: 'subscription' })), + replay: { latest: upgraded ? '8' : '7', events }, + error: { code: -32000, message: 'unchanged error', data: null }, + }; +} + +const options = () => ({ baseRevision: releaseTransition.base, baselineRevision: releaseTransition.baseline, baseline: transcript(true), label: 'test release' }); + +test('accepts the exact reviewed historical transition without modifying inputs', () => { + const parent = transcript(false), candidate = transcript(true); + assert.equal(comparePRBase(parent, candidate, options()), 'reviewed 6.8 to 6.9 transition'); + assert.deepEqual(parent, transcript(false)); + assert.deepEqual(candidate, transcript(true)); +}); + +for (const [name, mutate] of Object.entries({ + effort: value => { value.snapshot.meta.effort = 'high'; }, + configuration: value => { value.configuration.revision = 'unexpected'; }, + provider: value => { value.provider.name = 'changed provider'; }, + error: value => { value.error.data = {}; }, + title: value => { value.live[1].payload.title = 'different title'; }, + eventOrder: value => { value.live.reverse(); }, + eventCount: value => { value.replay.events.splice(1, 1); }, + cursor: value => { value.snapshot.cursor = '9'; }, + identity: value => { value.history.messages[0].message.presentation.turn_id = 'other-turn'; }, + addedField: value => { value.snapshot.meta.new_field = true; }, +})) test(`rejects an additional candidate ${name} change`, () => { + const candidate = transcript(true); mutate(candidate); + assert.throws(() => comparePRBase(transcript(false), candidate, options()), assert.AssertionError); +}); + +test('an unexpected historical value cannot be overwritten by a release expectation', () => { + const parent = transcript(false); parent.initial.meta.effort = 'off'; + assert.throws(() => comparePRBase(parent, transcript(true), options()), /unexpected release source field effort/); +}); + +test('release expectations must describe the independently built frozen baseline', () => { + const config = options(); config.baseline.updated.compact_model = 'another-model'; + assert.throws(() => comparePRBase(transcript(false), transcript(true), config), /reviewed pre-reorganization transition/); + assert.throws(() => comparePRBase(transcript(false), transcript(true), { ...options(), baselineRevision: '0'.repeat(40) }), /different frozen baseline/); +}); + +test('every other PR base retains exact transcript equality', () => { + const config = { ...options(), baseRevision: '1'.repeat(40) }; + assert.throws(() => comparePRBase(transcript(false), transcript(true), config), assert.AssertionError); + assert.equal(comparePRBase(transcript(true), transcript(true), config), 'strict'); +}); diff --git a/scripts/backend-compat/run.mjs b/scripts/backend-compat/run.mjs index 63a7b271..05469140 100644 --- a/scripts/backend-compat/run.mjs +++ b/scripts/backend-compat/run.mjs @@ -7,6 +7,7 @@ import { join, resolve } from 'node:path'; import { parseArgs } from 'node:util'; import { fileURLToPath } from 'node:url'; import { compare } from './compare.mjs'; +import { comparePRBase } from './release-transition.mjs'; import { loadSDK, smoke, lifecycle, rollback } from './scenarios.mjs'; const { values } = parseArgs({ options: { candidate: { type: 'string' }, base: { type: 'string' }, output: { type: 'string' } } }); @@ -83,7 +84,7 @@ try { for (const path of manifest.generatedPaths) compare(await tree(join(baseline, path)), await tree(join(candidate, path)), `fresh generated ${path}`); compare(await tree(join(baseline, 'packages/sdk/dist')), await tree(join(candidate, 'packages/sdk/dist')), 'compiled SDK artifact freeze'); evidence.artifacts = { baselineBinary: digest(await readFile(baseBinary)), parentBinary: digest(await readFile(parentBinary)), candidateBinary: digest(await readFile(candidateBinary)), sdk: await tree(join(baseline, 'packages/sdk/dist')), protocol: await tree(join(baseline, 'packages/protocol/generated')), lockfile: digest(await readFile(join(baseline, 'package-lock.json'))) }; - await command(process.execPath, ['--test', fileURLToPath(new URL('./compare.test.mjs', import.meta.url))]); + await command(process.execPath, ['--test', ...['compare.test.mjs', 'release-transition.test.mjs'].map(name => fileURLToPath(new URL(name, import.meta.url)))]); const sdk = await loadSDK(baseline); // Every invocation gets new state, including repeated calibration runs. const runs = await mkdtemp(join(output, 'runs-')); @@ -103,7 +104,10 @@ try { compare(reference, actual, `${transport} fixed-SDK differential`); const parentTranscript = parentBinary === baseBinary ? reference : await smoke(sdk, parentBinary, join(runs, `parent-${transport}`), transport); await writeFile(join(runs, `parent-${transport}.json`), JSON.stringify(parentTranscript, null, 2)); - compare(parentTranscript, actual, `${transport} immediate-PR-base differential`); + evidence.results[`${transport}PRBase`] = comparePRBase(parentTranscript, actual, { + baseRevision: evidence.parentRevision, baselineRevision: manifest.revision, + baseline: reference, label: `${transport} immediate-PR-base differential`, + }); evidence.results[transport] = 'passed: calibrated response/error/event transcript'; } for (const [name, binary] of [['baseline', baseBinary], ['candidate', candidateBinary]]) {