From 5f8e0d70855fdd3eacd6fbb7d2fe797e44dd83f0 Mon Sep 17 00:00:00 2001 From: Ross Miller Date: Tue, 21 Jul 2026 08:04:50 -0500 Subject: [PATCH 01/18] docs(plans): commit the reconciled phase 6 plan The phase plan lands before the work it describes, so the branch reads in the order it was executed. Section 0 records the five facts found in the code that contradict what the plan was written against, rather than letting the plan and the tree drift apart silently. The largest of those is the precondition. Phase 2 was expected to leave four modules importing model-client directly, which would have blocked three journeys. The grep is empty on master, so every journey is in scope. Two smaller corrections matter for implementation. key-store.ts is gone, so the data directory env var is no longer load bearing, and the scripted adapter has to satisfy FakeTextGeneration rather than the bare port, because the contract suite types its subject to the fake's scripting and recording surface. Claude-Session: https://claude.ai/code/session_01FSkESkJTRpt3Ye2cM7c5n5 --- docs/plans/refactor/phase-6.md | 133 +++++++++++++++++++++++++++++++++ 1 file changed, 133 insertions(+) create mode 100644 docs/plans/refactor/phase-6.md diff --git a/docs/plans/refactor/phase-6.md b/docs/plans/refactor/phase-6.md new file mode 100644 index 0000000..fb2a0ae --- /dev/null +++ b/docs/plans/refactor/phase-6.md @@ -0,0 +1,133 @@ +# Phase 6 — Committed E2E Journey Suite on Fake Adapters + +> **PARALLEL-EXECUTION ADDENDUM (architect, owner-approved):** Phase 7 develops CONCURRENTLY with this phase, in the main checkout, while Phase 6 runs in its own worktree off post-P2 master. Phase 6 MERGES FIRST. Consequences: land S1 (walking skeleton) and push it as early as possible and message the architect when it is on the branch, Phase 7 wants it for local verification. Journey (h)'s per-error-class variants may be added by Phase 7's lead as its final integration task after this phase merges, so build journey (h) with a parameterizable failure-injection point. Do not touch server/ or shared/ source (Phase 7 owns them this window); if a journey needs a server change, report it, do not make it. + +## Objective + +Prove the whole app works end to end, deterministically, with zero live-provider traffic and zero API keys, by driving the real browser UI against the real Fastify server whose edges are fakes. Pure test addition. No production file changes except config lines and two devDependencies. The suite is the regression net Phase 7's durability work will lean on, and the thing a cold reader opens to see that the hexagon is not decoration. + +## 0. Reconciled during execution + +Facts found in the code that contradict what this plan was written against. The plan below is corrected in place, and this list records what moved and why, so the difference between plan and reality is never silent. + +1. **The Phase 2 precondition is clean on master.** `rg -l "model-client" server --glob '!server/adapters/**' --glob '!**/*.test.ts'` returns nothing at `e8a5dd1`. Phase 2 finished the job. No journey is blocked, so (a), (b), and (f) are all in scope. +2. **`server/services/key-store.ts` no longer exists.** The plan's harness section warns that it builds its vault at module load, which forces `TUTOR_DATA_DIR` to be set before the first server import. Phase 2 replaced it with `createFileKeyVault({ dataDir })`, constructed inside `createPorts` from a `getDataDir()` call that reads the environment every time. The env var is still set before the dynamic import, because the guarantee costs nothing and `server/test/setup-env.ts` documents the same reasoning, but the constraint is now belt and braces rather than load bearing. +3. **The config surface is four lines, not three.** `pnpm lint` globs `client/ server/ electron/ shared/` explicitly, so an ESLint boundary block for `e2e/` would never run without adding `e2e/` to that script. The four touches are the vitest `exclude`, the tsconfig `include`, the ESLint boundary block, and the lint script glob. The phase gate's real invariant, that `git diff --stat master -- client/ server/ shared/ electron/` shows nothing, is unaffected, and it is the invariant this phase holds to. +4. **The scripted adapter must satisfy `FakeTextGeneration`, not bare `TextGeneration`.** `describeTextGenerationContract` types its `makeSubject` as `() => FakeTextGeneration`, because two of the behaviours it pins are only observable through the fake's scripting and recording surface. So the scripted adapter keeps the FIFO `scriptStreamText` / `scriptGenerateObject` / `scriptToolConversation` methods and the `requests` record, and adds intent rules underneath them. A queued script wins, rules answer when the queue is empty, and an unmatched call throws. That ordering is what lets one object satisfy the contract suite and the journeys at once. +5. **The renamed data directory guard.** Per-test isolation is a fresh `buildServer()` plus a fresh books directory, and the fixture removes `${dataDir}/books` on teardown rather than the whole worker directory, because the worker directory also holds the key vault file the real adapter would write to if anything ever asked it to. + +## Precondition on Phase 2 (hard gate, verified before starting) + +The harness premise is that overriding `textGeneration` makes generation deterministic. Verified empty at `e8a5dd1`: + +``` +rg -l "model-client" server --glob '!server/adapters/**' --glob '!**/*.test.ts' +``` + +## Runner decision: `@playwright/test`, standalone, `e2e/` top level + +Recommended over extending Phase 3's hand-rolled CDP harness, and over vitest browser mode. + +- **Maturity.** Playwright is the boring industry default. Phase 3's CDP harness was correct for what it did, a one-off pixel measurement where a screenshot and a byte compare were the whole job. A committed journey suite needs auto-waiting locators, per-test isolation, traces, and reporting, and hand-rolling those is precisely the clever-over-simple trap the owner's principles reject. +- **Deterministic waits come free.** Web-first assertions such as `await expect(locator).toContainText(...)` retry to a deadline. That is the answer to SSE timing flake, and it is why no journey contains a sleep. +- **One tool covers Electron.** `_electron.launch()` gives journey (g) without a second harness. That single fact decides it, because the alternative is CDP-attaching to Electron by hand. +- **Vitest browser mode is rejected** as the newest, least battle-tested option, and mixing it into `pnpm test` would put a 90-second suite behind the 2.5-second unit run. `pnpm test` stays fast and E2E is `pnpm e2e`. + +Cost is honest. One devDependency, one Chromium download in CI which is cached, and a second test runner in the repo. Two Playwright projects, `web` for chromium and all journeys, and `electron` for one smoke. + +## Harness design + +Server boots in-process inside a Playwright fixture, using the real `buildServer` from `server/index.ts`, so the fake objects stay addressable from the test and can be scripted and asserted mid-journey. + +```ts +// e2e/support/app.ts (worker-scoped dataDir, test-scoped server) +process.env.TUTOR_DATA_DIR ??= mkdtempSync(join(tmpdir(), 'tutor-e2e-')) +const { buildServer } = await import('@server/index.js') // AFTER the env var + +const model = createScriptedTextGeneration() +const app = await buildServer({ + textGeneration: model, + keyVault: createFakeKeyVault({ anthropic: 'e2e-key' }), + speechSynthesis: createFakeSpeechSynthesis(), + audioAssembly: createFakeAudioAssembly(), + imageGeneration: createFakeImageGeneration(), +}) +await app.register(fastifyStatic, { root: distDir }) // test-only, serves dist/ +await app.listen({ port: 0, host: '127.0.0.1' }) +``` + +Why this shape: + +- **`buildServer(overrides)` is Phase 2's real signature** at `server/index.ts:61`, the same one `server/test/route-harness.ts` uses. `startServer` cannot be used because Fastify forbids registering the static plugin after `listen()`. The harness calls `recoverFromCrash()` itself if a journey needs boot parity. +- **Single origin.** In web mode `client/api/http.ts` resolves `_base = ''` and issues relative `/api/...` paths, because `window.electronAPI` is absent. Serving `dist/` from the same instance means no proxy, no CORS, and no `electronAPI` stub. `@fastify/static` is a first-party plugin and a new devDependency, and it replaces the hand-rolled static server Phase 3 used. No SPA fallback is needed because the client routes on view state rather than on URLs. +- **Data isolation.** `TUTOR_DATA_DIR` is worker-scoped and set once, before the first dynamic import of any server module. Per-test isolation comes from a fresh `buildServer()` plus a fresh books directory per test. +- **Zero live traffic is structural, not a promise.** No provider key exists on the real vault, the fake vault is in-memory, and any un-scripted model call throws by construction. + +**Scripted model adapter** (`e2e/support/scripted-text-generation.ts`) implements `FakeTextGeneration` and answers by intent rather than by call order. Rules match on request shape, meaning `schemaName` or a substring of the prompt, and return fixture content while recording every request. Phase 2's FIFO `createFakeTextGeneration` stays untouched for unit and contract tests. A journey makes N calls in an order the UI decides, and coupling fixtures to that order is the single largest flake source. Fidelity is enforced two ways, by `req.schema.parse(value)` before returning so a drifted fixture fails loudly exactly as a malformed model response would, and by running the adapter against Phase 2's own `text-generation.contract.ts` suite in vitest. + +**Fixtures** (`e2e/fixtures/`) are TS modules rather than JSON, so `tsc` catches schema drift. Plus `sample-book.epub`, a small binary generated once by a committed script using the same `epub-gen-memory` the export adapter uses, and `redux-state.json` for the Electron experiment. + +**Journey helpers** (`e2e/support/journeys/*.ts`) are thin page objects exposing intent, such as `wizard.createBook({topic})` and `reader.finishChapter()`. Locators are role- and text-based and never CSS. Phase 3 gave every icon-only button an accessible name, which makes this possible and makes the suite double as an accessibility net. No `data-testid` is added to production components. If one journey genuinely cannot address an element, that is a finding to report rather than a silent client edit. + +## Journeys and acceptance checks + +| # | Journey | Acceptance | +|---|---|---| +| a | Create to TOC streams to edit/approve to chapter 1 streams to read | Wizard accepts topic, `toc` events render chapter titles from the fixture, editing a title persists to `toc.yml`, approve triggers `/start`, chapter 1 prose appears in the reader, `books/{id}/meta.yml` shows `generatedUpTo: 1` | +| b | Read to quiz to submit to feedback to next chapter | Quiz renders the fixture questions, a wrong answer records in the feedback record, submitting feedback triggers generation, chapter 2 text renders, and the scripted adapter's recorded chapter-2 prompt contains the feedback text, which proves the adaptive loop rather than just the plumbing | +| c | EPUB import preview to confirm | `setInputFiles` with `sample-book.epub`, preview dialog shows the fixture title and chapter count, confirm returns to the library with the book present | +| d | EPUB export | Export from the library, background task reaches done, `waitForEvent('download')` yields a non-empty `.epub`, and the file exists under the temp data dir | +| e | Library CRUD and search | Rename persists across reload, tag add shows in the filter surface, search by title filters the grid, delete removes the book from disk | +| f | Audiobook install gate | With an empty data dir, status shows not-installed and the gate appears, and the install button is never clicked because it downloads roughly 90MB from evermeet.cx. A second test seeds the binary and model directory in the temp dir, then generates chapter audio through the fake synthesis and asserts the player appears | +| g | Electron rehydration smoke (`@electron`) | Two runs, same seeded library, differing only in `redux-state.json`. Seeded position opens the reader at the seeded chapter, removed falls back to the server-derived position. Fixture-seeded and generation-free by construction, since fakes cannot be injected into the packaged main process | +| h | Failure journey | Scripted adapter raises with a distinctive message on the chapter stream, and the UI surfaces that message rather than a generic fallback. The injection point is parameterized so Phase 7 can add per-error-class variants | + +## Implementer tasks + +- **S1 — Walking skeleton.** The `e2e/` scaffold, `playwright.config.ts`, the `app.ts` fixture, the scripted adapter, `pnpm e2e`, plus one journey that creates a book and asserts the fixture TOC titles render. Proves static serving, api-base resolution, SSE, the scripted model, and the temp data dir. Also the vitest `exclude: ['e2e/**']` because the default include collects `*.spec.ts` and would otherwise break `pnpm test`, the tsconfig `include`, the ESLint boundary allowing `e2e` to reach `@server` and `@shared` while forbidding `e2e` to reach `@client`, and the lint script glob. +- **S2 — Contract-test the scripted adapter** against `server/ports/text-generation.contract.ts` in vitest. Lands with S1 or immediately after, and this is the fidelity guarantee. +- **S3** — journey (a) completed through approve to chapter 1 to read. +- **S4** — journey (b), including the prompt-contains-feedback assertion. +- **S5** — fixture builder script and committed `sample-book.epub`, plus journey (c). +- **S6** — journey (d). +- **S7** — journey (e). +- **S8** — journey (f), both halves. +- **S9** — journey (h). +- **S10** — Electron project and journey (g), CI jobs, and `e2e/README.md` covering how to run, how to add a journey, why fakes, and why no testids. + +One commit per task, `test(e2e): ...`. S3 to S9 are independent once S1 lands and can fan out across worktrees. + +## CI integration + +Append to the existing `.github/workflows/ci.yml` rather than creating a file, per consolidation delta 5. + +- **`e2e-web`** runs on macos-14 with node 24, keeps `ELECTRON_SKIP_BINARY_DOWNLOAD: 1`, caches `~/Library/Caches/ms-playwright`, runs `pnpm exec playwright install chromium`, `pnpm build`, and `pnpm e2e --project=web`, and uploads `playwright-report/` on failure. +- **`e2e-electron`** is the same but without `ELECTRON_SKIP_BINARY_DOWNLOAD`, because the binary is required. It caches `~/Library/Caches/electron` and runs `pnpm e2e --project=electron`. + +Both run in parallel with `verify`. Budget is roughly 30s for the build, 60 to 90s for the web suite, and 30s for electron, all inside the 3 to 4 minute target. Flake policy is `retries: 0`, `trace: 'retain-on-failure'`, and `workers: 2`. With in-process fakes and no network there is no legitimate source of nondeterminism, so a retry would hide a real bug. A journey that flakes twice gets quarantined with `test.fixme` plus an issue, never a blanket retry. + +## Risks + +1. **Fake-adapter fidelity drift** is the highest. Fixtures could satisfy the tests while diverging from what a real model returns. Mitigated by `schema.parse` on every scripted object, by running Phase 2's contract suite against the scripted adapter in S2, and by fixtures being TS modules typed against the real schemas. Residual risk stays on prompt content, which no fake can cover, and the manual pass in the Phase 2 and Phase 7 gates remains the backstop. +2. **Playwright resolving the server's TS.** Server modules import with `.js` specifiers and `@server` and `@shared` aliases. Playwright's loader handles tsconfig `paths` and extension substitution, but that is unverified here and is S1's first checkpoint. The fallback if it fails is to boot the server in a `tsx` child process and drive the scripted adapter through a test-only control route, which has more moving parts, so it is only used if forced. +3. **Electron in CI.** A GUI app on a hosted macOS runner, plus a large binary download. Isolated in its own job and tagged `@electron` so it can be excluded with one flag without touching the web suite. +4. **SSE timing.** Handled by auto-retrying assertions and zero-delay scripted chunks. The adapter takes an optional `chunkDelayMs` for the one or two journeys where streaming render order matters. +5. **Audiobook install gate touching the network.** Structural, because the install button is never clicked and the journey asserts the gate rather than the install. +6. **Suite runtime creep** as journeys accrete. The gate below pins a number. + +## Phase gate + +- `pnpm e2e` green twice consecutively, locally and in CI, with `retries: 0`. +- Web suite under 120s wall clock, and both e2e jobs under 4 minutes each. +- `pnpm test`, `pnpm typecheck`, and `pnpm lint` unchanged and green, with E2E excluded from vitest and included in tsc and ESLint. +- **Mutation evidence.** With one deliberate break, the suite fails and names the journey, recorded in the PR body. A test suite that has never failed is not yet evidence. +- Zero production source files changed. `git diff --stat master -- client/ server/ shared/ electron/` shows nothing. +- Zero outbound provider traffic. No key exists in either vault, and the scripted adapter has no SDK import. + +### Critical Files for Implementation + +- `server/index.ts` holds `buildServer(overrides)`, the exact boot mechanism the harness calls +- `server/composition-root.ts` holds the `Ports` interface and the override semantics +- `server/ports/text-generation.ts` and its `.fake.ts` and `.contract.ts` siblings define the shape the scripted adapter implements and is contract-tested against +- `client/api/http.ts` explains why single-origin static serving works in web mode +- `.github/workflows/ci.yml` is the file the two E2E jobs append to From d75ad01388e802dcd978ae87e9373d32c2ec2948 Mon Sep 17 00:00:00 2001 From: Ross Miller Date: Tue, 21 Jul 2026 08:16:43 -0500 Subject: [PATCH 02/18] test(e2e): land the walking skeleton for the journey suite One Playwright test creates a book and watches the fixture table of contents stream into the wizard. It looks small and it proves the entire harness. The built client is served from the same Fastify instance that answers its API calls, so client/api/http.ts resolves its base to empty and issues relative requests exactly as it does in production. The server-sent-event stream reaches the browser and reassembles. The scripted model answers the create-book call site. Everything writes into a throwaway data directory rather than the reader's real library. The biggest unknown going in was whether Playwright's loader could resolve the server's TypeScript, since every server module imports with a .js specifier through an @server or @shared alias. It can, unmodified. The tsx child process fallback the plan held in reserve is not needed. The scripted adapter answers by intent rather than by call order. A journey makes model calls in whatever order the UI decides, and coupling fixtures to that order is the largest available source of flake. Rules match on the prompt or the schema and are consulted newest first, which makes failure injection a one line addition rather than a rewrite. Phase 2's first-in first-out scripting surface is kept on top of the rules, both so a journey can force a one-off answer and so the adapter can satisfy the port's own contract suite, which types its subject to that surface. A call matching neither a queued script nor a rule throws. There is no default answer on purpose, so zero live-provider traffic is a property of the design rather than a promise. Four config lines carry it. Vitest excludes the journeys only, not all of e2e, because the scripted adapter's contract test belongs to vitest. Tsconfig and the lint glob take the new zone in. The ESLint boundary lets a journey reach the server it boots and forbids it reaching the client it is supposed to drive from the outside. Claude-Session: https://claude.ai/code/session_01FSkESkJTRpt3Ye2cM7c5n5 --- .gitignore | 4 +- e2e/fixtures/chapter-stream.ts | 32 ++++ e2e/fixtures/quiz.ts | 64 +++++++ e2e/fixtures/skills.ts | 25 +++ e2e/fixtures/toc-stream.ts | 43 +++++ e2e/global-setup.ts | 22 +++ e2e/journeys/create-book.spec.ts | 47 +++++ e2e/support/app.ts | 143 +++++++++++++++ e2e/support/default-script.ts | 92 ++++++++++ e2e/support/journeys/library.ts | 35 ++++ e2e/support/journeys/wizard.ts | 46 +++++ e2e/support/scripted-text-generation.ts | 225 ++++++++++++++++++++++++ eslint.config.mjs | 29 ++- package.json | 7 +- playwright.config.ts | 59 +++++++ pnpm-lock.yaml | 113 ++++++++++++ tsconfig.json | 2 +- vitest.config.ts | 7 + 18 files changed, 990 insertions(+), 5 deletions(-) create mode 100644 e2e/fixtures/chapter-stream.ts create mode 100644 e2e/fixtures/quiz.ts create mode 100644 e2e/fixtures/skills.ts create mode 100644 e2e/fixtures/toc-stream.ts create mode 100644 e2e/global-setup.ts create mode 100644 e2e/journeys/create-book.spec.ts create mode 100644 e2e/support/app.ts create mode 100644 e2e/support/default-script.ts create mode 100644 e2e/support/journeys/library.ts create mode 100644 e2e/support/journeys/wizard.ts create mode 100644 e2e/support/scripted-text-generation.ts create mode 100644 playwright.config.ts diff --git a/.gitignore b/.gitignore index cf8be6e..0d7d5fe 100644 --- a/.gitignore +++ b/.gitignore @@ -7,4 +7,6 @@ books/ *.log .worktrees/ .claude/worktrees/ -.DS_Store \ No newline at end of file +.DS_Store +playwright-report/ +test-results/ \ No newline at end of file diff --git a/e2e/fixtures/chapter-stream.ts b/e2e/fixtures/chapter-stream.ts new file mode 100644 index 0000000..36cc041 --- /dev/null +++ b/e2e/fixtures/chapter-stream.ts @@ -0,0 +1,32 @@ +import { TOC_CHAPTERS } from './toc-stream.js' + +/** + * The chapter prose the scripted model streams back. + * + * Every chapter carries a distinctive marker sentence, so a journey can + * assert it is looking at chapter 2 and not a stale chapter 1 without + * matching on a heading the UI might also render in a sidebar. + */ + +/** The sentence that appears only in chapter `num`. Journeys locate on this. */ +export function chapterMarker(num: number): string { + return `This is the seeded prose for chapter ${num}.` +} + +/** The full markdown for chapter `num`, matching the shape the reader renders. */ +export function chapterMarkdown(num: number): string { + const chapter = TOC_CHAPTERS[num - 1] + const title = chapter ? chapter.title : `Chapter ${num}` + return [ + `# ${title}\n\n`, + `${chapterMarker(num)}\n\n`, + '## Why it matters\n\n', + 'A body that spins loses energy to the tide it raises on its partner, and it keeps losing energy until the spin and the orbit agree.\n\n', + '> The brake never releases, it only runs out of spin to remove.\n', + ].join('') +} + +/** The same markdown split into stream chunks, so the journey exercises the real SSE reassembly. */ +export function chapterStreamChunks(num: number): string[] { + return chapterMarkdown(num).split(/(?<=\n\n)/) +} diff --git a/e2e/fixtures/quiz.ts b/e2e/fixtures/quiz.ts new file mode 100644 index 0000000..add373b --- /dev/null +++ b/e2e/fixtures/quiz.ts @@ -0,0 +1,64 @@ +/** + * The quiz the scripted model returns after each chapter. + * + * Note for journeys: `server/services/generate-quiz.ts` shuffles the options + * with `Math.random()` before saving, so the stored `correctIndex` is not the + * one written here and the on-screen option order is not this order. A + * journey must therefore locate an option by its TEXT, never by its position, + * and must decide right from wrong by comparing against these strings. + */ + +export interface QuizFixtureQuestion { + question: string + options: string[] + /** Index into `options` as written here, before the service shuffles them. */ + correctIndex: number +} + +export const QUIZ_QUESTIONS: QuizFixtureQuestion[] = [ + { + question: 'What removes a body\'s rotational energy during tidal locking?', + options: [ + 'The tidal bulge dragging against the body\'s own rotation', + 'The solar wind stripping momentum from the upper atmosphere', + 'Magnetic coupling between the two bodies\' iron cores', + 'Radioactive decay slowly redistributing mass in the mantle', + ], + correctIndex: 0, + }, + { + question: 'What state does a tidally locked body settle into?', + options: [ + 'Its rotation period matches its orbital period around the partner', + 'Its rotation stops entirely relative to the background stars', + 'Its orbit becomes perfectly circular and stops precessing', + 'Its axis of rotation aligns exactly with the orbital plane normal', + ], + correctIndex: 0, + }, + { + question: 'What mainly sets how long locking takes?', + options: [ + 'The separation between the bodies, which the tidal force depends on steeply', + 'The absolute temperature of the smaller body at the time of formation', + 'The total number of other satellites sharing the same orbital resonance', + 'The chemical composition of the atmosphere retained by the larger body', + ], + correctIndex: 0, + }, +] + +/** The value the scripted adapter returns for a quiz generateObject call. */ +export const QUIZ_FIXTURE = { questions: QUIZ_QUESTIONS } + +/** The correct option text for a question, located by its text rather than its shuffled position. */ +export function correctOptionFor(question: QuizFixtureQuestion): string { + return question.options[question.correctIndex] +} + +/** Any option that is not the correct one, for journeys that need to answer wrongly on purpose. */ +export function wrongOptionFor(question: QuizFixtureQuestion): string { + const wrong = question.options.find((_option, index) => index !== question.correctIndex) + if (!wrong) throw new Error('quiz fixture: every option is the correct one, so there is no wrong answer to pick') + return wrong +} diff --git a/e2e/fixtures/skills.ts b/e2e/fixtures/skills.ts new file mode 100644 index 0000000..16f7035 --- /dev/null +++ b/e2e/fixtures/skills.ts @@ -0,0 +1,25 @@ +import { TOC_CHAPTERS } from './toc-stream.js' + +/** + * The skill classification `server/services/start-book.ts` asks for right + * before it streams chapter 1. Its schema is declared inline in that service, + * so this fixture is validated against it at run time by the scripted + * adapter's `schema.parse`, which is what catches drift if that schema + * changes. + * + * Classification failure is non-fatal in the service, so a drifted fixture + * would not fail the journey, it would silently skip the skills. The + * `skills_classified` assertion in journey (a) is what keeps that honest. + */ +export const SKILL_CLASSIFICATION = { + skills: [ + { name: 'Orbital Mechanics', weight: 5 }, + { name: 'Classical Dynamics', weight: 3 }, + ], + chapters: TOC_CHAPTERS.map((_chapter, index) => ({ + chapterIndex: index, + skills: [ + { skill: 'Orbital Mechanics', subskill: `Concept ${index + 1}`, weight: 2 }, + ], + })), +} diff --git a/e2e/fixtures/toc-stream.ts b/e2e/fixtures/toc-stream.ts new file mode 100644 index 0000000..dfe23e7 --- /dev/null +++ b/e2e/fixtures/toc-stream.ts @@ -0,0 +1,43 @@ +/** + * The table of contents the scripted model streams back for every journey + * that creates a book. + * + * It is a TS module rather than JSON so `tsc` sees it, and it is split into + * chunks rather than one string so the journey exercises the real SSE path, + * many `toc` events accumulating in the client, rather than a single write + * that would hide a reassembly bug. + * + * The markdown shape is the one `server/services/toc-parser.ts` accepts, a + * `#` title, an italic subtitle, then numbered `**bold** — description` + * lines. Changing the shape here without checking that parser is the fastest + * way to make every creation journey fail at once. + */ + +export const TOC_BOOK_TITLE = 'Tidal Locking' +export const TOC_BOOK_SUBTITLE = 'How Worlds Stop Spinning' + +export interface TocFixtureChapter { + title: string + description: string +} + +/** The chapters the fixture TOC parses into, in order. Journeys assert against these. */ +export const TOC_CHAPTERS: TocFixtureChapter[] = [ + { title: 'Angular Momentum', description: 'Why a spinning body keeps spinning until something takes the spin away.' }, + { title: 'Raising the Bulge', description: 'The tidal force that deforms a body and drags its own rotation backwards.' }, + { title: 'The Slow Brake', description: 'Working out how long a world takes to fall into a locked rotation.' }, +] + +/** + * The fixture streamed as the model would produce it, one chunk per line, so + * the client accumulates the same partial markdown a real run would. + */ +export const TOC_STREAM_CHUNKS: string[] = [ + `# ${TOC_BOOK_TITLE}\n`, + `*${TOC_BOOK_SUBTITLE}*\n`, + '\n', + ...TOC_CHAPTERS.map((chapter, index) => `${index + 1}. **${chapter.title}** — ${chapter.description}\n`), +] + +/** The whole fixture as one string, for assertions that want the parsed result rather than the stream. */ +export const TOC_MARKDOWN = TOC_STREAM_CHUNKS.join('') diff --git a/e2e/global-setup.ts b/e2e/global-setup.ts new file mode 100644 index 0000000..73ea266 --- /dev/null +++ b/e2e/global-setup.ts @@ -0,0 +1,22 @@ +import { execFileSync } from 'node:child_process' +import { fileURLToPath } from 'node:url' + +/** + * Builds the client bundle the app fixture serves out of `dist/`. + * + * This runs once per `playwright test` invocation rather than being a + * separate CI step, so there is exactly one code path and no way to run the + * journeys against a stale bundle. `vite build` also emits `dist-electron/`, + * which the Electron project needs. + * + * Set `E2E_SKIP_BUILD=1` when iterating on a journey and the client has not + * changed, which turns a 15 second startup into an instant one. + */ +export default function globalSetup(): void { + if (process.env.E2E_SKIP_BUILD) { + console.log('[e2e] E2E_SKIP_BUILD set, reusing the existing dist/ build') + return + } + const repoRoot = fileURLToPath(new URL('..', import.meta.url)) + execFileSync('pnpm', ['exec', 'vite', 'build'], { cwd: repoRoot, stdio: 'inherit' }) +} diff --git a/e2e/journeys/create-book.spec.ts b/e2e/journeys/create-book.spec.ts new file mode 100644 index 0000000..fc0b467 --- /dev/null +++ b/e2e/journeys/create-book.spec.ts @@ -0,0 +1,47 @@ +import { TOC_BOOK_TITLE, TOC_CHAPTERS } from '../fixtures/toc-stream.js' +import { expect, test } from '../support/app.js' +import { library } from '../support/journeys/library.js' +import { wizard } from '../support/journeys/wizard.js' + +/** + * Journey (a), first half: the walking skeleton. + * + * This one test is doing more than it looks like. Passing it proves the + * whole harness at once, that the built client is served from the same + * origin as the API, that `client/api/http.ts` resolves its base to empty + * and issues relative requests, that the server-sent-event stream reaches + * the browser and reassembles, that the scripted model answers the + * create-book call site by intent, and that everything wrote into a + * throwaway data directory instead of the reader's real library. + * + * S3 extends this file through approval, chapter 1, and the reader. + */ +test('creates a book and streams the table of contents into the wizard', async ({ page, model, app }) => { + await page.goto('/') + await library(page).waitForReady() + await library(page).openWizard() + + await wizard(page).submit({ topic: 'Tidal locking', details: 'Aim it at a physics graduate.' }) + + // Web-first assertions retry to a deadline, which is how this suite waits + // for a stream to finish without ever sleeping. + for (const chapter of TOC_CHAPTERS) { + await expect(wizard(page).tocEntry(chapter.title)).toBeVisible() + } + await wizard(page).waitForTocApproval() + + // The prompt the server actually sent, read off the same object the server + // called. This is what an in-process fake buys over a child process. + expect(model.requests.streamText).toHaveLength(1) + expect(model.requests.streamText[0].prompt).toContain('Tidal locking') + expect(model.requests.streamText[0].prompt).toContain('Aim it at a physics graduate.') + + // And the book landed on disk, in the throwaway directory, with the title + // parsed out of the fixture rather than the topic the reader typed. + const books = await app.fastify.inject({ method: 'GET', url: '/api/books' }) + expect(books.statusCode).toBe(200) + const library_ = books.json() as Array<{ id: string; title: string; status: string }> + expect(library_).toHaveLength(1) + expect(library_[0].title).toBe(TOC_BOOK_TITLE) + expect(library_[0].status).toBe('toc_review') +}) diff --git a/e2e/support/app.ts b/e2e/support/app.ts new file mode 100644 index 0000000..83ec4f1 --- /dev/null +++ b/e2e/support/app.ts @@ -0,0 +1,143 @@ +import { mkdtempSync, rmSync, existsSync } from 'node:fs' +import { rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { fileURLToPath } from 'node:url' +import { test as base } from '@playwright/test' +import type { FastifyInstance } from 'fastify' +import { applyDefaultScript } from './default-script.js' +import { createScriptedTextGeneration, type ScriptedTextGeneration } from './scripted-text-generation.js' + +/** + * The Playwright fixture that boots the real app for one test. + * + * The server runs in-process, not as a child process, and that is the whole + * design. A journey needs to reach into the fake model mid-flight, to add a + * failure rule or to read back the prompt the server actually sent, and a + * child process would put a serialisation boundary in the way of both. In + * process, `model` is the same object the running server is calling. + * + * Three properties fall out of the shape below. + * + * Single origin. `client/api/http.ts` leaves its base empty when + * `window.electronAPI` is absent, so the built client issues relative + * `/api/...` requests. Serving `dist/` from the same Fastify instance that + * answers those requests means no proxy, no CORS, and no Electron stub. The + * client under test is the real production bundle, unmodified. + * + * Isolation. `TUTOR_DATA_DIR` points at a temp directory created once per + * worker, and the books directory inside it is removed before every test, so + * a journey always starts from an empty library and never sees the real one. + * + * Zero live-provider traffic, structurally. The real key vault is never + * built, because `keyVault` is overridden with an in-memory fake before + * `createPorts` runs, and the two ports that read keys through the vault + * take whichever vault won. The scripted model has no SDK import and throws + * on any call nobody wrote a fixture for. + */ + +const REPO_ROOT = fileURLToPath(new URL('../..', import.meta.url)) +const DIST_DIR = join(REPO_ROOT, 'dist') + +export interface TutorApp { + /** `http://127.0.0.1:` for the instance serving this test. */ + origin: string + /** The temp data directory this worker's books are written under. */ + dataDir: string + /** Absolute path to one book's directory on disk, for filesystem assertions. */ + bookDir(bookId: string): string + /** The running instance, for the rare journey that needs to close or inspect it. */ + fastify: FastifyInstance +} + +interface WorkerFixtures { + /** One temp data directory per worker, created before any server module is imported. */ + dataDir: string +} + +interface TestFixtures { + /** The scripted model this test's server is wired to. Add rules to it before the action that triggers a call. */ + model: ScriptedTextGeneration + /** The booted app. Depend on this (or on `page`, which does) to get a server. */ + app: TutorApp +} + +export const test = base.extend({ + dataDir: [async ({}, use, workerInfo) => { + const dir = mkdtempSync(join(tmpdir(), `tutor-e2e-w${workerInfo.workerIndex}-`)) + // Set before any server module is imported. Every adapter resolves the + // directory through getDataDir() on each call, so this is belt and + // braces rather than load bearing, but server/test/setup-env.ts makes + // the same guarantee for the same reason and consistency is cheap. + process.env.TUTOR_DATA_DIR = dir + await use(dir) + rmSync(dir, { recursive: true, force: true }) + }, { scope: 'worker' }], + + model: async ({}, use) => { + const model = createScriptedTextGeneration() + applyDefaultScript(model) + await use(model) + }, + + app: async ({ dataDir, model }, use) => { + if (!existsSync(join(DIST_DIR, 'index.html'))) { + throw new Error( + `No client build at ${DIST_DIR}. The e2e run builds it in e2e/global-setup.ts, so seeing this means that step was skipped or failed.`, + ) + } + + // Every test starts from an empty library. Removing only books/ rather + // than the whole worker directory leaves anything else the run wrote, + // which matters for the audiobook journey's seeded binaries. + await rm(join(dataDir, 'books'), { recursive: true, force: true }) + + // Imported after TUTOR_DATA_DIR is set, and after the check above, so a + // missing build fails with the message rather than with a static-plugin + // error from deep inside Fastify. + const [{ buildServer }, { createFakeKeyVault }, { createFakeSpeechSynthesis }, { createFakeAudioAssembly }, { createFakeImageGeneration }, fastifyStatic] = await Promise.all([ + import('@server/index.js'), + import('@server/ports/key-vault.fake.js'), + import('@server/ports/speech-synthesis.fake.js'), + import('@server/ports/audio-assembly.fake.js'), + import('@server/ports/image-generation.fake.js'), + import('@fastify/static'), + ]) + + const fastify = await buildServer({ + textGeneration: model, + // A key so the client's provider gate opens, on a vault that only + // exists in memory. The real file vault is never constructed. + keyVault: createFakeKeyVault({ anthropic: 'e2e-key' }), + speechSynthesis: createFakeSpeechSynthesis(), + audioAssembly: createFakeAudioAssembly(), + imageGeneration: createFakeImageGeneration(), + }) + + // Test-only, and registered here rather than in the app because the app + // is served by Electron from a file:// URL in production and has no use + // for a static route. `buildServer` is used rather than `startServer` + // because Fastify forbids registering a plugin after listen(). + await fastify.register(fastifyStatic.default, { root: DIST_DIR }) + await fastify.listen({ port: 0, host: '127.0.0.1' }) + + const address = fastify.server.address() + if (!address || typeof address === 'string') throw new Error('e2e server did not bind a TCP port') + + await use({ + origin: `http://127.0.0.1:${address.port}`, + dataDir, + bookDir: (bookId: string) => join(dataDir, 'books', bookId), + fastify, + }) + + await fastify.close() + }, + + // Points every page in this test at the instance the fixture just booted. + baseURL: async ({ app }, use) => { + await use(app.origin) + }, +}) + +export const expect = test.expect diff --git a/e2e/support/default-script.ts b/e2e/support/default-script.ts new file mode 100644 index 0000000..00c58eb --- /dev/null +++ b/e2e/support/default-script.ts @@ -0,0 +1,92 @@ +import { chapterStreamChunks } from '../fixtures/chapter-stream.js' +import { QUIZ_FIXTURE } from '../fixtures/quiz.js' +import { SKILL_CLASSIFICATION } from '../fixtures/skills.js' +import { TOC_STREAM_CHUNKS } from '../fixtures/toc-stream.js' +import { promptIncludes, systemIncludes, type ScriptedTextGeneration } from './scripted-text-generation.js' + +/** + * Wires every model call site the journeys traverse to its fixture. + * + * The matchers below are the one place this suite knows what the server's + * prompts say. Each is a phrase lifted verbatim from the service that builds + * that prompt, and the comment names the service, so when a prompt is + * reworded the fix is a one line edit here rather than an archaeology + * expedition. A reworded prompt does not fail silently either, because an + * unmatched call throws with the request summary attached. + * + * Rules are consulted newest first, so a journey adds its own rule on top to + * override any of these. That is how journey (h) injects a failure and how + * Phase 7 will add its per-error-class variants. + */ + +/** Phrase from `server/services/create-book.ts`'s system prompt. */ +const TOC_SYSTEM_PHRASE = 'creating a table of contents for a personalized learning book' + +/** Phrase shared by `server/services/start-book.ts` and `server/services/generate-next-chapter.ts`. */ +const CHAPTER_SYSTEM_PHRASE = 'writing a chapter for a personalized learning book' + +/** Phrase from `server/services/start-book.ts`'s skill classification prompt. */ +const SKILLS_PROMPT_PHRASE = 'classifying the learning content of a book' + +/** Phrase from `server/services/generate-quiz.ts`'s prompt. */ +const QUIZ_PROMPT_PHRASE = 'multiple-choice quiz questions to test comprehension' + +/** Phrase from `server/services/revise-toc.ts`'s system prompt. */ +const REVISE_SYSTEM_PHRASE = 'revising an existing table of contents' + +/** + * Which chapter a chapter-stream request is for. + * + * Both chapter services write `This is Chapter N of M.` into the prompt, and + * that number is the only thing distinguishing the two streams, so it is read + * back out rather than tracked as call-order state. + */ +export function chapterNumberFrom(prompt: string | undefined): number { + const match = (prompt ?? '').match(/This is Chapter (\d+) of/) + return match ? Number(match[1]) : 1 +} + +/** Matches any chapter-generation stream, chapter 1 or chapter N. */ +export const isChapterStream = systemIncludes(CHAPTER_SYSTEM_PHRASE) + +/** Matches the chapter-generation stream for one specific chapter. */ +export const isChapterStreamFor = (num: number) => (req: { system?: string; prompt?: string }): boolean => + isChapterStream(req) && chapterNumberFrom(req.prompt) === num + +/** Matches the table-of-contents stream that `POST /api/books` opens. */ +export const isTocStream = systemIncludes(TOC_SYSTEM_PHRASE) + +/** Installs the default rule set. Journeys may add rules on top to shadow any of these. */ +export function applyDefaultScript(model: ScriptedTextGeneration): void { + // Registered oldest first so the resulting precedence reads top to bottom + // in this function, since each call unshifts onto the front of the list. + model.onStreamText({ + name: 'chapter stream (start-book and generate-next-chapter)', + match: isChapterStream, + respond: req => ({ chunks: chapterStreamChunks(chapterNumberFrom(req.prompt)) }), + }) + + model.onStreamText({ + name: 'table of contents revision (revise-toc)', + match: systemIncludes(REVISE_SYSTEM_PHRASE), + respond: { chunks: TOC_STREAM_CHUNKS }, + }) + + model.onStreamText({ + name: 'table of contents (create-book)', + match: isTocStream, + respond: { chunks: TOC_STREAM_CHUNKS }, + }) + + model.onGenerateObject({ + name: 'quiz (generate-quiz)', + match: promptIncludes(QUIZ_PROMPT_PHRASE), + respond: { value: QUIZ_FIXTURE }, + }) + + model.onGenerateObject({ + name: 'skill classification (start-book)', + match: promptIncludes(SKILLS_PROMPT_PHRASE), + respond: { value: SKILL_CLASSIFICATION }, + }) +} diff --git a/e2e/support/journeys/library.ts b/e2e/support/journeys/library.ts new file mode 100644 index 0000000..b539180 --- /dev/null +++ b/e2e/support/journeys/library.ts @@ -0,0 +1,35 @@ +import type { Page } from '@playwright/test' + +/** + * The library screen, expressed as intent rather than as selectors. + * + * Every locator here is role- or text-based. None is a CSS class and none is + * a `data-testid`, because Phase 3 gave every icon-only control an accessible + * name and addressing the UI the way a screen reader does makes this suite + * double as an accessibility net. If a journey cannot reach a control this + * way, the finding is that the control has no accessible name, and the fix + * belongs in the component rather than here. + */ +export function library(page: Page) { + // Two controls open the wizard, one in the header and one in the empty + // state. The header one is always present, so it is the one journeys use + // and the empty-state duplicate never makes a locator ambiguous. + const newBook = () => page.getByRole('banner').getByRole('button', { name: 'New Book' }) + + return { + /** Waits for the library to be interactive, which is when the New Book control exists. */ + async waitForReady(): Promise { + await newBook().waitFor() + }, + + /** Opens the create-book wizard. */ + async openWizard(): Promise { + await newBook().click() + }, + + /** The card for one book, located by its title. */ + card(title: string) { + return page.getByRole('button', { name: new RegExp(title, 'i') }) + }, + } +} diff --git a/e2e/support/journeys/wizard.ts b/e2e/support/journeys/wizard.ts new file mode 100644 index 0000000..7b2efff --- /dev/null +++ b/e2e/support/journeys/wizard.ts @@ -0,0 +1,46 @@ +import type { Page } from '@playwright/test' + +/** + * The create-book flow, from the New Book dialog through the streamed table + * of contents to the approval that generates chapter 1. + * + * Two screens, one intent. `WizardModal` collects the topic and + * `CreationView` shows the streaming result, but a reader experiences them as + * one act, so they share one page object. + * + * The table of contents renders as markdown with no landmark role of its + * own, so it is addressed by the text it contains rather than by the class + * on its container. Text is what a reader sees, a class is an implementation + * detail, and matching the class would make a purely visual refactor break + * the suite. + */ +export function wizard(page: Page) { + return { + /** Fills the topic (and optional details) and submits, which opens the table-of-contents stream. */ + async submit({ topic, details }: { topic: string; details?: string }): Promise { + await page.getByLabel('Topic').fill(topic) + if (details !== undefined) await page.getByLabel(/^Details/).fill(details) + await page.getByRole('button', { name: 'Create' }).click() + }, + + /** Resolves once the stream has finished and the approve control is offered. */ + async waitForTocApproval(): Promise { + await page.getByRole('button', { name: /Generate Chapter 1/ }).waitFor() + }, + + /** Approves the table of contents, which starts chapter 1. */ + async approveToc(): Promise { + await page.getByRole('button', { name: /Generate Chapter 1/ }).click() + }, + + /** Opens the revision panel for editing the table of contents. */ + async openTocEditor(): Promise { + await page.getByRole('button', { name: 'Edit Table of Contents' }).click() + }, + + /** One chapter entry in the rendered table of contents, located by its title. */ + tocEntry(title: string) { + return page.getByText(title, { exact: false }).first() + }, + } +} diff --git a/e2e/support/scripted-text-generation.ts b/e2e/support/scripted-text-generation.ts new file mode 100644 index 0000000..1c98082 --- /dev/null +++ b/e2e/support/scripted-text-generation.ts @@ -0,0 +1,225 @@ +import type { + GenerateObjectRequest, + RunToolConversationRequest, + StreamTextRequest, + TextChunk, +} from '@server/ports/text-generation.js' +import type { FakeTextGeneration, ToolConversationStep } from '@server/ports/text-generation.fake.js' + +/** + * The TextGeneration a journey drives. + * + * Phase 2's `createFakeTextGeneration` answers strictly in call order, which + * is right for a unit test that knows exactly how many model calls the code + * under test makes. A journey does not know that. Clicking "Generate Chapter + * 1" triggers a skill classification, a chapter stream, and a quiz, in an + * order the UI decides and may reorder tomorrow, and a fixture list coupled + * to that order is the single largest source of flake in a suite like this. + * + * So this adapter answers by intent. A rule matches on the shape of the + * request, meaning the system prompt, the user prompt, or the schema name, + * and returns the fixture for that call site. Rules are consulted + * most-recently-added first, which is what makes failure injection a one + * line addition rather than a rewrite: a journey adds a rule that matches + * the chapter stream and throws, and it shadows the default rule underneath + * without removing it. + * + * The FIFO surface from Phase 2's fake is kept, and a queued script always + * beats a rule. Two reasons. It lets one journey force a specific one-off + * answer without inventing a rule for it, and it lets this adapter satisfy + * `describeTextGenerationContract`, whose subject is typed to the fake's + * scripting and recording surface rather than to the bare port. That + * contract run is the fidelity guarantee, see + * `e2e/support/scripted-text-generation.contract.test.ts`. + * + * A call that matches neither a queued script nor a rule throws. There is no + * default answer on purpose. Zero live-provider traffic is then structural + * rather than a promise, because the only way to get a response out of this + * object is to have written the response down first. + */ + +/** What a matched streamText rule produces: either chunks to yield, or something to throw. */ +export type StreamOutcome = + | { chunks: string[]; chunkDelayMs?: number } + | { throws: unknown } + +/** What a matched generateObject rule produces. The value is still parsed through the caller's schema. */ +export type ObjectOutcome = + | { value: unknown } + | { throws: unknown } + +/** What a matched runToolConversation rule produces. */ +export type ToolOutcome = + | { steps: ToolConversationStep[] } + | { throws: unknown } + +export interface Rule { + /** Named so an unmatched-call error can list what was on offer. */ + name: string + match: (req: Req) => boolean + respond: Outcome | ((req: Req) => Outcome) +} + +export type StreamRule = Rule +export type ObjectRule = Rule, ObjectOutcome> +export type ToolRule = Rule + +export interface ScriptedTextGeneration extends FakeTextGeneration { + /** Adds a streamText rule. Later rules shadow earlier ones, so this is also the failure-injection point. */ + onStreamText(rule: StreamRule): void + /** Adds a generateObject rule. Later rules shadow earlier ones. */ + onGenerateObject(rule: ObjectRule): void + /** Adds a runToolConversation rule. Later rules shadow earlier ones. */ + onToolConversation(rule: ToolRule): void +} + +/** True when the request's user prompt contains `needle`. */ +export const promptIncludes = (needle: string) => (req: { prompt?: string }): boolean => + (req.prompt ?? '').includes(needle) + +/** True when the request's system prompt contains `needle`. */ +export const systemIncludes = (needle: string) => (req: { system?: string }): boolean => + (req.system ?? '').includes(needle) + +/** True when the request carries this `schemaName`. Only one real call site sets it, so prefer prompt matching. */ +export const schemaNamed = (name: string) => (req: GenerateObjectRequest): boolean => + req.schemaName === name + +/** True when every one of `matchers` is true. */ +export const allOf = (...matchers: Array<(req: Req) => boolean>) => (req: Req): boolean => + matchers.every(matcher => matcher(req)) + +function resolve(rule: Rule, req: Req): Outcome { + return typeof rule.respond === 'function' + ? (rule.respond as (req: Req) => Outcome)(req) + : rule.respond +} + +/** The diagnostic an unmatched call raises. Names every rule that was on offer, because "no rule matched" alone is unactionable. */ +function unmatched(method: string, rules: Array<{ name: string }>, detail: string): Error { + const offered = rules.length > 0 ? rules.map(rule => rule.name).join(', ') : '(none)' + return new Error( + `scripted-text-generation: no rule matched a ${method} call, and nothing was queued. ` + + `Rules on offer: ${offered}. Request: ${detail}`, + ) +} + +/** A short, greppable summary of a request, for the unmatched-call diagnostic. */ +function summarise(req: { system?: string; prompt?: string }): string { + const head = (text: string | undefined) => (text ?? '').replace(/\s+/g, ' ').slice(0, 120) + return `system="${head(req.system)}" prompt="${head(req.prompt)}"` +} + +export function createScriptedTextGeneration(): ScriptedTextGeneration { + const requests: FakeTextGeneration['requests'] = { + streamText: [], + generateObject: [], + runToolConversation: [], + } + + // Rules are unshifted rather than pushed, so the newest rule is consulted + // first and a journey can shadow a default without deleting it. + const streamRules: StreamRule[] = [] + const objectRules: ObjectRule[] = [] + const toolRules: ToolRule[] = [] + + const streamQueue: string[][] = [] + const objectQueue: unknown[] = [] + const toolQueue: ToolConversationStep[][] = [] + + async function* yieldChunks(chunks: string[], signal?: AbortSignal, chunkDelayMs?: number): AsyncGenerator { + for (const chunk of chunks) { + if (signal?.aborted) throw signal.reason + if (chunkDelayMs) await new Promise(done => setTimeout(done, chunkDelayMs)) + yield chunk + } + } + + /** + * A stream that rejects the moment it is pulled, which is how a failure + * rule reaches the caller. Written as an iterable rather than as an async + * generator that throws, because a generator with no `yield` is a lint + * error and, more to the point, this shape says what it does. + */ + function throwOnly(thrown: unknown): AsyncIterable { + return { + [Symbol.asyncIterator]: () => ({ next: () => Promise.reject(thrown) }), + } + } + + function streamText(req: StreamTextRequest): AsyncIterable { + requests.streamText.push(req) + + const queued = streamQueue.shift() + if (queued) return yieldChunks(queued, req.signal) + + const rule = streamRules.find(candidate => candidate.match(req)) + if (!rule) throw unmatched('streamText', streamRules, summarise(req)) + + const outcome = resolve(rule, req) + if ('throws' in outcome) return throwOnly(outcome.throws) + return yieldChunks(outcome.chunks, req.signal, outcome.chunkDelayMs) + } + + async function generateObject(req: GenerateObjectRequest): Promise { + requests.generateObject.push(req as GenerateObjectRequest) + if (req.signal?.aborted) throw req.signal.reason + + if (objectQueue.length > 0) { + return req.schema.parse(objectQueue.shift()) + } + + const wide = req as GenerateObjectRequest + const rule = objectRules.find(candidate => candidate.match(wide)) + if (!rule) throw unmatched('generateObject', objectRules, summarise(req)) + + const outcome = resolve(rule, wide) + if ('throws' in outcome) throw outcome.throws + // Parsed through the caller's own schema, so a fixture that drifts away + // from the shape the app asks for fails here, loudly, exactly as a real + // model returning malformed output would. + return req.schema.parse(outcome.value) + } + + async function* runSteps(steps: ToolConversationStep[], req: RunToolConversationRequest): AsyncGenerator { + for (const step of steps.slice(0, req.maxSteps)) { + if (req.signal?.aborted) throw req.signal.reason + if (step.type === 'text') { + yield { type: 'text', text: step.text } + continue + } + const tool = req.tools[step.tool] + if (!tool) { + throw new Error(`scripted-text-generation: scripted tool call names "${step.tool}", which is not in this request's tools.`) + } + await tool.execute(tool.inputSchema.parse(step.input)) + } + } + + function runToolConversation(req: RunToolConversationRequest): AsyncIterable { + requests.runToolConversation.push(req) + + const queued = toolQueue.shift() + if (queued) return runSteps(queued, req) + + const rule = toolRules.find(candidate => candidate.match(req)) + if (!rule) throw unmatched('runToolConversation', toolRules, summarise({ system: req.system })) + + const outcome = resolve(rule, req) + if ('throws' in outcome) return throwOnly(outcome.throws) + return runSteps(outcome.steps, req) + } + + return { + requests, + scriptStreamText: chunks => { streamQueue.push(chunks) }, + scriptGenerateObject: value => { objectQueue.push(value) }, + scriptToolConversation: steps => { toolQueue.push(steps) }, + onStreamText: rule => { streamRules.unshift(rule) }, + onGenerateObject: rule => { objectRules.unshift(rule) }, + onToolConversation: rule => { toolRules.unshift(rule) }, + streamText, + generateObject, + runToolConversation, + } +} diff --git a/eslint.config.mjs b/eslint.config.mjs index bce0e5f..f98b3a1 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -105,6 +105,33 @@ const boundaries = [ }, ]), }, + { + // The end-to-end journey suite drives the client through a browser, never + // by importing it. It may reach @server and @shared, because it boots the + // real server in process and wires Phase 2's own fakes into it, but a + // direct @client import would mean a journey had reached past the browser + // into the code it is supposed to be testing from the outside. + files: ['e2e/**/*.ts', 'playwright.config.ts'], + rules: forbid([ + { + group: ['@client/**', '**/client/**'], + message: 'A journey drives the client through the browser. Assert on what a reader can see, not on client internals.', + }, + ]), + }, + { + files: ['e2e/**/*.ts'], + rules: { + // Playwright fixtures that take no dependencies are declared with an + // empty destructuring pattern, which is the framework's own documented + // signature and not the mistake this rule usually catches. + 'no-empty-pattern': 'off', + // A Playwright fixture hands its value to the test by calling `use`, + // which is an unrelated function that happens to share a name with + // React 19's hook. There is no React in this zone at all. + 'react-hooks/rules-of-hooks': 'off', + }, + }, ] export default tseslint.config( @@ -134,7 +161,7 @@ export default tseslint.config( languageOptions: { globals: globals.browser }, }, { - files: ['server/**/*.ts', 'electron/**/*.ts', 'shared/node/**/*.ts'], + files: ['server/**/*.ts', 'electron/**/*.ts', 'shared/node/**/*.ts', 'e2e/**/*.ts', 'playwright.config.ts'], languageOptions: { globals: globals.node }, }, { diff --git a/package.json b/package.json index 30a8392..fd3880c 100644 --- a/package.json +++ b/package.json @@ -15,8 +15,9 @@ "electron:preview": "vite build && electron .", "test": "vitest run", "test:watch": "vitest", - "lint": "eslint --max-warnings 0 client/ server/ electron/ shared/", - "lint:fix": "eslint --fix --max-warnings 0 client/ server/ electron/ shared/", + "e2e": "playwright test", + "lint": "eslint --max-warnings 0 client/ server/ electron/ shared/ e2e/ playwright.config.ts", + "lint:fix": "eslint --fix --max-warnings 0 client/ server/ electron/ shared/ e2e/ playwright.config.ts", "typecheck": "tsc --noEmit", "release": "scripts/release.sh", "mcp:dev": "tsx server/mcp-server.ts", @@ -27,6 +28,8 @@ "devDependencies": { "@eslint/js": "^10.0.1", "@evilmartians/lefthook": "^2.1.3", + "@fastify/static": "^10.1.0", + "@playwright/test": "^1.61.1", "@tailwindcss/vite": "^4.2.1", "@types/culori": "^4.0.1", "@types/diff": "^8.0.0", diff --git a/playwright.config.ts b/playwright.config.ts new file mode 100644 index 0000000..1902c0e --- /dev/null +++ b/playwright.config.ts @@ -0,0 +1,59 @@ +import { defineConfig, devices } from '@playwright/test' + +/** + * The end-to-end journey suite. See `e2e/README.md` for how to run it and + * how to add a journey. + * + * `retries: 0` is deliberate and is the most load-bearing line in this file. + * The server runs in-process against fake adapters, the model is a scripted + * object, and nothing in the suite touches a network, so there is no + * legitimate source of nondeterminism left for a retry to paper over. A + * retry here would convert a real bug into a green run. A journey that + * flakes twice gets quarantined with `test.fixme` and an issue, never a + * blanket retry. + */ +export default defineConfig({ + testDir: './e2e/journeys', + + // Builds the client bundle the fixture serves. Always, rather than only + // when dist/ is missing, because a stale bundle would silently test + // yesterday's client. Skippable with E2E_SKIP_BUILD for a fast re-run. + globalSetup: './e2e/global-setup.ts', + + fullyParallel: true, + forbidOnly: !!process.env.CI, + retries: 0, + workers: 2, + + timeout: 60_000, + expect: { timeout: 15_000 }, + + // Each test boots a Fastify instance whose logger writes a JSON line per + // request, which drowns the reporter. The trace holds the browser side of + // a failure; set E2E_VERBOSE=1 to see the server side too. + quiet: !process.env.E2E_VERBOSE, + + reporter: process.env.CI + ? [['github'], ['html', { open: 'never' }], ['list']] + : [['list']], + + use: { + trace: 'retain-on-failure', + // baseURL is supplied per test by the `app` fixture in e2e/support/app.ts, + // because each test binds its own ephemeral port. + }, + + projects: [ + { + name: 'web', + grepInvert: /@electron/, + use: { ...devices['Desktop Chrome'], viewport: { width: 1440, height: 900 } }, + }, + { + // Launches the packaged main process, so it needs the Electron binary + // and gets its own CI job. One flag excludes it everywhere else. + name: 'electron', + grep: /@electron/, + }, + ], +}) diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 0356789..21192be 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -174,6 +174,12 @@ importers: '@evilmartians/lefthook': specifier: ^2.1.3 version: 2.1.3 + '@fastify/static': + specifier: ^10.1.0 + version: 10.1.0 + '@playwright/test': + specifier: ^1.61.1 + version: 1.61.1 '@tailwindcss/vite': specifier: ^4.2.1 version: 4.2.1(vite@7.3.1(@types/node@25.3.5)(jiti@2.6.1)(lightningcss@1.31.1)(tsx@4.21.0)(yaml@2.8.2)) @@ -754,6 +760,9 @@ packages: os: [darwin, linux, win32] hasBin: true + '@fastify/accept-negotiator@2.0.1': + resolution: {integrity: sha512-/c/TW2bO/v9JeEgoD/g1G5GxGeCF1Hafdf79WPmUlgYiBXummY0oX3VVq4yFkKKVBKDNlaDUYoab7g38RpPqCQ==} + '@fastify/ajv-compiler@4.0.5': resolution: {integrity: sha512-KoWKW+MhvfTRWL4qrhUwAAZoaChluo0m0vbiJlGMt2GXvL4LVPQEjt8kSpHI3IBq5Rez8fg+XeH3cneztq+C7A==} @@ -778,6 +787,12 @@ packages: '@fastify/rate-limit@10.3.0': resolution: {integrity: sha512-eIGkG9XKQs0nyynatApA3EVrojHOuq4l6fhB4eeCk4PIOeadvOJz9/4w3vGI44Go17uaXOWEcPkaD8kuKm7g6Q==} + '@fastify/send@4.1.0': + resolution: {integrity: sha512-TMYeQLCBSy2TOFmV95hQWkiTYgC/SEx7vMdV+wnZVX4tt8VBLKzmH8vV9OzJehV0+XBfg+WxPMt5wp+JBUKsVw==} + + '@fastify/static@10.1.0': + resolution: {integrity: sha512-iK/8TvRM/EgNOyQL+EpWu+x3aR6o4GWt+UI+27zmE7w6t/6d80mXqOtWLdEKQ13vL/g1Jry0ae2icj6GP7tGzA==} + '@floating-ui/core@1.7.5': resolution: {integrity: sha512-1Ih4WTWyw0+lKyFMcBHGbb5U5FtuHJuujoyyr5zTaWS5EYMeT6Jb2AuDeftsCsEuchO+mM2ij5+q9crhydzLhQ==} @@ -1129,6 +1144,11 @@ packages: resolution: {integrity: sha512-+1VkjdD0QBLPodGrJUeqarH8VAIvQODIbwh9XpP5Syisf7YoQgsJKPNFoqqLQlu+VQ/tVSshMR6loPMn8U+dPg==} engines: {node: '>=14'} + '@playwright/test@1.61.1': + resolution: {integrity: sha512-8nKv6+0RJSL9FE4jYOEGXnPeM/Hg12qZpmqzZjRh3qM0Y7c3z1mrOTfFLids72RDQYVh9WpLEfR5WdpNX4fkig==} + engines: {node: '>=18'} + hasBin: true + '@protobufjs/aspromise@1.1.2': resolution: {integrity: sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==} @@ -2332,6 +2352,10 @@ packages: resolution: {integrity: sha512-oIXISMynqSqm241k6kcQ5UwttDILMK4BiurCfGEREw6+X9jkkpEe5T9FZaApyLGGOnFuyMWZpdolTXMtvEJ08Q==} engines: {node: '>=18'} + content-disposition@2.0.1: + resolution: {integrity: sha512-e+H0ZXHSWYrENhQzw1LPuP4oF5MzVKmDU6d3hxlvaPEYLLg62MxtQNPRx4SYSuYJSBUgnQIG4HIN2tEtNv7Dog==} + engines: {node: '>=18'} + content-type@1.0.5: resolution: {integrity: sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==} engines: {node: '>= 0.6'} @@ -2985,6 +3009,9 @@ packages: fastify-plugin@5.1.0: resolution: {integrity: sha512-FAIDA8eovSt5qcDgcBvDuX/v0Cjz0ohGhENZ/wpc3y+oZCY2afZ9Baqql3g/lC+OHRnciQol4ww7tuthOb9idw==} + fastify-plugin@6.0.0: + resolution: {integrity: sha512-fZOty7z3O7vOliF6d8bHE3wiEh1KcNnKEQensSgTk9C1DvN6nRLS++XVd86v33Hw/8u9Un8A1zDrQ8ujcQDHEg==} + fastify@5.8.1: resolution: {integrity: sha512-y0kicFvvn7CYWoPOVLOcvn4YyKQz03DIY7UxmyOy21/J8eXm09R+tmb+tVDBW5h+pja30cHI5dqUcSlvY86V2A==} @@ -3096,6 +3123,11 @@ packages: fs.realpath@1.0.0: resolution: {integrity: sha512-OO0pH2lK6a0hZnAdau5ItzHPI6pUlvI7jMVnxUQRtw4owF2wk8lOSabtGDCTP4Ggrg2MbGnWO9X8K1t4+fGMDw==} + fsevents@2.3.2: + resolution: {integrity: sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==} + engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0} + os: [darwin] + fsevents@2.3.3: resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==} engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0} @@ -3166,6 +3198,10 @@ packages: deprecated: Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me hasBin: true + glob@13.0.6: + resolution: {integrity: sha512-Wjlyrolmm8uDpm/ogGyXZXb1Z+Ca2B8NbJwqBVg0axK9GbBeoS7yGV6vjXnYdGm6X53iehEuxxbyiKp8QmN4Vw==} + engines: {node: 18 || 20 || >=22} + glob@7.2.3: resolution: {integrity: sha512-nFR0zLpU2YCaRxwoCJvL6UvCH2JFyFVIvwTLsIf21AuHlMskA1hhTdk+LlYJtOlYt9v6dvszD2BGRqBL+iQK9Q==} deprecated: Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me @@ -3730,6 +3766,10 @@ packages: lru-cache@10.4.3: resolution: {integrity: sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ==} + lru-cache@11.5.2: + resolution: {integrity: sha512-4pfM1Ff0x50o0tQwb5ucw/RzNyD0/YJME6IVcStalZuMWxdt3sR3huStTtxz4PUmvZfRguvDejasvQ2kifR11g==} + engines: {node: 20 || >=22} + lru-cache@5.1.1: resolution: {integrity: sha512-KpNARQA3Iwv+jTA0utUVVbrh+Jlrr1Fv0e56GGzAFOXN7dk/FviaDW8LHmK52DlcH4WP2n6gI8vN1aesBFgo9w==} @@ -3947,6 +3987,11 @@ packages: engines: {node: '>=4.0.0'} hasBin: true + mime@3.0.0: + resolution: {integrity: sha512-jSCU7/VB1loIWBZe14aEYHU/+1UMEHoaO7qxCOVJOw9GgH72VAWppxNcjU+x9a2k3GSIBXNKxXQFqRvvZ7vr3A==} + engines: {node: '>=10.0.0'} + hasBin: true + mimic-fn@2.1.0: resolution: {integrity: sha512-OqbOk5oEQeAZ8WXWydlu9HJjz9WVdEIvamMCcXmuqUYjTknH/sqsWvhQ3vgwKFRR1HpjvNBKQ37nbJgYzGqGcg==} engines: {node: '>=6'} @@ -4259,6 +4304,10 @@ packages: resolution: {integrity: sha512-Xa4Nw17FS9ApQFJ9umLiJS4orGjm7ZzwUrwamcGQuHSzDyth9boKDaycYdDcZDuqYATXw4HFXgaqWTctW/v1HA==} engines: {node: '>=16 || 14 >=14.18'} + path-scurry@2.0.2: + resolution: {integrity: sha512-3O/iVVsJAPsOnpwWIeD+d6z/7PmqApyQePUtCndjatj/9I5LylHvt5qluFaBT3I5h3r1ejfR056c+FCv+NnNXg==} + engines: {node: 18 || 20 || >=22} + path-to-regexp@6.3.0: resolution: {integrity: sha512-Yhpw4T9C6hPpgPeA28us07OJeqZ5EzQTkbfwuhsUg0c237RomFoETJgmp2sa3F/41gfLE6G5cqcYwznmeEeOlQ==} @@ -4309,6 +4358,16 @@ packages: platform@1.3.6: resolution: {integrity: sha512-fnWVljUchTro6RiCFvCXBbNhJc2NijN7oIQxbwsyL0buWJPG85v81ehlHI9fXrJsMNgTofEoWIQeClKpgxFLrg==} + playwright-core@1.61.1: + resolution: {integrity: sha512-h7Qlt6m4REp25qvIdvbDtVmD4LqVXfpRxhORv9L0jzETM05p4fuPJ3dKyuSXQxDSbXnmS79HAgi9589lGSpLkg==} + engines: {node: '>=18'} + hasBin: true + + playwright@1.61.1: + resolution: {integrity: sha512-DWnY5o3YbLWK4GovuAVwpqL+1VwGNdUGrRr++8j8PtQQzvAVZUIMjKQ90fY689sEJZJBbZVw1rXaOKSTitkzPQ==} + engines: {node: '>=18'} + hasBin: true + plist@3.1.0: resolution: {integrity: sha512-uysumyrvkUX0rX/dEVqt8gC3sTBzd4zoWfLeS29nb53imdaXVvLINYXTI2GNqzaMuvacNx4uJQ8+b3zXR0pkgQ==} engines: {node: '>=10.4.0'} @@ -5988,6 +6047,8 @@ snapshots: '@evilmartians/lefthook@2.1.3': {} + '@fastify/accept-negotiator@2.0.1': {} + '@fastify/ajv-compiler@4.0.5': dependencies: ajv: 8.18.0 @@ -6022,6 +6083,24 @@ snapshots: fastify-plugin: 5.1.0 toad-cache: 3.7.0 + '@fastify/send@4.1.0': + dependencies: + '@lukeed/ms': 2.0.2 + escape-html: 1.0.3 + fast-decode-uri-component: 1.0.1 + http-errors: 2.0.1 + mime: 3.0.0 + + '@fastify/static@10.1.0': + dependencies: + '@fastify/accept-negotiator': 2.0.1 + '@fastify/error': 4.2.0 + '@fastify/send': 4.1.0 + content-disposition: 2.0.1 + fastify-plugin: 6.0.0 + fastq: 1.20.1 + glob: 13.0.6 + '@floating-ui/core@1.7.5': dependencies: '@floating-ui/utils': 0.2.11 @@ -6356,6 +6435,10 @@ snapshots: '@pkgjs/parseargs@0.11.0': optional: true + '@playwright/test@1.61.1': + dependencies: + playwright: 1.61.1 + '@protobufjs/aspromise@1.1.2': {} '@protobufjs/base64@1.1.2': {} @@ -7537,6 +7620,8 @@ snapshots: content-disposition@1.0.1: {} + content-disposition@2.0.1: {} + content-type@1.0.5: {} convert-source-map@2.0.0: {} @@ -8346,6 +8431,8 @@ snapshots: fastify-plugin@5.1.0: {} + fastify-plugin@6.0.0: {} + fastify@5.8.1: dependencies: '@fastify/ajv-compiler': 4.0.5 @@ -8491,6 +8578,9 @@ snapshots: fs.realpath@1.0.0: {} + fsevents@2.3.2: + optional: true + fsevents@2.3.3: optional: true @@ -8560,6 +8650,12 @@ snapshots: package-json-from-dist: 1.0.1 path-scurry: 1.11.1 + glob@13.0.6: + dependencies: + minimatch: 10.2.4 + minipass: 7.1.3 + path-scurry: 2.0.2 + glob@7.2.3: dependencies: fs.realpath: 1.0.0 @@ -9109,6 +9205,8 @@ snapshots: lru-cache@10.4.3: {} + lru-cache@11.5.2: {} + lru-cache@5.1.1: dependencies: yallist: 3.1.1 @@ -9572,6 +9670,8 @@ snapshots: mime@2.6.0: {} + mime@3.0.0: {} + mimic-fn@2.1.0: {} mimic-function@5.0.1: {} @@ -9910,6 +10010,11 @@ snapshots: lru-cache: 10.4.3 minipass: 7.1.3 + path-scurry@2.0.2: + dependencies: + lru-cache: 11.5.2 + minipass: 7.1.3 + path-to-regexp@6.3.0: {} path-to-regexp@8.3.0: {} @@ -9958,6 +10063,14 @@ snapshots: platform@1.3.6: {} + playwright-core@1.61.1: {} + + playwright@1.61.1: + dependencies: + playwright-core: 1.61.1 + optionalDependencies: + fsevents: 2.3.2 + plist@3.1.0: dependencies: '@xmldom/xmldom': 0.8.11 diff --git a/tsconfig.json b/tsconfig.json index 470ea08..c5b6b5e 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -21,6 +21,6 @@ "@shared/*": ["./shared/*"] } }, - "include": ["shared/**/*.ts", "server/**/*.ts", "client/**/*.ts", "client/**/*.tsx", "electron/**/*.ts"], + "include": ["shared/**/*.ts", "server/**/*.ts", "client/**/*.ts", "client/**/*.tsx", "electron/**/*.ts", "e2e/**/*.ts", "playwright.config.ts"], "exclude": ["node_modules", "dist", "dist-electron"] } diff --git a/vitest.config.ts b/vitest.config.ts index f3a2b44..a8e522a 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -20,6 +20,13 @@ export default defineConfig({ '**/dist-electron/**', '.worktrees/**', '.claude/worktrees/**', + // The Playwright journey suite. Vitest's default include collects + // *.spec.ts, so without this line `pnpm test` would try to run the + // journeys, and `pnpm test` is deliberately the fast one. Journeys + // are `pnpm e2e`. Only the journeys are excluded, not all of e2e/, + // because the scripted adapter's contract test lives under e2e/ as a + // *.test.ts and vitest is exactly the runner that should own it. + 'e2e/**/*.spec.ts', ], }, resolve: { From 98beb7e9945412075ec19ddaa20a78f27184bd66 Mon Sep 17 00:00:00 2001 From: Ross Miller Date: Tue, 21 Jul 2026 08:19:35 -0500 Subject: [PATCH 03/18] test(e2e): run the port contract against the scripted model The journeys are only worth anything if the object standing in for the AI provider behaves the way the port says a TextGeneration behaves. So Phase 2's own contract suite runs against this adapter with no exemptions, covering cancellation mid stream, schema validation of every generated object, request recording, and the tool conversation's step limit. That is the fidelity guarantee the plan asks for, and it is why the adapter kept the fake's scripting surface rather than inventing its own. Ten further tests cover the rule surface the contract does not know about. Rules answer by intent regardless of call order, a later rule shadows an earlier one, a failure rule throws the exact value it names so a typed error class survives to the caller, a queued script beats a matching rule, and an unmatched call throws while naming every rule that was on offer. That last one is what makes zero live provider traffic a property of the design. Writing the allOf test found two real defects in what shipped with the skeleton. The default script's comment claimed precedence reads top to bottom, when rules are pushed onto the front of the list and precedence therefore reads bottom to top. The default rules are disjoint so nothing behaved wrongly, but anyone adding a narrow rule would have put it in the wrong place. And the two matcher helpers took different request shapes, so combining a system matcher with a prompt matcher failed to compile. Both now read one shared shape, which is what allOf needs to infer. Claude-Session: https://claude.ai/code/session_01FSkESkJTRpt3Ye2cM7c5n5 --- e2e/support/default-script.ts | 17 ++- e2e/support/scripted-text-generation.test.ts | 140 +++++++++++++++++++ e2e/support/scripted-text-generation.ts | 14 +- 3 files changed, 165 insertions(+), 6 deletions(-) create mode 100644 e2e/support/scripted-text-generation.test.ts diff --git a/e2e/support/default-script.ts b/e2e/support/default-script.ts index 00c58eb..6865ad0 100644 --- a/e2e/support/default-script.ts +++ b/e2e/support/default-script.ts @@ -2,7 +2,12 @@ import { chapterStreamChunks } from '../fixtures/chapter-stream.js' import { QUIZ_FIXTURE } from '../fixtures/quiz.js' import { SKILL_CLASSIFICATION } from '../fixtures/skills.js' import { TOC_STREAM_CHUNKS } from '../fixtures/toc-stream.js' -import { promptIncludes, systemIncludes, type ScriptedTextGeneration } from './scripted-text-generation.js' +import { + promptIncludes, + systemIncludes, + type MatchableRequest, + type ScriptedTextGeneration, +} from './scripted-text-generation.js' /** * Wires every model call site the journeys traverse to its fixture. @@ -50,7 +55,7 @@ export function chapterNumberFrom(prompt: string | undefined): number { export const isChapterStream = systemIncludes(CHAPTER_SYSTEM_PHRASE) /** Matches the chapter-generation stream for one specific chapter. */ -export const isChapterStreamFor = (num: number) => (req: { system?: string; prompt?: string }): boolean => +export const isChapterStreamFor = (num: number) => (req: MatchableRequest): boolean => isChapterStream(req) && chapterNumberFrom(req.prompt) === num /** Matches the table-of-contents stream that `POST /api/books` opens. */ @@ -58,8 +63,12 @@ export const isTocStream = systemIncludes(TOC_SYSTEM_PHRASE) /** Installs the default rule set. Journeys may add rules on top to shadow any of these. */ export function applyDefaultScript(model: ScriptedTextGeneration): void { - // Registered oldest first so the resulting precedence reads top to bottom - // in this function, since each call unshifts onto the front of the list. + // Each call pushes onto the front of the list, so precedence reads BOTTOM + // to top in this function and a narrow rule must be registered after the + // broad rule it refines. The rules below are deliberately disjoint, each + // matching a phrase that appears in exactly one service's prompt, so the + // order among them does not matter and only a journey's own added rule + // ever shadows one. model.onStreamText({ name: 'chapter stream (start-book and generate-next-chapter)', match: isChapterStream, diff --git a/e2e/support/scripted-text-generation.test.ts b/e2e/support/scripted-text-generation.test.ts new file mode 100644 index 0000000..8664e2c --- /dev/null +++ b/e2e/support/scripted-text-generation.test.ts @@ -0,0 +1,140 @@ +import { describe, expect, it } from 'vitest' +import { z } from 'zod' +import { describeTextGenerationContract } from '@server/ports/text-generation.contract.js' +import { + allOf, + createScriptedTextGeneration, + promptIncludes, + schemaNamed, + systemIncludes, +} from './scripted-text-generation.js' + +/** + * The fidelity guarantee for the journey suite's model. + * + * The journeys are only worth anything if the object standing in for the AI + * provider behaves like the port says a TextGeneration behaves. So the same + * contract Phase 2 wrote for its own fake runs against this adapter, with no + * exemptions: cancellation mid-stream, schema validation of every generated + * object, request recording, and the tool conversation's step limit. + * + * The contract exercises the first-in first-out surface, because that is + * what it was written against. The rule surface this adapter adds on top is + * covered by the block below it, and the two together are what the journeys + * actually lean on. + * + * This file is a vitest test rather than a Playwright journey on purpose. It + * needs no browser and no server, so it belongs in the fast suite, which is + * why `vitest.config.ts` excludes only `e2e/**` + '/*.spec.ts' rather than + * all of `e2e/`. + */ +describeTextGenerationContract('scripted (e2e)', () => createScriptedTextGeneration()) + +describe('scripted TextGeneration rules', () => { + const model = { provider: 'anthropic', model: 'claude-test' } as const + + async function drain(stream: AsyncIterable): Promise { + let text = '' + for await (const chunk of stream) text += chunk + return text + } + + it('answers a streamText call from the rule whose matcher fits, regardless of call order', async () => { + const ai = createScriptedTextGeneration() + ai.onStreamText({ name: 'toc', match: systemIncludes('table of contents'), respond: { chunks: ['# A TOC'] } }) + ai.onStreamText({ name: 'chapter', match: systemIncludes('writing a chapter'), respond: { chunks: ['Chapter prose'] } }) + + // Deliberately the reverse of the order the rules were registered in. + expect(await drain(ai.streamText({ model, system: 'writing a chapter', prompt: 'x' }))).toBe('Chapter prose') + expect(await drain(ai.streamText({ model, system: 'a table of contents', prompt: 'y' }))).toBe('# A TOC') + }) + + it('lets a later rule shadow an earlier one, which is how a journey injects a failure', async () => { + const ai = createScriptedTextGeneration() + ai.onStreamText({ name: 'default chapter', match: systemIncludes('chapter'), respond: { chunks: ['prose'] } }) + ai.onStreamText({ name: 'chapter 2 fails', match: promptIncludes('Chapter 2'), respond: { throws: new Error('provider is overloaded') } }) + + expect(await drain(ai.streamText({ model, system: 'chapter', prompt: 'Chapter 1' }))).toBe('prose') + await expect(drain(ai.streamText({ model, system: 'chapter', prompt: 'Chapter 2' }))).rejects.toThrow('provider is overloaded') + }) + + it('throws the exact value a failure rule names, so a typed error class survives', async () => { + class RateLimited extends Error { + readonly retryAfterMs = 1000 + } + const ai = createScriptedTextGeneration() + ai.onStreamText({ name: 'rate limited', match: () => true, respond: { throws: new RateLimited('slow down') } }) + + await expect(drain(ai.streamText({ model, prompt: 'x' }))).rejects.toBeInstanceOf(RateLimited) + }) + + it('prefers a queued script over a matching rule, so a journey can force a one-off answer', async () => { + const ai = createScriptedTextGeneration() + ai.onStreamText({ name: 'default', match: () => true, respond: { chunks: ['from the rule'] } }) + ai.scriptStreamText(['from the queue']) + + expect(await drain(ai.streamText({ model, prompt: 'x' }))).toBe('from the queue') + expect(await drain(ai.streamText({ model, prompt: 'x' }))).toBe('from the rule') + }) + + it('parses a rule-supplied object through the caller\'s schema, so a drifted fixture fails loudly', async () => { + const ai = createScriptedTextGeneration() + ai.onGenerateObject({ name: 'quiz', match: promptIncludes('quiz'), respond: { value: { questions: 'not an array' } } }) + + await expect(ai.generateObject({ + model, + schema: z.object({ questions: z.array(z.string()) }), + prompt: 'write a quiz', + })).rejects.toThrow() + }) + + it('matches a generateObject rule on schemaName as well as on the prompt', async () => { + const ai = createScriptedTextGeneration() + ai.onGenerateObject({ name: 'next book', match: schemaNamed('nextBook'), respond: { value: { title: 'Tides' } } }) + + const result = await ai.generateObject({ + model, + schema: z.object({ title: z.string() }), + schemaName: 'nextBook', + prompt: 'anything at all', + }) + + expect(result.title).toBe('Tides') + }) + + it('combines matchers with allOf, which is how one call site is told apart from another', async () => { + const ai = createScriptedTextGeneration() + // Broad rule first, narrow rule second. Rules are consulted newest first, + // so a narrow rule must be registered AFTER the broad one it refines. + ai.onStreamText({ name: 'any chapter', match: systemIncludes('writing a chapter'), respond: { chunks: ['first'] } }) + ai.onStreamText({ + name: 'chapter 2 only', + match: allOf(systemIncludes('writing a chapter'), promptIncludes('This is Chapter 2 of')), + respond: { chunks: ['second'] }, + }) + + expect(await drain(ai.streamText({ model, system: 'writing a chapter', prompt: 'This is Chapter 1 of 3.' }))).toBe('first') + expect(await drain(ai.streamText({ model, system: 'writing a chapter', prompt: 'This is Chapter 2 of 3.' }))).toBe('second') + }) + + it('throws on an unmatched call, naming the rules on offer, so no journey can reach a live provider', async () => { + const ai = createScriptedTextGeneration() + ai.onStreamText({ name: 'table of contents', match: systemIncludes('table of contents'), respond: { chunks: ['x'] } }) + + expect(() => ai.streamText({ model, system: 'something else entirely', prompt: 'hello' })) + .toThrow(/no rule matched a streamText call.*table of contents/s) + await expect(ai.generateObject({ model, schema: z.object({}), prompt: 'hello' })) + .rejects.toThrow(/no rule matched a generateObject call/) + }) + + it('records every request whether it was answered by a rule or by the queue', async () => { + const ai = createScriptedTextGeneration() + ai.onStreamText({ name: 'any', match: () => true, respond: { chunks: ['ok'] } }) + ai.scriptStreamText(['queued']) + + await drain(ai.streamText({ model, prompt: 'first' })) + await drain(ai.streamText({ model, prompt: 'second' })) + + expect(ai.requests.streamText.map(req => req.prompt)).toEqual(['first', 'second']) + }) +}) diff --git a/e2e/support/scripted-text-generation.ts b/e2e/support/scripted-text-generation.ts index 1c98082..881689f 100644 --- a/e2e/support/scripted-text-generation.ts +++ b/e2e/support/scripted-text-generation.ts @@ -73,12 +73,22 @@ export interface ScriptedTextGeneration extends FakeTextGeneration { onToolConversation(rule: ToolRule): void } +/** + * The part of a request every matcher below reads. Deliberately one shape + * rather than one per method, so `allOf` can combine a system matcher with a + * prompt matcher without either being widened at the call site. + */ +export interface MatchableRequest { + system?: string + prompt?: string +} + /** True when the request's user prompt contains `needle`. */ -export const promptIncludes = (needle: string) => (req: { prompt?: string }): boolean => +export const promptIncludes = (needle: string) => (req: MatchableRequest): boolean => (req.prompt ?? '').includes(needle) /** True when the request's system prompt contains `needle`. */ -export const systemIncludes = (needle: string) => (req: { system?: string }): boolean => +export const systemIncludes = (needle: string) => (req: MatchableRequest): boolean => (req.system ?? '').includes(needle) /** True when the request carries this `schemaName`. Only one real call site sets it, so prefer prompt matching. */ From 5de26e6e246f27b14fd9c6e15f133c6d2437e2c1 Mon Sep 17 00:00:00 2001 From: Ross Miller Date: Tue, 21 Jul 2026 08:25:18 -0500 Subject: [PATCH 04/18] test(e2e): seed books on disk and read them in the reader Journey (a)'s second half, plus the two shared helpers the rest of the suite needs before it can fan out. Seeding writes a book straight to the data directory through the same filesystem repository production uses, never through raw file writes, so a fixture takes the same validation and atomic write path a real save takes and a book the app cannot read fails at the seed rather than three assertions later. Most journeys are not about creating a book, they are about exporting one or renaming one or listening to one, and making each of them drive the whole wizard first would mean one wizard regression failed six journeys that have nothing to do with the wizard. The reader page object hides the fact that a chapter is paged one section at a time, so a journey asks to finish a chapter rather than clicking Next in a loop of its own. It also offers the chapter tab strip as a separate way to move, because the end of chapter control opens the quiz and a journey about reading should not have to answer questions to turn a page. Three locator facts were found by running this rather than by reading the markup, and each is written down where it was needed. The section controls are named Previous section and Next section, so a loose match on Next would take the end of chapter control instead. The chapter tab strip holds a Chapter N button per chapter, so the article's own Chapter N label has to be scoped to the article. And a book card is a div with an onClick, carrying no role and no accessible name, so a journey reaches it by its title text. That last one is a real accessibility gap rather than a test inconvenience, since a keyboard user cannot reach a book card at all. It is reported to the architect rather than patched here, because this phase changes no production file. Claude-Session: https://claude.ai/code/session_01FSkESkJTRpt3Ye2cM7c5n5 --- e2e/journeys/reading.spec.ts | 38 +++++++++++++++ e2e/support/journeys/library.ts | 17 ++++++- e2e/support/journeys/reader.ts | 83 +++++++++++++++++++++++++++++++++ e2e/support/seed.ts | 78 +++++++++++++++++++++++++++++++ 4 files changed, 214 insertions(+), 2 deletions(-) create mode 100644 e2e/journeys/reading.spec.ts create mode 100644 e2e/support/journeys/reader.ts create mode 100644 e2e/support/seed.ts diff --git a/e2e/journeys/reading.spec.ts b/e2e/journeys/reading.spec.ts new file mode 100644 index 0000000..568155b --- /dev/null +++ b/e2e/journeys/reading.spec.ts @@ -0,0 +1,38 @@ +import { chapterMarker } from '../fixtures/chapter-stream.js' +import { TOC_BOOK_TITLE } from '../fixtures/toc-stream.js' +import { expect, test } from '../support/app.js' +import { library } from '../support/journeys/library.js' +import { reader } from '../support/journeys/reader.js' +import { seedBook } from '../support/seed.js' + +/** + * Journey (a), second half: a book on disk opens and reads. + * + * This starts from a seeded book rather than from the wizard, deliberately. + * Creating a book is `create-book.spec.ts`'s subject, and making every other + * journey drive the whole wizard first would mean one wizard regression + * failed the entire suite instead of the one journey that is actually about + * the wizard. Seeding also gives this journey a two-chapter book, which is + * what it needs to page from one chapter to the next. + * + * Navigation here goes through the reader's own chapter tab strip rather + * than through the end-of-chapter control, because that control opens the + * quiz. The chapter to quiz to feedback to next chapter loop is journey (b) + * and has its own file. + */ +test('opens a seeded book in the reader and pages from one chapter to the next', async ({ page, app }) => { + await seedBook(app.dataDir, { generatedUpTo: 2 }) + + await page.goto('/') + await library(page).waitForReady() + await library(page).openBook(TOC_BOOK_TITLE) + + await reader(page).waitForChapter() + await expect(reader(page).prose(chapterMarker(1))).toBeVisible() + expect(await reader(page).currentChapterNumber()).toBe(1) + + await reader(page).goToChapter(2) + + await expect(reader(page).prose(chapterMarker(2))).toBeVisible() + expect(await reader(page).currentChapterNumber()).toBe(2) +}) diff --git a/e2e/support/journeys/library.ts b/e2e/support/journeys/library.ts index b539180..161c955 100644 --- a/e2e/support/journeys/library.ts +++ b/e2e/support/journeys/library.ts @@ -27,9 +27,22 @@ export function library(page: Page) { await newBook().click() }, - /** The card for one book, located by its title. */ + /** + * The card for one book, located by its title. + * + * Located by text and not by role, because a book card is a `div` with an + * `onClick` and carries no role and no accessible name. That is a real + * accessibility gap, reported to the architect rather than patched here, + * since this phase changes no production file. Clicking the title works + * because the click bubbles to the card that handles it. + */ card(title: string) { - return page.getByRole('button', { name: new RegExp(title, 'i') }) + return page.getByText(title, { exact: true }).first() + }, + + /** Opens a book in the reader. */ + async openBook(title: string): Promise { + await this.card(title).click() }, } } diff --git a/e2e/support/journeys/reader.ts b/e2e/support/journeys/reader.ts new file mode 100644 index 0000000..f28ab60 --- /dev/null +++ b/e2e/support/journeys/reader.ts @@ -0,0 +1,83 @@ +import { expect, type Page } from '@playwright/test' + +/** + * The chapter reader. + * + * A chapter is shown one section at a time rather than as one scroll, and + * `client/lib/split-sections.ts` guarantees at least two sections per + * chapter, so "finish this chapter" means paging to the last section and + * then taking the control that appears there. `finishChapter` does that + * paging, which is why no journey should ever click Next in a loop itself. + * + * The section controls are named Previous and Next, and the control at the + * end of a chapter is named Next Chapter or Finish Book. Every locator below + * matches exactly, because a loose match on "Next" would also match "Next + * Chapter" and page one section too far. + */ +export function reader(page: Page) { + const nextSection = () => page.getByRole('button', { name: 'Next section', exact: true }) + const nextChapter = () => page.getByRole('button', { name: 'Next Chapter', exact: true }) + const finishBook = () => page.getByRole('button', { name: 'Finish Book', exact: true }) + + // Scoped to the article, because the chapter tab strip above it holds a + // "Chapter N" button per chapter and an unscoped match would find those + // too. The article's own label is the one that says what is being read. + const chapterLabel = () => page.getByRole('article').getByText(/^Chapter \d+$/) + + return { + /** Resolves once a chapter is on screen. */ + async waitForChapter(): Promise { + await expect(chapterLabel()).toBeVisible() + }, + + /** The "Chapter N" indicator's number, which is what the reader believes it is showing. */ + async currentChapterNumber(): Promise { + return Number((await chapterLabel().innerText()).replace(/\D+/g, '')) + }, + + /** Pages forward through the chapter's sections until the end-of-chapter control appears. */ + async readToEndOfChapter(): Promise { + // Bounded rather than while(true), so a UI that stops advancing fails + // as a clear assertion instead of hanging until the test timeout. + for (let step = 0; step < 20; step++) { + if (await nextChapter().isVisible() || await finishBook().isVisible()) return + await nextSection().click() + } + throw new Error('reader: paged 20 sections without reaching the end of the chapter') + }, + + /** + * Pages to the end of the chapter and takes the control that continues + * past it, which opens the quiz rather than the next chapter. The quiz + * and the feedback form stand between one chapter and the next by + * design, and `quiz.ts` drives them. + */ + async finishChapter(): Promise { + await this.readToEndOfChapter() + if (await finishBook().isVisible()) { + await finishBook().click() + return + } + await nextChapter().click() + }, + + /** + * Jumps straight to a chapter through the tab strip above the prose. + * This is the reader's own navigation and skips the quiz, so it is what a + * journey uses when the subject is reading rather than the adaptive loop. + */ + async goToChapter(num: number): Promise { + await page.getByRole('button', { name: `Chapter ${num}`, exact: true }).click() + }, + + /** Returns to the library. */ + async backToLibrary(): Promise { + await page.getByRole('button', { name: 'Back to library' }).click() + }, + + /** Chapter prose, addressed by the text it contains rather than by its container's class. */ + prose(text: string | RegExp) { + return page.getByText(text).first() + }, + } +} diff --git a/e2e/support/seed.ts b/e2e/support/seed.ts new file mode 100644 index 0000000..f528f00 --- /dev/null +++ b/e2e/support/seed.ts @@ -0,0 +1,78 @@ +import { randomUUID } from 'node:crypto' +import type { BookMeta, Quiz } from '@shared/domain.js' +import { createFsBookRepository } from '@server/adapters/fs-book-repository.js' +import { chapterMarkdown } from '../fixtures/chapter-stream.js' +import { QUIZ_FIXTURE } from '../fixtures/quiz.js' +import { TOC_BOOK_SUBTITLE, TOC_BOOK_TITLE, TOC_CHAPTERS } from '../fixtures/toc-stream.js' + +/** + * Puts a book on disk without going through the UI. + * + * Most journeys are not about creating a book, they are about exporting one, + * renaming one, or listening to one, and driving the whole creation wizard + * first would make each of those slower and would make an unrelated wizard + * regression fail six journeys at once. Seeding gets a journey to its + * subject in one call. + * + * Seeding goes through `createFsBookRepository`, never through raw `fs`, so a + * fixture takes the same validation and atomic-write path production takes. + * A seeded book that the app cannot read is then a failure here rather than a + * mystery three assertions later. This mirrors `seedBook` in + * `server/test/route-harness.ts`, which does the same for the inject suite. + */ + +export interface SeedBookOptions extends Partial { + /** How many chapters to write markdown for. Defaults to `generatedUpTo`. */ + chaptersOnDisk?: number + /** Whether to write a quiz alongside each chapter. Defaults to false. */ + withQuizzes?: boolean +} + +/** + * Writes one book into `dataDir` and returns its metadata. + * + * Defaults describe a book part-way through being read: three chapters in the + * table of contents, the first two written, status `reading`. + */ +export async function seedBook(dataDir: string, options: SeedBookOptions = {}): Promise { + const { chaptersOnDisk, withQuizzes = false, ...overrides } = options + const books = createFsBookRepository({ dataDir }) + + const id = overrides.id ?? `e2e-${randomUUID().slice(0, 8)}` + const now = new Date().toISOString() + const meta: BookMeta = { + id, + title: TOC_BOOK_TITLE, + subtitle: TOC_BOOK_SUBTITLE, + prompt: 'Tidal locking\n\nAim it at a physics graduate.', + status: 'reading', + totalChapters: TOC_CHAPTERS.length, + generatedUpTo: 2, + createdAt: now, + updatedAt: now, + tags: [], + audioGeneratedChapters: [], + ...overrides, + } + + await books.saveBook(meta) + await books.saveToc(id, { chapters: TOC_CHAPTERS.map(chapter => ({ ...chapter })) }) + + const written = chaptersOnDisk ?? meta.generatedUpTo + for (let num = 1; num <= written; num++) { + await books.saveChapter(id, num, chapterMarkdown(num)) + if (withQuizzes) await books.saveQuiz(id, num, QUIZ_FIXTURE as Quiz) + } + + return meta +} + +/** Reads a book's metadata back off disk, for assertions that a journey persisted something. */ +export async function readBook(dataDir: string, bookId: string): Promise { + return createFsBookRepository({ dataDir }).getBook(bookId) +} + +/** The repository pointed at a journey's temp data directory, for any read or write the helpers above do not cover. */ +export function bookRepository(dataDir: string) { + return createFsBookRepository({ dataDir }) +} From 41fc28600fe8f18ea9e3de9f27039cae89663bad Mon Sep 17 00:00:00 2001 From: Ross Miller Date: Tue, 21 Jul 2026 08:38:50 -0500 Subject: [PATCH 05/18] test(e2e): smoke the packaged app's rehydration, wire both CI jobs Journey (g) plus the two CI jobs and the suite's README, which is the last of the plan's tasks that does not belong to a single journey. The Electron project exists for exactly one thing the web project cannot reach. In the browser redux-persist writes to localStorage, and in the packaged app it writes through IPC to a file the main process owns, so the rehydration path is only observable with a real main process running. Two runs over the same seeded library, differing only in redux-state.json. With a remembered position the reader opens there, and without one it derives a position from the server's own progress record. Both are correct and getting them backwards is the bug a smoke test should catch. Writing it surfaced a redux-persist behaviour worth recording rather than working around. The default reconciler merges one level deep, so a persisted slice replaces that slice's initial state wholesale instead of being folded into it. A fixture that wrote a partial settings slice therefore deleted the provider configuration the client reads on its first render, and the renderer died before painting anything. The fixture builder now persists only the reading position and says why in a comment, since that is the slice this journey is actually about. The Electron project runs its two tests serially. Two Electron applications launching at once each start their own server and load the audiobook stack, and on a loaded machine that turned a six second test into a thirty four second one. Neither CI job builds the client as a separate step. Global setup builds it, so there is one code path and no way for a job to run the journeys against a bundle it did not build itself. The web job keeps the skip flag for the Electron binary and the Electron job deliberately drops it, which is the whole reason the two are separate jobs. Claude-Session: https://claude.ai/code/session_01FSkESkJTRpt3Ye2cM7c5n5 --- .github/workflows/ci.yml | 76 +++++++++++++ e2e/README.md | 96 +++++++++++++++++ e2e/fixtures/redux-state.ts | 47 ++++++++ e2e/journeys/electron-rehydration.spec.ts | 126 ++++++++++++++++++++++ playwright.config.ts | 5 + 5 files changed, 350 insertions(+) create mode 100644 e2e/README.md create mode 100644 e2e/fixtures/redux-state.ts create mode 100644 e2e/journeys/electron-rehydration.spec.ts diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5aa7557..fe5b8f2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -40,3 +40,79 @@ jobs: - run: pnpm lint - run: pnpm test + + # The end-to-end journey suite, in its own job so it runs alongside verify + # rather than behind it. `pnpm test` stays the fast one and never waits on a + # browser. See e2e/README.md for what the journeys cover and why the server + # is faked at its edges rather than mocked at its HTTP boundary. + e2e-web: + runs-on: macos-14 + env: + # The web project never launches Electron, so the binary is dead weight + # here. The e2e-electron job below deliberately omits this. + ELECTRON_SKIP_BINARY_DOWNLOAD: 1 + steps: + - uses: actions/checkout@v4 + + - uses: pnpm/action-setup@v4 + + - uses: actions/setup-node@v4 + with: + node-version: 24 + cache: pnpm + + - run: pnpm install --frozen-lockfile + + - name: Cache Playwright browsers + uses: actions/cache@v4 + with: + path: ~/Library/Caches/ms-playwright + key: ${{ runner.os }}-playwright-${{ hashFiles('pnpm-lock.yaml') }} + restore-keys: ${{ runner.os }}-playwright- + + - run: pnpm exec playwright install chromium + + # No separate build step. e2e/global-setup.ts runs `vite build` itself, + # so there is exactly one code path and no way for a job to run the + # journeys against a bundle it did not build. + - run: pnpm e2e --project=web + + - uses: actions/upload-artifact@v4 + if: failure() + with: + name: playwright-report-web + path: playwright-report/ + retention-days: 7 + + # Isolated from e2e-web because it launches a real GUI application and needs + # the Electron binary, which is the one thing every other job skips + # downloading. One `--project` flag keeps it out of the web suite. + e2e-electron: + runs-on: macos-14 + steps: + - uses: actions/checkout@v4 + + - uses: pnpm/action-setup@v4 + + - uses: actions/setup-node@v4 + with: + node-version: 24 + cache: pnpm + + - name: Cache Electron binary + uses: actions/cache@v4 + with: + path: ~/Library/Caches/electron + key: ${{ runner.os }}-electron-${{ hashFiles('pnpm-lock.yaml') }} + restore-keys: ${{ runner.os }}-electron- + + - run: pnpm install --frozen-lockfile + + - run: pnpm e2e --project=electron + + - uses: actions/upload-artifact@v4 + if: failure() + with: + name: playwright-report-electron + path: playwright-report/ + retention-days: 7 diff --git a/e2e/README.md b/e2e/README.md new file mode 100644 index 0000000..07ca0be --- /dev/null +++ b/e2e/README.md @@ -0,0 +1,96 @@ +# End-to-end journeys + +Up: [../README.md](../README.md) + +This suite drives the real browser UI against the real Fastify server, with the server's outermost adapters swapped for fakes. It is the answer to a fair question a cold reader asks about a hexagonal codebase, which is whether the ports are load bearing or decorative. Every journey here boots the production `buildServer` and hands it fakes through the same `overrides` argument Electron uses to hand it a diagram renderer, and it works. + +## Running it + +```bash +pnpm e2e # everything, both projects +pnpm e2e --project=web # the browser journeys +pnpm e2e --project=electron # the packaged-app smoke, needs the Electron binary +pnpm e2e create-book # one file +pnpm e2e --ui # Playwright's time-travel debugger +``` + +The run builds `dist/` first, every time, in `global-setup.ts`. That is one code path rather than two, so a journey can never test yesterday's client. Set `E2E_SKIP_BUILD=1` while iterating on a journey to skip it. + +Set `E2E_VERBOSE=1` to see the Fastify server's log lines. They are suppressed by default because each test boots its own instance and the JSON request log drowns the reporter. + +`pnpm test` is a different thing and stays fast. It is the vitest unit and contract suite and it never opens a browser. The one file under `e2e/` that vitest DOES own is `support/scripted-text-generation.test.ts`, because that test needs no browser. + +## What a journey gets + +```ts +import { expect, test } from '../support/app.js' + +test('...', async ({ page, model, app }) => { /* ... */ }) +``` + +| Fixture | What it is | +|---|---| +| `page` | A browser page already pointed at this test's own server instance, so `page.goto('/')` just works | +| `app` | `{ origin, dataDir, bookDir(id), fastify }`. `app.fastify.inject(...)` gives an API-level assertion without a second HTTP client | +| `model` | The scripted `TextGeneration` this test's server is wired to. Add rules before the action that triggers a call, read `model.requests` after | + +Plus the helpers: + +- `support/seed.ts` puts books on disk through the real filesystem repository. +- `support/journeys/*.ts` are page objects that expose intent, such as `wizard.approveToc()` and `reader.finishChapter()`. +- `fixtures/*.ts` are the content the scripted model returns, as TypeScript so `tsc` catches drift. + +## Adding a journey + +1. Add a `*.spec.ts` under `e2e/journeys/`, importing `test` and `expect` from `../support/app.js`, never from `@playwright/test`. The re-export is what carries the fixtures. +2. Get to your subject the cheap way. If the journey is about exporting a book, `seedBook(app.dataDir, ...)` rather than driving the whole creation wizard, so a wizard regression fails the wizard journey and not yours. +3. Address the UI the way a screen reader does. `getByRole` first, `getByText` second, and never a CSS class. +4. Assert with `await expect(locator)...`, which retries to a deadline. Never sleep. +5. If your journey needs a model call nobody has scripted yet, add a rule to `support/default-script.ts` matching a phrase lifted verbatim from the server prompt that produces it, and put the content in `fixtures/`. + +## Why fakes rather than mocks + +The server is faked at its ports, not mocked at its HTTP boundary. Everything between the browser and the AI provider is the real thing, meaning real routes, real services, real domain rules, real YAML on a real disk. Only the four edges that would cost money, need a network, or need a 90MB binary are swapped, and they are swapped through `buildServer(overrides)`, the same seam production already has. + +Mocking `fetch` in the browser instead would have tested the client against a fiction of the server. This tests it against the server. + +Zero live-provider traffic is a property of the design rather than a promise. The real key vault is never constructed, the fake one lives only in memory, and the scripted model has no SDK import and throws on any call nobody wrote a fixture for. There is no path from a journey to a provider even if someone wanted one. + +## Why the model answers by intent + +`support/scripted-text-generation.ts` matches on the shape of a request, not on the order calls arrive in. Phase 2's `createFakeTextGeneration` answers strictly first-in first-out, which is right for a unit test that knows exactly how many calls the code under test makes. A journey does not know that. Approving a table of contents fires a skill classification, a chapter stream, and a quiz, in an order the UI decides and may reorder tomorrow, and a fixture list coupled to that order is the largest available source of flake. + +Rules are consulted newest first, so a test shadows a default with one line, which is also how failure injection works: + +```ts +model.onStreamText({ + name: 'chapter 2 is rate limited', + match: isChapterStreamFor(2), + respond: { throws: new Error('rate limit exceeded') }, +}) +``` + +`throws` takes any value and rethrows it unchanged, so a typed error class survives all the way to the caller. + +The fidelity guarantee is `support/scripted-text-generation.test.ts`, which runs the port's own contract suite against this adapter with no exemptions, plus `req.schema.parse(value)` on every generated object so a drifted fixture fails as loudly as a malformed model response would. + +## Why no test ids + +There is not one `data-testid` in the client, and adding one would be the easy way out of every hard locator in this suite. The rule holds because addressing the UI by role and by name means the suite doubles as an accessibility net. Two real gaps were found by writing these journeys and reported rather than papered over, a book card that is a `div` with an `onClick` and therefore carries no role and no accessible name, and a context menu built from plain buttons with no menu semantics. + +The one sanctioned exception is a hidden ``, which has no role and no accessible name by construction. It is commented where it appears. + +## Flake policy + +`retries: 0`, and that is the most load-bearing line in `playwright.config.ts`. The server is in process, the adapters are fakes, and nothing touches a network, so there is no legitimate source of nondeterminism left. A retry could only convert a real bug into a green run. A journey that flakes twice gets quarantined with `test.fixme` and an issue, never a blanket retry. + +One real source of nondeterminism does exist inside the product and journeys have to respect it. `server/services/generate-quiz.ts` shuffles a quiz's options with `Math.random()` before saving, so a quiz option must be located by its text and never by its position. + +## Projects + +| Project | Selects | Runs where | +|---|---|---| +| `web` | everything not tagged `@electron` | the `e2e-web` CI job, Chromium, headless | +| `electron` | tests tagged `@electron` | the `e2e-electron` CI job, which is the only one that downloads the Electron binary | + +The Electron project exists for one thing that the web project cannot reach at all. In the browser redux-persist writes to localStorage, and in the packaged app it writes through IPC to a file the main process owns, so the rehydration path is only observable with a real main process. Fakes cannot be injected into a packaged main process, so that journey seeds its library rather than generating one. diff --git a/e2e/fixtures/redux-state.ts b/e2e/fixtures/redux-state.ts new file mode 100644 index 0000000..b8bb535 --- /dev/null +++ b/e2e/fixtures/redux-state.ts @@ -0,0 +1,47 @@ +/** + * Builds the `redux-state.json` file the Electron main process reads on + * behalf of redux-persist. + * + * The nesting is redux-persist's, not this project's, and it is worth + * spelling out because it is easy to get wrong from the outside. The file is + * an object keyed by storage key. Under `persist:tutor` sits a JSON STRING + * whose parsed value is an object with one entry per persisted slice, and + * each of THOSE values is itself a JSON string. Two levels of stringification, + * because redux-persist serialises each slice independently so that one + * corrupt slice cannot take the rest down with it. + * + * `client/store/persist.ts` sets `key: 'tutor'` and swaps its storage for + * Electron IPC when `window.electronAPI.storageGet` exists, and + * `electron/main.ts` answers that IPC out of this file. + * + * Only the reading position is written here, deliberately. redux-persist's + * default reconciler merges one level deep, meaning a persisted slice + * REPLACES that slice's initial state wholesale rather than being merged into + * it. Writing a partial `settings` slice therefore deletes the provider + * configuration the client reads on its first render, and the renderer dies + * with "Cannot read properties of undefined". Found the hard way. Persist a + * slice here only if the fixture supplies every field that slice's initial + * state declares. + */ + +export interface ReduxStateFixture { + /** Where the reader resumes. Omit it to make the reader fall back to the server-derived position. */ + position?: { bookId: string; chapter: number; section: number } +} + +/** The parsed contents of `redux-state.json` for the given fixture. */ +export function buildReduxState(fixture: ReduxStateFixture): Record { + const slices: Record = { + _persist: JSON.stringify({ version: -1, rehydrated: true }), + } + + if (fixture.position) { + const { bookId, chapter, section } = fixture.position + slices.readingProgress = JSON.stringify({ + positions: { [bookId]: { chapter, section, lastReadAt: new Date().toISOString() } }, + furthest: { [bookId]: chapter }, + }) + } + + return { 'persist:tutor': JSON.stringify(slices) } +} diff --git a/e2e/journeys/electron-rehydration.spec.ts b/e2e/journeys/electron-rehydration.spec.ts new file mode 100644 index 0000000..f0d661d --- /dev/null +++ b/e2e/journeys/electron-rehydration.spec.ts @@ -0,0 +1,126 @@ +import { mkdtempSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { fileURLToPath } from 'node:url' +import { _electron as electron, expect, test, type ElectronApplication, type Page } from '@playwright/test' +import { chapterMarker } from '../fixtures/chapter-stream.js' +import { buildReduxState, type ReduxStateFixture } from '../fixtures/redux-state.js' +import { TOC_BOOK_TITLE } from '../fixtures/toc-stream.js' +import { bookRepository, seedBook } from '../support/seed.js' + +/** + * Journey (g): the packaged app rehydrates where the reader left off. + * + * This is the only journey that runs the real Electron main process, and it + * is deliberately the smallest thing worth running there. Fakes cannot be + * injected into a packaged main process, so anything that would call a model + * is out of reach by construction, which is why this journey seeds its + * library rather than generating one. + * + * What IS only observable here is the rehydration path. In the browser + * redux-persist writes to localStorage, and in Electron it writes through IPC + * to a file the main process owns, so the file is a surface the web project + * cannot reach at all. Two runs over the same seeded library, differing only + * in `redux-state.json`. With a saved position the reader opens where the + * reader was, and without one it falls back to a position derived from the + * server's own progress record. Both are correct, and getting them backwards + * is exactly the bug a smoke test should catch. + * + * Tagged `@electron` so the `electron` Playwright project selects it and the + * `web` project excludes it with one flag, which is what keeps the Electron + * binary out of every other CI job. + */ + +const REPO_ROOT = fileURLToPath(new URL('../..', import.meta.url)) + +/** How many chapters the seeded book records as read. The fallback position is derived from this. */ +const CHAPTERS_READ = 3 + +/** Electron has a real window to open and a real server to start, so it gets longer than a browser page load. */ +const BOOT_TIMEOUT_MS = 40_000 + +interface LaunchedApp { + app: ElectronApplication + page: Page + dataDir: string + bookId: string +} + +/** + * Seeds a data directory with one readable book and a `redux-state.json` + * built from `fixture`, then launches the packaged app against it. + */ +async function launchWithState(fixture: (bookId: string) => ReduxStateFixture): Promise { + const dataDir = mkdtempSync(join(tmpdir(), 'tutor-e2e-electron-')) + const book = await seedBook(dataDir, { + status: 'reading', + totalChapters: CHAPTERS_READ, + generatedUpTo: CHAPTERS_READ, + chaptersOnDisk: CHAPTERS_READ, + }) + + // The server-derived fallback reads the progress record rather than + // generatedUpTo, so the record has to exist for the second run to have + // anything to fall back to. + const books = bookRepository(dataDir) + for (let num = 1; num <= CHAPTERS_READ; num++) { + await books.saveChapterProgress(book.id, num, { scroll: 1, completed: true, completedAt: new Date().toISOString() }) + } + + writeFileSync(join(dataDir, 'redux-state.json'), JSON.stringify(buildReduxState(fixture(book.id)))) + + const app = await electron.launch({ + args: [REPO_ROOT], + // A temp data directory, so the packaged app never opens the reader's + // real library, and no provider key exists in it, so nothing this app + // does can reach a live provider. + env: { ...process.env, TUTOR_DATA_DIR: dataDir }, + }) + + return { app, page: await app.firstWindow(), dataDir, bookId: book.id } +} + +/** The reader's own "Chapter N" label, scoped past the chapter tab strip that repeats it. */ +function chapterLabel(page: Page, num: number) { + return page.getByRole('article').getByText(`Chapter ${num}`, { exact: true }) +} + +async function openTheSeededBook(page: Page): Promise { + await expect(page.getByRole('banner').getByRole('button', { name: 'New Book' })).toBeVisible({ timeout: BOOT_TIMEOUT_MS }) + await page.getByText(TOC_BOOK_TITLE, { exact: true }).first().click() +} + +test.describe('@electron rehydration', () => { + test('opens the reader at the position redux-state.json remembers', async () => { + const { app, page, dataDir, bookId } = await launchWithState(id => ({ + position: { bookId: id, chapter: 0, section: 0 }, + })) + + try { + await openTheSeededBook(page) + // Chapter 0 was remembered, so the reader must land there and must NOT + // take the server-derived fallback, which would be chapter 3. + await expect(chapterLabel(page, 1)).toBeVisible({ timeout: BOOT_TIMEOUT_MS }) + await expect(page.getByText(chapterMarker(1)).first()).toBeVisible() + expect(bookId).toBeTruthy() + } finally { + await app.close() + rmSync(dataDir, { recursive: true, force: true }) + } + }) + + test('falls back to the server-derived position when no saved position exists', async () => { + const { app, page, dataDir } = await launchWithState(() => ({})) + + try { + await openTheSeededBook(page) + // Three chapters are recorded as read and nothing was remembered, so + // `ReaderPage.tsx` derives the position from `chaptersRead - 1`. + await expect(chapterLabel(page, CHAPTERS_READ)).toBeVisible({ timeout: BOOT_TIMEOUT_MS }) + await expect(page.getByText(chapterMarker(CHAPTERS_READ)).first()).toBeVisible() + } finally { + await app.close() + rmSync(dataDir, { recursive: true, force: true }) + } + }) +}) diff --git a/playwright.config.ts b/playwright.config.ts index 1902c0e..83d95a7 100644 --- a/playwright.config.ts +++ b/playwright.config.ts @@ -54,6 +54,11 @@ export default defineConfig({ // and gets its own CI job. One flag excludes it everywhere else. name: 'electron', grep: /@electron/, + // Serial. Two Electron apps launching at once each start their own + // Fastify server and load the audiobook stack, and on a loaded machine + // that turned a 6 second test into a 34 second one. There are two tests + // here and there is nothing to gain from overlapping them. + fullyParallel: false, }, ], }) From 75f13470f17f875e1cf3fca9b144cf03f24e9d86 Mon Sep 17 00:00:00 2001 From: Ross Miller Date: Tue, 21 Jul 2026 08:36:35 -0500 Subject: [PATCH 06/18] test(e2e): complete journey a, revision through chapter 1 approval The first half of journey a proved the wizard and the toc stream, this half proves the rest of the acceptance, that editing the table of contents actually persists, that approving it calls the real start route, and that chapter 1 lands on disk with generatedUpTo at 1. A revised toc fixture with visibly different chapter titles gives the revision something to prove, since streaming the same titles back would pass even if persistence were broken. default-script.ts now points the revise-toc rule at that fixture instead of at the original, so revising demonstrably changes the table of contents rather than replaying it. The revise panel's submit control has no shared page object yet, since wizard.ts is owned by another concurrent phase-6 agent, so this adds a small helper local to the spec file rather than editing that file. Claude-Session: https://claude.ai/code/session_01FSkESkJTRpt3Ye2cM7c5n5 --- e2e/fixtures/toc-stream.ts | 23 ++++++++++ e2e/journeys/create-book.spec.ts | 74 +++++++++++++++++++++++++++++++- e2e/support/default-script.ts | 4 +- 3 files changed, 98 insertions(+), 3 deletions(-) diff --git a/e2e/fixtures/toc-stream.ts b/e2e/fixtures/toc-stream.ts index dfe23e7..cc8a6f5 100644 --- a/e2e/fixtures/toc-stream.ts +++ b/e2e/fixtures/toc-stream.ts @@ -41,3 +41,26 @@ export const TOC_STREAM_CHUNKS: string[] = [ /** The whole fixture as one string, for assertions that want the parsed result rather than the stream. */ export const TOC_MARKDOWN = TOC_STREAM_CHUNKS.join('') + +/** + * A revised table of contents, with visibly different chapter titles than + * `TOC_CHAPTERS` but the same descriptions, title, and subtitle, standing in + * for a reader who asked to simplify the chapter titles without touching + * their content. `default-script.ts` wires the `revise-toc` stream to this + * fixture rather than to `TOC_STREAM_CHUNKS`, so a revision journey can + * assert that revising actually changed something instead of the stream + * just replaying the original TOC back. + */ +export const TOC_REVISED_CHAPTERS: TocFixtureChapter[] = [ + { title: 'Why It Keeps Turning', description: 'Why a spinning body keeps spinning until something takes the spin away.' }, + { title: 'The Bulge Explained', description: 'The tidal force that deforms a body and drags its own rotation backwards.' }, + { title: 'Braking To Zero', description: 'Working out how long a world takes to fall into a locked rotation.' }, +] + +/** `TOC_REVISED_CHAPTERS`, streamed the same way `TOC_STREAM_CHUNKS` streams `TOC_CHAPTERS`. */ +export const TOC_REVISED_STREAM_CHUNKS: string[] = [ + `# ${TOC_BOOK_TITLE}\n`, + `*${TOC_BOOK_SUBTITLE}*\n`, + '\n', + ...TOC_REVISED_CHAPTERS.map((chapter, index) => `${index + 1}. **${chapter.title}** — ${chapter.description}\n`), +] diff --git a/e2e/journeys/create-book.spec.ts b/e2e/journeys/create-book.spec.ts index fc0b467..608959c 100644 --- a/e2e/journeys/create-book.spec.ts +++ b/e2e/journeys/create-book.spec.ts @@ -1,7 +1,26 @@ -import { TOC_BOOK_TITLE, TOC_CHAPTERS } from '../fixtures/toc-stream.js' +import type { Page } from '@playwright/test' +import { chapterMarker } from '../fixtures/chapter-stream.js' +import { TOC_BOOK_TITLE, TOC_CHAPTERS, TOC_REVISED_CHAPTERS } from '../fixtures/toc-stream.js' import { expect, test } from '../support/app.js' import { library } from '../support/journeys/library.js' import { wizard } from '../support/journeys/wizard.js' +import { bookRepository, readBook } from '../support/seed.js' + +/** + * The revise-toc panel's own input and send control, addressed locally + * rather than added to `wizard()` because `e2e/support/journeys/wizard.ts` + * is owned by another concurrent phase-6 agent. Promote this into that file + * if a later journey needs it too. + */ +function reviseTocPanel(page: Page) { + return { + /** Types a revision instruction and submits it. */ + async submit(feedback: string): Promise { + await page.getByPlaceholder('Change chapter 1 to be simpler...').fill(feedback) + await page.getByRole('button', { name: 'Send revision' }).click() + }, + } +} /** * Journey (a), first half: the walking skeleton. @@ -45,3 +64,56 @@ test('creates a book and streams the table of contents into the wizard', async ( expect(library_[0].title).toBe(TOC_BOOK_TITLE) expect(library_[0].status).toBe('toc_review') }) + +/** + * Journey (a), second half: revise the table of contents, approve it, and + * land on chapter 1. + * + * Revising first is what makes the approval meaningful to test here — it + * proves the book that gets started is the one on disk after the edit, not a + * stale in-memory copy of the original streamed TOC. `default-script.ts` + * points the revise-toc stream at `TOC_REVISED_CHAPTERS`, a fixture with + * visibly different titles than `TOC_CHAPTERS`, so asserting those titles + * render and persist actually exercises the revision rather than coincidence. + */ +test('revises the table of contents, persists it, and generates chapter 1 on approval', async ({ page, model, app }) => { + await page.goto('/') + await library(page).waitForReady() + await library(page).openWizard() + await wizard(page).submit({ topic: 'Tidal locking', details: 'Aim it at a physics graduate.' }) + await wizard(page).waitForTocApproval() + + await wizard(page).openTocEditor() + await reviseTocPanel(page).submit('Make every chapter title a single short, punchy phrase.') + + // The revised titles rendering is the stream having reassembled the new + // fixture, the same way TOC_CHAPTERS rendering proves it in the first test. + for (const chapter of TOC_REVISED_CHAPTERS) { + await expect(wizard(page).tocEntry(chapter.title)).toBeVisible() + } + + const booksRes = await app.fastify.inject({ method: 'GET', url: '/api/books' }) + const id = (booksRes.json() as Array<{ id: string }>)[0].id + + // Persistence, checked on disk rather than on screen: the revision wrote + // through to toc.yml, not just into the streaming buffer the page renders. + const toc = await bookRepository(app.dataDir).getToc(id) + expect(toc.chapters.map(chapter => chapter.title)).toEqual(TOC_REVISED_CHAPTERS.map(chapter => chapter.title)) + + await wizard(page).approveToc() + + await expect(page.getByText(chapterMarker(1)).first()).toBeVisible() + + const book = await readBook(app.dataDir, id) + expect(book.generatedUpTo).toBe(1) + expect(book.status).toBe('reading') + + // And the model path /start actually drives: a skill classification call + // and a chapter-1 stream, not just the revise-toc stream from earlier. + expect(model.requests.generateObject.some( + req => (req.prompt ?? '').includes('classifying the learning content of a book'), + )).toBe(true) + expect(model.requests.streamText.some( + req => (req.prompt ?? '').includes('This is Chapter 1 of'), + )).toBe(true) +}) diff --git a/e2e/support/default-script.ts b/e2e/support/default-script.ts index 6865ad0..8d7abab 100644 --- a/e2e/support/default-script.ts +++ b/e2e/support/default-script.ts @@ -1,7 +1,7 @@ import { chapterStreamChunks } from '../fixtures/chapter-stream.js' import { QUIZ_FIXTURE } from '../fixtures/quiz.js' import { SKILL_CLASSIFICATION } from '../fixtures/skills.js' -import { TOC_STREAM_CHUNKS } from '../fixtures/toc-stream.js' +import { TOC_REVISED_STREAM_CHUNKS, TOC_STREAM_CHUNKS } from '../fixtures/toc-stream.js' import { promptIncludes, systemIncludes, @@ -78,7 +78,7 @@ export function applyDefaultScript(model: ScriptedTextGeneration): void { model.onStreamText({ name: 'table of contents revision (revise-toc)', match: systemIncludes(REVISE_SYSTEM_PHRASE), - respond: { chunks: TOC_STREAM_CHUNKS }, + respond: { chunks: TOC_REVISED_STREAM_CHUNKS }, }) model.onStreamText({ From 67caa08032e0bdd772821112597c2ffc25ba0712 Mon Sep 17 00:00:00 2001 From: Ross Miller Date: Tue, 21 Jul 2026 08:35:24 -0500 Subject: [PATCH 07/18] test(e2e): cover the epub export journey from library to downloaded file This seeds a finished three chapter book, right clicks its card, and clicks Export EPUB, then waits for the browser download event the client's auto download anchor fires once the background task completes. The export port is deliberately not faked here. Every other journey stubs the AI port with a scripted model, but an epub export makes no AI call at all, it only converts markdown to HTML and hands it to the real epub gen memory library. Faking that library would only prove the test calls a fake correctly, running it for real is what proves the double default handling in server/adapters/epub-gen-export.ts still works end to end, since that handling exists specifically for the Electron production build. The test asserts the downloaded file is a non empty real zip by checking its PK header, that book.epub landed on disk under the book's own directory before the download fired, and that the background task reached status done through GET /api/tasks. Claude-Session: https://claude.ai/code/session_01FSkESkJTRpt3Ye2cM7c5n5 --- e2e/journeys/epub-export.spec.ts | 82 ++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 e2e/journeys/epub-export.spec.ts diff --git a/e2e/journeys/epub-export.spec.ts b/e2e/journeys/epub-export.spec.ts new file mode 100644 index 0000000..576f4b6 --- /dev/null +++ b/e2e/journeys/epub-export.spec.ts @@ -0,0 +1,82 @@ +import { readFile } from 'node:fs/promises' +import { join } from 'node:path' +import { expect, test } from '../support/app.js' +import { library } from '../support/journeys/library.js' +import { seedBook } from '../support/seed.js' + +/** + * Journey (d): exporting a finished book to EPUB from the library. + * + * The book is seeded complete, three chapters written against a + * three-chapter table of contents, so the context menu's Export EPUB entry + * is enabled immediately. Getting a book to that state through the UI is + * the generation journeys' subject, not this one's. + * + * The export itself is deliberately not faked. Every other journey stubs its + * AI port with the scripted model, but an EPUB export makes no AI call at + * all, only markdown-to-HTML conversion and the real epub-gen-memory library + * assembling a zip. Faking that library would only prove this test calls a + * fake correctly. Running it for real is what proves the double-default + * handling in server/adapters/epub-gen-export.ts, which exists specifically + * for the Electron production build, still works end to end. + * + * The flow this journey rides: the click starts a background task (see + * createExportEpub in server/services/export-epub.ts), the task's + * `task_done` SSE event tells useBackgroundTaskEffects.ts to fetch the EPUB + * and click a hidden anchor carrying a `download` attribute, and that + * anchor click is what Playwright surfaces as a `download` event on the + * page. + */ +test('exports a finished book to EPUB from the library context menu', async ({ page, app }) => { + const book = await seedBook(app.dataDir, { generatedUpTo: 3, totalChapters: 3 }) + + await page.goto('/') + await library(page).waitForReady() + + // The card is a plain div with an onContextMenu handler and no role or + // accessible name (see card()'s doc comment in library.ts), so + // right-clicking its title is how a reader opens this menu too. + await library(page).card(book.title).click({ button: 'right' }) + + // The context menu is a plain div of buttons, not an ARIA menu, confirmed + // against test-results/*/error-context.md on a real run: "Export EPUB"'s + // accessible role is "button", never "menuitem". + const exportItem = page.getByRole('button', { name: 'Export EPUB' }) + await expect(exportItem).toBeVisible() + + // The download only fires once the background task finishes, well after + // this click resolves, but Promise.all still guards against a + // pathological fast path firing it before the listener attaches. + const [download] = await Promise.all([ + page.waitForEvent('download'), + exportItem.click(), + ]) + + expect(download.suggestedFilename()).toMatch(/\.epub$/) + + const savedPath = join(app.dataDir, 'downloaded-book.epub') + await download.saveAs(savedPath) + const savedBytes = await readFile(savedPath) + expect(savedBytes.length).toBeGreaterThan(0) + // A real EPUB is a zip archive, so its first two bytes are the local file + // header signature "PK". A stubbed or truncated export would still pass a + // bare non-empty check, so this is the one that actually proves a real + // archive came back. + expect(savedBytes.subarray(0, 2).toString('latin1')).toBe('PK') + + // createExportEpub writes book.epub to disk (artifactStore.writeEpub) + // before it ever calls backgroundTasks.succeed(), and succeed() is what + // the client waits on before it fetches the file for download, so this + // on-disk copy is guaranteed to already exist by the time the download + // above resolved. + const onDiskBytes = await readFile(join(app.bookDir(book.id), 'book.epub')) + expect(onDiskBytes.length).toBeGreaterThan(0) + + // And the background task the export ran as reached done, the same + // status the client's own auto-download waited on. + const tasksResponse = await app.fastify.inject({ method: 'GET', url: '/api/tasks' }) + expect(tasksResponse.statusCode).toBe(200) + const task = (tasksResponse.json() as Array<{ bookId: string; type: string; status: string }>) + .find(t => t.bookId === book.id && t.type === 'generate-epub') + expect(task?.status).toBe('done') +}) From e7d7077c08b5d9e7575e723bc6b155e9b4d84b1e Mon Sep 17 00:00:00 2001 From: Ross Miller Date: Tue, 21 Jul 2026 08:38:20 -0500 Subject: [PATCH 08/18] test(e2e): cover library rename, tagging, search, and delete This adds journey e, library crud and search, as its own spec file owned by this task. Each operation gets its own test so a failure names the specific operation that broke, rather than an assertion partway through one long scenario. Every mutation asserts on screen and on disk. A card can show a change that a failed request never actually persisted, and the server can also persist a change the grid fails to re render because fetchBooks did not run. Checking only the screen would miss the first case, checking only disk would miss the second, so rename, tag add, and delete each read the book back off disk through readBook or bookRepository after the on screen assertion passes. Delete opens an in app confirmation dialog that requires typing delete, not a native window.confirm, so no dialog auto accept handler was needed. Search reaches the toolbar's search box through the documented Cmd or Ctrl plus F shortcut rather than clicking its own toggle button, because that button is icon only with no aria label, so no role or text locator can name it. The Rename dialog's Title field has the same kind of gap, its label is plain text with no htmlFor or id pairing to the input, so the test reaches it by position among the dialog's textboxes instead of by label. Claude-Session: https://claude.ai/code/session_01FSkESkJTRpt3Ye2cM7c5n5 --- e2e/journeys/library-crud.spec.ts | 113 ++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 e2e/journeys/library-crud.spec.ts diff --git a/e2e/journeys/library-crud.spec.ts b/e2e/journeys/library-crud.spec.ts new file mode 100644 index 0000000..45d8537 --- /dev/null +++ b/e2e/journeys/library-crud.spec.ts @@ -0,0 +1,113 @@ +import { expect, test } from '../support/app.js' +import { library } from '../support/journeys/library.js' +import { bookRepository, readBook, seedBook } from '../support/seed.js' + +/** + * Journey (e): library CRUD and search. + * + * Every mutation here asserts twice, once on what the reader sees and once + * on the YAML the fixture wrote. The two can drift apart in either + * direction: a card can show a change that a failed request never actually + * persisted, or the server can write a change the grid fails to re-render + * because `fetchBooks()` didn't run. Checking only the screen would miss the + * first, checking only disk would miss the second, so both are asserted for + * every operation below. + * + * Split into one `test` per operation rather than one long scenario, so a + * failure names the operation that broke rather than an assertion three + * steps into an unrelated one. + */ + +test('rename persists across reload', async ({ page, app }) => { + const book = await seedBook(app.dataDir, { title: 'Rename Me' }) + + await page.goto('/') + await library(page).waitForReady() + + await library(page).card('Rename Me').click({ button: 'right' }) + await page.getByRole('button', { name: 'Rename' }).click() + + const dialog = page.getByRole('dialog', { name: 'Rename Book' }) + // The Title field's