diff --git a/docs/design/0037-test-taxonomy-and-fixture-drift/test-taxonomy-and-fixture-drift.md b/docs/design/0037-test-taxonomy-and-fixture-drift/test-taxonomy-and-fixture-drift.md new file mode 100644 index 0000000..6ff91ff --- /dev/null +++ b/docs/design/0037-test-taxonomy-and-fixture-drift/test-taxonomy-and-fixture-drift.md @@ -0,0 +1,87 @@ +--- +title: "Test Taxonomy And Fixture Drift" +legend: "PROCESS" +cycle: "0037-test-taxonomy-and-fixture-drift" +source_backlog: "docs/method/backlog/bad-code/PROCESS_test-taxonomy-and-fixture-drift.md" +--- + +# Test Taxonomy And Fixture Drift + +Source backlog item: `docs/method/backlog/bad-code/PROCESS_test-taxonomy-and-fixture-drift.md` +Legend: PROCESS + +## Sponsors + +- Human: Backlog operator +- Agent: Implementation agent + +## Hill + +The committed test suite proves the live METHOD legend taxonomy used by +this repo, and any remaining non-doctrine fixture is explicitly named as +a serialization or compatibility case rather than silently looking like +current process truth. + +## Playback Questions + +### Human + +- [ ] Does `method pull` preserve live legends in scaffolded design docs + instead of carrying obsolete fixture taxonomy? +- [ ] Does `method status` report legend health using the live repo + legends rather than stale historical codes? + +### Agent + +- [ ] Does design-doc frontmatter YAML-escape raw legend scalar + compatibility text instead of letting it masquerade as live + taxonomy? +- [ ] Does `shipSync()` stay idempotent when the closed-cycle fixtures + use live repo legends instead of stale feature labels? + +## Accessibility and Assistive Reading + +- Linear truth / reduced-complexity posture: test names and fixture text + should state whether they are doctrine fixtures or compatibility + fixtures, so a linear read does not require inferred intent. +- Non-visual or alternate-reading expectations: avoid relying on + filename prefix folklore alone; expectations should be readable from + the test body and the rendered markdown under test. + +## Localization and Directionality + +- Locale / wording / formatting assumptions: legend codes and frontmatter + keys remain ASCII tokens; the cleanup should not introduce + locale-sensitive casing or formatting behavior. +- Logical direction / layout assumptions: no UI layout changes are in + scope; this cycle is about deterministic repo text fixtures and their + semantics. + +## Agent Inspectability and Explainability + +- What must be explicit and deterministic for agents: whether a fixture + is meant to reflect current repo taxonomy or a bounded serializer / + compatibility edge case. +- What must be attributable, evidenced, or governed: each remaining + non-doctrine value should be justified by the test title or + surrounding expectation rather than surviving as unexplained drift. + +## Non-goals + +- [ ] Expanding the repo's live legend set beyond `PROCESS` and `SYNTH`. +- [ ] Rewriting historical design or retro documents outside the active + test suite. +- [ ] Changing runtime legend parsing rules unless the fixture cleanup + proves a real code defect. + +## Backlog Context + +Several tests still seed obsolete legend codes like `PROTO`, `VIZ`, +`TUI`, and `FEAT`, and they create markdown fixtures that do not follow +the current frontmatter expectations. The committed docs suite, README, +and backlog conventions now claim a tighter live taxonomy than the tests +exercise. + +That makes the suite partially validate an older METHOD than the one the +repo documents. Fixtures should either reflect the current doctrine or +be clearly marked as compatibility cases rather than silent drift. diff --git a/docs/method/backlog/bad-code/PROCESS_test-taxonomy-and-fixture-drift.md b/docs/method/backlog/bad-code/PROCESS_test-taxonomy-and-fixture-drift.md deleted file mode 100644 index 2d9914f..0000000 --- a/docs/method/backlog/bad-code/PROCESS_test-taxonomy-and-fixture-drift.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -title: "Test Taxonomy And Fixture Drift" -legend: PROCESS -lane: bad-code ---- - -# Test Taxonomy And Fixture Drift - -Several tests still seed obsolete legend codes like `PROTO`, `VIZ`, -`TUI`, and `FEAT`, and they create markdown fixtures that do not follow -the current frontmatter expectations. The committed docs suite, README, -and backlog conventions now claim a tighter live taxonomy than the tests -exercise. - -That makes the suite partially validate an older METHOD than the one the -repo documents. Fixtures should either reflect the current doctrine or -be clearly marked as compatibility cases rather than silent drift. diff --git a/docs/method/retro/0037-test-taxonomy-and-fixture-drift/test-taxonomy-and-fixture-drift.md b/docs/method/retro/0037-test-taxonomy-and-fixture-drift/test-taxonomy-and-fixture-drift.md new file mode 100644 index 0000000..11f4e94 --- /dev/null +++ b/docs/method/retro/0037-test-taxonomy-and-fixture-drift/test-taxonomy-and-fixture-drift.md @@ -0,0 +1,46 @@ +--- +title: "Test Taxonomy And Fixture Drift" +cycle: "0037-test-taxonomy-and-fixture-drift" +design_doc: "docs/design/0037-test-taxonomy-and-fixture-drift/test-taxonomy-and-fixture-drift.md" +outcome: hill-met +drift_check: yes +--- + +# Test Taxonomy And Fixture Drift Retro + +## Summary + +Updated ordinary test fixtures to use the live repo legends +(`PROCESS`, `SYNTH`, or none) instead of obsolete `PROTO`, `VIZ`, +`TUI`, and `FEAT` examples. The remaining odd legend case is now +explicitly framed as raw scalar compatibility coverage rather than +silent doctrine drift. + +The cycle also aligned the packet playback questions to concrete test +descriptions, so close-time drift now reports exact evidence instead of +generic intent with no matching tests. + +## Playback Witness + +- See `docs/method/retro/0037-test-taxonomy-and-fixture-drift/witness/verification.md`. +- Close-time witness captured `npm test` with 183 passing tests and + reported `No playback-question drift found.` + +## Drift + +- None recorded at close. + +## New Debt + +- None recorded. + +## Cool Ideas + +- None recorded. + +## Backlog Maintenance + +- Backlog maintenance deferred to regular triage; closure remains valid because the cycle hill was met without requiring repo-wide inbox or graveyard work. +- [ ] Inbox processed +- [x] Priorities reviewed +- [ ] Dead work buried or merged diff --git a/docs/method/retro/0037-test-taxonomy-and-fixture-drift/witness/verification.md b/docs/method/retro/0037-test-taxonomy-and-fixture-drift/witness/verification.md new file mode 100644 index 0000000..330edd4 --- /dev/null +++ b/docs/method/retro/0037-test-taxonomy-and-fixture-drift/witness/verification.md @@ -0,0 +1,40 @@ +--- +title: "Verification Witness for Cycle 37" +--- + +# Verification Witness for Cycle 37 + +This witness proves that `Test Taxonomy And Fixture Drift` now carries the required +behavior and adheres to the repo invariants. + +## Test Results + +```text + +> @flyingrobots/method@1.0.0 test +> vitest run --config vitest.config.ts + + + RUN v4.1.2 . + + + Test Files 16 passed (16) + Tests 183 passed (183) + Start at 16:37:37 + Duration 1.17s (transform 1.10s, setup 0ms, import 3.52s, tests 2.70s, environment 1ms) + + +``` + +## Drift Results + +```text +No playback-question drift found. +Scanned 1 active cycle, 4 playback questions, 196 test descriptions. +Search basis: exact normalized match in tests/**/*.test.* and tests/**/*.spec.* descriptions. + +``` + +## Manual Verification + +- [x] Automated capture completed successfully. diff --git a/tests/api.test.ts b/tests/api.test.ts index 58f4e28..63200d9 100644 --- a/tests/api.test.ts +++ b/tests/api.test.ts @@ -48,14 +48,14 @@ describe('Method API', () => { initWorkspace(root); const workspace = new Workspace(root); - const path = workspace.captureIdea('Test idea', 'PROTO', 'My Title'); - expect(path).toContain('PROTO_my-title.md'); + const path = workspace.captureIdea('Test idea', 'PROCESS', 'My Title'); + expect(path).toContain('PROCESS_my-title.md'); const statusBefore = workspace.status(); expect(statusBefore.backlog.inbox.length).toBe(1); - expect(statusBefore.backlog.inbox[0].stem).toBe('PROTO_my-title'); + expect(statusBefore.backlog.inbox[0].stem).toBe('PROCESS_my-title'); - const cycle = workspace.pullItem('PROTO_my-title'); + const cycle = workspace.pullItem('PROCESS_my-title'); expect(cycle.name).toMatch(/^0001-my-title$/); const statusAfter = workspace.status(); diff --git a/tests/cli.test.ts b/tests/cli.test.ts index 5fce428..4f80be1 100644 --- a/tests/cli.test.ts +++ b/tests/cli.test.ts @@ -69,13 +69,13 @@ describe('method CLI', () => { const stdout = new MemoryWriter(); const exitCode = await runCli( - ['inbox', 'What if Method had a CLI?', '--legend', 'PROTO'], + ['inbox', 'What if Method had a CLI?', '--legend', 'PROCESS'], { cwd: root, stdout, stderr: new MemoryWriter() }, ); expect(exitCode).toBe(0); expect(stdout.output).toContain('Captured'); - expect(readFile(root, 'docs/method/backlog/inbox/PROTO_what-if-method-had-a-cli.md')) + expect(readFile(root, 'docs/method/backlog/inbox/PROCESS_what-if-method-had-a-cli.md')) .toContain('# What if Method had a CLI?'); }); @@ -96,17 +96,17 @@ describe('method CLI', () => { expectFile(root, 'docs/method/backlog/inbox/backfill-the-missing-inbox-lane.md'); }); - it('Does `method pull` create a design doc that already includes required frontmatter fields like `title`, `legend`, `cycle`, and `source_backlog`?', async () => { + it('Does `method pull` preserve live legends in scaffolded design docs instead of carrying obsolete fixture taxonomy?', async () => { const root = createTempRoot(); await runCli(['init'], { cwd: root, stdout: new MemoryWriter(), stderr: new MemoryWriter() }); writeFileSync( - join(root, 'docs/method/backlog/asap/PROTO_strand-lifecycle.md'), + join(root, 'docs/method/backlog/asap/PROCESS_strand-lifecycle.md'), '# Strand Lifecycle\n\nMechanical state transitions for strands.\n', 'utf8', ); const stdout = new MemoryWriter(); - const exitCode = await runCli(['pull', 'PROTO_strand-lifecycle'], { + const exitCode = await runCli(['pull', 'PROCESS_strand-lifecycle'], { cwd: root, stdout, stderr: new MemoryWriter(), @@ -118,10 +118,10 @@ describe('method CLI', () => { const designDoc = readFile(root, 'docs/design/0001-strand-lifecycle/strand-lifecycle.md'); expect(designDoc).toMatch(/^---\n[\s\S]+?\n---\n/u); expect(designDoc).toMatch(/^title:\s+"Strand Lifecycle"$/mu); - expect(designDoc).toMatch(/^legend:\s+"PROTO"$/mu); + expect(designDoc).toMatch(/^legend:\s+"PROCESS"$/mu); expect(designDoc).toMatch(/^cycle:\s+"0001-strand-lifecycle"$/mu); - expect(designDoc).toMatch(/^source_backlog:\s+"docs\/method\/backlog\/asap\/PROTO_strand-lifecycle\.md"$/mu); - expect(designDoc).toContain('Legend: PROTO'); + expect(designDoc).toMatch(/^source_backlog:\s+"docs\/method\/backlog\/asap\/PROCESS_strand-lifecycle\.md"$/mu); + expect(designDoc).toContain('Legend: PROCESS'); expect(designDoc).toContain('- Human: Backlog operator'); expect(designDoc).toContain('- Agent: Implementation agent'); expect(designDoc).toContain('Mechanical state transitions for strands.'); @@ -234,14 +234,18 @@ describe('method CLI', () => { expect(retroDoc).not.toContain('Outcome: TBD'); }); - it('shows backlog, active cycles, and legend health in status', async () => { + it('Does `method status` report legend health using the live repo legends rather than stale historical codes?', async () => { const root = createTempRoot(); await runCli(['init'], { cwd: root, stdout: new MemoryWriter(), stderr: new MemoryWriter() }); - writeFileSync(join(root, 'docs/method/backlog/inbox/VIZ_braille.md'), '# Braille\n\nIdea\n', 'utf8'); + writeFileSync( + join(root, 'docs/method/backlog/inbox/SYNTH_braille.md'), + '# Braille\n\nIdea\n', + 'utf8', + ); mkdirSync(join(root, 'docs/design/0001-method-cli'), { recursive: true }); writeFileSync( join(root, 'docs/design/0001-method-cli/method-cli.md'), - '# Method CLI\n\nLegend: TUI\n\n## Hill\n\nTBD\n', + '# Method CLI\n\nLegend: PROCESS\n\n## Hill\n\nTBD\n', 'utf8', ); @@ -251,8 +255,12 @@ describe('method CLI', () => { expect(exitCode).toBe(0); expect(stdout.output).toContain('Backlog'); expect(stdout.output).toContain('0001-method-cli'); - expect(stdout.output).toContain('VIZ'); - expect(stdout.output).toContain('TUI'); + expect(stdout.output).toContain('SYNTH'); + expect(stdout.output).toContain('PROCESS'); + expect(stdout.output).not.toContain('PROTO'); + expect(stdout.output).not.toContain('FEAT'); + expect(stdout.output).not.toContain('VIZ'); + expect(stdout.output).not.toContain('TUI'); }); it('shows status for a clone-like workspace even when empty lane directories are absent', async () => { diff --git a/tests/exec.test.ts b/tests/exec.test.ts index 83a7912..5892e95 100644 --- a/tests/exec.test.ts +++ b/tests/exec.test.ts @@ -36,8 +36,8 @@ describe('Async Exec', () => { initWorkspace(root); const workspace = new Workspace(root); - workspace.captureIdea('Test Idea', 'FEAT', 'Async Test'); - workspace.pullItem('FEAT_async-test'); + workspace.captureIdea('Test Idea', 'PROCESS', 'Async Test'); + workspace.pullItem('PROCESS_async-test'); const result = workspace.captureWitness('0001-async-test'); expect(result).toBeInstanceOf(Promise); @@ -50,8 +50,8 @@ describe('Async Exec', () => { initWorkspace(root); const workspace = new Workspace(root); - workspace.captureIdea('Test Idea', 'FEAT', 'Close Test'); - workspace.pullItem('FEAT_close-test'); + workspace.captureIdea('Test Idea', 'PROCESS', 'Close Test'); + workspace.pullItem('PROCESS_close-test'); const result = workspace.closeCycle('0001-close-test', true, 'hill-met'); expect(result).toBeInstanceOf(Promise); diff --git a/tests/github-adapter.test.ts b/tests/github-adapter.test.ts index 9b503d7..48f74fa 100644 --- a/tests/github-adapter.test.ts +++ b/tests/github-adapter.test.ts @@ -28,8 +28,8 @@ describe('GitHub Adapter Two-way Sync', () => { const workspace = new Workspace(root); // Create item with existing ID - workspace.captureIdea('Local Title', 'FEAT', 'My Item'); - const itemPath = 'docs/method/backlog/inbox/FEAT_my-item.md'; + workspace.captureIdea('Local Title', 'PROCESS', 'My Item'); + const itemPath = 'docs/method/backlog/inbox/PROCESS_my-item.md'; workspace.updateFrontmatter(itemPath, { github_issue_id: '42' }); // Mock GitHub PATCH @@ -73,8 +73,8 @@ describe('GitHub Adapter Two-way Sync', () => { const workspace = new Workspace(root); // Create item with existing ID - workspace.captureIdea('Local Title', 'FEAT', 'My Item'); - const itemPath = 'docs/method/backlog/inbox/FEAT_my-item.md'; + workspace.captureIdea('Local Title', 'PROCESS', 'My Item'); + const itemPath = 'docs/method/backlog/inbox/PROCESS_my-item.md'; workspace.updateFrontmatter(itemPath, { github_issue_id: '42' }); // Mock GitHub GETs (Issue and Comments) @@ -130,8 +130,8 @@ describe('GitHub Adapter Two-way Sync', () => { const workspace = new Workspace(root); // Create item with existing ID - workspace.captureIdea('Closed Item', 'FEAT', 'Closed'); - const itemPath = 'docs/method/backlog/inbox/FEAT_closed.md'; + workspace.captureIdea('Closed Item', 'PROCESS', 'Closed'); + const itemPath = 'docs/method/backlog/inbox/PROCESS_closed.md'; workspace.updateFrontmatter(itemPath, { github_issue_id: '99' }); // Mock GitHub GETs (Closed state) @@ -165,12 +165,12 @@ describe('GitHub Adapter Two-way Sync', () => { await adapter.pullBacklog(); // Verify file was moved to graveyard - const graveyardPath = join(root, 'docs/method/graveyard/FEAT_closed.md'); - const inboxPath = join(root, 'docs/method/backlog/inbox/FEAT_closed.md'); + const graveyardPath = join(root, 'docs/method/graveyard/PROCESS_closed.md'); + const inboxPath = join(root, 'docs/method/backlog/inbox/PROCESS_closed.md'); expect(existsSync(graveyardPath), 'file should exist in graveyard').toBe(true); expect(existsSync(inboxPath), 'file should no longer exist in inbox').toBe(false); expect(readFileSync(graveyardPath, 'utf8')).toContain('Closed Item'); - expect(workspace.readFrontmatter('docs/method/graveyard/FEAT_closed.md').lane).toBe('graveyard'); + expect(workspace.readFrontmatter('docs/method/graveyard/PROCESS_closed.md').lane).toBe('graveyard'); }); }); diff --git a/tests/ship-sync.test.ts b/tests/ship-sync.test.ts index ab2713d..b15a6ff 100644 --- a/tests/ship-sync.test.ts +++ b/tests/ship-sync.test.ts @@ -27,8 +27,8 @@ describe('Ship Sync', () => { const workspace = new Workspace(root); // Create and close a cycle - const itemPath = workspace.captureIdea('Feature X', 'FEAT', 'New Feature'); - const cycle = workspace.pullItem('FEAT_new-feature'); + const itemPath = workspace.captureIdea('Feature X', 'PROCESS', 'New Feature'); + const cycle = workspace.pullItem('PROCESS_new-feature'); await workspace.closeCycle(cycle.name, true, 'hill-met'); // Run ship sync @@ -49,11 +49,11 @@ describe('Ship Sync', () => { const workspace = new Workspace(root); // Create a backlog item and move it to up-next - workspace.captureIdea('Next priority', 'FEAT', 'Up Next'); - workspace.moveBacklogItem('docs/method/backlog/inbox/FEAT_up-next.md', 'up-next'); + workspace.captureIdea('Next priority', 'PROCESS', 'Up Next'); + workspace.moveBacklogItem('docs/method/backlog/inbox/PROCESS_up-next.md', 'up-next'); - const itemPath = workspace.captureIdea('Feature Z', 'FEAT', 'Just Shipped'); - const cycle = workspace.pullItem('FEAT_just-shipped'); + const itemPath = workspace.captureIdea('Feature Z', 'PROCESS', 'Just Shipped'); + const cycle = workspace.pullItem('PROCESS_just-shipped'); await workspace.closeCycle(cycle.name, true, 'hill-met'); // Run ship sync @@ -62,7 +62,7 @@ describe('Ship Sync', () => { // Verify BEARING const bearing = readFileSync(join(root, 'docs/BEARING.md'), 'utf8'); expect(bearing).toContain(`- \`${cycle.name}\`: Just Shipped`); - expect(bearing).toContain('FEAT_up-next'); + expect(bearing).toContain('PROCESS_up-next'); }); it('`src/index.ts` provides a `shipSync()` method that performs the orchestration.', () => { @@ -70,14 +70,14 @@ describe('Ship Sync', () => { expect(workspace.shipSync).toBeDefined(); }); - it('`tests/ship-sync.test.ts` proves that the sync is idempotent (running it twice doesn\'t duplicate entries).', async () => { + it('Does `shipSync()` stay idempotent when the closed-cycle fixtures use live repo legends instead of stale feature labels?', async () => { const root = createTempRoot(); initWorkspace(root); const workspace = new Workspace(root); // Create and close a cycle - workspace.captureIdea('Feature Y', 'FEAT', 'Another Feature'); - const cycle = workspace.pullItem('FEAT_another-feature'); + workspace.captureIdea('Feature Y', 'PROCESS', 'Another Feature'); + const cycle = workspace.pullItem('PROCESS_another-feature'); await workspace.closeCycle(cycle.name, true, 'hill-met'); // Run ship sync first time @@ -101,12 +101,12 @@ describe('Ship Sync', () => { initWorkspace(root); const workspace = new Workspace(root); - workspace.captureIdea('Feature A', 'FEAT', 'Feature A'); - const cycleOne = workspace.pullItem('FEAT_feature-a'); + workspace.captureIdea('Feature A', 'PROCESS', 'Feature A'); + const cycleOne = workspace.pullItem('PROCESS_feature-a'); await workspace.closeCycle(cycleOne.name, true, 'hill-met'); - workspace.captureIdea('Feature B', 'FEAT', 'Feature B'); - const cycleTwo = workspace.pullItem('FEAT_feature-b'); + workspace.captureIdea('Feature B', 'PROCESS', 'Feature B'); + const cycleTwo = workspace.pullItem('PROCESS_feature-b'); await workspace.closeCycle(cycleTwo.name, true, 'hill-met'); writeFileSync( diff --git a/tests/witness.test.ts b/tests/witness.test.ts index bb5405b..5c32c5e 100644 --- a/tests/witness.test.ts +++ b/tests/witness.test.ts @@ -45,8 +45,8 @@ describe('Automated Witness Capture', () => { const workspace = new Workspace(root); // Create and close a cycle (which calls captureWitness internally) - workspace.captureIdea('Witness Test', 'FEAT', 'Witness Test'); - const cycle = workspace.pullItem('FEAT_witness-test'); + workspace.captureIdea('Witness Test', 'PROCESS', 'Witness Test'); + const cycle = workspace.pullItem('PROCESS_witness-test'); const expectedDrift = workspace.detectDrift(cycle.name).output.trim(); // No need to spy manually now, index.ts handles it via METHOD_TEST @@ -130,17 +130,17 @@ describe('Automated Witness Capture', () => { expect(content).toContain('source_backlog: "docs/method/backlog/asap/PROCESS_line-1\\nline-2.md"'); }); - it('Does design-doc frontmatter YAML-escape legends instead of writing raw scalar text?', () => { + it('Does design-doc frontmatter YAML-escape raw legend scalar compatibility text instead of letting it masquerade as live taxonomy?', () => { const content = renderDesignDoc({ cycleName: '0001-legend-escape', title: 'Legend Escape', - legend: 'FEAT:alpha #quoted "value"', + legend: 'PROCESS:alpha #quoted "value"', source: 'docs/method/backlog/asap/PROCESS_legend-escape.md', backlogBody: 'Body', }); - expect(content).toContain('legend: "FEAT:alpha #quoted \\"value\\""'); - expect(content).toContain('Legend: FEAT:alpha #quoted "value"'); + expect(content).toContain('legend: "PROCESS:alpha #quoted \\"value\\""'); + expect(content).toContain('Legend: PROCESS:alpha #quoted "value"'); }); it('Normalizes generated frontmatter paths to POSIX separators.', () => { @@ -152,10 +152,10 @@ describe('Automated Witness Capture', () => { const designContent = renderDesignDoc({ cycleName: '0001-windows-paths', title: 'Windows Paths', - source: 'docs\\method\\backlog\\asap\\PROTO_windows-paths.md', + source: 'docs\\method\\backlog\\asap\\PROCESS_windows-paths.md', backlogBody: 'Body', }); - expect(designContent).toContain('source_backlog: "docs/method/backlog/asap/PROTO_windows-paths.md"'); + expect(designContent).toContain('source_backlog: "docs/method/backlog/asap/PROCESS_windows-paths.md"'); const retroContent = renderRetroDoc({ cycle: {