Skip to content

Commit c3959ea

Browse files
Narratorclaude
andauthored
Sprint 3071 Task A — RFC 0001 baseline + @domscribe/core schema (v3) + @domscribe/verify scaffold (#54)
* docs(sprints): publish 3071 RFC 0001 baseline + positioning verdict (#51 A1) Harness self-test passes 10/10 at 0 pixel diff — proves the measurement mechanism is sound. Agent one-shot styling completion is unmeasured: no agent-integration harness exists on main to drive the falsifier's `--mode=measure` path. Conservative default applies — verify is positioned as a self-correction layer (the <85% branch); no Slack alert (threshold neither met nor measurable). Documents the agent-integration harness as the follow-up sprint task that retires this measurement gap. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(core): VerifyResultSchema + verifyHistory; ANNOTATION_SCHEMA_VERSION 2 → 3 (#51 A2) Adds the canonical shape for the RFC 0002 verify_after_edit MCP tool output: - VerifyVerdictSchema: match | partial | no_change | regression. - VerifyResultSchema: verdict + timestamp + structured deltas (componentStylesDelta, computedStyleDelta, boundingRectDelta) + optional pixelDiffRatio, screenshotRef (relay blob ref, never inline bytes — preserves the 4 KB-per-element budget), and notes. - AnnotationContextSchema.verifyHistory: optional VerifyResult[]; older clients silently ignore it. - ANNOTATION_SCHEMA_VERSION bumped 2 → 3; additive-only v2 → v3 migration step registered. Schema additions are deliberately structured (not a binary pass/fail) — the RFC 0002 retry-resolution falsifier requires actionable deltas for the agent to reason about its own edit. All additions are optional; pre-v3 clients see no behaviour change. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(verify): scaffold @domscribe/verify; lift comparator from RFC 0001 harness (#51 A3) New package owns the pixel-diff comparator shared between the test- fixtures falsifier (RFC 0001) and the upcoming relay verify_after_edit MCP tool (RFC 0002): - Lifted verbatim from styling/scripts/falsifier.ts: PER_PIXEL_THRESHOLD, MAX_DIFF_RATIO, loadPng, diff. Numeric defaults unchanged so the committed baselines stay valid. - compareScreenshots(actual, baseline, { maxDiffRatio? }) wraps the loadPng + diff pair for the common one-shot call site (the RFC 0002 retry round uses a stricter tolerance — opt in per call). - Pure-TS (pixelmatch + pngjs only); no DOM; consumable from Node CI and the relay runtime. - Tagged scope:infra; depends only on @domscribe/core. eslint enforce-module-boundaries: scope:test now permitted to consume scope:infra (test-fixtures is the first such consumer). - @domscribe/test-fixtures swaps inline comparator + pixelmatch/pngjs dev-deps for `@domscribe/verify`. Falsifier self-test rerun: 10/10 passes, 0 pixel diff — lift is behavior-preserving. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
1 parent a171724 commit c3959ea

22 files changed

Lines changed: 970 additions & 58 deletions
Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
2+
> @domscribe/test-fixtures@0.0.1 test:falsifier /tmp/wt-sprint-3071-task-a/packages/domscribe-test-fixtures
3+
> tsx styling/scripts/falsifier.ts
4+
5+
{
6+
"mode": "self-test",
7+
"total": 10,
8+
"passes": 10,
9+
"fails": 0,
10+
"oneShotRate": 1,
11+
"annotations": [
12+
{
13+
"id": "A001",
14+
"fixture": "tailwind",
15+
"passed": true,
16+
"pixelDiffRatio": 0,
17+
"diffPixels": 0
18+
},
19+
{
20+
"id": "A002",
21+
"fixture": "tailwind",
22+
"passed": true,
23+
"pixelDiffRatio": 0,
24+
"diffPixels": 0
25+
},
26+
{
27+
"id": "A003",
28+
"fixture": "tailwind",
29+
"passed": true,
30+
"pixelDiffRatio": 0,
31+
"diffPixels": 0
32+
},
33+
{
34+
"id": "A004",
35+
"fixture": "tailwind",
36+
"passed": true,
37+
"pixelDiffRatio": 0,
38+
"diffPixels": 0
39+
},
40+
{
41+
"id": "A005",
42+
"fixture": "tailwind",
43+
"passed": true,
44+
"pixelDiffRatio": 0,
45+
"diffPixels": 0
46+
},
47+
{
48+
"id": "A101",
49+
"fixture": "styled",
50+
"passed": true,
51+
"pixelDiffRatio": 0,
52+
"diffPixels": 0
53+
},
54+
{
55+
"id": "A102",
56+
"fixture": "styled",
57+
"passed": true,
58+
"pixelDiffRatio": 0,
59+
"diffPixels": 0
60+
},
61+
{
62+
"id": "A103",
63+
"fixture": "styled",
64+
"passed": true,
65+
"pixelDiffRatio": 0,
66+
"diffPixels": 0
67+
},
68+
{
69+
"id": "A104",
70+
"fixture": "styled",
71+
"passed": true,
72+
"pixelDiffRatio": 0,
73+
"diffPixels": 0
74+
},
75+
{
76+
"id": "A105",
77+
"fixture": "styled",
78+
"passed": true,
79+
"pixelDiffRatio": 0,
80+
"diffPixels": 0
81+
}
82+
]
83+
}
Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# Sprint 3071 — RFC 0001 baseline + positioning verdict
2+
3+
**Author:** Staff SWE (sprint run 3071, issue #51)
4+
**Date:** 2026-06-08
5+
**Decides:** the positioning language for `verify_after_edit` (RFC 0002).
6+
**Does not decide:** whether to ship `verify_after_edit` — that bet is made by the DOP memo and RFC 0002; this doc only sequences how it is framed.
7+
8+
---
9+
10+
## TL;DR
11+
12+
| Quantity | Value | Source |
13+
| ----------------------------------------------------------- | --------------------------------------- | ----------------------------------------------- |
14+
| RFC 0001 falsifier (≥70% agent one-shot styling completion) | **unmeasured** | no agent-integration harness exists on `main` |
15+
| RFC 0001 mechanism self-test | **10/10 (100%), 0 pixel diff** | `styling/scripts/falsifier.ts --mode=self-test` |
16+
| Positioning verdict | **self-correction layer (<85% branch)** | conservative default in absence of measurement |
17+
| Slack alert (≥85% trigger) | **not posted** | threshold neither met nor measurable |
18+
19+
The lift of the comparator into `@domscribe/verify` (this PR, Task A3) is independently validated by the self-test: the harness re-imports the comparator and continues to grade all 10 baselines at 0 pixel diff.
20+
21+
## What the harness can measure today
22+
23+
The RFC 0001 falsifier harness (`packages/domscribe-test-fixtures/styling/scripts/falsifier.ts`) supports three modes:
24+
25+
1. **`self-test`** — builds the Tailwind and styled-components fixture apps, screenshots each annotation's `afterRoute`, and diffs against the committed baseline. Expected pass rate is **100% by design** — this is the harness's own correctness check, not a measurement of agent capability. The README is explicit:
26+
27+
> It does not invoke an agent. The agent-integration loop is built on top of this — see `--mode=measure`.
28+
29+
2. **`record`** — re-captures the baseline PNGs from the canonical `/after` routes.
30+
31+
3. **`measure --agent-output=<dir>`** — production grading: reads one screenshot per annotation from an external directory (produced by an agent-integration harness) and diffs against the baseline. **This is the mode that would actually answer "what is the agent's one-shot styling completion rate?"**
32+
33+
## What is missing
34+
35+
The agent-integration loop required to run `--mode=measure` does **not** exist on `main`. Specifically, there is no harness that:
36+
37+
- Reads each annotation from `styling/annotations.json`,
38+
- Drives an agent (Claude / Codex / similar) through the edit using the intent + source-file context,
39+
- Boots the fixture from the post-edit source,
40+
- Screenshots the rendered element into a per-annotation PNG,
41+
- Hands the directory to `falsifier.ts --mode=measure`.
42+
43+
Until that loop exists, the inherited RFC 0001 falsifier (≥70% one-shot agent styling completion by sprint 2734+6) is **unmeasured**. The self-test pass rate is structurally **not** a substitute — the self-test screenshots the canonical-after route, not an agent's edit, so it cannot fall below 100% no matter how poorly an agent would perform.
44+
45+
## Self-test result (mechanism-only)
46+
47+
```
48+
mode=self-test, total=10, passes=10, fails=0, oneShotRate=1.0
49+
all annotations: pixelDiffRatio=0, diffPixels=0
50+
```
51+
52+
Raw JSON: [`3071-rfc-0001-baseline.harness-self-test.json`](./3071-rfc-0001-baseline.harness-self-test.json).
53+
54+
The 100% pass rate means:
55+
56+
- The Vite build for both fixtures is reproducible.
57+
- Chromium + screenshot capture is locale/font/viewport-deterministic in this CI environment.
58+
- The lifted comparator in `@domscribe/verify` (this PR) diffs identically to the inline version it replaces — none of the 10 baseline diffs shifted off zero.
59+
60+
The 100% pass rate **does not mean** the agent's one-shot styling completion rate is 100%. That number is unknown.
61+
62+
## Methodology
63+
64+
- **Where:** ephemeral dev sandbox; node v20.19.4; pnpm 9.12.0; playwright 1.58.2 (chrome-headless-shell 1208); locale `en-US`, timezone `UTC`, viewport `800×600`, scale 1, animations disabled (matches the harness defaults).
65+
- **Source:** worktree at `origin/main@a171724` (RFC 0001 Task B merge), plus the `@domscribe/verify` lift introduced by this PR.
66+
- **Command:** `pnpm --filter @domscribe/test-fixtures test:falsifier`.
67+
- **Reproducibility:** the same command on the same commit on a CI runner with the documented Playwright cache returns the same JSON. Re-recording baselines (`--mode=record`) would only be needed if the canonical-after routes or the Chromium build changes.
68+
69+
## Positioning verdict
70+
71+
Per RFC 0002 §Implications-for-PM and issue #51, the baseline gates how `verify_after_edit` is framed:
72+
73+
- **≥85% → trust layer.** Verify catches the long tail; the build is conservative; PM may consider deferring relay registration (Task B) if capacity is tight.
74+
- **<85% → self-correction layer.** Verify is load-bearing for the value loop; the full build proceeds.
75+
76+
The baseline is unmeasured. The conservative default in the absence of measurement is the **self-correction layer** branch — we cannot justify treating verify as a long-tail polish layer when we have no evidence the short tail is solved. The full build proceeds; the Slack alert (which fires only on ≥85%) is **not** posted.
77+
78+
## What this means for Task B
79+
80+
No change. Task B (runtime `ScreenshotCapturer` + relay `verify_after_edit` MCP tool) ships as planned, soft-recommended in MCP prompts, no lifecycle gate. The package-level value of `@domscribe/verify` is independent of the agent one-shot rate — the harness already consumes it, and the relay tool will consume it on the same contract.
81+
82+
## Follow-up — agent-integration harness (next sprint)
83+
84+
The cleanest way to retire this measurement gap is to add `--mode=agent` (or a separate driver script under `styling/scripts/`) that:
85+
86+
1. For each annotation in `annotations.json`, spawns the agent under test with a fixed prompt (intent + sourceFile + sourceLine + the merged RFC 0001 `styleSource` + `componentStyles`).
87+
2. Applies the agent's edit to a scratch copy of the fixture, builds it, screenshots `afterRoute`.
88+
3. Writes `<id>.png` into a deterministic agent-output directory.
89+
4. Invokes the existing `--mode=measure` with that directory.
90+
91+
This is the prerequisite for measuring both the inherited RFC 0001 falsifier (≥70% one-shot) **and** the RFC 0002 falsifier (≥60% retry-resolution rate). Sized as a separate sprint task; out of scope for issue #51 (per the issue's "Out of scope" enumeration, which lists agent-side work as a P1 follow-up rather than in-scope).
92+
93+
## References
94+
95+
- [RFC 0001 — Two-tier component-style attribution](../rfcs/0001-component-styles-capture.md)
96+
- [RFC 0002 — Post-edit verification as an MCP diagnostic tool](../rfcs/0002-post-edit-verify-mcp-tool.md)
97+
- Issue [#51](https://github.com/patchorbit/domscribe/issues/51), Issue [#52](https://github.com/patchorbit/domscribe/issues/52)
98+
- PRs [#49](https://github.com/patchorbit/domscribe/pull/49), [#50](https://github.com/patchorbit/domscribe/pull/50) (RFC 0001 Tasks A and B)
99+
- Harness source: [`packages/domscribe-test-fixtures/styling/scripts/falsifier.ts`](../../packages/domscribe-test-fixtures/styling/scripts/falsifier.ts)
100+
- Harness README: [`packages/domscribe-test-fixtures/styling/README.md`](../../packages/domscribe-test-fixtures/styling/README.md)

eslint.config.mjs

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,19 @@ export default [
4343
'scope:adapter',
4444
],
4545
},
46+
{
47+
// scope:test consumes the same packages adapters do — it
48+
// grades them. Notably, `@domscribe/test-fixtures` now imports
49+
// `@domscribe/verify` (scope:infra) so the harness and the
50+
// relay verify_after_edit tool share one comparator.
51+
sourceTag: 'scope:test',
52+
onlyDependOnLibsWithTags: [
53+
'scope:core',
54+
'scope:infra',
55+
'scope:build',
56+
'scope:adapter',
57+
],
58+
},
4659
],
4760
},
4861
],

packages/domscribe-core/src/lib/migrations/annotation-migrations.spec.ts

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,4 +141,31 @@ describe('migrateAnnotation', () => {
141141

142142
expect(result.context.runtimeContext).toBeUndefined();
143143
});
144+
145+
it('should migrate a v2 annotation up to v3 (additive verifyHistory, no field rewrite)', () => {
146+
// Simulates a v2 annotation persisted between RFC 0001 (v1→v2) and
147+
// RFC 0002 (v2→v3). The v2 → v3 step is purely additive (verifyHistory
148+
// is a new optional field) — pre-existing runtimeContext data must
149+
// survive untouched.
150+
const raw = buildRawAnnotation({
151+
metadata: { schemaVersion: 2 },
152+
context: {
153+
pageUrl: 'http://localhost:3000',
154+
pageTitle: 'Test',
155+
viewport: { width: 1920, height: 1080 },
156+
userAgent: 'test-agent',
157+
runtimeContext: {
158+
componentStyles: { computed: { padding: '16px' } },
159+
},
160+
},
161+
});
162+
163+
const result = migrateAnnotation(raw);
164+
165+
expect(result.metadata.schemaVersion).toBe(ANNOTATION_SCHEMA_VERSION);
166+
expect(result.context.runtimeContext).toEqual({
167+
componentStyles: { computed: { padding: '16px' } },
168+
});
169+
expect(result.context.verifyHistory).toBeUndefined();
170+
});
144171
});

packages/domscribe-core/src/lib/migrations/annotation-migrations.ts

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,16 @@ const migrationSteps: Record<number, (data: Record<string, unknown>) => void> =
3131
1: () => {
3232
// No-op: v1 → v2 is purely additive.
3333
},
34+
/**
35+
* v2 → v3: additive only (per RFC 0002).
36+
*
37+
* v3 adds optional `context.verifyHistory` (an array of `VerifyResult`
38+
* records emitted by the `verify_after_edit` MCP tool). The field is
39+
* absent on v2 payloads, so no field rewriting is required.
40+
*/
41+
2: () => {
42+
// No-op: v2 → v3 is purely additive.
43+
},
3444
};
3545

3646
/**
Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
/**
2+
* Schema tests for RFC 0002 additions to @domscribe/core.
3+
*
4+
* Covers the additive surface: VerifyResultSchema, AnnotationContext.verifyHistory,
5+
* and the v3 schema-version bump. The pre-RFC 0002 annotation shape is exercised
6+
* exhaustively in `annotation-migrations.spec.ts` and the wider integration
7+
* suites; this spec is scoped to the new fields.
8+
*/
9+
10+
import { describe, it, expect } from 'vitest';
11+
import {
12+
ANNOTATION_SCHEMA_VERSION,
13+
AnnotationContextSchema,
14+
VerifyResultSchema,
15+
VerifyVerdictSchema,
16+
} from './annotation.js';
17+
18+
describe('ANNOTATION_SCHEMA_VERSION', () => {
19+
it('is at v3 (RFC 0002 — verifyHistory)', () => {
20+
expect(ANNOTATION_SCHEMA_VERSION).toBe(3);
21+
});
22+
});
23+
24+
describe('VerifyVerdictSchema', () => {
25+
it.each(['match', 'partial', 'no_change', 'regression'] as const)(
26+
'accepts %s',
27+
(verdict) => {
28+
expect(VerifyVerdictSchema.parse(verdict)).toBe(verdict);
29+
},
30+
);
31+
32+
it('rejects unknown verdicts', () => {
33+
expect(() => VerifyVerdictSchema.parse('ok')).toThrow();
34+
});
35+
});
36+
37+
describe('VerifyResultSchema', () => {
38+
it('parses a minimal match result (verdict + timestamp only)', () => {
39+
const parsed = VerifyResultSchema.parse({
40+
verdict: 'match',
41+
timestamp: '2026-06-08T12:00:00.000Z',
42+
});
43+
expect(parsed.verdict).toBe('match');
44+
expect(parsed.componentStylesDelta).toBeUndefined();
45+
expect(parsed.screenshotRef).toBeUndefined();
46+
});
47+
48+
it('parses a fully-populated partial result with all delta arrays', () => {
49+
const parsed = VerifyResultSchema.parse({
50+
verdict: 'partial',
51+
timestamp: '2026-06-08T12:00:00.000Z',
52+
pixelDiffRatio: 0.012,
53+
componentStylesDelta: [
54+
{ property: 'padding', before: '16px', after: '24px' },
55+
],
56+
computedStyleDelta: [
57+
{ property: 'background-color', before: null, after: 'rgb(0, 0, 0)' },
58+
],
59+
boundingRectDelta: [{ field: 'height', before: 32, after: 40 }],
60+
screenshotRef: 'blob://relay/ann_x/post-edit-1.png',
61+
notes: 'padding matched intent; background-color regressed',
62+
});
63+
expect(parsed.componentStylesDelta).toHaveLength(1);
64+
expect(parsed.boundingRectDelta?.[0]?.field).toBe('height');
65+
expect(parsed.screenshotRef).toMatch(/^blob:\/\//);
66+
});
67+
68+
it('rejects out-of-range pixelDiffRatio', () => {
69+
expect(() =>
70+
VerifyResultSchema.parse({
71+
verdict: 'match',
72+
timestamp: '2026-06-08T12:00:00.000Z',
73+
pixelDiffRatio: 1.5,
74+
}),
75+
).toThrow();
76+
});
77+
78+
it('rejects unknown BoundingRectDelta fields', () => {
79+
expect(() =>
80+
VerifyResultSchema.parse({
81+
verdict: 'partial',
82+
timestamp: '2026-06-08T12:00:00.000Z',
83+
boundingRectDelta: [
84+
// @ts-expect-error — runtime rejection is the point
85+
{ field: 'depth', before: 0, after: 10 },
86+
],
87+
}),
88+
).toThrow();
89+
});
90+
});
91+
92+
describe('AnnotationContextSchema.verifyHistory', () => {
93+
it('accepts a context without verifyHistory (older clients silently ignore)', () => {
94+
const parsed = AnnotationContextSchema.parse({
95+
pageUrl: 'http://localhost:3000',
96+
pageTitle: 'Test',
97+
viewport: { width: 1920, height: 1080 },
98+
userAgent: 'test-agent',
99+
});
100+
expect(parsed.verifyHistory).toBeUndefined();
101+
});
102+
103+
it('accepts an append-only history of VerifyResults', () => {
104+
const parsed = AnnotationContextSchema.parse({
105+
pageUrl: 'http://localhost:3000',
106+
pageTitle: 'Test',
107+
viewport: { width: 1920, height: 1080 },
108+
userAgent: 'test-agent',
109+
verifyHistory: [
110+
{ verdict: 'partial', timestamp: '2026-06-08T12:00:00.000Z' },
111+
{ verdict: 'match', timestamp: '2026-06-08T12:00:05.000Z' },
112+
],
113+
});
114+
expect(parsed.verifyHistory).toHaveLength(2);
115+
expect(parsed.verifyHistory?.[1]?.verdict).toBe('match');
116+
});
117+
});

0 commit comments

Comments
 (0)