Skip to content

feat: add guided generation benchmark tooling #32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mikeylong merged 7 commits into from
Mar 13, 2026
Merged

feat: add guided generation benchmark tooling #32

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c56226e
feat: add generation bundle adapter commands
mikeylong Mar 13, 2026
b70dfc9
feat: add prepare-generation output schema
mikeylong Mar 13, 2026
8fdc82c
feat: own generation sessions and run artifacts
mikeylong Mar 13, 2026
39b21f0
chore: remove repo-level vercel preview
mikeylong Mar 13, 2026
c988b9e
chore: refresh checks after vercel unlink
mikeylong Mar 13, 2026
8aff2b9
feat: add guided generation benchmark tooling
mikeylong Mar 13, 2026
4718f97
feat: capture generation previews
mikeylong Mar 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,5 @@ Before proposing code changes, ensure the plan explicitly states:

Repo-specific durability rules:
- this repo is the canonical source of truth for CLI behavior, validator logic, extractor behavior, schema, and release provenance
- the static preview files in `public/index.html` and `vercel.json` are repo-local preview infrastructure, not the primary product website
- the public onboarding and richer product marketing site still live in `../surfaces-webapps/apps/interfacectl-web`
- if a cross-repo task also requires consumer wiring, land and test the source change here first, then update `surfaces-webapps`
268 changes: 264 additions & 4 deletions API.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -563,15 +563,19 @@ The output directory contains:
| Path | Description |
|------|-------------|
| `manifest.json` | Bundle manifest with version, contract id/version, tool info, inputs, and file hashes |
| `contract.normalized.json` | Normalized full contract (stable key order, same information as source) |
| `surfaces/<surfaceId>.json` | One file per surface; surface data plus minimal metadata for runtime loading |
| `constraints/motion.json` | Motion constraint category (other categories may be added later) |
| `contract/normalized.json` | Normalized full contract kept for traceability and downstream validation |
| `surfaces/<surfaceId>/generation.json` | Surface entrypoint for generators and adapter consumers |
| `surfaces/<surfaceId>/sections.json` | Surface-local section slices and anatomy references |
| `surfaces/<surfaceId>/components.json` | Shared component catalog referenced by sections |
| `surfaces/<surfaceId>/constraints.json` | Cross-cutting contract-authoritative constraints for the surface |
| `surfaces/<surfaceId>/repair-map.json` | Deterministic repair actions keyed by canonical finding codes |
| `surfaces/<surfaceId>/authoring.json` | Optional authoring hints when the surface declares them |

**Manifest fields**

| Field | Description |
|-------|-------------|
| `bundleVersion` | Format version for this bundle (e.g. `"1.0"`) |
| `bundleVersion` | Format version for this bundle (e.g. `"2.0"`) |
| `contractId` | From the contract |
| `contractVersion` | From the contract (`version` field) |
| `schemaVersion` | Schema bundle identifier used by the CLI (e.g. `surfaces.web.contract@1`) |
Expand All @@ -583,6 +587,262 @@ No timestamps are included in the manifest so that bundles remain deterministic.

---

### `prepare-generation`

Resolves one compiled surface bundle into a single, agent-ready JSON payload for local workspace agents.

**Synopsis:**
```bash
interfacectl prepare-generation --bundle-root <dir> --surface <id> [--out <path>]
```

**Description:**
- Loads a compiled generation bundle and validates that the requested surface exists.
- Resolves `generation.json`, `sections.json`, `components.json`, `constraints.json`, `repair-map.json`, optional `authoring.json`, and `contract/normalized.json`.
- Emits one deterministic JSON document with summary text, checklist items, resolved generation guidance, sections, components, constraints, repairs, provenance, and source file paths.
- Uses evidence refs only; it does not inline extracted observation payloads.

**Options:**

| Option | Description |
|--------|-------------|
| `--bundle-root <dir>` | Path to the compiled generation bundle directory (required) |
| `--surface <id>` | Surface identifier to resolve from the bundle (required) |
| `--out <path>` | Optional output file path. When provided, the command writes the JSON payload to disk and suppresses the full stdout payload |

**Exit Codes:**
- `0`: Prepared payload written successfully
- `10`: Invalid input, missing bundle files, unsupported bundle version, or unreadable compiled contract
- `1`: Unexpected internal error

**Output shape**

The generated JSON document includes:

| Field | Description |
|-------|-------------|
| `surface` | `{ surfaceId, displayName, type }` |
| `bundle` | `{ root, version, manifestPath, sourcePaths }` |
| `contract` | `{ id, version, normalizedPath }` |
| `summary` | Human-readable text plus checklist/focus items and top repair priorities |
| `generation` | Resolved `boundary`, `structure`, `layout`, `visual`, and `guidance` objects |
| `sections` | Resolved section list for the target surface |
| `components` | Resolved component catalog for the target surface |
| `constraints` | Cross-cutting contract-authoritative constraints |
| `repairMap` | Deterministic repair actions keyed by canonical finding codes |
| `authoring` | Optional authoring hints when present in the source bundle |
| `evidenceRefs` | Evidence refs only; no inline extracted payloads |

---

### `init-generation-session`

Creates a tracked local-agent session from an already compiled bundle. The command copies the bundle into a session-local directory, prepares the canonical agent payload once, and writes `session.json`.

**Synopsis:**
```bash
interfacectl init-generation-session --bundle-root <dir> --surface <id> --workspace-root <path> [--tool <codex|cursor>] [--guidance-mode <prepared|unguided>] [--brief-file <path>] [--session <id>] [--artifacts-root <path>]
```

**Description:**
- Requires a compiled bundle and does not call `compile` implicitly.
- Defaults session artifacts to `<workspaceRoot>/artifacts/generation-sessions/<surfaceId>/<sessionId>/`.
- Writes `bundle/`, optional `prepared-input.json`, and `session.json`.
- `--guidance-mode prepared` is the guided contract-aware arm. `--guidance-mode unguided` intentionally withholds the prepared payload from the generator while still freezing the same bundle for later validation.
- `--brief-file` freezes the implementation brief into the session so later benchmark comparisons can prove both sessions used the same task.
- Canonical session schema lives at `packages/interfacectl-cli/schemas/generation-session.schema.json`.

**Exit Codes:**
- `0`: Session created successfully
- `10`: Invalid input, unreadable bundle, duplicate session, or unreadable workspace root
- `1`: Unexpected internal error

---

### `record-generation-attempt`

Records one operator-reviewed attempt for a tracked session.

**Synopsis:**
```bash
interfacectl record-generation-attempt --session-dir <path> --assessment-file <path>
```

**Description:**
- Loads `session.json` from the provided session directory.
- Validates the workspace against the frozen session bundle.
- Requires a rubric assessment with `structure`, `components`, `boundary`, `visual`, `responsiveness`, `notes`, and optional `touchedFiles`.
- Writes:
- `attempts/<nnn>.validate.json`
- `attempts/<nnn>.assessment.json`
- `attempts/<nnn>.metadata.json`
- Emits a canonical `generation` run into `contracts/generated/contract-runs.json` and rebuilds `contract-lineage.json`.
- Canonical assessment schema lives at `packages/interfacectl-cli/schemas/generation-assessment.schema.json`.

**Exit Codes:**
- `0`: Attempt recorded successfully
- `10`: Invalid input, missing session, invalid assessment payload, or unreadable files
- `1`: Unexpected internal error

---

### `review-generation-attempt`

Records an explicit human review for one `warn` attempt.

**Synopsis:**
```bash
interfacectl review-generation-attempt --session-dir <path> --attempt <number> --review-file <path>
```

**Description:**
- Loads the validate payload for the selected attempt and requires that the attempt status is `warn`.
- Requires a review file with `status`, `findingCodes`, and `rationale`.
- When `status=accepted`, the reviewed finding-code set must cover every remaining finding on the attempt.
- Writes `attempts/<nnn>.review.json`.
- Canonical review schema lives at `packages/interfacectl-cli/schemas/generation-attempt-review.schema.json`.

**Exit Codes:**
- `0`: Review recorded successfully
- `10`: Invalid input, missing session/attempt, invalid review coverage, or unreadable files
- `1`: Unexpected internal error

---

### `summarize-generation-session`

Summarizes recorded attempts for one session.

**Synopsis:**
```bash
interfacectl summarize-generation-session --session-dir <path>
```

**Description:**
- Aggregates attempt count, first pass attempt, first acceptable attempt, latest status, latest outcome, recurring finding codes, recurring repair codes, latest assessment, and the latest review when present.
- Writes `summary.json` and `summary.md`.
- Canonical summary schema lives at `packages/interfacectl-cli/schemas/generation-session-summary.schema.json`.

**Exit Codes:**
- `0`: Latest outcome is `pass` or `accepted-warn`
- `30`: Latest outcome is `warn` or `block`
- `10`: Missing session or attempts, or invalid artifacts
- `1`: Unexpected internal error

---

### `compare-generation-sessions`

Compares one unguided baseline session against one prepared guided session for the same implementation brief.

**Synopsis:**
```bash
interfacectl compare-generation-sessions --baseline-session-dir <path> --guided-session-dir <path> [--out-dir <path>]
```

**Description:**
- Requires both sessions to target the same surface, use the same tool, and freeze the same brief file.
- Requires `guidanceMode=unguided` for the baseline session and `guidanceMode=prepared` for the guided session.
- Computes first-attempt finding deltas, attempts-to-acceptable-outcome delta, rubric deltas, and goal checks.
- Writes `comparison.json` and `comparison.md`.
- Canonical schema lives at `packages/interfacectl-cli/schemas/generation-session-comparison.schema.json`.

**Exit Codes:**
- `0`: Comparison artifacts written successfully
- `10`: Invalid session pairing, mismatched brief, or unreadable artifacts
- `1`: Unexpected internal error

---

### `suggest-contract-deltas`

Generates evidence-backed contract-refinement suggestions from a guided session without mutating the contract.

**Synopsis:**
```bash
interfacectl suggest-contract-deltas --session-dir <path> [--out <path>]
```

**Description:**
- Requires a guided `prepared` session.
- Uses the session repair map, repeated finding patterns, and latest validate evidence to propose contract paths for review.
- Emits deterministic suggestion JSON and markdown.
- Suggestions start as `status=proposed`.
- Canonical schema lives at `packages/interfacectl-cli/schemas/contract-delta-suggestions.schema.json`.

**Exit Codes:**
- `0`: Suggestion artifacts written successfully
- `10`: Invalid session state or unreadable artifacts
- `1`: Unexpected internal error

---

### `review-contract-delta-suggestions`

Applies human accept/reject decisions to suggestion artifacts without mutating the canonical contract.

**Synopsis:**
```bash
interfacectl review-contract-delta-suggestions --suggestions <path> --review-file <path> [--out <path>]
```

**Description:**
- Loads an existing suggestion artifact.
- Applies review decisions keyed by `suggestionId`.
- Preserves undecided suggestions as `proposed`.
- Rewrites markdown alongside the updated JSON output.

**Exit Codes:**
- `0`: Suggestion review recorded successfully
- `10`: Invalid decisions or unreadable files
- `1`: Unexpected internal error

---

### `summarize-generation-benchmark`

Aggregates one or more comparison artifacts and reviewed suggestion sets into one benchmark report.

**Synopsis:**
```bash
interfacectl summarize-generation-benchmark --comparisons <path[,path...]> [--suggestions <path[,path...]>] [--out-dir <path>]
```

**Description:**
- Summarizes whether guided sessions reduced first-attempt blocking findings, reached acceptable outcomes no later, and improved rubric dimensions.
- Aggregates accepted/rejected/proposed suggestion counts across surfaces.
- Writes `benchmark-report.json` and `benchmark-report.md`.
- Canonical schema lives at `packages/interfacectl-cli/schemas/generation-benchmark-report.schema.json`.

**Exit Codes:**
- `0`: Benchmark report written successfully
- `10`: Missing comparison inputs or unreadable artifacts
- `1`: Unexpected internal error

---

### `emit-run-artifact`

Writes one canonical run record into `contracts/generated/contract-runs.json` and rebuilds `contract-lineage.json`.

**Synopsis:**
```bash
interfacectl emit-run-artifact --workspace-root <path> --surface <id> --source <bootstrap|generation|ci|runtime> --status <pass|warn|fail|unknown> [--contract <path>] [--extraction-path <path>] [--report-path <path>] [--finding-codes <csv>] [--workspace-id <id>] [--idempotency-key <key>] [--created-at <timestamp>] [--run-id <id>]
```

**Description:**
- Creates `contract-runs.json` and `contract-lineage.json` if they do not yet exist.
- Uses canonical schema files:
- `packages/interfacectl-cli/schemas/contract-runs.schema.json`
- `packages/interfacectl-cli/schemas/contract-lineage.schema.json`
- Dedupes by `(workspaceId, runId)` and `(workspaceId, idempotencyKey)` when provided.

**Exit Codes:**
- `0`: Run artifact emitted successfully
- `10`: Invalid input

---

### `generate-contract` (Phase 0)

Extracts a deterministic contract artifact from a Next.js app by analyzing app code. **Extraction only** — no enforcement, no network calls. Output is schema-valid; extracted-only fields live under `x_extracted`.
Expand Down
Loading
Loading