From 57e7bd8b6a95973792d59cc58c5061610c9c8f9a Mon Sep 17 00:00:00 2001 From: Clayton Kim Date: Sat, 25 Jul 2026 07:40:32 -0700 Subject: [PATCH 1/4] Route visual verification through evaluated skill families --- .gitignore | 2 + CONTRIBUTING.md | 7 + README.md | 45 ++- docs/installation.md | 136 ++++++++ docs/plans/artifact-check-skill-router.md | 149 +++++++++ docs/plans/visual-eval-prefix-caching.md | 164 ++++++++++ .../artifact-slop-false-provenance/clean.json | 8 + .../artifact-slop-false-provenance/fire.json | 14 + .../artifact-slop-false-sequence/clean.json | 8 + .../artifact-slop-false-sequence/fire.json | 14 + .../artifact-slop-false-state/clean.json | 8 + .../artifact-slop-false-state/fire.json | 14 + .../layout-repeated-track-symmetry/clean.json | 8 + .../layout-repeated-track-symmetry/fire.json | 14 + evals/lib/canonical-json.mjs | 7 + evals/rubric-coverage.json | 24 +- evals/run-rubrics.mjs | 194 ++++++++++++ evals/run.mjs | 15 +- evals/visual-cache/prefix.mjs | 83 +++++ evals/visual-cache/prefix.test.mjs | 100 ++++++ evals/visual-model-policy/README.md | 101 ++++++ .../example-measurements.json | 92 ++++++ evals/visual-model-policy/select.mjs | 296 ++++++++++++++++++ evals/visual-model-policy/select.test.mjs | 170 ++++++++++ install-pi.sh | 24 +- package.json | 4 +- .../agents/ve-verifier-artifact-slop-gap.md | 31 ++ .../.claude/agents/ve-verifier-copy.md | 26 -- .../agents/ve-verifier-visual-tells.md | 27 -- plugins/visual-explainer/SKILL.md | 113 ++----- .../references/delegated-skills.md | 90 ++++++ .../references/legacy-html.md | 113 ++----- .../references/model-routing.md | 71 +++++ .../references/verification.md | 113 ++++++- .../visual-explainer/scripts/verify/SPEC.md | 32 +- .../scripts/verify/checks.json | 14 +- .../scripts/verify/lib/engine.mjs | 8 +- .../scripts/verify/lib/model-policy.mjs | 122 ++++++++ .../scripts/verify/lib/model-policy.test.mjs | 113 +++++++ .../scripts/verify/lib/report.mjs | 118 ++++++- .../scripts/verify/rubric-criteria.json | 25 ++ .../verify/rubrics/pass-artifact-slop-gap.md | 70 +++++ .../scripts/verify/rubrics/pass-copy.md | 42 --- .../scripts/verify/rubrics/pass-layout.md | 4 +- .../verify/rubrics/pass-visual-tells.md | 15 - scripts/check-manifests.mjs | 18 +- 46 files changed, 2503 insertions(+), 363 deletions(-) create mode 100644 docs/installation.md create mode 100644 docs/plans/artifact-check-skill-router.md create mode 100644 docs/plans/visual-eval-prefix-caching.md create mode 100644 evals/fixtures/rubrics/artifact-slop-false-provenance/clean.json create mode 100644 evals/fixtures/rubrics/artifact-slop-false-provenance/fire.json create mode 100644 evals/fixtures/rubrics/artifact-slop-false-sequence/clean.json create mode 100644 evals/fixtures/rubrics/artifact-slop-false-sequence/fire.json create mode 100644 evals/fixtures/rubrics/artifact-slop-false-state/clean.json create mode 100644 evals/fixtures/rubrics/artifact-slop-false-state/fire.json create mode 100644 evals/fixtures/rubrics/layout-repeated-track-symmetry/clean.json create mode 100644 evals/fixtures/rubrics/layout-repeated-track-symmetry/fire.json create mode 100644 evals/lib/canonical-json.mjs create mode 100644 evals/visual-cache/prefix.mjs create mode 100644 evals/visual-cache/prefix.test.mjs create mode 100644 evals/visual-model-policy/README.md create mode 100644 evals/visual-model-policy/example-measurements.json create mode 100644 evals/visual-model-policy/select.mjs create mode 100644 evals/visual-model-policy/select.test.mjs create mode 100644 plugins/visual-explainer/.claude/agents/ve-verifier-artifact-slop-gap.md delete mode 100644 plugins/visual-explainer/.claude/agents/ve-verifier-copy.md delete mode 100644 plugins/visual-explainer/.claude/agents/ve-verifier-visual-tells.md create mode 100644 plugins/visual-explainer/references/delegated-skills.md create mode 100644 plugins/visual-explainer/references/model-routing.md create mode 100644 plugins/visual-explainer/scripts/verify/lib/model-policy.mjs create mode 100644 plugins/visual-explainer/scripts/verify/lib/model-policy.test.mjs create mode 100644 plugins/visual-explainer/scripts/verify/rubric-criteria.json create mode 100644 plugins/visual-explainer/scripts/verify/rubrics/pass-artifact-slop-gap.md delete mode 100644 plugins/visual-explainer/scripts/verify/rubrics/pass-copy.md delete mode 100644 plugins/visual-explainer/scripts/verify/rubrics/pass-visual-tells.md diff --git a/.gitignore b/.gitignore index 6814a4a..08f0eaf 100644 --- a/.gitignore +++ b/.gitignore @@ -28,5 +28,7 @@ skills-lock.json # Publish hygiene (Artifacture) .codex-briefs/ evals/model-matrix/out/ +evals/visual-model-policy/runs/ +evals/visual-model-policy/policy.generated.json docs/goals/ artifacts/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 19caf21..ba5ce09 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -23,6 +23,13 @@ objects with `id`, `family`, `severity`, `spec`, etc.). To add one: 4. Add the fixture's expected result to `evals/expectations.json`. 5. Run `npm run ve:eval` and confirm the new fixture is caught. +Focused visual rubrics may also return criterion IDs that are narrower than a +top-level catalog check. Register those in +`plugins/visual-explainer/scripts/verify/rubric-criteria.json`, keep them scoped +to one routed pass, add fire/clean human-reviewed eval cases, and run +`npm run check:manifests`. Do not present an unregistered rubric label as a +verdict `check_id`. + ## Adding a shared component 1. Export the component from `visual-explainer-mdx/components.tsx`. diff --git a/README.md b/README.md index d7c23c4..bb88ebc 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ # Artifacture -**Verified visual explainers for coding agents.** Artifacture turns MDX/React sources into self-contained HTML artifacts: architecture diagrams, code walkthroughs, literate diff explainers, comparison tables, and slide decks. It then proves they hold up, with a deterministic 200+ check design-quality verifier (layout, contrast, responsiveness, AI-slop tells), small blind LLM rubric passes, and a multi-model eval harness. Kimi, GLM, DeepSeek, Claude, and Codex all produce passing output from the same ~3k-token authoring contract. +**Verified visual explainers for coding agents.** Artifacture turns MDX/React sources into self-contained HTML artifacts: architecture diagrams, code walkthroughs, literate diff explainers, comparison tables, and slide decks. Verification is a routed family of skills: Artifacture runs deterministic mechanics and artifact-specific checks, Impeccable owns general visual craft, and Unslop owns prose. Visual judgment is dispatched to the smallest model and batch size that clears the relevant eval suite; the main agent orchestrates and summarizes instead of spending a frontier-model turn inspecting every screenshot. Artifacture began as a fork of [nicobailon/visual-explainer](https://github.com/nicobailon/visual-explainer) and preserves its spirit; see Credits below. @@ -52,15 +52,36 @@ Video formats (9:16 reel, 16:9 long-form) render to MP4 through Hyperframes; sam ## What it adds to upstream visual-explainer -- **ve-verify** (`scripts/verify/`): a 200+ check deterministic design-quality gate — static scans, real-browser measurement (390px overflow, WCAG contrast in both themes, Mermaid render), and small blind LLM rubric passes. Exit codes and JSON reports make it usable as a CI gate. Its own eval suite of seeded-violation fixtures proves each check fires. +- **ve-verify** (`scripts/verify/`): a 200+ check deterministic design-quality gate — static scans, real-browser measurement (390px overflow, WCAG contrast in both themes, Mermaid render), and routed specialist judgment. Exit codes and JSON reports make it usable as a CI gate. Seeded-violation fixtures prove deterministic checks fire; the visual-model policy separately selects the smallest model and screenshot batch size that clear precision, recall, silence, grounding, schema, latency, and cost gates. - **Tiered agent docs**: SKILL.md is a ~2.5k-token bootstrap plus one ~300-token card per use case (`cards/`). A covered flow reads about 3,100 tokens instead of 62,000. Deep references load only on escalation. - **17 shared components** (`visual-explainer-mdx/components.tsx`): DiagramCanvas with computed layout and CSS-only mobile linearization, build-time Shiki CodeBlock, DiffBlock, TerminalBlock, JsonTree, an interactive Quiz, MermaidBlock with zoom/pan chrome, decks, posters, and more. Strict-export integrity checks catch bad edge ids and undefined components at build time. - **PresentationDeck** (`visual-explainer-mdx/presentation.tsx`): a second deck engine for presented (not scrolled) decks — a fixed 1920×1080 stage scaled to fit any screen, collapsible slide rail, keyboard nav, and drill-down primitives (click-to-expand cards/sheets with a click-anywhere-to-close guard, ladder/fanout diagrams, metrics, steppers). Fully `--ve-*` token-driven so every preset skins it; its behavioral contract is pinned by a headless eval suite (`npm run ve:eval-presentation`). See [docs/presentation-deck.md](docs/presentation-deck.md) for when to use it vs `SlideDeck`. - **`/explain-diff`**: a literate diff mode (background → intuition → walkthrough → quiz), adapted from Geoffrey Litt's prompt pattern. -- **Model-matrix eval harness** (`evals/model-matrix/`): the same briefs across Kimi, GLM, DeepSeek, Claude, and Codex, scored on deterministic compliance plus a blind screenshot judge. Point it at your own model in about ten minutes. +- **Two model eval harnesses**: `evals/model-matrix/` compares artifact generation, while `evals/visual-model-policy/` qualifies visual-verification models and screenshot batch sizes. Generation quality and review quality are deliberately not treated as the same benchmark. - **One-command team sharing**: `share.sh` deploys to Vercel (zero setup, public) or sharehtml on Cloudflare (stable update-in-place URLs, team SSO via Cloudflare Access, comments). See `docs/TEAM-SHARING.md`. - **External design systems + `ve:learn`**: brand token sets are user-owned artifacts resolved from a registry outside the skill (`$ARTIFACTURE_DESIGN_DIR` → `~/.artifacture/design-systems/` → repo `design-systems/`) and inlined into exports by preset name. `npm run ve:learn -- --name ` drafts a system from a token source, a live page, or an image palette; deterministic heuristics are pinned by their own eval suite (`evals/design-systems/`). The repo ships the mechanism only — systems (typically private brand tokens) live in your own registry. See `docs/design-systems.md`. +## Skill family + +Artifacture is the artifact-mechanics and evidence-routing member of a skill +family: + +| Skill | Owns | +|---|---| +| **Artifacture** | export, browser/state coverage, clipping, containment, layout candidates, diagrams, operating-model fidelity, and evidence routing | +| **Impeccable** | general visual craft, design specificity, typography, color, generic decoration, and visual AI tells | +| **Unslop** | prose cadence, voice, and AI-writing patterns | + +Artifacture does not copy the other skills' judgment prompts. Legacy +high-precision craft/prose detectors remain temporarily as candidate extractors: +they can route evidence, but they cannot fail Artifacture. If a companion skill +is missing, its pass is reported as skipped. Artifacture-owned visual judgment +uses the smallest eval-qualified model and screenshot batch size; a +frontier/main agent is not the automatic fallback. + +See [installation and family setup](docs/installation.md) and the +[visual-model policy harness](evals/visual-model-policy/README.md). + ## Install Any agent with skills support (Claude Code, Codex, Cursor, and others), via the [skills CLI](https://skills.sh): @@ -69,8 +90,16 @@ Any agent with skills support (Claude Code, Codex, Cursor, and others), via the npx skills add theclaymethod/artifacture ``` +Install the recommended companion skills: + +```bash +npx impeccable skills install +npx skills add theclaymethod/unslop +``` + First generation clones the render pipeline to `~/.artifacture` (one-time, -Node >= 22); full-clone installs use the repo in place. +Node >= 22); full-clone installs use the repo in place. The companion skills +remain independently versioned and keep ownership of their prompts. **Claude Code, as a plugin:** @@ -87,6 +116,12 @@ cp -r /tmp/artifacture/plugins/visual-explainer ~/.agents/skills/visual-explaine rm -rf /tmp/artifacture ``` +After installation, follow [docs/installation.md](docs/installation.md) to +verify the render pipeline, detect missing companion skills, and install or +generate a visual-model policy. Until a pass has an eval-qualified route, +Artifacture reports `no-eval-qualified-model` rather than consuming the current +large agent by default. + For the upstream project, see [nicobailon/visual-explainer](https://github.com/nicobailon/visual-explainer). ## Commands @@ -112,12 +147,14 @@ For private team sharing setup, see [`docs/TEAM-SHARING.md`](docs/TEAM-SHARING.m ## Docs - [Features](docs/features.md): the full capability reference, including all output modes, aesthetics, and how generation works. +- [Installation and skill family](docs/installation.md): core setup, companion skills, failure behavior, and visual-model policy. - [Design systems](docs/design-systems.md): the external design-system registry format, resolution order, `ve:learn` token learning, and the agent-assisted refinement flow. - [PresentationDeck vs SlideDeck](docs/presentation-deck.md): which deck engine to reach for, the drill-down primitives, and the eval-pinned behavioral contract. - [Team sharing](docs/TEAM-SHARING.md): one-command deploys to Vercel or a team-gated Cloudflare space. - [Skill docs](plugins/visual-explainer/SKILL.md): what an agent actually reads, plus the per-use-case [cards](plugins/visual-explainer/cards/). - [Verifier](plugins/visual-explainer/scripts/verify/): the deterministic design-quality gate and its [eval suite](evals/). - [Model-matrix harness](evals/model-matrix/): benchmark your own model or agent on the same briefs. +- [Visual-model policy harness](evals/visual-model-policy/): select the smallest qualified model and safe screenshot batch size per routed pass. ## Limitations diff --git a/docs/installation.md b/docs/installation.md new file mode 100644 index 0000000..cc2c420 --- /dev/null +++ b/docs/installation.md @@ -0,0 +1,136 @@ +# Installation and skill-family setup + +Artifacture is usable by itself for export and deterministic verification. Full +visual and prose review is a family-of-skills workflow: + +- Artifacture owns artifact mechanics, rendered-state evidence, and + artifact-specific visual semantics. +- Impeccable owns general visual craft and visual AI tells. +- Unslop owns prose quality and AI-writing patterns. + +The skills stay independently installable and independently versioned. +Artifacture routes judgment to them and does not vendor their prompts. Legacy +deterministic craft/prose matches are candidate signals only; they do not count +as Artifacture failures. + +## Requirements + +- Node.js 22 or newer +- a browser available to Playwright for rendered verification +- a skills-capable harness +- optional provider credentials for direct small-model visual evals + +## Recommended install + +```bash +npx skills add theclaymethod/artifacture +npx impeccable skills install +npx skills add theclaymethod/unslop +``` + +Artifacture's first generation clones the render pipeline into +`~/.artifacture`. A full repository clone uses itself as the pipeline. + +The installation commands do not merge the three skills. This is intentional: +updating Artifacture must not silently change Impeccable's design rubric or +Unslop's prose rubric. + +## Claude Code plugin install + +```bash +git clone https://github.com/theclaymethod/artifacture.git +/plugin marketplace add ./artifacture +``` + +Install Impeccable and Unslop separately with the recommended commands above. +The Artifacture plugin records a delegated pass as skipped when its owning skill +is unavailable. + +## Manual file-based install + +```bash +git clone --depth 1 https://github.com/theclaymethod/artifacture.git /tmp/artifacture +cp -r /tmp/artifacture/plugins/visual-explainer ~/.agents/skills/visual-explainer +rm -rf /tmp/artifacture +``` + +Then install Impeccable and Unslop into the skill directory used by the same +harness. Do not copy their instructions into the Artifacture folder. + +## Pi install + +From a full clone: + +```bash +./install-pi.sh +``` + +The installer copies Artifacture's skill and prompts, then reports whether +companion skill directories are visible to Pi. It does not delete, replace, or +silently install the companion skills. + +## Verify the core + +From the Artifacture repository: + +```bash +npm install +npm run ve:check +npm run ve:eval-visual-model-policy +``` + +The last command verifies policy selection and cache-prefix identity. It does +not claim that a real provider/model is qualified. + +## Qualify visual models + +Artifacture should not spend the main/frontier agent on routine screenshot +review. Model and screenshot batch size are selected empirically per pass. + +1. Read `evals/visual-model-policy/README.md`. +2. Run human-reviewed fire/clean cases against candidate vision models and + batch sizes. +3. Record the measurements. +4. Generate a policy: + + ```bash + npm run ve:select-visual-model-policy -- \ + --input evals/visual-model-policy/measurements.json \ + --out ~/.artifacture/visual-model-policy.json + ``` + +5. Set `ARTIFACTURE_VISUAL_MODEL_POLICY` only when the policy is stored + somewhere else. + +No model graduates from its name, size claim, or self-reported confidence. +Qualification requires measured precision, recall, silence accuracy, correct +image/region grounding, valid JSON, latency, and cost. A batch size is valid +only for the pass/model combination that was tested. + +## Missing capability behavior + +Artifacture never hides missing verification by substituting the current agent: + +| Missing capability | Result | +|---|---| +| Browser automation | browser checks skipped with explicit disclosure | +| Impeccable | `impeccable:critique` skipped | +| Unslop | `unslop:cleanup-report` skipped | +| Eval-qualified visual model | pass skipped as `no-eval-qualified-model` | + +The artifact may still be delivered with a clear incomplete-verification +receipt. It must not be called fully verified. + +## Updating + +Update each family member independently: + +```bash +git -C ~/.artifacture pull --ff-only +npm install --prefix ~/.artifacture +npx impeccable skills install +npx skills add theclaymethod/unslop +``` + +After changing models, prompts, image detail, rubrics, or batching, rerun the +visual evals before replacing the generated policy. diff --git a/docs/plans/artifact-check-skill-router.md b/docs/plans/artifact-check-skill-router.md new file mode 100644 index 0000000..4c8a646 --- /dev/null +++ b/docs/plans/artifact-check-skill-router.md @@ -0,0 +1,149 @@ +# Artifact-check skill router + +## Problem + +The verifier currently exposes a large catalog of deterministic and LLM checks. +Although individual catalog entries have `applies_when` prose, the high-level +LLM protocol can still feel like one universal battery. That is expensive, hard +to explain, and encourages weak reviewers to answer questions that the artifact +never earned. + +Artifact checks should work like narrow Impeccable skills: route the artifact to +the few visual disciplines its structure, task, and evidence make relevant. + +## Product contract + +Every rendered state receives one cheap mechanical baseline: + +- runtime and failed requests; +- viewport/canvas fit; +- text, arrow, and component clipping candidates. + +Everything deeper is routed. A routed check must name: + +1. the visual skill; +2. the applicability signal; +3. the artifact or named region; +4. the criterion/rubric; +5. the evidence input; +6. the silence case. + +The verifier returns independent findings with screenshot evidence. It does not +average unrelated findings into a generic design score. + +## Initial skill families + +| Skill | Example applicability signals | Example checks | +|---|---|---| +| Layout | repeated tracks, split panes, dense dashboards, large empty regions | symmetry, row alignment, hierarchy, crowding, intentional dead space | +| Typography | multiple type roles, long prose, narrow labels, custom fonts | role contrast, clipping, measure, hierarchy, fallback stability | +| Clarify | action labels, status language, technical terms, competing calls to action | ambiguity, undefined jargon, redundant labels, distinct commitments | +| Color and contrast | colored surfaces, status pairs, custom palette | readable contrast, status semantics, reflexive accent use | +| Motion | transitions, reveals, timed states | reduced motion, geometry animation, continuity, safe resting states | +| Diagram | SVG/Mermaid or relational visual structure | label clipping, arrow endpoints, proportional honesty, diagram necessity | +| Delegated visual craft | general aesthetic or visual-AI-tell candidate | route evidence to Impeccable critique; do not reproduce its detector or taste rubric | +| Delegated prose | prose-pattern candidate | route excluded-filtered prose to Unslop `cleanup --report`; do not rewrite during verification | +| Artifact semantic slop gap | explicit named-region review | false sequence, false state/confidence, or false provenance encoded by decoration | + +## Routing sequence + +1. Extract cheap structural signals from source and rendered DOM. +2. Build a candidate list of skills, not a flat list of every check. +3. For each candidate skill, select only criteria whose applicability signal is + present. +4. Batch screenshots only when the selected criteria and viewport requirements + match. +5. Run each skill in fresh context with only its rubric and required evidence. +6. Merge findings by artifact/region; preserve `silent` results for regression + calibration. + +## First missing check: repeated-track symmetry + +The layout pass must catch a composition that establishes peer columns or rows +but renders them with unexplained unequal tracks, broken divider/baseline +alignment, or whitespace that makes one peer region feel missing. + +Applicability: + +- a repeated grid/flex topology is visible; or +- the artifact declares an input/process/output, before/after, comparison, or + peer-panel relationship. + +Flag: + +- peer tracks have unexplained unequal widths; +- corresponding rows or dividers do not align; +- one track contains accidental dead space that changes its apparent weight. + +Stay silent: + +- the asymmetry is visibly intentional; +- content hierarchy assigns the tracks different roles; +- the composition is editorial rather than a peer mapping. + +Evidence: + +- full-state screenshot; +- repeated-track bounding boxes and computed grid/flex values when available; +- the smallest crop that demonstrates the drift. + +## Slop ownership boundary + +Artifacture is designed to run alongside Impeccable and Unslop. It must route +to those skills rather than copying their taxonomies into a second, drifting +implementation. + +Route general visual craft and visual AI tells to `impeccable:critique`. +Examples include reflex fonts or palettes, gradients, glass, glow, nested +cards, fake window chrome, decorative text, template scaffolding, aesthetic +costume, ornament without information, and manufactured hierarchy. + +Route prose-pattern candidates to `unslop:cleanup-report`. Supply only +excluded-filtered prose and run Unslop's read-only `cleanup --report` path. +Artifacture verification never rewrites prose. + +Artifacture retains one deliberately small `artifacture:slop-gap` pass for +artifact-semantic decoration outside both skills: + +- false sequence: numbering, connectors, or arrows imply an order that the + source does not establish; +- false state or confidence: badges, meters, progress, or status ornament + implies measured state or certainty without evidence; and +- false provenance: citation-like, source-like, or validation-like decoration + implies an origin or verification the artifact cannot support. + +The gap pass is explicit opt-in only: +`data-ve-checks="artifacture:slop-gap"`. It receives one nominated crop, its +visible text, and the smallest truth/source excerpt needed to judge the claim. +It stays silent on typography, color, generic decoration, prose cadence, +clipping, crowding, symmetry, and dead space. + +If Impeccable or Unslop is unavailable, disclose the skipped delegated check. +Do not emulate it with an embedded fallback rubric. + +## Eval-driven rollout + +1. Add fire/clean screenshot pairs for symmetry: + - unequal three-column peer mapping → fire; + - equal three-column mapping → clean; + - intentional 60/40 editorial split → clean. +2. Measure false positives separately from catch rate. +3. Compare one-image and grouped-image passes with the same rubric. +4. Increase batch size only while per-image evidence grounding and silence + accuracy remain stable. +5. Add the next skill family only after the router correctly excludes it from + unrelated fixtures. + +## Implementation slices + +1. Rubric slice: conditional repeated-track symmetry question in `pass-layout`. +2. Candidate slice: browser collector emits repeated-track candidates and + bounding boxes. +3. Router slice: report lists selected skill passes and why each was selected. +4. Harness slice: `evals/visual-model-policy/` records catch, false-positive, + silence, evidence-grounding, schema validity, latency, and cost by model and + batch size, then generates the smallest-qualified runtime policy. Direct API + measurement population remains provider-specific. +5. Protocol slice: delivery reports the selected skills, skipped skills, and + check-level findings—never just a global pass/fail. +6. Boundary slice: test visual delegation, prose delegation, mixed candidates, ambiguous generic `slop` silence, explicit gap opt-in, and excluded-profile routing. diff --git a/docs/plans/visual-eval-prefix-caching.md b/docs/plans/visual-eval-prefix-caching.md new file mode 100644 index 0000000..4ff9c00 --- /dev/null +++ b/docs/plans/visual-eval-prefix-caching.md @@ -0,0 +1,164 @@ +# Visual eval prefix caching + +## Decision + +Use one immutable evidence prefix per exact screenshot set, then append one +criterion-specific suffix per judgment. Treat coding-agent caching as +best-effort; use direct model API calls when cache telemetry or controlled +small-model selection matters. + +The cacheable request order is: + +1. fixed tool definitions; +2. fixed system/verdict contract; +3. fixed evidence interpretation instructions; +4. optional active design-system exception excerpt; +5. exact image blocks in stable state-id order; +6. cache breakpoint; +7. selected check ids and criterion prompt. + +The image belongs before the variable criterion. The same PNG bytes, image +order, detail setting, model, tools, and shared instructions must be reused. + +## What is possible + +OpenAI prompt caching supports exact prompt-prefix reuse, including images. +Images and their `detail` setting must be identical. Direct Responses API calls +can place an explicit breakpoint on an `input_image` block and reuse a stable +`prompt_cache_key`. + +Anthropic prompt caching also supports image blocks. The cache covers tools, +system, and messages through the selected breakpoint. The first request must +begin returning before concurrent requests can read the newly written cache. + +Primary references: + +- https://developers.openai.com/api/docs/guides/prompt-caching +- https://platform.claude.com/docs/en/build-with-claude/prompt-caching +- https://code.claude.com/docs/en/sub-agents#how-forks-differ-from-named-subagents + +## Coding-agent constraints + +### Claude Code + +Named subagents start with fresh context and a separate prompt cache. Separate +`ve-verifier-*` agents therefore cannot be assumed to share an image prefix. +Conversation forks share the parent's cache, but inherit the parent's system +prompt, tools, model, and history. That makes forks useful for same-model review +branches, not for switching every visual judgment to a cheaper model. + +Artifacture's current `ve-verifier-*` agents are not image-prefix shaped. Their +agent definitions differ before the task begins, and each agent receives a +screenshot path in its variable task prompt, then reads the image through a tool. +The image tool result therefore arrives after the divergent prompt content. +Changing only rubric wording cannot make those calls share an image prefix. A +generic agent improves system/tool stability, but the image must still be +attached in the first request or loaded once before forking. + +A cache-friendly Claude Code experiment is: + +1. Load one evidence package in the parent. +2. Wait until that turn has begun returning. +3. Fork one branch per criterion, appending only the criterion suffix. +4. Compare with fresh named subagents. + +Do not call this a cache hit unless provider usage exposes +`cache_read_input_tokens`. + +### Codex + +Codex is engineered around exact prefix reuse, including images and tools, but +changing the model, tool list, sandbox, approval mode, or working directory can +break the prefix. Full-history subagent forks preserve more of the prefix but +inherit the parent model. A fresh small-model subagent may be cheaper, but it +does not provide a measurable shared-image cache contract. + +Do not infer cache success from elapsed time. Codex subagent results currently +do not expose `cached_tokens` or `cache_write_tokens` to Artifacture. + +Codex source tests show root and child threads can share a session-derived +`prompt_cache_key`, which is necessary but not sufficient. Exact model, +instructions, tools, history, and image content must still match. A thin +evidence-loading root followed by `fork_turns=all` children is the strongest +agent-native experiment; treat it as unverified until token telemetry confirms +the read. + +## Artifacture request families + +`evals/visual-cache/prefix.mjs` defines a provider-neutral prefix manifest. A +request belongs to the same cache family only when its `prefix_id` matches. +The identity includes: + +- provider and model; +- exact system, shared-instruction, design-system, and tool-definition hashes; +- image byte hashes; +- image state ids, stable order, and detail settings. + +Criterion prompts are suffixes and do not affect `prefix_id`. + +Delegated skills are separate request families. Artifacture may share an exact +image prefix only among its own compatible visual passes. +`impeccable:critique` retains Impeccable's skill context and cache family; +Artifacture does not flatten or copy that rubric merely to manufacture cache +reuse. `unslop:cleanup-report` is text-only and never shares an image prefix. +The explicit `artifacture:slop-gap` pass may share an Artifacture image prefix +only when its crop and source/truth evidence are byte-for-byte identical to the +other requests in that family. + +Recommended grouping: + +- group by one rendered state whenever possible; +- use the same image tuple only for criteria that genuinely need every image; +- never add a “helpful” extra screenshot to only one request in a family; +- keep timestamps, output paths, candidate ids, and rubric questions after the + breakpoint; +- warm one request before parallel suffix fan-out. + +## Eval matrix + +For each model and screenshot family, compare: + +| Mode | Shape | +|---|---| +| cold-independent | separate requests with intentionally distinct cache keys | +| warm-suffix | one cache write, then identical prefix plus criterion suffixes | +| one-grouped-pass | one image prefix and all compatible criteria in one output | +| coding-agent-named | one fresh named subagent per criterion | +| coding-agent-fork | one evidence-loading turn, then one fork per criterion | + +Test suffix counts `1, 2, 4, 8` and image counts `1, 2, 4`. Run the warm write +before parallel reads. + +Use at least 10 randomized cold replicates and 20 randomized warm replicates. +For Claude, use `DISABLE_PROMPT_CACHING=1` for the cold agent control. For +Codex/direct API, use a new session or cache key for the cold condition. + +Record: + +- catch rate and false-positive rate per criterion; +- silence accuracy; +- evidence grounding to the correct image and region; +- cross-criterion contamination; +- input, cache-write, cache-read, and output tokens; +- time to first token and total latency; +- provider/model/detail/image tuple and `prefix_id`; +- primer cost separately and included in end-to-end cost. + +A configuration graduates only when: + +1. cache-read tokens are non-zero on warm suffix requests; +2. the warm path reduces measured input cost or latency; +3. criterion accuracy stays within the accepted tolerance of cold-independent; +4. evidence grounding and silence do not degrade as suffix count increases. + +## Current recommendation + +Use direct API suffix fan-out for the controlled small-model eval harness. +Keep coding-agent subagents as the orchestration fallback and quality +cross-check, not as the mechanism used to prove cache savings. + +Model qualification and runtime routing now live in +`evals/visual-model-policy/`. The selector chooses the smallest model and +cheapest qualified batch size per pass after quality gates. The main agent does +not perform screenshot judgment as an implicit fallback; an unqualified or +unavailable route is reported as `no-eval-qualified-model`. diff --git a/evals/fixtures/rubrics/artifact-slop-false-provenance/clean.json b/evals/fixtures/rubrics/artifact-slop-false-provenance/clean.json new file mode 100644 index 0000000..16897ce --- /dev/null +++ b/evals/fixtures/rubrics/artifact-slop-false-provenance/clean.json @@ -0,0 +1,8 @@ +{ + "case": "The footnote marker resolves to a supplied source with title and URL.", + "human_label": "clean", + "judge_response": { + "pass": true, + "findings": [] + } +} diff --git a/evals/fixtures/rubrics/artifact-slop-false-provenance/fire.json b/evals/fixtures/rubrics/artifact-slop-false-provenance/fire.json new file mode 100644 index 0000000..06f82c8 --- /dev/null +++ b/evals/fixtures/rubrics/artifact-slop-false-provenance/fire.json @@ -0,0 +1,14 @@ +{ + "case": "A claim carries a citation-style [4] marker but the artifact contains no source list or target.", + "human_label": "fire", + "judge_response": { + "pass": false, + "findings": [ + { + "check_id": "artifact-slop-false-provenance", + "evidence": "The reference marker implies traceable evidence that the artifact cannot supply.", + "fix": "Link a real source or remove the citation treatment." + } + ] + } +} diff --git a/evals/fixtures/rubrics/artifact-slop-false-sequence/clean.json b/evals/fixtures/rubrics/artifact-slop-false-sequence/clean.json new file mode 100644 index 0000000..15354d1 --- /dev/null +++ b/evals/fixtures/rubrics/artifact-slop-false-sequence/clean.json @@ -0,0 +1,8 @@ +{ + "case": "Numbered steps correspond to the documented workflow and each arrow represents a real dependency.", + "human_label": "clean", + "judge_response": { + "pass": true, + "findings": [] + } +} diff --git a/evals/fixtures/rubrics/artifact-slop-false-sequence/fire.json b/evals/fixtures/rubrics/artifact-slop-false-sequence/fire.json new file mode 100644 index 0000000..6b87e18 --- /dev/null +++ b/evals/fixtures/rubrics/artifact-slop-false-sequence/fire.json @@ -0,0 +1,14 @@ +{ + "case": "Three independent options are numbered 1–3 and connected by arrows even though no order exists.", + "human_label": "fire", + "judge_response": { + "pass": false, + "findings": [ + { + "check_id": "artifact-slop-false-sequence", + "evidence": "Numbers and arrows falsely imply an ordered workflow.", + "fix": "Remove sequential cues or document the actual dependency." + } + ] + } +} diff --git a/evals/fixtures/rubrics/artifact-slop-false-state/clean.json b/evals/fixtures/rubrics/artifact-slop-false-state/clean.json new file mode 100644 index 0000000..6438130 --- /dev/null +++ b/evals/fixtures/rubrics/artifact-slop-false-state/clean.json @@ -0,0 +1,8 @@ +{ + "case": "The confidence value is calculated from the displayed test results and has an accessible explanation.", + "human_label": "clean", + "judge_response": { + "pass": true, + "findings": [] + } +} diff --git a/evals/fixtures/rubrics/artifact-slop-false-state/fire.json b/evals/fixtures/rubrics/artifact-slop-false-state/fire.json new file mode 100644 index 0000000..ac044b9 --- /dev/null +++ b/evals/fixtures/rubrics/artifact-slop-false-state/fire.json @@ -0,0 +1,14 @@ +{ + "case": "A static mockup shows 87% confidence although no calculation or source supplies that value.", + "human_label": "fire", + "judge_response": { + "pass": false, + "findings": [ + { + "check_id": "artifact-slop-false-state", + "evidence": "The confidence meter implies a measured value with no supporting data.", + "fix": "Remove the value or bind it to a real calculation and source." + } + ] + } +} diff --git a/evals/fixtures/rubrics/layout-repeated-track-symmetry/clean.json b/evals/fixtures/rubrics/layout-repeated-track-symmetry/clean.json new file mode 100644 index 0000000..0911e02 --- /dev/null +++ b/evals/fixtures/rubrics/layout-repeated-track-symmetry/clean.json @@ -0,0 +1,8 @@ +{ + "case": "A dominant narrative column is intentionally wider than a narrow evidence rail and the hierarchy is clear.", + "human_label": "clean", + "judge_response": { + "pass": true, + "findings": [] + } +} diff --git a/evals/fixtures/rubrics/layout-repeated-track-symmetry/fire.json b/evals/fixtures/rubrics/layout-repeated-track-symmetry/fire.json new file mode 100644 index 0000000..8c5a08d --- /dev/null +++ b/evals/fixtures/rubrics/layout-repeated-track-symmetry/fire.json @@ -0,0 +1,14 @@ +{ + "case": "Three peer columns use visibly different widths and the middle divider drifts despite equivalent content roles.", + "human_label": "fire", + "judge_response": { + "pass": false, + "findings": [ + { + "check_id": "layout-repeated-track-symmetry", + "evidence": "Peer columns have unexplained width and divider drift.", + "fix": "Use equal tracks and align the shared divider baseline." + } + ] + } +} diff --git a/evals/lib/canonical-json.mjs b/evals/lib/canonical-json.mjs new file mode 100644 index 0000000..e4e578a --- /dev/null +++ b/evals/lib/canonical-json.mjs @@ -0,0 +1,7 @@ +export function canonicalJson(value) { + if (Array.isArray(value)) return `[${value.map(canonicalJson).join(',')}]`; + if (value && typeof value === 'object') { + return `{${Object.keys(value).sort().map((key) => `${JSON.stringify(key)}:${canonicalJson(value[key])}`).join(',')}}`; + } + return JSON.stringify(value); +} diff --git a/evals/rubric-coverage.json b/evals/rubric-coverage.json index 6b62f4a..297a219 100644 --- a/evals/rubric-coverage.json +++ b/evals/rubric-coverage.json @@ -1,8 +1,20 @@ { - "note": "Tracks executable-fixture coverage for the 32 llm-pass/transcript catalog checks that evals/run.mjs cannot exercise (see plans/007-DESIGN-NOTE.md). Separate from evals/parity-allowlist.json, which is scoped to deterministic-stage (static-text/static-dom/browser) checks only. operating-model-fit also has a JSONL routing set covering slide and long-form section decisions.", + "note": "Tracks executable-fixture coverage for the 32 llm-pass/transcript catalog checks that evals/run.mjs cannot exercise (see plans/007-DESIGN-NOTE.md). Delegated checks are evaluated by their owning installed skill and excluded from Artifacture rubric coverage. operating-model-fit also has a JSONL routing set covering slide and long-form section decisions.", "behavioral_fixture_sets": { "operating-model-fit": "evals/fixtures/rubrics/operating-model-fit/routing-cases.jsonl" }, + "delegated_skill_checks": { + "impeccable:critique": [ + "font-pairing-same-classification", + "default-accent-reflex" + ], + "unslop:cleanup-report": [ + "unslop-prose-style", + "title-claim-function", + "copy-redundancy", + "jargon-undefined" + ] + }, "total_llm_pass_and_transcript_checks": 32, "covered_by_evals_run_rubrics": [ "text-visibly-clipped", @@ -10,14 +22,12 @@ "diagram-type-coherent", "preset-both-mode-visual", "deck-content-completeness", - "title-claim-function", "operating-model-fit", "reel-caption-matches-transcript" ], "remaining_uncovered": [ "real-table-for-tabular-data", "hierarchy-squint-test", - "unslop-prose-style", "diagram-legend-matches-figure", "diagram-focal-single-dominant", "diagram-proportional-honesty-visual", @@ -34,11 +44,7 @@ "poster-visual-fit-squint", "reel-no-mid-word-cuts", "demo-aesthetic-match", - "demo-adds-value", - "font-pairing-same-classification", - "default-accent-reflex", - "copy-redundancy", - "jargon-undefined" + "demo-adds-value" ], - "remaining_count": 24 + "remaining_count": 19 } diff --git a/evals/run-rubrics.mjs b/evals/run-rubrics.mjs index 6f3567d..1dc027a 100644 --- a/evals/run-rubrics.mjs +++ b/evals/run-rubrics.mjs @@ -42,6 +42,9 @@ const OPERATING_MODEL_ROUTING_CASES = join( const checksCatalog = JSON.parse( readFileSync(resolve(REPO_ROOT, 'plugins/visual-explainer/scripts/verify/checks.json'), 'utf8'), ).checks; +const rubricCriteria = JSON.parse( + readFileSync(resolve(REPO_ROOT, 'plugins/visual-explainer/scripts/verify/rubric-criteria.json'), 'utf8'), +).criteria; const severityById = new Map(checksCatalog.map((check) => [check.id, check.severity])); @@ -100,6 +103,27 @@ function runStubCheck(id) { return rows; } +function runCriterionCheck(id) { + const registered = rubricCriteria.find((criterion) => criterion.id === id); + if (!registered) throw new Error(`rubric-criteria.json has no entry for ${id}`); + const dir = join(RUBRICS_FIXTURES_ROOT, id); + return ['fire', 'clean'].map((caseName) => { + const fixturePath = join(dir, `${caseName}.json`); + if (!existsSync(fixturePath)) { + return { id, caseName, status: 'error', message: `missing fixture ${fixturePath}` }; + } + const fixture = JSON.parse(readFileSync(fixturePath, 'utf8')); + const result = deriveStatus(id, fixture.judge_response, 'warn'); + const expected = caseName === 'fire' ? 'warn' : 'pass'; + return { + id, + caseName, + status: result.status === expected ? 'pass' : 'FAIL', + message: `human-reviewed expected=${expected} derived=${result.status}`, + }; + }); +} + // --- Real algorithm: word-overlap scorer for caption vs. transcript -------- // Per checks.json's impl_hint: "Normalize both streams; sliding-window // overlap ratio per caption." A full sliding-window implementation is @@ -275,6 +299,173 @@ function runOperatingModelPassSelectionCases() { }); } +function runVisualFamilyPassSelectionCases() { + const baseCtx = { + profile: 'page', + preset: 'custom', + presetHint: '', + html: '
', + filePath: 'artifact.html', + }; + const cases = [ + { + name: 'diagram-check-emits-diagram-family', + ctx: baseCtx, + check: { id: 'diagram-type-coherent', stage: 'llm-pass', status: 'llm-required', severity: 'error' }, + expected: 'diagram', + }, + { + name: 'poster-check-emits-poster-family', + ctx: { ...baseCtx, profile: 'poster' }, + check: { id: 'poster-visual-fit-squint', stage: 'llm-pass', status: 'llm-required', severity: 'error' }, + expected: 'poster', + }, + ]; + return cases.map(({ name, ctx, check, expected }) => { + const report = buildReport(ctx, [check]); + const actual = report.llm_passes_required.includes(expected); + return { + id: 'visual-family-pass-selection', + caseName: name, + status: actual ? 'pass' : 'FAIL', + message: `expected=${expected} actual=${report.llm_passes_required.join('|')}`, + }; + }); +} + +function runDelegatedSkillSelectionCases() { + const visualCandidate = { + id: 'nested-cards', + stage: 'static-dom', + status: 'warn', + severity: 'warn', + }; + const proseCandidate = { + id: 'copy-slop-phrases', + stage: 'static-text', + status: 'warn', + severity: 'warn', + }; + const unrelatedFinding = { + id: 'body-text-contrast-aa', + stage: 'browser', + status: 'fail', + severity: 'error', + }; + const baseCtx = { + profile: 'slides', + preset: 'custom', + presetHint: '', + html: '
', + filePath: 'slides.html', + }; + const cases = [ + { + name: 'visual-candidate-routes-only-impeccable', + ctx: baseCtx, + checks: [visualCandidate], + expected: ['impeccable:critique'], + }, + { + name: 'prose-candidate-routes-only-unslop', + ctx: baseCtx, + checks: [proseCandidate], + expected: ['unslop:cleanup-report'], + }, + { + name: 'mixed-candidates-route-each-owner-once', + ctx: baseCtx, + checks: [visualCandidate, proseCandidate], + expected: ['impeccable:critique', 'unslop:cleanup-report'], + }, + { + name: 'clean-candidates-route-neither-owner', + ctx: baseCtx, + checks: [ + { ...visualCandidate, status: 'pass' }, + { ...proseCandidate, status: 'pass' }, + ], + expected: [], + }, + { + name: 'unrelated-failure', + ctx: baseCtx, + checks: [unrelatedFinding], + expected: [], + }, + { + name: 'impeccable-owned-llm-candidate-routes-to-impeccable', + ctx: baseCtx, + checks: [{ + id: 'font-pairing-same-classification', + stage: 'llm-pass', + status: 'llm-required', + severity: 'warn', + }], + expected: ['impeccable:critique'], + }, + { + name: 'unslop-owned-llm-candidate-routes-to-unslop', + ctx: baseCtx, + checks: [{ + id: 'unslop-prose-style', + stage: 'llm-pass', + status: 'llm-required', + severity: 'warn', + }], + expected: ['unslop:cleanup-report'], + }, + { + name: 'explicit-impeccable-marker', + ctx: { ...baseCtx, html: '
' }, + checks: [], + expected: ['impeccable:critique'], + }, + { + name: 'explicit-unslop-marker', + ctx: { ...baseCtx, html: "
" }, + checks: [], + expected: ['unslop:cleanup-report'], + }, + { + name: 'explicit-artifacture-slop-gap-marker', + ctx: { ...baseCtx, html: '
' }, + checks: [], + expected: ['artifacture:slop-gap'], + }, + { + name: 'ambiguous-slop-marker-routes-nothing', + ctx: { ...baseCtx, html: '
' }, + checks: [], + expected: [], + }, + { + name: 'video-visual-candidate-does-not-route-ui-critique', + ctx: { ...baseCtx, profile: 'video-comp' }, + checks: [visualCandidate], + expected: [], + }, + { + name: 'video-prose-candidate-can-route-unslop', + ctx: { ...baseCtx, profile: 'video-comp' }, + checks: [proseCandidate], + expected: ['unslop:cleanup-report'], + }, + ]; + + return cases.map(({ name, ctx, checks, expected }) => { + const report = buildReport(ctx, checks); + const actual = [...report.llm_passes_required].sort(); + const wanted = [...expected].sort(); + return { + id: 'delegated-skill-selection', + caseName: name, + status: JSON.stringify(actual) === JSON.stringify(wanted) ? 'pass' : 'FAIL', + message: `expected=${wanted.join('|') || 'none'} actual=${actual.join('|') || 'none'}`, + }; + }); +} + // --- Driver ------------------------------------------------------------ const catalogIds = new Set(checksCatalog.map((check) => check.id)); @@ -289,8 +480,11 @@ for (const entry of SLICE) { const rows = entry.mode === 'algorithm' ? runTranscriptCheck(entry.id) : runStubCheck(entry.id); allRows.push(...rows); } +for (const criterion of rubricCriteria) allRows.push(...runCriterionCheck(criterion.id)); allRows.push(...runOperatingModelRoutingCases()); allRows.push(...runOperatingModelPassSelectionCases()); +allRows.push(...runVisualFamilyPassSelectionCases()); +allRows.push(...runDelegatedSkillSelectionCases()); console.log('check_id,case,status,detail'); for (const row of allRows) { diff --git a/evals/run.mjs b/evals/run.mjs index 467683a..2576727 100644 --- a/evals/run.mjs +++ b/evals/run.mjs @@ -58,7 +58,12 @@ function runVerifier(filePath, stage) { function failedChecks(report) { if (!report || !Array.isArray(report.checks)) return []; - return report.checks.filter((check) => check.status === 'fail' || check.status === 'warn'); + return report.checks.filter( + (check) => + check.status === 'fail' || + check.status === 'warn' || + check.status === 'delegated-candidate', + ); } function severityFor(report, id) { @@ -76,7 +81,11 @@ function checkViolation(fileName, expected) { const allowedCoFires = new Set(expected.allowed_co_fires || []); const missing = expected.must_fire.filter((id) => !firedIds.has(id)); const otherErrors = fired.filter( - (check) => !expected.must_fire.includes(check.id) && !allowedCoFires.has(check.id) && check.severity === 'error', + (check) => + !expected.must_fire.includes(check.id) && + !allowedCoFires.has(check.id) && + check.severity === 'error' && + check.status !== 'delegated-candidate', ); return { @@ -98,7 +107,7 @@ function checkClean(fileName) { const filePath = join(cleanRoot, fileName); const stage = 'browser'; const { result, report, args } = runVerifier(filePath, stage); - const fired = failedChecks(report); + const fired = failedChecks(report).filter((check) => check.status !== 'delegated-candidate'); return { fileName, status: result.status === 0 && fired.length === 0 ? 'pass' : 'fail', diff --git a/evals/visual-cache/prefix.mjs b/evals/visual-cache/prefix.mjs new file mode 100644 index 0000000..667ef99 --- /dev/null +++ b/evals/visual-cache/prefix.mjs @@ -0,0 +1,83 @@ +import crypto from 'node:crypto'; +import fs from 'node:fs/promises'; +import path from 'node:path'; +import { canonicalJson } from '../lib/canonical-json.mjs'; + +const ALLOWED_DETAIL = new Set(['low', 'high', 'original']); + +export async function buildPrefixManifest({ + provider, + model, + systemText, + sharedInstructions, + designSystemText = '', + toolsJson = '[]', + images, +}) { + if (!provider) throw new Error('provider is required'); + if (!model) throw new Error('model is required'); + if (!systemText) throw new Error('systemText is required'); + if (!sharedInstructions) throw new Error('sharedInstructions is required'); + if (!Array.isArray(images) || images.length === 0) { + throw new Error('at least one image is required'); + } + + const seenIds = new Set(); + const imageRecords = []; + for (const image of images) { + if (!image?.id || !image?.path) throw new Error('every image requires id and path'); + if (seenIds.has(image.id)) throw new Error(`duplicate image id: ${image.id}`); + seenIds.add(image.id); + const detail = image.detail || 'high'; + if (!ALLOWED_DETAIL.has(detail)) throw new Error(`unsupported image detail: ${detail}`); + const bytes = await fs.readFile(image.path); + imageRecords.push({ + id: image.id, + basename: path.basename(image.path), + detail, + bytes: bytes.byteLength, + sha256: digest(bytes), + }); + } + + // Stable state ids, not filesystem enumeration order, define image order. + imageRecords.sort((a, b) => a.id.localeCompare(b.id)); + + const identity = { + schema_version: 1, + provider, + model, + tools_sha256: digest(toolsJson), + system_sha256: digest(systemText), + shared_instructions_sha256: digest(sharedInstructions), + design_system_sha256: digest(designSystemText), + images: imageRecords, + }; + return { + prefix_id: digest(canonicalJson(identity)), + ...identity, + }; +} + +export function attachCriterionSuffix(prefixManifest, { + pass, + criteria, + prompt, +}) { + if (!prefixManifest?.prefix_id) throw new Error('prefixManifest is required'); + if (!pass) throw new Error('pass is required'); + if (!Array.isArray(criteria) || criteria.length === 0) { + throw new Error('at least one criterion is required'); + } + if (!prompt) throw new Error('prompt is required'); + return { + prefix_id: prefixManifest.prefix_id, + pass, + criteria: [...criteria].sort(), + suffix_sha256: digest(prompt), + }; +} + +function digest(value) { + return crypto.createHash('sha256').update(value).digest('hex'); +} diff --git a/evals/visual-cache/prefix.test.mjs b/evals/visual-cache/prefix.test.mjs new file mode 100644 index 0000000..1458b88 --- /dev/null +++ b/evals/visual-cache/prefix.test.mjs @@ -0,0 +1,100 @@ +import assert from 'node:assert/strict'; +import fs from 'node:fs/promises'; +import os from 'node:os'; +import path from 'node:path'; +import test from 'node:test'; +import { attachCriterionSuffix, buildPrefixManifest } from './prefix.mjs'; + +async function fixture() { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'artifacture-prefix-')); + const first = path.join(dir, 'first.png'); + const second = path.join(dir, 'second.png'); + await fs.writeFile(first, Buffer.from('image-one')); + await fs.writeFile(second, Buffer.from('image-two')); + return { dir, first, second }; +} + +function options(paths) { + return { + provider: 'openai', + model: 'small-vision-model', + systemText: 'Return verdict JSON only.', + sharedInstructions: 'Inspect only the supplied evidence.', + designSystemText: 'Use the active design system as an exception source.', + toolsJson: '[]', + images: [ + { id: 's01:dark', path: paths.first, detail: 'high' }, + { id: 's01:light', path: paths.second, detail: 'high' }, + ], + }; +} + +test('identical evidence produces an identical prefix id', async (t) => { + const paths = await fixture(); + t.after(() => fs.rm(paths.dir, { recursive: true, force: true })); + const a = await buildPrefixManifest(options(paths)); + const b = await buildPrefixManifest(options(paths)); + assert.equal(a.prefix_id, b.prefix_id); +}); + +test('stable state ids canonicalize image order', async (t) => { + const paths = await fixture(); + t.after(() => fs.rm(paths.dir, { recursive: true, force: true })); + const a = await buildPrefixManifest(options(paths)); + const reversed = options(paths); + reversed.images.reverse(); + const b = await buildPrefixManifest(reversed); + assert.equal(a.prefix_id, b.prefix_id); + assert.deepEqual(a.images.map((image) => image.id), ['s01:dark', 's01:light']); +}); + +test('changing image bytes invalidates the prefix', async (t) => { + const paths = await fixture(); + t.after(() => fs.rm(paths.dir, { recursive: true, force: true })); + const a = await buildPrefixManifest(options(paths)); + await fs.writeFile(paths.first, Buffer.from('image-one-changed')); + const b = await buildPrefixManifest(options(paths)); + assert.notEqual(a.prefix_id, b.prefix_id); +}); + +test('changing image detail invalidates the prefix', async (t) => { + const paths = await fixture(); + t.after(() => fs.rm(paths.dir, { recursive: true, force: true })); + const a = await buildPrefixManifest(options(paths)); + const changed = options(paths); + changed.images[0].detail = 'low'; + const b = await buildPrefixManifest(changed); + assert.notEqual(a.prefix_id, b.prefix_id); +}); + +test('changing model or tools invalidates the prefix', async (t) => { + const paths = await fixture(); + t.after(() => fs.rm(paths.dir, { recursive: true, force: true })); + const a = await buildPrefixManifest(options(paths)); + const modelChanged = options(paths); + modelChanged.model = 'another-model'; + const b = await buildPrefixManifest(modelChanged); + const toolsChanged = options(paths); + toolsChanged.toolsJson = '[{"name":"read"}]'; + const c = await buildPrefixManifest(toolsChanged); + assert.notEqual(a.prefix_id, b.prefix_id); + assert.notEqual(a.prefix_id, c.prefix_id); +}); + +test('criterion suffixes share the prefix without sharing verdict content', async (t) => { + const paths = await fixture(); + t.after(() => fs.rm(paths.dir, { recursive: true, force: true })); + const prefix = await buildPrefixManifest(options(paths)); + const layout = attachCriterionSuffix(prefix, { + pass: 'layout', + criteria: ['layout-repeated-track-symmetry'], + prompt: 'Judge repeated-track symmetry.', + }); + const slop = attachCriterionSuffix(prefix, { + pass: 'artifacture:slop-gap', + criteria: ['artifact-slop-false-state'], + prompt: 'Judge decorative text scaffolding.', + }); + assert.equal(layout.prefix_id, slop.prefix_id); + assert.notEqual(layout.suffix_sha256, slop.suffix_sha256); +}); diff --git a/evals/visual-model-policy/README.md b/evals/visual-model-policy/README.md new file mode 100644 index 0000000..9c26b4a --- /dev/null +++ b/evals/visual-model-policy/README.md @@ -0,0 +1,101 @@ +# Visual verifier model policy + +This harness answers one operational question: + +> What is the smallest model, and the cheapest safe screenshot batch size, that +> can run each routed visual check family? + +It does not benchmark artifact generation. `evals/model-matrix/` owns that +separate problem. + +## Policy + +Artifacture's main thread is an orchestrator. It gathers deterministic evidence, +dispatches visual judgment, merges findings, and reports uncertainty. It does +not inspect every screenshot with the host model. + +Model choice is per pass, not global. A small model may qualify for clipping and +symmetry while a larger model remains necessary for operating-model fidelity. +Batch size is also per pass. Four screenshots in one request are allowed only +when the measured grounding and silence gates remain green. + +The selector always: + +1. filters out configurations that miss a quality or sample-size gate; +2. chooses the lowest-ranked (smallest) qualified model; +3. chooses the lowest-cost qualified batch size within that model; and +4. records larger qualified models only as runtime escalation fallbacks. + +A cheaper larger model does not displace a qualified smaller model. Change the +candidate `rank` ordering when deployment policy defines “smallest” differently. + +## Required eval data + +Use human-reviewed fire/clean cases for each pass. Include hard negatives: +intentional asymmetry, authored dead space, dense expert surfaces, decorative +but truthful states, and screenshots containing several unrelated regions. + +For every `model × pass × batch_size` cell, record: + +- at least 3 independent runs; +- at least 10 positive and 10 negative cases; +- TP, FP, TN, and FN; +- correct image-and-region grounding; +- JSON validity and explicit abstentions; +- p95 latency and actual provider cost; and +- the exact model, image detail, and stable prefix identity used. + +The default graduation gates are: + +| Metric | Gate | +|---|---:| +| Precision | ≥ 0.90 | +| Recall | ≥ 0.90 | +| Silence accuracy | ≥ 0.95 | +| Image/region grounding | ≥ 0.95 | +| JSON validity | ≥ 0.99 | +| Abstention rate | ≤ 0.05 | +| p95 latency | ≤ 30 s | +| Cost per case | ≤ $0.01 | + +These are floors, not universal truth. High-consequence families can override +them in the measurement file. + +## Generate a policy + +Copy `example-measurements.json`, replace the placeholder candidates with the +actual direct-API models under test, and populate results from the eval run: + +```bash +npm run ve:select-visual-model-policy -- \ + --input evals/visual-model-policy/measurements.json \ + --out ~/.artifacture/visual-model-policy.json +``` + +The command exits `1` if any measured pass has no qualified route. Do not turn +that into an automatic frontier-model fallback. Add cases, test the next model, +or disclose `no-eval-qualified-model`. + +Set `ARTIFACTURE_VISUAL_MODEL_POLICY` when the policy lives elsewhere. + +## Runtime escalation + +The selected model gets one normal attempt and one schema-only retry. Escalate +to the next eval-qualified model only for: + +- invalid JSON after that retry; +- explicit abstention; +- evidence that cannot be assigned to one image and region; +- provider failure or timeout after one retry; or +- an explicit out-of-distribution signal. + +Do not escalate merely because the main thread prefers a more articulate +critique. The model's job is to find and ground violations, not to write the +final narrative. + +## Prefix caching + +Use the immutable evidence-prefix contract in `evals/visual-cache/`. Cache +measurements and model qualification are independent gates: a cache hit does not +make an inaccurate model acceptable, and a qualified model does not justify an +unmeasured batch size. diff --git a/evals/visual-model-policy/example-measurements.json b/evals/visual-model-policy/example-measurements.json new file mode 100644 index 0000000..f259fa1 --- /dev/null +++ b/evals/visual-model-policy/example-measurements.json @@ -0,0 +1,92 @@ +{ + "schema_version": 1, + "thresholds": { + "min_runs": 3, + "min_positives": 10, + "min_negatives": 10, + "precision": 0.9, + "recall": 0.9, + "silence_accuracy": 0.95, + "grounding_accuracy": 0.95, + "json_validity": 0.99, + "max_abstention_rate": 0.05, + "max_p95_latency_ms": 30000, + "max_cost_per_case_usd": 0.01 + }, + "candidates": [ + { + "id": "provider-small-vision", + "rank": 1, + "class": "small", + "provider": "replace-me" + }, + { + "id": "provider-medium-vision", + "rank": 2, + "class": "medium", + "provider": "replace-me" + } + ], + "results": [ + { + "pass": "layout", + "model": "provider-small-vision", + "batch_size": 2, + "runs": 3, + "cases": 100, + "positives": 40, + "negatives": 60, + "tp": 38, + "fp": 2, + "tn": 58, + "fn": 2, + "grounded": 39, + "grounding_total": 40, + "json_valid": 100, + "responses": 100, + "abstentions": 1, + "total_cost_usd": 0.1, + "p95_latency_ms": 5000 + }, + { + "pass": "layout", + "model": "provider-small-vision", + "batch_size": 4, + "runs": 3, + "cases": 100, + "positives": 40, + "negatives": 60, + "tp": 31, + "fp": 2, + "tn": 58, + "fn": 9, + "grounded": 35, + "grounding_total": 40, + "json_valid": 100, + "responses": 100, + "abstentions": 2, + "total_cost_usd": 0.06, + "p95_latency_ms": 6500 + }, + { + "pass": "layout", + "model": "provider-medium-vision", + "batch_size": 4, + "runs": 3, + "cases": 100, + "positives": 40, + "negatives": 60, + "tp": 40, + "fp": 1, + "tn": 59, + "fn": 0, + "grounded": 40, + "grounding_total": 40, + "json_valid": 100, + "responses": 100, + "abstentions": 0, + "total_cost_usd": 0.4, + "p95_latency_ms": 9000 + } + ] +} diff --git a/evals/visual-model-policy/select.mjs b/evals/visual-model-policy/select.mjs new file mode 100644 index 0000000..c71234c --- /dev/null +++ b/evals/visual-model-policy/select.mjs @@ -0,0 +1,296 @@ +#!/usr/bin/env node +import crypto from 'node:crypto'; +import fs from 'node:fs/promises'; +import path from 'node:path'; +import process from 'node:process'; +import { pathToFileURL } from 'node:url'; +import { canonicalJson } from '../lib/canonical-json.mjs'; + +export const DEFAULT_THRESHOLDS = Object.freeze({ + min_runs: 3, + min_positives: 10, + min_negatives: 10, + precision: 0.9, + recall: 0.9, + silence_accuracy: 0.95, + grounding_accuracy: 0.95, + json_validity: 0.99, + max_abstention_rate: 0.05, + max_p95_latency_ms: 30000, + max_cost_per_case_usd: 0.01, +}); + +const RUNTIME_ESCALATION_TRIGGERS = Object.freeze([ + 'invalid-json-after-one-retry', + 'explicit-abstain', + 'evidence-not-attributable-to-one-image-and-region', + 'provider-error-or-timeout-after-one-retry', + 'out-of-distribution-input', +]); + +export function metricsFor(result) { + const responses = positiveInt(result.responses, 'responses'); + const cases = positiveInt(result.cases, 'cases'); + return { + precision: ratio(result.tp, Number(result.tp) + Number(result.fp), 1), + recall: ratio(result.tp, Number(result.tp) + Number(result.fn), 1), + silence_accuracy: ratio(result.tn, Number(result.tn) + Number(result.fp), 1), + grounding_accuracy: ratio(result.grounded, result.grounding_total, 0), + json_validity: ratio(result.json_valid, responses, 0), + abstention_rate: ratio(result.abstentions, responses, 0), + cost_per_case_usd: Number(result.total_cost_usd || 0) / cases, + p95_latency_ms: Number(result.p95_latency_ms), + }; +} + +export function qualificationFor(result, thresholds = DEFAULT_THRESHOLDS) { + const metrics = metricsFor(result); + const reasons = []; + const requireAtLeast = (field, actual, minimum) => { + if (Number(actual) < Number(minimum)) reasons.push(`${field} ${actual} < ${minimum}`); + }; + const requireAtMost = (field, actual, maximum) => { + if (!Number.isFinite(Number(actual)) || Number(actual) > Number(maximum)) { + reasons.push(`${field} ${actual} > ${maximum}`); + } + }; + + requireAtLeast('runs', result.runs, thresholds.min_runs); + requireAtLeast('positives', result.positives, thresholds.min_positives); + requireAtLeast('negatives', result.negatives, thresholds.min_negatives); + requireAtLeast('precision', metrics.precision, thresholds.precision); + requireAtLeast('recall', metrics.recall, thresholds.recall); + requireAtLeast('silence_accuracy', metrics.silence_accuracy, thresholds.silence_accuracy); + requireAtLeast('grounding_accuracy', metrics.grounding_accuracy, thresholds.grounding_accuracy); + requireAtLeast('json_validity', metrics.json_validity, thresholds.json_validity); + requireAtMost('abstention_rate', metrics.abstention_rate, thresholds.max_abstention_rate); + requireAtMost('p95_latency_ms', metrics.p95_latency_ms, thresholds.max_p95_latency_ms); + requireAtMost('cost_per_case_usd', metrics.cost_per_case_usd, thresholds.max_cost_per_case_usd); + + return { qualified: reasons.length === 0, reasons, metrics }; +} + +export function selectVisualModelPolicy(input) { + validateInput(input); + const thresholds = { ...DEFAULT_THRESHOLDS, ...(input.thresholds || {}) }; + const candidateById = new Map(input.candidates.map((candidate) => [candidate.id, candidate])); + const passes = [...new Set(input.results.map((result) => result.pass))].sort(); + const routes = {}; + const blocked = {}; + + for (const pass of passes) { + const evaluated = input.results + .filter((result) => result.pass === pass) + .map((result) => { + const candidate = candidateById.get(result.model); + if (!candidate) throw new Error(`result references unknown model ${result.model}`); + return { + ...result, + model_rank: candidate.rank, + model_class: candidate.class || null, + provider: candidate.provider || null, + ...qualificationFor(result, thresholds), + }; + }); + + const qualified = evaluated.filter((row) => row.qualified); + if (qualified.length === 0) { + blocked[pass] = { + reason: 'no-eval-qualified-model', + evaluated: evaluated.map(compactEvaluation), + }; + continue; + } + + const bestByModel = new Map(); + for (const row of qualified) { + const prior = bestByModel.get(row.model); + if (!prior || compareWithinModel(row, prior) < 0) bestByModel.set(row.model, row); + } + const ladder = [...bestByModel.values()].sort(compareModels); + const selected = ladder[0]; + routes[pass] = { + model: selected.model, + model_class: selected.model_class, + provider: selected.provider, + batch_size: selected.batch_size, + metrics: roundMetrics(selected.metrics), + evidence: { + runs: selected.runs, + cases: selected.cases, + positives: selected.positives, + negatives: selected.negatives, + }, + escalation_chain: ladder.slice(1).map((row) => ({ + model: row.model, + model_class: row.model_class, + provider: row.provider, + batch_size: row.batch_size, + })), + escalation_triggers: [...RUNTIME_ESCALATION_TRIGGERS], + }; + } + + const canonicalInput = canonicalJson(input); + return { + schema_version: 1, + generated_at: new Date().toISOString(), + source_sha256: crypto.createHash('sha256').update(canonicalInput).digest('hex'), + policy: 'smallest-eval-qualified-model-first', + thresholds, + routes, + blocked, + }; +} + +function validateInput(input) { + if (!input || !Array.isArray(input.candidates) || !Array.isArray(input.results)) { + throw new Error('input requires candidates[] and results[]'); + } + const ids = new Set(); + const ranks = new Set(); + for (const candidate of input.candidates) { + if (!candidate.id || !Number.isInteger(Number(candidate.rank)) || Number(candidate.rank) <= 0) { + throw new Error('each candidate requires id and positive integer rank'); + } + if (ids.has(candidate.id)) throw new Error(`duplicate candidate ${candidate.id}`); + if (ranks.has(Number(candidate.rank))) throw new Error(`duplicate candidate rank ${candidate.rank}`); + ids.add(candidate.id); + ranks.add(Number(candidate.rank)); + } + for (const result of input.results) { + for (const field of [ + 'pass', 'model', 'batch_size', 'runs', 'cases', 'positives', 'negatives', + 'tp', 'fp', 'tn', 'fn', 'grounded', 'grounding_total', 'json_valid', + 'responses', 'abstentions', 'total_cost_usd', 'p95_latency_ms', + ]) { + if (result[field] === undefined || result[field] === null || result[field] === '') { + throw new Error(`result requires ${field}`); + } + } + positiveInt(result.batch_size, 'batch_size'); + positiveInt(result.runs, 'runs'); + positiveInt(result.cases, 'cases'); + positiveInt(result.responses, 'responses'); + for (const field of [ + 'positives', 'negatives', 'tp', 'fp', 'tn', 'fn', 'grounded', + 'json_valid', 'abstentions', + ]) nonNegativeInt(result[field], field); + positiveInt(result.grounding_total, 'grounding_total'); + nonNegativeFinite(result.total_cost_usd, 'total_cost_usd'); + nonNegativeFinite(result.p95_latency_ms, 'p95_latency_ms'); + if (Number(result.positives) + Number(result.negatives) !== Number(result.cases)) { + throw new Error('positives + negatives must equal cases'); + } + if (Number(result.tp) + Number(result.fn) !== Number(result.positives)) { + throw new Error('tp + fn must equal positives'); + } + if (Number(result.tn) + Number(result.fp) !== Number(result.negatives)) { + throw new Error('tn + fp must equal negatives'); + } + if (Number(result.json_valid) > Number(result.responses)) { + throw new Error('json_valid cannot exceed responses'); + } + if (Number(result.abstentions) > Number(result.responses)) { + throw new Error('abstentions cannot exceed responses'); + } + if (Number(result.grounded) > Number(result.grounding_total)) { + throw new Error('grounded cannot exceed grounding_total'); + } + } +} + +function compareWithinModel(a, b) { + return ( + Number(a.metrics.cost_per_case_usd) - Number(b.metrics.cost_per_case_usd) + || Number(b.batch_size) - Number(a.batch_size) + || Number(a.metrics.p95_latency_ms) - Number(b.metrics.p95_latency_ms) + ); +} + +function compareModels(a, b) { + return ( + Number(a.model_rank) - Number(b.model_rank) + || compareWithinModel(a, b) + || String(a.model).localeCompare(String(b.model)) + ); +} + +function compactEvaluation(row) { + return { + model: row.model, + model_class: row.model_class, + provider: row.provider, + batch_size: row.batch_size, + qualified: row.qualified, + reasons: row.reasons, + metrics: roundMetrics(row.metrics), + }; +} + +function roundMetrics(metrics) { + return Object.fromEntries( + Object.entries(metrics).map(([key, value]) => [ + key, + Number.isFinite(value) ? Number(value.toFixed(key === 'cost_per_case_usd' ? 8 : 4)) : value, + ]), + ); +} + +function positiveInt(value, field) { + const number = Number(value); + if (!Number.isInteger(number) || number <= 0) throw new Error(`${field} must be a positive integer`); + return number; +} + +function nonNegativeInt(value, field) { + const number = Number(value); + if (!Number.isInteger(number) || number < 0) throw new Error(`${field} must be a non-negative integer`); + return number; +} + +function nonNegativeFinite(value, field) { + const number = Number(value); + if (!Number.isFinite(number) || number < 0) throw new Error(`${field} must be a non-negative number`); + return number; +} + +function ratio(numerator, denominator, whenEmpty) { + const den = Number(denominator); + if (den === 0) return whenEmpty; + return Number(numerator) / den; +} + +function parseArgs(argv) { + const out = { input: null, output: null }; + for (let index = 0; index < argv.length; index += 1) { + if (argv[index] === '--input') out.input = argv[++index]; + else if (argv[index] === '--out') out.output = argv[++index]; + else throw new Error(`unknown argument ${argv[index]}`); + } + if (!out.input) throw new Error('usage: select.mjs --input measurements.json [--out policy.json]'); + return out; +} + +async function main() { + const args = parseArgs(process.argv.slice(2)); + const source = JSON.parse(await fs.readFile(path.resolve(args.input), 'utf8')); + const policy = selectVisualModelPolicy(source); + const rendered = `${JSON.stringify(policy, null, 2)}\n`; + if (args.output) { + const output = path.resolve(args.output); + await fs.mkdir(path.dirname(output), { recursive: true }); + await fs.writeFile(output, rendered); + console.log(output); + } else { + process.stdout.write(rendered); + } + if (Object.keys(policy.blocked).length > 0) process.exitCode = 1; +} + +if (import.meta.url === pathToFileURL(process.argv[1] || '').href) { + main().catch((error) => { + console.error(error.message || error); + process.exit(2); + }); +} diff --git a/evals/visual-model-policy/select.test.mjs b/evals/visual-model-policy/select.test.mjs new file mode 100644 index 0000000..50ca0b7 --- /dev/null +++ b/evals/visual-model-policy/select.test.mjs @@ -0,0 +1,170 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { + DEFAULT_THRESHOLDS, + metricsFor, + qualificationFor, + selectVisualModelPolicy, +} from './select.mjs'; + +const candidates = [ + { id: 'nano', rank: 1, class: 'small', provider: 'test' }, + { id: 'mini', rank: 2, class: 'medium', provider: 'test' }, + { id: 'frontier', rank: 3, class: 'large', provider: 'test' }, +]; + +function result(overrides = {}) { + return { + pass: 'layout', + model: 'nano', + batch_size: 1, + runs: 3, + cases: 100, + positives: 40, + negatives: 60, + tp: 38, + fp: 2, + tn: 58, + fn: 2, + grounded: 39, + grounding_total: 40, + json_valid: 100, + responses: 100, + abstentions: 1, + total_cost_usd: 0.1, + p95_latency_ms: 5000, + ...overrides, + }; +} + +test('computes precision, recall, silence, grounding, validity, and cost', () => { + assert.deepEqual(metricsFor(result()), { + precision: 0.95, + recall: 0.95, + silence_accuracy: 58 / 60, + grounding_accuracy: 39 / 40, + json_validity: 1, + abstention_rate: 0.01, + cost_per_case_usd: 0.001, + p95_latency_ms: 5000, + }); +}); + +test('rejects a cheap batch when recall falls below the gate', () => { + const outcome = qualificationFor(result({ batch_size: 4, tp: 32, fn: 8 })); + assert.equal(outcome.qualified, false); + assert.match(outcome.reasons.join('\n'), /recall/); +}); + +test('selects the smallest qualified model before a cheaper larger model', () => { + const policy = selectVisualModelPolicy({ + candidates, + results: [ + result({ model: 'nano', batch_size: 2, total_cost_usd: 0.2 }), + result({ model: 'mini', batch_size: 8, total_cost_usd: 0.05 }), + ], + }); + assert.equal(policy.routes.layout.model, 'nano'); + assert.equal(policy.routes.layout.batch_size, 2); + assert.equal(policy.routes.layout.escalation_chain[0].model, 'mini'); +}); + +test('selects the empirically cheapest qualified batch within one model', () => { + const policy = selectVisualModelPolicy({ + candidates, + results: [ + result({ model: 'nano', batch_size: 1, total_cost_usd: 0.2 }), + result({ model: 'nano', batch_size: 2, total_cost_usd: 0.08 }), + result({ model: 'nano', batch_size: 4, total_cost_usd: 0.04, tp: 30, fn: 10 }), + ], + }); + assert.equal(policy.routes.layout.model, 'nano'); + assert.equal(policy.routes.layout.batch_size, 2); +}); + +test('blocks a pass when no model has enough positive and negative evidence', () => { + const policy = selectVisualModelPolicy({ + candidates, + results: [result({ + cases: 8, + positives: 4, + negatives: 4, + tp: 4, + fp: 0, + tn: 4, + fn: 0, + grounded: 4, + grounding_total: 4, + json_valid: 8, + responses: 8, + abstentions: 0, + })], + }); + assert.equal(policy.routes.layout, undefined); + assert.equal(policy.blocked.layout.reason, 'no-eval-qualified-model'); + assert.match(policy.blocked.layout.evaluated[0].reasons.join('\n'), /positives/); +}); + +test('allows stricter per-run thresholds without changing the selector', () => { + const policy = selectVisualModelPolicy({ + candidates, + thresholds: { ...DEFAULT_THRESHOLDS, recall: 0.98 }, + results: [ + result({ model: 'nano' }), + result({ model: 'mini', tp: 40, fn: 0 }), + ], + }); + assert.equal(policy.routes.layout.model, 'mini'); +}); + +test('keeps routes independent by check family', () => { + const policy = selectVisualModelPolicy({ + candidates, + results: [ + result({ pass: 'layout', model: 'nano' }), + result({ pass: 'operating-model', model: 'nano', tp: 30, fn: 10 }), + result({ pass: 'operating-model', model: 'mini' }), + ], + }); + assert.equal(policy.routes.layout.model, 'nano'); + assert.equal(policy.routes['operating-model'].model, 'mini'); +}); + +test('rejects internally inconsistent empirical counts', () => { + assert.throws( + () => selectVisualModelPolicy({ + candidates, + results: [result({ tp: 40, fn: 2 })], + }), + /tp \+ fn must equal positives/, + ); +}); + +test('requires explicit finite cost measurements', () => { + const missingCost = result(); + delete missingCost.total_cost_usd; + assert.throws( + () => selectVisualModelPolicy({ candidates, results: [missingCost] }), + /result requires total_cost_usd/, + ); +}); + +test('blocks an otherwise accurate model when cost exceeds the gate', () => { + const policy = selectVisualModelPolicy({ + candidates, + thresholds: { max_cost_per_case_usd: 0.0005 }, + results: [result()], + }); + assert.equal(policy.routes.layout, undefined); + assert.match(policy.blocked.layout.evaluated[0].reasons.join('\n'), /cost_per_case_usd/); +}); + +test('requires grounding observations', () => { + assert.throws( + () => selectVisualModelPolicy({ + candidates, + results: [result({ grounded: 0, grounding_total: 0 })], + }), + /grounding_total must be a positive integer/, + ); +}); diff --git a/install-pi.sh b/install-pi.sh index 9f443ab..99c3254 100755 --- a/install-pi.sh +++ b/install-pi.sh @@ -1,16 +1,17 @@ #!/bin/bash -# install-pi.sh - Install visual-explainer for Pi +# install-pi.sh - Install Artifacture's visual-explainer skill for Pi set -e SKILL_DIR="$HOME/.pi/agent/skills/visual-explainer" +PI_SKILLS_DIR="$HOME/.pi/agent/skills" PROMPTS_DIR="$HOME/.pi/agent/prompts" # Check if we're in the repo or need to clone if [ ! -f "plugins/visual-explainer/SKILL.md" ]; then - echo "Cloning visual-explainer..." + echo "Cloning Artifacture..." TEMP_DIR=$(mktemp -d) - git clone --depth 1 https://github.com/nicobailon/visual-explainer.git "$TEMP_DIR" + git clone --depth 1 https://github.com/theclaymethod/artifacture.git "$TEMP_DIR" cd "$TEMP_DIR" CLEANUP=true else @@ -41,7 +42,22 @@ if [ "$CLEANUP" = true ]; then fi echo "" -echo "Done! Restart pi to use visual-explainer." +echo "Artifacture core installed. Restart Pi to use visual-explainer." +echo "" +echo "Companion skill family:" +if [ -d "$PI_SKILLS_DIR/impeccable" ]; then + echo " [found] Impeccable — general visual craft" +else + echo " [missing] Impeccable — install separately for general visual craft" +fi +if [ -d "$PI_SKILLS_DIR/unslop" ]; then + echo " [found] Unslop — prose review" +else + echo " [missing] Unslop — install separately for prose review" +fi +echo " Missing companions are reported as skipped; Artifacture does not copy their rubrics." +echo " Visual screenshot passes also require an eval-qualified model policy." +echo " See: https://github.com/theclaymethod/artifacture/blob/main/docs/installation.md" echo "" echo "Commands available:" echo " /diff-review, /plan-review, /project-recap, /fact-check" diff --git a/package.json b/package.json index 8d27840..9dfc80b 100644 --- a/package.json +++ b/package.json @@ -19,9 +19,11 @@ "ve:eval": "node evals/run.mjs && node evals/design-systems/run.mjs", "ve:eval-presentation": "node evals/run-presentation.mjs", "ve:eval-rubrics": "node evals/run-rubrics.mjs", + "ve:eval-visual-model-policy": "node --test evals/visual-model-policy/select.test.mjs evals/visual-cache/prefix.test.mjs plugins/visual-explainer/scripts/verify/lib/model-policy.test.mjs", + "ve:select-visual-model-policy": "node evals/visual-model-policy/select.mjs", "check:manifests": "node scripts/check-manifests.mjs", "ve:learn": "node scripts/ve-mdx/learn.mjs", - "test": "node --test visual-explainer-mdx/diagram-layout.test.mjs visual-explainer-mdx/presentation-core.test.mjs scripts/ve-mdx/design-systems.test.mjs scripts/ve-mdx/export-react-singleton.test.mjs scripts/ve-mdx/learn-extractors.test.mjs" + "test": "node --test visual-explainer-mdx/diagram-layout.test.mjs visual-explainer-mdx/presentation-core.test.mjs scripts/ve-mdx/design-systems.test.mjs scripts/ve-mdx/export-react-singleton.test.mjs scripts/ve-mdx/learn-extractors.test.mjs evals/visual-model-policy/select.test.mjs evals/visual-cache/prefix.test.mjs plugins/visual-explainer/scripts/verify/lib/model-policy.test.mjs" }, "dependencies": { "@mdx-js/rollup": "^3.1.1", diff --git a/plugins/visual-explainer/.claude/agents/ve-verifier-artifact-slop-gap.md b/plugins/visual-explainer/.claude/agents/ve-verifier-artifact-slop-gap.md new file mode 100644 index 0000000..e113160 --- /dev/null +++ b/plugins/visual-explainer/.claude/agents/ve-verifier-artifact-slop-gap.md @@ -0,0 +1,31 @@ +--- +name: ve-verifier-artifact-slop-gap +description: Judge only curated explanatory-artifact decorations that falsely imply sequence, state, confidence, or provenance. Returns only verdict JSON. +tools: Read +--- + +# ve-verifier-artifact-slop-gap + +You run one narrow verification pass. You do not fix files. + +## Read + +1. `plugins/visual-explainer/scripts/verify/rubrics/pass-artifact-slop-gap.md` +2. Only the candidate-region crops, visible text, accessibility labels, and + minimal truth excerpt named by the orchestrator. + +Do not read Impeccable or Unslop source. Do not judge generic aesthetic slop, +prose cadence, typography, color, cards, glow, glass, layout, or clipping. + +A finding requires a decoration that communicates an unsupported semantic about +sequence, state, confidence, or provenance. + +## Output + +Your final message must be only one valid JSON object matching the rubric schema: + +```json +{"pass":true,"findings":[]} +``` + +Do not include prose, markdown, code fences, summaries, or recommendations outside `findings`. diff --git a/plugins/visual-explainer/.claude/agents/ve-verifier-copy.md b/plugins/visual-explainer/.claude/agents/ve-verifier-copy.md deleted file mode 100644 index 54061b1..0000000 --- a/plugins/visual-explainer/.claude/agents/ve-verifier-copy.md +++ /dev/null @@ -1,26 +0,0 @@ ---- -name: ve-verifier-copy -description: Run the visual-explainer copy/slop LLM verification pass over extracted prose only. Returns only verdict JSON. -tools: Read ---- - -# ve-verifier-copy - -You run one verification pass. You do not fix files. - -## Read - -1. `plugins/visual-explainer/scripts/verify/rubrics/pass-copy.md` -2. The excluded-filtered prose extract named by the orchestrator. - -Do not read screenshots, source code, tables, identifiers, labels, counters, timestamps, status strings, Mermaid labels, or square-bracket system messages. - -## Output - -Your final message must be only one valid JSON object matching the rubric schema: - -```json -{"pass":true,"findings":[]} -``` - -Do not include prose, markdown, code fences, summaries, or recommendations outside `findings`. diff --git a/plugins/visual-explainer/.claude/agents/ve-verifier-visual-tells.md b/plugins/visual-explainer/.claude/agents/ve-verifier-visual-tells.md deleted file mode 100644 index 22aa7bd..0000000 --- a/plugins/visual-explainer/.claude/agents/ve-verifier-visual-tells.md +++ /dev/null @@ -1,27 +0,0 @@ ---- -name: ve-verifier-visual-tells -description: Run the visual-explainer visual-tells LLM verification pass over screenshots and candidate extracts. Returns only verdict JSON. -tools: Read ---- - -# ve-verifier-visual-tells - -You run one verification pass. You do not fix files. - -## Read - -1. `plugins/visual-explainer/scripts/verify/rubrics/pass-visual-tells.md` -2. The `ve-verify` report JSON named by the orchestrator. -3. The light/dark screenshots and candidate extracts named by the orchestrator. - -Do not apply copy, diagram, layout, completeness, or preset-aesthetic questions in this pass. - -## Output - -Your final message must be only one valid JSON object matching the rubric schema: - -```json -{"pass":true,"findings":[]} -``` - -Do not include prose, markdown, code fences, summaries, or recommendations outside `findings`. diff --git a/plugins/visual-explainer/SKILL.md b/plugins/visual-explainer/SKILL.md index 7ecf4c9..9169f88 100644 --- a/plugins/visual-explainer/SKILL.md +++ b/plugins/visual-explainer/SKILL.md @@ -106,101 +106,24 @@ Presets: mono-industrial, nothing, blueprint, editorial, paper-ink, terminal, cu |code|cards/code-walkthrough.md|| |explain-diff|cards/explain-diff.md|| -Clarify: `./references/clarify.md`. Use components/tokens, not hand CSS/coords. Run /unslop (or apply `./scripts/verify/rubrics/pass-copy.md`) on drafted copy; poster/video/brand/bespoke -> `./references/legacy-html.md`. +Clarify: `./references/clarify.md`. Use components/tokens, not hand CSS/coords. Run Unslop on drafted copy; do not emulate it with an embedded fallback. Delegated verification ownership: `./references/delegated-skills.md`. Visual judgment model policy: `./references/model-routing.md`. Poster/video/brand/bespoke -> `./references/legacy-html.md`. ## 6. Verify -Read `./references/verification.md`; run `node {{skill_dir}}/scripts/verify/ve-verify.mjs --json --screens ` (in this repo, `{{skill_dir}}` = `plugins/visual-explainer`); run the LLM passes including visual-tells; report the artifact path, report JSON, per-pass pass/fail, and an explicit disclosure if anything could not be verified. - -## Anti-Patterns (AI Slop) - -These patterns are explicitly forbidden. They signal "AI-generated template" and undermine the skill's purpose of producing distinctive, high-quality diagrams. Review every generated page against this list. Each ban includes the positive alternative to use instead. - -### Typography - -- Do not use Inter, Roboto, Arial, Helvetica, or `system-ui, sans-serif` alone as the primary `--font-body`; pick from the font pairings in `./references/libraries.md` or preserve a real project font stack. -- Cap each page at four named font families. Prefer one family in three or four weights over fake variety. -- Do not pair two fonts from the same class, such as two geometric sans or two humanist sans faces. Contrast on a real axis, such as serif plus sans or mono plus sans, or stay within one family. -- Do not reach for reflex webfonts such as Fraunces, Playfair Display, Cormorant, Lora, Syne, Space Grotesk, DM Sans/Serif, Outfit, Plus Jakarta Sans, or Instrument Sans/Serif as the voice face on a custom page. Pick against three brand-voice words from a real catalog; mono variants inside code are fine. -- Keep a committed type scale. Adjacent levels need at least a 1.25x difference carried by size plus weight or color; avoid 14/15/16px muddle and lone 500-vs-400 weight bumps. -- Set body and running prose at 16px or larger in `rem`. Never ship `user-scalable=no` or `maximum-scale=1`. -- Bound `clamp()` type: max/min should stay at or below about 2.5, and hero/display max should stay at or below about `6rem`. -- Add `0.05em` to `0.12em` tracking to all-caps labels unless a house aesthetic sets its own value. -- Load webfonts with `font-display: swap`; Google Fonts links must include `&display=swap`. - -### Color - -- Do not use indigo/violet defaults (`#8b5cf6`, `#7c3aed`, `#a78bfa`) or the cyan + magenta + pink neon set (`#06b6d4` -> `#d946ef` -> `#f472b6`). Build palettes from reference templates or from a real named theme. -- Do not default body or surface neutrals to warm cream/sand (`OKLCH` L `.84-.99`, C `<.06`, hue `40-100`, or tokens such as `--paper`, `--cream`, `--sand`, `--linen`) unless the aesthetic explicitly owns paper. Tint neutrals toward the real brand hue or stay chroma-0; carry warmth through accent, type, or imagery. -- Do not use indigo/blue around hue 250 or warm orange around hue 60 as the only accent without a brand reason. Choose accents that match the subject, theme, or data semantics. -- Cap decorative chrome at about four hue families. Chart series, syntax highlighting, and semantic status colors are exempt. -- Never put mid-gray text on a saturated background. Use the ink color at reduced alpha or a shade of the background hue. -- Verify body-text contrast at 4.5:1 or better, and large/bold text at 3:1 or better, against the real rendered background in both themes. -- Never encode meaning by color alone. Add a label, icon, shape, or texture; avoid legends made only of bare swatches. - -### Backgrounds & Effects - -- Do not use full-bleed violet-to-blue/cyan or purple-to-pink gradient washes as backgrounds. Use a solid brand color, a real image/chart, or an intentional non-default multi-stop gradient. -- Do not add ambient decorative layers such as blurred gradient orbs, particle canvases, floating dot grids, or multiple radial glows. Tie atmosphere to real content, once and restrained. -- Do not use gradient text on headings. Use actual type scale, weight, layout, and color contrast. -- Do not use glassmorphism (`backdrop-blur` plus translucent surface) as the default card/nav/modal treatment. Reserve one frosted moment over real imagery; otherwise use opaque surfaces with a hairline or shadow. -- Do not use animated glowing box-shadows or pulsing/breathing effects on static content. Use entrance reveals, hover feedback, or user-initiated transitions only. -- Do not use a colored one-sided `border-left` or `border-right` stripe as a card accent. Use a full hairline border, a background tint, or a leading glyph. -- Do not copy-paste one box-shadow onto every raised element. Use two or three elevation levels tied to importance, or rely on spacing and borders. - -### Layout & Structure - -- Do not center everything with uniform padding. Build hierarchy with visible differences in scale, alignment, density, and section rhythm. -- Do not style every card identically. Hero, primary, secondary, and reference material need different weight. -- Do not nest a card directly inside another card. Use spacing, a divider, or a subheading for subgrouping. -- Do not box every block by default. Render standalone paragraphs, images, and lone stats as plain flow content. -- Do not stamp out big-number plus small-label stat tiles unless the numbers are real, sourced data. Use narrative cards, tables, or diagrams when the values are illustrative. -- Do not dump a wall of eight or more undifferentiated bullets. Group the content into two to four labeled clusters, a table, or a diagram; flat glossaries and changelogs are exceptions. -- Do not append a tiny lowercase descriptor gloss to every item in a grid/list/catalog (`collapsible data`, `one canvas`, `titled region`). -- Name things once; add a descriptor only when it disambiguates a specific item. -- Derive spacing from one small scale such as `4/8/12/16/24/32/48/64`. Avoid scattered one-off values like `13px`, `17px`, and `22px`. -- Create rhythm: tight spacing within groups, generous spacing between sections. -- Vary section structure to match content type. Do not force a diff, timeline, and comparison into one grid mold. Break the grid once for a deliberate focal point. -- Avoid symmetric layouts where both halves mirror each other without a reason. - -### Motion - -- Reveal animations must enhance already-visible content. Never default content to `opacity: 0` gated on a JavaScript class; headless, PDF, and PNG exports can ship blank. -- Gate reveals behind CSS scroll timelines or provide reduced-motion and `noscript` fallbacks. -- Do not loop the same fade-and-rise on every section or choreograph header, sections, and footer on load. Pick one hero moment; stagger only siblings within a list. -- Use ease-out quart/quint/expo curves. Do not use bounce or elastic curves, including `cubic-bezier()` control points outside `[0,1]`. -- Animate `transform` and `opacity`, not `width`, `height`, `top`, `left`, or `margin`. -- Keep hover and press feedback at or below 300ms. -- Never write `outline: none` on a focusable element without a `:focus-visible` replacement. Give interactive controls at least a `44px` by `44px` hit area. -- Do not stage a spinner or skeleton in a static artifact that fetches nothing. - -### Iconography - -- Do not use emoji icons in section headers or inline body bullets. Use styled monospace labels, numbered badges, asymmetric section dividers, or inline SVG that matches the palette. -- Pick one icon system per role and use it uniformly. Do not mix emoji, inline SVG, and unicode glyphs for equivalent items. -- Do not repeat the same icon-in-rounded-box pattern for every section header. - -### Copy - -- Open with the claim. Cut throat-clearing such as "Here's the thing:", chatbot artifacts such as "Great question!" and "I hope this helps", and significance inflation such as "stands as a testament to". -- Do not let a heading restate its own next sentence. The first sentence after a heading must add a fact, number, or mechanism. -- Do not repeat a point across sections. Say it once in the strongest place. -- Define load-bearing jargon on first use. -- Keep one capitalization convention per heading level. -- Use zero or one em dash per section. Avoid colon-before-dramatic-reveal and manufactured "not just X, it's Y" parallelism. -- Run `/unslop` on drafted copy when available. If it is unavailable, apply the rubric in `./scripts/verify/rubrics/pass-copy.md` before writing prose into HTML. -- Code blocks should use a simple header with filename or language label, never three-dot window chrome. - -### The Slop Test - -Before delivering, apply this test: **Would a developer looking at this page immediately think "AI generated this"?** The telltale signs: - -1. Inter or Roboto font with purple/violet gradient accents -2. Every heading has `background-clip: text` gradient -3. Emoji icons leading every section -4. Glowing cards with animated shadows -5. Cyan-magenta-pink color scheme on dark background -6. Perfectly uniform card grid with no visual hierarchy -7. Three-dot code block chrome - -If two or more of these are present, the page is slop. Regenerate with a different aesthetic direction — Editorial, Blueprint, Paper/ink, or a specific IDE theme. These constrained aesthetics are harder to mess up because they have specific visual requirements that prevent defaulting to generic patterns. +Read `./references/verification.md`, `./references/delegated-skills.md`, and `./references/model-routing.md`; run `node {{skill_dir}}/scripts/verify/ve-verify.mjs --json --screens ` (in this repo, `{{skill_dir}}` = `plugins/visual-explainer`); run only the routed Artifacture passes and delegated Impeccable/Unslop checks. The main thread orchestrates and summarizes; it does not perform screenshot judgment. Dispatch each Artifacture-owned visual pass to the smallest eval-qualified model and exact batch size in the generated policy. If no qualified route can run, skip with `no-eval-qualified-model` instead of silently using the host/frontier model. Report the artifact path, report JSON, actual model/batch per pass, per-pass pass/fail, and an explicit disclosure if anything could not be verified. + +## Design-craft delegation + +Artifacture does not maintain a general AI-slop checklist. Use the ownership +contract in `./references/delegated-skills.md`: + +- Impeccable owns general visual craft, design specificity, typography, color, + generic decoration, and visual AI tells. +- Unslop owns prose patterns, cadence, voice, and AI-writing tells. +- Artifacture owns mechanical artifact checks and the explicit + `artifacture:slop-gap` pass. + +The Artifacture gap is limited to decoration that falsely implies sequence, +measured state/confidence, or provenance/verification. Do not expand that list +with generic taste rules. If Impeccable or Unslop is unavailable, disclose the +skipped delegated check instead of recreating its rubric here. diff --git a/plugins/visual-explainer/references/delegated-skills.md b/plugins/visual-explainer/references/delegated-skills.md new file mode 100644 index 0000000..300e91b --- /dev/null +++ b/plugins/visual-explainer/references/delegated-skills.md @@ -0,0 +1,90 @@ +# Delegated verification skills + +Artifacture is one layer in a verification system. It owns artifact mechanics, +evidence extraction, state coverage, and artifact-specific semantics. It routes +general visual craft to Impeccable and prose-pattern judgment to Unslop. + +The main thread is an orchestrator, not the default visual judge. Artifacture- +owned visual passes use the smallest eval-qualified model and batch size from +`./model-routing.md`. Impeccable and Unslop retain their own skill context and +model policy. + +## Ownership + +### Artifacture + +Artifacture owns: + +- runtime, request, viewport, and canvas failures; +- text, arrow, and component clipping; +- crowding, repeated-track symmetry, and intentional versus accidental dead + space; +- state enumeration and screenshot evidence; +- source-versus-render completeness; +- diagram and operating-model fidelity; and +- the curated artifact-semantic slop gap below. + +### Impeccable + +Impeccable owns general visual craft and generic visual AI tells, including +typography, color, gradients, glow, glass, nested cards, fake chrome, +decorative text, template scaffolding, aesthetic costume, ornament without +information, and manufactured hierarchy. + +Artifacture emits `impeccable:critique` with the relevant screenshot evidence +and candidate locations. Invoke the installed Impeccable skill in read-only +critique/audit mode. Do not copy its detector taxonomy or taste rules into +Artifacture. + +### Unslop + +Unslop owns prose patterns, cadence, voice, and AI-writing tells. + +Artifacture emits `unslop:cleanup-report` and supplies excluded-filtered prose +only. Invoke Unslop as `cleanup --report`. Verification is report-only: do not +rewrite text while determining whether the artifact passes. + +## Curated Artifacture gap + +`artifacture:slop-gap` is explicit opt-in for three artifact-semantic cases +that neither general visual craft nor prose linting can establish: + +1. `artifact-slop-false-sequence`: decorative numbering, connectors, or arrows + imply an order unsupported by the source. +2. `artifact-slop-false-state`: badges, meters, progress, or status ornament + imply measured state, confidence, or completion without evidence. +3. `artifact-slop-false-provenance`: citation-like, source-like, or + validation-like decoration implies provenance or verification the artifact + cannot support. + +Declare the pass only on a named region: + +```html +
...
+``` + +Supply one crop, its visible text, and the smallest source/truth excerpt needed +to judge the semantic claim. The pass must stay silent on typography, color, +generic decoration, prose cadence, clipping, crowding, symmetry, and dead +space. + +## Failure and cache behavior + +If a delegated skill is unavailable, disclose that its check was skipped. +Never emulate it with a local fallback rubric. + +Treat the three routes as separate request families: + +- Artifacture visual passes may share an exact screenshot prefix when their + evidence package, model, image order, detail level, tools, and shared + instructions are identical. Only the final criterion suffix should vary. +- Impeccable receives the relevant screenshots and candidate regions in its + own skill context. Its rubric is not appended to an Artifacture request. +- Unslop receives excluded-filtered prose in report-only mode without + screenshots or visual-rubric text. + +Legacy deterministic craft/prose detectors in `checks.json` are candidate +extractors, not Artifacture verdicts. A match has status +`delegated-candidate`, does not increment Artifacture errors or warnings, and +only causes the owning skill route to appear. The owning skill decides whether +the candidate is a real issue. diff --git a/plugins/visual-explainer/references/legacy-html.md b/plugins/visual-explainer/references/legacy-html.md index a9b47ac..7fa3afb 100644 --- a/plugins/visual-explainer/references/legacy-html.md +++ b/plugins/visual-explainer/references/legacy-html.md @@ -377,7 +377,7 @@ Before you render any prose into the page, **run the copy through the `/unslop` 1. Draft the full set of prose copy for the page as a plain-text block before writing HTML. 2. Invoke `/unslop` on that block. The skill runs its two-pass diagnosis-then-reconstruction and returns revised copy. -3. If `/unslop` is unavailable in the current surface (for example, Codex CLI), apply the embedded de-slop rubric in `../scripts/verify/rubrics/pass-copy.md` yourself, using the same instruction hierarchy as the verification protocol: deterministic checks first, then fresh-context copy judgment. +3. If Unslop is unavailable, disclose that the prose check was skipped. Do not emulate it with an Artifacture fallback rubric. 4. Paste the unslopped copy into the HTML template. Do not paraphrase it again or "polish" it further — `/unslop` already did that work, and re-editing reintroduces the patterns it removed. If you skip this step and your prose still reads as AI-generated (telltale phrases like "it's important to note", "let that sink in", "in today's fast-paced landscape", predictable three-item lists, transitional "however"s and "moreover"s), the output has failed the craft bar regardless of how good the visual design is. @@ -404,7 +404,7 @@ Exit `0` means no error-severity failures; warnings still require judgment. Exit Do not run LLM visual passes until `ve-verify` exits `0`. Use at most 3 deterministic repair cycles. If failures remain, deliver only with explicit unresolved check IDs and evidence. -After `ve-verify` passes, run the LLM passes in `./verification.md`: hierarchy, aesthetic, visual-tells, diagram, completeness, copy, and poster as required by the report/profile. Claude Code should dispatch the `ve-verifier-*` agents in parallel. Single-agent environments should run the same files in `../scripts/verify/rubrics/` sequentially. +After `ve-verify` passes, run only the Artifacture passes and delegated skills named by `./verification.md`. General visual craft routes to Impeccable, prose routes to Unslop report-only, and Artifacture's explicit slop-gap pass covers only false sequence, state/confidence, or provenance. Claude Code should dispatch compatible selected `ve-verifier-*` agents in parallel. Single-agent environments should run the same selected Artifacture rubric files sequentially. Each LLM pass returns only: @@ -689,101 +689,26 @@ The full protocol lives in `./verification.md`; the executable rubric questions The irreducible judgment checks are: - **Layout judgment:** hierarchy, visible clipping, fixed chrome obstruction, semantic table need, slide focal clarity, composition repetition, and sparse slide diagnosis. - **Aesthetic judgment:** active-preset fidelity, both-mode visual inversion, status-color meaning, moment-of-surprise, grid-break intent, red-accent intent, and demo embed fit. -- **Visual-tells judgment:** whether the visual choices communicate the source material through specific hierarchy, motifs, data emphasis, and format-appropriate tells. +- **Delegated Impeccable judgment:** general visual craft, design specificity, hierarchy, typography, color, and generic visual AI tells. - **Diagram judgment:** legend fidelity, focal dominance, proportional honesty, type coherence, removal simplicity, and whether a diagram is necessary. - **Completeness judgment:** source inventory mapped to rendered content, plus whether a demo adds motion/interaction value. -- **Copy judgment:** extracted prose passes `/unslop` standards. +- **Delegated Unslop judgment:** excluded-filtered prose passes `cleanup --report`. - **Poster judgment:** exported PNG preserves canvas fit, hierarchy, and the intended hero/moment-of-surprise. Deterministic checks cover file structure, console/runtime errors, placeholder leaks, body overflow, link/style contracts, Mermaid container mechanics, preset tokens, slide sizing, poster bounds, video/transcript rules, and other machine-checkable constraints. -## Anti-Patterns (AI Slop) - -These patterns are explicitly forbidden. They signal "AI-generated template" and undermine the skill's purpose of producing distinctive, high-quality diagrams. Review every generated page against this list. Each ban includes the positive alternative to use instead. - -### Typography - -- Do not use Inter, Roboto, Arial, Helvetica, or `system-ui, sans-serif` alone as the primary `--font-body`; pick from the font pairings in `./libraries.md` or preserve a real project font stack. -- Cap each page at four named font families. Prefer one family in three or four weights over fake variety. -- Do not pair two fonts from the same class, such as two geometric sans or two humanist sans faces. Contrast on a real axis, such as serif plus sans or mono plus sans, or stay within one family. -- Do not reach for reflex webfonts such as Fraunces, Playfair Display, Cormorant, Lora, Syne, Space Grotesk, DM Sans/Serif, Outfit, Plus Jakarta Sans, or Instrument Sans/Serif as the voice face on a custom page. Pick against three brand-voice words from a real catalog; mono variants inside code are fine. -- Keep a committed type scale. Adjacent levels need at least a 1.25x difference carried by size plus weight or color; avoid 14/15/16px muddle and lone 500-vs-400 weight bumps. -- Set body and running prose at 16px or larger in `rem`. Never ship `user-scalable=no` or `maximum-scale=1`. -- Bound `clamp()` type: max/min should stay at or below about 2.5, and hero/display max should stay at or below about `6rem`. -- Add `0.05em` to `0.12em` tracking to all-caps labels unless a house aesthetic sets its own value. -- Load webfonts with `font-display: swap`; Google Fonts links must include `&display=swap`. - -### Color - -- Do not use indigo/violet defaults (`#8b5cf6`, `#7c3aed`, `#a78bfa`) or the cyan + magenta + pink neon set (`#06b6d4` -> `#d946ef` -> `#f472b6`). Build palettes from reference templates or from a real named theme. -- Do not default body or surface neutrals to warm cream/sand (`OKLCH` L `.84-.99`, C `<.06`, hue `40-100`, or tokens such as `--paper`, `--cream`, `--sand`, `--linen`) unless the aesthetic explicitly owns paper. Tint neutrals toward the real brand hue or stay chroma-0; carry warmth through accent, type, or imagery. -- Do not use indigo/blue around hue 250 or warm orange around hue 60 as the only accent without a brand reason. Choose accents that match the subject, theme, or data semantics. -- Cap decorative chrome at about four hue families. Chart series, syntax highlighting, and semantic status colors are exempt. -- Never put mid-gray text on a saturated background. Use the ink color at reduced alpha or a shade of the background hue. -- Verify body-text contrast at 4.5:1 or better, and large/bold text at 3:1 or better, against the real rendered background in both themes. -- Never encode meaning by color alone. Add a label, icon, shape, or texture; avoid legends made only of bare swatches. - -### Backgrounds & Effects - -- Do not use full-bleed violet-to-blue/cyan or purple-to-pink gradient washes as backgrounds. Use a solid brand color, a real image/chart, or an intentional non-default multi-stop gradient. -- Do not add ambient decorative layers such as blurred gradient orbs, particle canvases, floating dot grids, or multiple radial glows. Tie atmosphere to real content, once and restrained. -- Do not use gradient text on headings. Use actual type scale, weight, layout, and color contrast. -- Do not use glassmorphism (`backdrop-blur` plus translucent surface) as the default card/nav/modal treatment. Reserve one frosted moment over real imagery; otherwise use opaque surfaces with a hairline or shadow. -- Do not use animated glowing box-shadows or pulsing/breathing effects on static content. Use entrance reveals, hover feedback, or user-initiated transitions only. -- Do not use a colored one-sided `border-left` or `border-right` stripe as a card accent. Use a full hairline border, a background tint, or a leading glyph. -- Do not copy-paste one box-shadow onto every raised element. Use two or three elevation levels tied to importance, or rely on spacing and borders. - -### Layout & Structure - -- Do not center everything with uniform padding. Build hierarchy with visible differences in scale, alignment, density, and section rhythm. -- Do not style every card identically. Hero, primary, secondary, and reference material need different weight. -- Do not nest a card directly inside another card. Use spacing, a divider, or a subheading for subgrouping. -- Do not box every block by default. Render standalone paragraphs, images, and lone stats as plain flow content. -- Do not stamp out big-number plus small-label stat tiles unless the numbers are real, sourced data. Use narrative cards, tables, or diagrams when the values are illustrative. -- Do not dump a wall of eight or more undifferentiated bullets. Group the content into two to four labeled clusters, a table, or a diagram; flat glossaries and changelogs are exceptions. -- Derive spacing from one small scale such as `4/8/12/16/24/32/48/64`. Avoid scattered one-off values like `13px`, `17px`, and `22px`. -- Create rhythm: tight spacing within groups, generous spacing between sections. -- Vary section structure to match content type. Do not force a diff, timeline, and comparison into one grid mold. Break the grid once for a deliberate focal point. -- Avoid symmetric layouts where both halves mirror each other without a reason. - -### Motion - -- Reveal animations must enhance already-visible content. Never default content to `opacity: 0` gated on a JavaScript class; headless, PDF, and PNG exports can ship blank. -- Gate reveals behind CSS scroll timelines or provide reduced-motion and `noscript` fallbacks. -- Do not loop the same fade-and-rise on every section or choreograph header, sections, and footer on load. Pick one hero moment; stagger only siblings within a list. -- Use ease-out quart/quint/expo curves. Do not use bounce or elastic curves, including `cubic-bezier()` control points outside `[0,1]`. -- Animate `transform` and `opacity`, not `width`, `height`, `top`, `left`, or `margin`. -- Keep hover and press feedback at or below 300ms. -- Never write `outline: none` on a focusable element without a `:focus-visible` replacement. Give interactive controls at least a `44px` by `44px` hit area. -- Do not stage a spinner or skeleton in a static artifact that fetches nothing. - -### Iconography - -- Do not use emoji icons in section headers or inline body bullets. Use styled monospace labels, numbered badges, asymmetric section dividers, or inline SVG that matches the palette. -- Pick one icon system per role and use it uniformly. Do not mix emoji, inline SVG, and unicode glyphs for equivalent items. -- Do not repeat the same icon-in-rounded-box pattern for every section header. - -### Copy - -- Open with the claim. Cut throat-clearing such as "Here's the thing:", chatbot artifacts such as "Great question!" and "I hope this helps", and significance inflation such as "stands as a testament to". -- Do not let a heading restate its own next sentence. The first sentence after a heading must add a fact, number, or mechanism. -- Do not repeat a point across sections. Say it once in the strongest place. -- Define load-bearing jargon on first use. -- Keep one capitalization convention per heading level. -- Use zero or one em dash per section. Avoid colon-before-dramatic-reveal and manufactured "not just X, it's Y" parallelism. -- Run `/unslop` on drafted copy when available. If it is unavailable, apply the rubric in `../scripts/verify/rubrics/pass-copy.md` before writing prose into HTML. -- Code blocks should use a simple header with filename or language label, never three-dot window chrome. - -### The Slop Test - -Before delivering, apply this test: **Would a developer looking at this page immediately think "AI generated this"?** The telltale signs: - -1. Inter or Roboto font with purple/violet gradient accents -2. Every heading has `background-clip: text` gradient -3. Emoji icons leading every section -4. Glowing cards with animated shadows -5. Cyan-magenta-pink color scheme on dark background -6. Perfectly uniform card grid with no visual hierarchy -7. Three-dot code block chrome - -If two or more of these are present, the page is slop. Regenerate with a different aesthetic direction — Editorial, Blueprint, Paper/ink, or a specific IDE theme. These constrained aesthetics are harder to mess up because they have specific visual requirements that prevent defaulting to generic patterns. +## Design-craft delegation + +Artifacture does not maintain a general AI-slop checklist. Use the ownership +contract in `./delegated-skills.md`: + +- Impeccable owns general visual craft, design specificity, typography, color, + generic decoration, and visual AI tells. +- Unslop owns prose patterns, cadence, voice, and AI-writing tells. +- Artifacture owns mechanical artifact checks and the explicit + `artifacture:slop-gap` pass. + +The Artifacture gap is limited to decoration that falsely implies sequence, +measured state/confidence, or provenance/verification. Do not expand that list +with generic taste rules. If Impeccable or Unslop is unavailable, disclose the +skipped delegated check instead of recreating its rubric here. diff --git a/plugins/visual-explainer/references/model-routing.md b/plugins/visual-explainer/references/model-routing.md new file mode 100644 index 0000000..0b35e00 --- /dev/null +++ b/plugins/visual-explainer/references/model-routing.md @@ -0,0 +1,71 @@ +# Visual judgment model routing + +Artifacture uses the smallest eval-qualified model for each routed visual pass. +The host/main agent orchestrates evidence collection and merges results; it does +not perform screenshot judgment merely because it is already in context. + +## Resolve the policy + +Use the first existing policy: + +1. `$ARTIFACTURE_VISUAL_MODEL_POLICY`; +2. `~/.artifacture/visual-model-policy.json`; or +3. `REPO/evals/visual-model-policy/policy.generated.json`. + +Each route names a pass, model, batch size, measured quality, and an escalation +chain. A route is valid only when it was generated by +`evals/visual-model-policy/select.mjs`. + +`ve-verify` resolves this policy and emits `llm_dispatch_plan` in `report.json`. +Hosts dispatch only entries whose status is `ready`; skipped entries are not +permission to substitute the host or main-thread model. + +If the selected pass has no route, return: + +```json +{ + "pass": null, + "status": "skipped", + "reason": "no-eval-qualified-model", + "findings": [] +} +``` + +Do not silently run the host model, a frontier model, or every visual check as a +fallback. + +## Runtime contract + +1. Dispatch only the selected check family and evidence package. +2. Use the policy's exact `batch_size`; do not opportunistically add screenshots. +3. Require verdict JSON with one image/state id and one region for every finding. +4. Allow one schema-only retry that does not add reasoning or examples. +5. Escalate to the next eval-qualified model only for a recorded trigger: + invalid JSON, explicit abstention, ungrounded evidence, provider failure, or + out-of-distribution input. +6. Record actual model, batch size, retry/escalation reason, latency, tokens, + cache reads, cost, and verdict. +7. Feed those records back into the eval dataset before changing the policy. + +Self-reported confidence is not a quality metric and does not justify escalation +by itself. + +## Main-thread boundary + +The main thread may inspect structured findings to merge duplicates, preserve +ownership, and explain the result. It must not reopen screenshots and replace a +small model's verdict with an unmeasured large-model critique. When a result is +ambiguous, mark it as such or trigger the policy's explicit escalation path. + +## Delegated skills + +`impeccable:critique` and `unslop:cleanup-report` remain independent skill +families. They own their prompts and model policies. Artifacture supplies only +the routed evidence and records which skill/model actually ran. It does not +flatten their rubrics into its own prompt to gain cache reuse. + +## Calibrating a new route + +Follow `REPO/evals/visual-model-policy/README.md`. Test model and screenshot +batch size together; a model qualified at batch size 1 is not automatically +qualified at 2, 4, or 8. diff --git a/plugins/visual-explainer/references/verification.md b/plugins/visual-explainer/references/verification.md index 64e45a0..51edf5a 100644 --- a/plugins/visual-explainer/references/verification.md +++ b/plugins/visual-explainer/references/verification.md @@ -40,7 +40,21 @@ The report JSON contains: } ], "screenshots": ["..."], - "llm_passes_required": ["hierarchy", "aesthetic-mono-industrial", "completeness", "copy", "visual-tells"] + "llm_passes_required": ["hierarchy", "impeccable:critique"], + "llm_dispatch_plan": [ + { + "pass": "hierarchy", + "owner": "artifacture", + "status": "ready", + "model": "provider-small-vision", + "batch_size": 2 + }, + { + "pass": "impeccable:critique", + "owner": "impeccable", + "status": "delegate-to-installed-skill" + } + ] } ``` @@ -48,9 +62,11 @@ The browser stage renders the required matrix for the detected profile. Page, sl ### Producing pass inputs -In single-agent mode, YOU produce the declared inputs before running each rubric. +The orchestrator produces the declared inputs before dispatching each rubric. +Input extraction and screenshot capture may run in the main thread; visual +judgment may not. -1. Extract prose text for P-copy with the exclusion filter in that rubric: remove code, identifiers, filenames, table headers, labels, counters, timestamps, status strings, Mermaid labels, and square-bracket system messages. A documented Node one-liner is acceptable, for example: +1. Extract prose text for delegated Unslop review: remove code, identifiers, filenames, table headers, labels, counters, timestamps, status strings, Mermaid labels, and square-bracket system messages. A documented Node one-liner is acceptable, for example: ```bash node -e "let s=require('fs').readFileSync(process.argv[1],'utf8'); s=s.replace(/<(script|style|pre|code|svg)\b[\s\S]*?<\/\1>/gi,' '); s=s.replace(/\[[^\]]*\]/g,' ').replace(/<[^>]+>/g,' '); console.log(s.replace(/\s+/g,' ').trim())" @@ -60,46 +76,95 @@ In single-agent mode, YOU produce the declared inputs before running each rubric 3. Capture P-diagram inputs one figure at a time. Scroll to each element carrying `data-diagram-role` or `.mermaid`; screenshot that element's bounding region; extract its visible labels; write a one-line content brief for the figure. 4. For P-operating-model, build a review map with one row per slide or coherent long-form section: stable unit id, unit type (`slide` or `section`), visible title, one-sentence narrative job, and screenshot path. If the source has no narrative job, use the visible claim and mark the route low-confidence. Do not use implementation source to infer intent. 5. Use candidate element lists from each deterministic check's `evidence` and `where` fields when a pass asks for candidate extracts. +6. For `impeccable:critique`, capture only the candidate screenshots and locations named by the report. For `unslop:cleanup-report`, supply only the excluded-filtered prose extract. +7. For `artifacture:slop-gap`, capture only the explicitly nominated region, its visible text, and the smallest source/truth excerpt needed to judge sequence, state/confidence, or provenance. ## 2. Run LLM Passes Run each required pass as a separate context. Use only the inputs named by the pass. Do not let screenshots, source text, or rubric sections leak across passes. -Claude Code: spawn the matching `ve-verifier-*` agents in one parallel Task batch. Each agent reads one rubric and returns only verdict JSON. - -Single-agent environments such as Codex CLI: run the same rubric files sequentially. Run rubrics in order; for each, load only that rubric file and its declared inputs; do not carry prior rubrics' questions, screenshots, or findings forward; emit the verdict JSON, then proceed. The files in `{{skill_dir}}/scripts/verify/rubrics/` are the single source of truth for both execution modes. +Before dispatch, read `./model-routing.md` and consume the report's +`llm_dispatch_plan`. Use the exact model and batch size recorded for every +`ready` Artifacture pass. The current/main agent must not inspect the +screenshots itself merely because a subagent or direct model route is +inconvenient. + +Prefer direct model API calls when controlled model selection, cost telemetry, +or cache measurement matters. Coding-agent subagents are an orchestration +fallback only when they can actually target the selected model. If the host +cannot run the policy's model, mark the pass skipped with +`no-eval-qualified-model`; do not silently substitute the host/frontier model. + +Invoke Impeccable and Unslop through their installed skills, not copied +Artifacture agents. Load only the selected rubric/skill and declared inputs; do +not carry prior questions, screenshots, or findings forward. Artifacture rubric +files are authoritative only for Artifacture-owned passes. + +### Cache-friendly visual dispatch + +When multiple selected criteria consume the exact same image set, build one +immutable evidence prefix: fixed tools and verdict contract, shared evidence +instructions, optional design-system exception excerpt, then exact image blocks +in stable state-id order. Append selected check ids and rubric questions only +after that prefix. + +Do not claim prefix-cache savings from coding-agent subagents without provider +usage telemetry. Claude Code named subagents use separate caches; forks share +the parent cache but inherit its model. Codex can reuse exact prefixes, but +model, tools, sandbox, approval, or working-directory changes can invalidate +them and its subagent protocol does not expose cache token counts. Use the +direct-API experiment in `docs/plans/visual-eval-prefix-caching.md` when measured +cache reads and controlled small-model selection are required. Every pass returns: ```json { "pass": true, + "execution": { + "model": "provider/model", + "batch_size": 2, + "policy_source": "~/.artifacture/visual-model-policy.json", + "escalated_from": null + }, "findings": [ - { "check_id": "text-visibly-clipped", "evidence": "390-dark screenshot: heading is cut at right edge", "fix": "Allow wrapping or widen the container, then rerun verification." } + { "check_id": "text-visibly-clipped", "state_id": "390-dark", "region": "hero heading", "evidence": "Heading is cut at right edge", "fix": "Allow wrapping or widen the container, then rerun verification." } ] } ``` +When no eval-qualified route can run: + +```json +{ + "pass": null, + "status": "skipped", + "reason": "no-eval-qualified-model", + "findings": [] +} +``` + Use these passes: | Pass | Run When | Inputs | Rubric | |---|---|---|---| -| P-layout | `llm_passes_required` includes `hierarchy`, or any layout candidate exists | `report.json`; the 4 standard screenshots; candidate lists for clipping, fixed chrome, div-grid tables, slide screenshots, demo frames when referenced | `{{skill_dir}}/scripts/verify/rubrics/pass-layout.md` | +| P-layout | `llm_passes_required` includes `hierarchy`, or any layout candidate exists | `report.json`; the 4 standard screenshots; candidate lists for clipping, fixed chrome, div-grid tables, repeated-track layouts, slide screenshots, demo frames when referenced | `{{skill_dir}}/scripts/verify/rubrics/pass-layout.md` | | P-aesthetic | `llm_passes_required` includes any `aesthetic-*` token, a preset is detected or declared, or any preset candidate exists | `report.json`; active preset name; light/dark screenshots; candidate extracts named in the report | `{{skill_dir}}/scripts/verify/rubrics/pass-aesthetic.md` | | P-diagram | diagrams are present | `report.json`; one screenshot per figure; extracted diagram labels; one-line content brief for each figure | `{{skill_dir}}/scripts/verify/rubrics/pass-diagram.md` | | P-operating-model | `llm_passes_required` includes `operating-model` | one screenshot per review unit; review map with unit id, unit type, visible title, and one-sentence narrative job; only the brief excerpts needed to resolve intent | `{{skill_dir}}/scripts/verify/rubrics/pass-operating-model.md` | | P-completeness | source material or demo evidence exists | source inventory; extracted rendered headings, bullets, table rows, cards, and demo-frame summary; no page screenshots | `{{skill_dir}}/scripts/verify/rubrics/pass-completeness.md` | -| P-copy | extracted prose exists | excluded-filtered prose text only | `{{skill_dir}}/scripts/verify/rubrics/pass-copy.md` | -| P-visual-tells | `llm_passes_required` includes `visual-tells` (page/slides/magazine) | `report.json`; light/dark screenshots; candidate extracts | `{{skill_dir}}/scripts/verify/rubrics/pass-visual-tells.md` | +| D-Impeccable | `llm_passes_required` includes `impeccable:critique` | candidate screenshots and locations from `report.json`; relevant design-system excerpt only | installed Impeccable skill; read-only critique/audit | +| D-Unslop | `llm_passes_required` includes `unslop:cleanup-report` | excluded-filtered prose text only | installed Unslop skill; `cleanup --report` | +| P-artifact-slop-gap | `llm_passes_required` includes `artifacture:slop-gap` | one nominated screenshot/crop; visible text; smallest source/truth excerpt needed to judge the claim | `{{skill_dir}}/scripts/verify/rubrics/pass-artifact-slop-gap.md` | | P-poster | profile is `poster` | exported PNG only | `{{skill_dir}}/scripts/verify/rubrics/pass-poster.md` | ### P-layout Questions -Ask the questions tagged in `pass-layout.md`: semantic table need, global hierarchy, visible text clipping, mobile fixed-chrome obstruction, slide focal clarity, repeated slide composition, sparse diagram slide. +Ask only the applicable questions tagged in `pass-layout.md`: semantic table need, global hierarchy, visible text clipping, mobile fixed-chrome obstruction, slide focal clarity, repeated-track symmetry, repeated slide composition, and sparse diagram slide. Repeated-track symmetry is conditional: run it only when the artifact visibly establishes equivalent columns or rows. ### P-aesthetic Questions -Load only the active preset section in `pass-aesthetic.md`. Do not judge Nothing rules on a Mono-Industrial page or generic custom pages against a named preset. Ask the generic both-mode question for any named preset. +Load only the active named-preset section in `pass-aesthetic.md`. Do not judge generic custom pages against a named preset; general visual craft and visual AI tells route to Impeccable. ### P-diagram Questions @@ -123,9 +188,23 @@ or misstates the unit's narrative job. Compare source inventory to extracted rendered content. Do not use screenshots. Missing source sections, decision cards, table rows, collapsible details, footnotes, or demonstrably low-value demo embeds fail this pass. -### P-copy Questions +### Delegated Impeccable Questions + +Invoke the installed Impeccable skill in read-only critique/audit mode with only +the routed screenshot evidence. Do not paste or paraphrase Impeccable's rubric +into Artifacture. + +### Delegated Unslop Questions + +Invoke Unslop as `cleanup --report` on excluded-filtered prose only. Do not +penalize identifiers, labels, counters, timestamps, status strings, Mermaid +labels, or system messages. Do not rewrite prose during verification. + +### P-artifact-slop-gap Questions -Judge only extracted prose. Do not penalize code, identifiers, filenames, table headers, labels, counters, timestamps, status strings, Mermaid labels, or square-bracket system messages. +Run only for explicit `data-ve-checks="artifacture:slop-gap"` regions. Ask +whether decoration falsely implies sequence, measured state/confidence, or +provenance/verification. Stay silent on every generic aesthetic or prose issue. ### P-poster Questions @@ -133,7 +212,7 @@ Inspect the exact exported PNG after every `poster export`. Check edge clipping, ## 3. Merge Verdicts And Re-Fix -1. Merge all pass verdicts into one table: pass name, `pass` boolean, finding count, check IDs. +1. Merge Artifacture and delegated verdicts into one table: pass/skill name, `pass` boolean, finding count, check IDs. 2. If every pass returns `"pass": true`, continue to Step 4. 3. If any pass returns `"pass": false`, fix only the defects named in `findings`. 4. Re-export from source. Never hand-edit generated HTML when an MDX/TSX source exists. @@ -147,7 +226,7 @@ The delivery message must include: 1. The artifact path. 2. The `ve-verify` report path. -3. A pass/fail line for each LLM pass that ran. +3. A pass/fail line for every Artifacture pass and delegated skill that ran. 4. One of these exact disclosure shapes: - `Verified: ve-verify passed and required LLM verification passes passed.` - `Could not fully verify: .` @@ -157,7 +236,7 @@ Never imply browser or visual verification happened when it did not. ## Process Boundaries -- Clarify-tier gates are defined in `{{skill_dir}}/references/clarify.md`; apply them before generation and do not duplicate them here. +- Delegated ownership is defined in `{{skill_dir}}/references/delegated-skills.md`; apply it without copying Impeccable or Unslop rubrics. Clarify-tier gates are defined in `{{skill_dir}}/references/clarify.md`. - Slide, magazine, poster, and video formats are opt-in only. Do not choose them without an explicit user request or flag. - For pages with 3 or more sections, use the fan-out policy and section retry limit in `{{skill_dir}}/references/section-contract.md`. - For video, run the Hyperframes workflow in order: doctor, build, lint, validate, draft render, extract 3 meaningful keyframes, show the user, wait for explicit approval, then final render. Reject invalid `--fps`, `--quality`, and `--aspect` flags before rendering. diff --git a/plugins/visual-explainer/scripts/verify/SPEC.md b/plugins/visual-explainer/scripts/verify/SPEC.md index b4cf08e..917afa6 100644 --- a/plugins/visual-explainer/scripts/verify/SPEC.md +++ b/plugins/visual-explainer/scripts/verify/SPEC.md @@ -21,7 +21,7 @@ plugins/visual-explainer/scripts/verify/ pass-hierarchy.md pass-aesthetic-.md # one per preset; verifier loads ONLY the active preset's file pass-completeness.md - pass-copy.md + pass-artifact-slop-gap.md # explicit semantic gap only; Impeccable/Unslop stay external evals/ fixtures/violations/.html # exactly one seeded violation each fixtures/clean/.html # must pass everything (false-positive guard) @@ -54,10 +54,21 @@ node plugins/visual-explainer/scripts/verify/ve-verify.mjs \ "status": "pass|fail|warn|skip", "evidence": "...", "where": "selector/line/screenshot ref", "fix_hint": "..." } ], "screenshots": ["..."], - "llm_passes_required": ["hierarchy", "aesthetic-mono-industrial", "completeness", "copy"] + "llm_passes_required": ["hierarchy", "aesthetic-mono-industrial", "completeness", "impeccable:critique", "unslop:cleanup-report", "artifacture:slop-gap"], + "llm_dispatch_plan": [ + {"pass":"hierarchy","owner":"artifacture","status":"ready","model":"provider-small-vision","batch_size":2}, + {"pass":"impeccable:critique","owner":"impeccable","status":"delegate-to-installed-skill"} + ] } ``` +`llm_dispatch_plan` is the executable routing boundary. Artifacture-owned +passes are `ready` only when the resolved eval policy contains a qualified +route; otherwise they are explicitly `skipped` with +`reason: "no-eval-qualified-model"`. A host must not replace a skipped route +with its current/main-thread model. Companion-skill entries retain their own +skill and model policies. + - Human output: compact table, failures first, each with fix_hint quoting the doc rule. ## Profile & preset auto-detection (lib/profile.mjs) @@ -87,15 +98,18 @@ Sequenced so a weak model cannot skip or blend steps: 1. Run ve-verify. If exit 1 → fix root cause → re-run. Max 3 cycles, then deliver with explicit failure disclosure. (Deterministic gate FIRST; no LLM review of pages that fail mechanics.) -2. LLM passes, each a separate small context consuming ONLY its named inputs: +2. LLM passes and delegated skills, each a separate small context consuming ONLY its named inputs: - P1 hierarchy/layout: 4 screenshots + pass-hierarchy.md (squint, weight, moment-of-surprise) - P2 aesthetic fidelity: 4 screenshots + the ACTIVE preset rubric only (swap test, motif rules) - P3 completeness: source material inventory + extracted section list/headings (NO screenshots) - - P4 copy: extracted prose text only (slop patterns; unslop escalation) - Claude Code: 4 parallel Task subagents (`.claude/agents/ve-verifier-*.md`, JSON verdict contract - like section-contract.md). Codex/single-agent: same rubric files run as 4 sequential - fresh-context passes. Rubric files are the single source of truth for both paths. -3. Verdict merge: any P1–P4 fail → fix → re-run affected pass only. Max 2 cycles. + - D1 visual craft: candidate screenshots routed to installed Impeccable critique/audit + - D2 prose: excluded-filtered prose routed to Unslop `cleanup --report` + - P4 artifact slop gap: an explicitly nominated crop plus truth excerpt, limited to false + sequence, state/confidence, or provenance + Claude Code: parallelize compatible Artifacture `ve-verifier-*` agents. Invoke Impeccable and + Unslop through their installed skills. Codex/single-agent: keep every pass/skill in a fresh + context. Artifacture rubric files are authoritative only for Artifacture-owned paths. +3. Verdict merge: any local or delegated check fails → fix → re-run only that path. Max 2 cycles. 4. Delivery message MUST include: report path, pass/fail per LLM pass, and either "verified" or the explicit could-not-verify disclosure. (Protocol requirement, reviewable from transcript.) @@ -104,7 +118,7 @@ Sequenced so a weak model cannot skip or blend steps: - SKILL.md §6 rewritten (shorter than today: point at ve-verify + verification.md). - references/verification.md: full protocol incl. rubrics index + report schema. - references/*.md: add rule-ID anchors where checks cite docs (only where curation demands). -- .claude/agents/ve-verifier-{hierarchy,aesthetic,completeness,copy}.md +- .claude/agents/ve-verifier-{layout,aesthetic,completeness,artifact-slop-gap}.md - check.mjs: keep as build smoke test; add `ve:verify` + `ve:eval` npm scripts (documented as direct-node invocations too, given broken npm shim). diff --git a/plugins/visual-explainer/scripts/verify/checks.json b/plugins/visual-explainer/scripts/verify/checks.json index 311f02a..838456d 100644 --- a/plugins/visual-explainer/scripts/verify/checks.json +++ b/plugins/visual-explainer/scripts/verify/checks.json @@ -1,7 +1,7 @@ { "version": 2, "generated": "2026-07-04", - "note": "v2: AI-tell/copy checks from impeccable+unslop distillation. Stage counts: {\"static-text\":70,\"static-dom\":69,\"llm-pass\":30,\"browser\":37,\"transcript\":2}", + "note": "Legacy deterministic craft/prose detectors are candidate extractors only: matches route to Impeccable or Unslop and do not fail Artifacture. Artifacture owns mechanics, evidence, and artifact semantics. Stage counts: {\"static-text\":70,\"static-dom\":69,\"llm-pass\":30,\"browser\":37,\"transcript\":2}", "checks": [ { "family": "F1_slop_global", @@ -424,7 +424,7 @@ "stage": "llm-pass", "severity": "warn", "applies_when": "Page has prose (same excluded-filter set as unslop-prose-phrases).", - "spec": "Concatenate the excluded-filtered prose text as the artifact slice. Rubric question: 'Beyond fixed phrases, does this prose show AI-slop stylistic tells — manufactured emphasis, predictable three-item list rhythm, hollow however/moreover transitions, empty summary sentences — that /unslop would rewrite?' YES = warn.", + "spec": "delegate: unslop:cleanup-report. Supply excluded-filtered prose only and use Unslop's installed report-only audit; Artifacture does not reproduce the prose rubric.", "fixture_violation": "Prose with a forced three-item list plus hollow transition rhythm that no fixed phrase catches.", "fp_guards": "Terse technical labels/captions excluded; judge only sustained prose, not one-line UI copy.", "source_rules": [ @@ -4069,7 +4069,7 @@ "stage": "llm-pass", "severity": "warn", "applies_when": "Page uses two different (non-code) font families for heading vs body.", - "spec": "pass: pass-visual-tells. Rubric question: 'Do the heading and body fonts fall in the same typographic classification (both geometric sans, both humanist sans, both transitional serif — e.g. Poppins+Montserrat, Inter+Manrope, DM Sans+Roboto) and provide no real contrast axis?' YES = warn. One family in multiple weights, or a genuine contrast pairing (serif+sans, mono+sans, condensed display+wide body), passes.", + "spec": "delegate: impeccable:critique. Supply the relevant screenshot and font-family evidence; Impeccable owns the visual-craft judgment.", "fixture_violation": "Heading in Poppins, body in Montserrat — two near-identical geometric sans presented as a 'pairing'.", "fp_guards": "A single family across weights is fine. Only flag two DIFFERENT families in the same bucket with no discernible contrast; genuine cross-axis pairings pass.", "source_rules": [ @@ -4089,7 +4089,7 @@ "stage": "llm-pass", "severity": "warn", "applies_when": "Page has a single dominant saturated accent color and no documented brand palette in source.", - "spec": "pass: pass-visual-tells. Rubric question: 'Is the only meaningfully-saturated accent hue a reflexive default — indigo/blue near hue 245–265 or warm orange near hue 45–65 — reached for without any brand reason, rather than a color justified by the artifact's actual subject or brand?' YES = warn.", + "spec": "delegate: impeccable:critique. Supply the relevant screenshot, palette evidence, and brand exception excerpt; Impeccable owns the visual-craft judgment.", "fixture_violation": "A page whose sole accent is #6366f1 indigo with no brand rationale.", "fp_guards": "If source material establishes blue/orange as a real brand identity (existing logo/brand assets), it is not a reflex — pass. Judgment call, warn only.", "source_rules": [ @@ -4108,7 +4108,7 @@ "stage": "llm-pass", "severity": "warn", "applies_when": "The artifact has slide titles or section headings that do narrative work. Covers, literal dividers, agendas, quotes, and reference labels are reviewed under the exception rules.", - "spec": "pass: pass-copy. For each narrative title, ask whether it names the concrete test, mechanism, result, decision, action, or completion criterion the unit earns. Warn on calendar/program containers made to deliver outcomes ('Week 1 ends with a calibrated eval'), tool/data/score/package personification ('the judge lies', 'the score hides', 'the package outlives'), abstract uplift transformations ('taste becomes infrastructure'), journey/paradigm/pillars framing, generic topic handles when a stronger claim is present, or counting titles that describe deck structure instead of the point.", + "spec": "delegate: unslop:cleanup-report. Supply excluded-filtered narrative titles with their adjacent prose; Artifacture records the applicability signal but does not reproduce the prose rubric.", "fixture_violation": "A slide titled 'Week 1 ends with a calibrated eval' when the body actually defines the completion evidence: a clean package run with judge agreement recorded.", "fp_guards": "Do not ban inanimate subjects categorically. Accurate causal mechanisms ('Examples calibrate the judge', 'The gate fails the build'), literal temporal facts ('Week 1 ends on Friday'), exact technical noun phrases, short questions that state the decision, and cover/divider/agenda labels may pass. Judge title function against the unit's narrative job, not a fixed grammar template.", "source_rules": [ @@ -4129,7 +4129,7 @@ "stage": "llm-pass", "severity": "warn", "applies_when": "Page has section headings each followed by prose.", - "spec": "pass: pass-copy. Rubric question: 'Does any section heading merely rephrase its own first sentence with no added fact/number/mechanism, or is the same explanatory point repeated across two or more sections?' YES = warn. A heading that just names its topic ('Architecture' over an architecture diagram) is fine — flag same information content restated, not a shared topic word.", + "spec": "delegate: unslop:cleanup-report. Supply excluded-filtered headings with adjacent prose; Artifacture records the applicability signal but does not reproduce the prose rubric.", "fixture_violation": "Heading 'Faster deploys' followed by 'Deploys are now faster.' with no new specifics.", "fp_guards": "A topic sentence that states the claim AND immediately substantiates it with a number/mechanism passes. Only zero-added-specificity restatement or cross-section duplication fires here; a generic topic handle may still be flagged separately by title-claim-function.", "source_rules": [ @@ -4149,7 +4149,7 @@ "stage": "llm-pass", "severity": "warn", "applies_when": "Page is an explainer aimed at a broader/onboarding audience (not an internal team-only reference).", - "spec": "pass: pass-copy. Rubric question: 'Does the document rely on 2+ acronyms or internal system names that are load-bearing for understanding the diagram yet never expanded on first use (no parenthetical, no glossary, no title attribute) anywhere?' YES = warn.", + "spec": "delegate: unslop:cleanup-report. Supply excluded-filtered prose plus the intended-audience note; Artifacture records the applicability signal but does not reproduce the prose rubric.", "fixture_violation": "An explainer using 'the CRDT merges via the WAL' with neither term ever expanded.", "fp_guards": "Terms the explicit target audience already knows (an internal architecture doc for the team that coined them) are exempt — only flag when the artifact reads as onboarding/broad audience or the term is never defined anywhere.", "source_rules": [ diff --git a/plugins/visual-explainer/scripts/verify/lib/engine.mjs b/plugins/visual-explainer/scripts/verify/lib/engine.mjs index 4404590..9906467 100644 --- a/plugins/visual-explainer/scripts/verify/lib/engine.mjs +++ b/plugins/visual-explainer/scripts/verify/lib/engine.mjs @@ -2,7 +2,7 @@ import fs from 'node:fs/promises'; import path from 'node:path'; import { fileURLToPath } from 'node:url'; import { buildContext } from './context.mjs'; -import { buildReport } from './report.mjs'; +import { buildReport, delegatedOwnerForCheck } from './report.mjs'; import { checks as staticTextChecks } from './checks/static-text.mjs'; import { checks as staticDomChecks } from './checks/static-dom.mjs'; @@ -65,10 +65,14 @@ async function runOne(definition, ctx, registries, options) { const actionable = results.filter(Boolean).filter((result) => result.status !== 'pass'); if (actionable.length === 0) return { ...base, status: 'pass', evidence: 'ok' }; const first = actionable[0]; - const status = first.status === 'warn' || definition.severity === 'warn' ? 'warn' : 'fail'; + const delegatedOwner = delegatedOwnerForCheck(definition.id); + const status = delegatedOwner + ? 'delegated-candidate' + : first.status === 'warn' || definition.severity === 'warn' ? 'warn' : 'fail'; return { ...base, status, + delegated_owner: delegatedOwner || undefined, evidence: first.evidence || definition.title || 'violation', where: first.where || '', fix_hint: first.fix_hint || definition.spec || definition.title || '', diff --git a/plugins/visual-explainer/scripts/verify/lib/model-policy.mjs b/plugins/visual-explainer/scripts/verify/lib/model-policy.mjs new file mode 100644 index 0000000..22c45a8 --- /dev/null +++ b/plugins/visual-explainer/scripts/verify/lib/model-policy.mjs @@ -0,0 +1,122 @@ +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const REPO_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../../..'); + +const PASS_ALIASES = Object.freeze({ + hierarchy: 'layout', + 'artifacture:slop-gap': 'artifact-slop-gap', +}); + +const ALLOWED_ESCALATION_TRIGGERS = new Set([ + 'invalid-json-after-one-retry', + 'explicit-abstain', + 'evidence-not-attributable-to-one-image-and-region', + 'provider-error-or-timeout-after-one-retry', + 'out-of-distribution-input', +]); + +export function resolveVisualModelPolicy({ + env = process.env, + homeDir = os.homedir(), + repoRoot = REPO_ROOT, +} = {}) { + const candidates = [ + env.ARTIFACTURE_VISUAL_MODEL_POLICY, + path.join(homeDir, '.artifacture', 'visual-model-policy.json'), + path.join(repoRoot, 'evals', 'visual-model-policy', 'policy.generated.json'), + ].filter(Boolean); + + for (const candidate of candidates) { + if (!fs.existsSync(candidate)) continue; + const policy = JSON.parse(fs.readFileSync(candidate, 'utf8')); + validateGeneratedPolicy(policy, candidate); + return { source: path.resolve(candidate), policy }; + } + return { source: null, policy: null }; +} + +export function buildLlmDispatchPlan(passes, resolved = resolveVisualModelPolicy()) { + return passes.map((pass) => { + if (pass === 'impeccable:critique') { + return { pass, owner: 'impeccable', status: 'delegate-to-installed-skill' }; + } + if (pass === 'unslop:cleanup-report') { + return { pass, owner: 'unslop', status: 'delegate-to-installed-skill' }; + } + + const routeKey = routeKeyFor(pass); + const route = resolved.policy?.routes?.[routeKey]; + if (!route) { + return { + pass, + owner: 'artifacture', + status: 'skipped', + reason: 'no-eval-qualified-model', + policy_source: resolved.source, + }; + } + return { + pass, + owner: 'artifacture', + status: 'ready', + policy_source: resolved.source, + route_key: routeKey, + model: route.model, + model_class: route.model_class || null, + provider: route.provider || null, + batch_size: route.batch_size, + escalation_chain: route.escalation_chain || [], + escalation_triggers: route.escalation_triggers || [], + }; + }); +} + +function routeKeyFor(pass) { + return PASS_ALIASES[pass] || pass; +} + +function validateGeneratedPolicy(policy, source) { + if ( + policy.schema_version !== 1 || + policy.policy !== 'smallest-eval-qualified-model-first' || + !/^[a-f0-9]{64}$/i.test(policy.source_sha256 || '') || + !policy.thresholds || + !policy.routes || + typeof policy.routes !== 'object' + ) { + throw new Error(`invalid or non-selector visual model policy: ${source}`); + } + for (const [pass, route] of Object.entries(policy.routes)) { + if ( + !pass || + !route?.model || + !Number.isInteger(Number(route.batch_size)) || + Number(route.batch_size) <= 0 || + !route.metrics || + !route.evidence || + !Array.isArray(route.escalation_chain) || + !Array.isArray(route.escalation_triggers) + ) { + throw new Error(`invalid route ${pass} in visual model policy: ${source}`); + } + for (const metric of [ + 'precision', 'recall', 'silence_accuracy', 'grounding_accuracy', + 'json_validity', 'abstention_rate', 'cost_per_case_usd', 'p95_latency_ms', + ]) { + if (!Number.isFinite(Number(route.metrics[metric]))) { + throw new Error(`route ${pass} lacks measured ${metric}: ${source}`); + } + } + for (const field of ['runs', 'cases', 'positives', 'negatives']) { + if (!Number.isInteger(Number(route.evidence[field])) || Number(route.evidence[field]) <= 0) { + throw new Error(`route ${pass} lacks positive evidence ${field}: ${source}`); + } + } + if (route.escalation_triggers.some((trigger) => !ALLOWED_ESCALATION_TRIGGERS.has(trigger))) { + throw new Error(`route ${pass} contains an unapproved escalation trigger: ${source}`); + } + } +} diff --git a/plugins/visual-explainer/scripts/verify/lib/model-policy.test.mjs b/plugins/visual-explainer/scripts/verify/lib/model-policy.test.mjs new file mode 100644 index 0000000..c812917 --- /dev/null +++ b/plugins/visual-explainer/scripts/verify/lib/model-policy.test.mjs @@ -0,0 +1,113 @@ +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import test from 'node:test'; +import { + buildLlmDispatchPlan, + resolveVisualModelPolicy, +} from './model-policy.mjs'; + +function generatedPolicy(routes) { + return { + schema_version: 1, + policy: 'smallest-eval-qualified-model-first', + source_sha256: 'a'.repeat(64), + thresholds: { recall: 0.9 }, + routes, + }; +} + +function measuredRoute(overrides = {}) { + return { + model: 'tiny-vision', + batch_size: 2, + metrics: { + precision: 0.95, + recall: 0.95, + silence_accuracy: 0.96, + grounding_accuracy: 0.97, + json_validity: 1, + abstention_rate: 0, + cost_per_case_usd: 0.001, + p95_latency_ms: 5000, + }, + evidence: { runs: 3, cases: 100, positives: 40, negatives: 60 }, + escalation_chain: [], + escalation_triggers: ['explicit-abstain'], + ...overrides, + }; +} + +test('resolves the explicit policy before home and repository defaults', () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'artifacture-policy-')); + const explicit = path.join(root, 'explicit.json'); + fs.writeFileSync(explicit, JSON.stringify(generatedPolicy({ + layout: measuredRoute(), + }))); + const resolved = resolveVisualModelPolicy({ + env: { ARTIFACTURE_VISUAL_MODEL_POLICY: explicit }, + homeDir: path.join(root, 'home'), + repoRoot: path.join(root, 'repo'), + }); + assert.equal(resolved.source, explicit); + assert.equal(resolved.policy.routes.layout.model, 'tiny-vision'); + fs.rmSync(root, { recursive: true, force: true }); +}); + +test('builds ready Artifacture routes and independent companion routes', () => { + const plan = buildLlmDispatchPlan( + ['hierarchy', 'impeccable:critique', 'unslop:cleanup-report'], + { + source: '/policy.json', + policy: generatedPolicy({ + layout: measuredRoute({ model_class: 'small', provider: 'test' }), + }), + }, + ); + assert.equal(plan[0].status, 'ready'); + assert.equal(plan[0].model, 'tiny-vision'); + assert.equal(plan[1].status, 'delegate-to-installed-skill'); + assert.equal(plan[2].owner, 'unslop'); +}); + +test('skips an Artifacture pass instead of falling back to the host model', () => { + const [entry] = buildLlmDispatchPlan( + ['operating-model'], + { source: null, policy: null }, + ); + assert.deepEqual(entry, { + pass: 'operating-model', + owner: 'artifacture', + status: 'skipped', + reason: 'no-eval-qualified-model', + policy_source: null, + }); +}); + +test('does not borrow layout qualification for an aesthetic pass', () => { + const [entry] = buildLlmDispatchPlan( + ['aesthetic-nothing'], + { source: '/policy.json', policy: generatedPolicy({ layout: measuredRoute() }) }, + ); + assert.equal(entry.status, 'skipped'); + assert.equal(entry.reason, 'no-eval-qualified-model'); +}); + +test('rejects a hand-written route without selector provenance and measurements', () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'artifacture-policy-')); + const explicit = path.join(root, 'hand-written.json'); + fs.writeFileSync(explicit, JSON.stringify({ + schema_version: 1, + routes: { layout: { model: 'tiny-vision', batch_size: 2 } }, + })); + assert.throws( + () => resolveVisualModelPolicy({ + env: { ARTIFACTURE_VISUAL_MODEL_POLICY: explicit }, + homeDir: path.join(root, 'home'), + repoRoot: path.join(root, 'repo'), + }), + /invalid or non-selector visual model policy/, + ); + fs.rmSync(root, { recursive: true, force: true }); +}); diff --git a/plugins/visual-explainer/scripts/verify/lib/report.mjs b/plugins/visual-explainer/scripts/verify/lib/report.mjs index 0436820..bfdb771 100644 --- a/plugins/visual-explainer/scripts/verify/lib/report.mjs +++ b/plugins/visual-explainer/scripts/verify/lib/report.mjs @@ -1,16 +1,71 @@ import fs from 'node:fs/promises'; +import { buildLlmDispatchPlan } from './model-policy.mjs'; + +// Artifacture extracts evidence and delegates general craft/prose judgment to +// the installed skills that own it. These ids are routing signals, not a second +// implementation of Impeccable or Unslop. +const IMPECCABLE_CRITIQUE_CANDIDATE_IDS = new Set([ + 'forbidden-body-font', + 'forbidden-accent-colors', + 'forbidden-gradient-text-headings', + 'forbidden-glow-pulse-animations', + 'no-emoji-in-ui-chrome', + 'no-three-dot-window-chrome', + 'prose-accent-overuse', + 'gradient-hero-background', + 'decorative-blur-orbs', + 'glassmorphism-default-surface', + 'nested-cards', + 'side-stripe-border', + 'cream-sand-background', + 'reflex-reject-fonts', + 'rainbow-accent-palette', + 'scroll-reveal-spam', + 'bounce-elastic-easing', + 'fake-loading-theater', + 'mixed-icon-systems', + 'inline-emoji-bullets', + 'copy-paste-drop-shadow', + 'uniform-descriptor-gloss', + 'hero-metric-template', + 'card-as-universal-wrapper', + 'decorative-svg-sparing-use', + 'font-pairing-same-classification', + 'default-accent-reflex', +]); + +const UNSLOP_CANDIDATE_IDS = new Set([ + 'unslop-prose-phrases', + 'copy-slop-phrases', + 'unslop-prose-style', + 'title-claim-function', + 'copy-redundancy', + 'jargon-undefined', +]); + +const DELEGATED_CANDIDATE_IDS = new Set([ + ...IMPECCABLE_CRITIQUE_CANDIDATE_IDS, + ...UNSLOP_CANDIDATE_IDS, +]); + +export function delegatedOwnerForCheck(id) { + if (IMPECCABLE_CRITIQUE_CANDIDATE_IDS.has(id)) return 'impeccable'; + if (UNSLOP_CANDIDATE_IDS.has(id)) return 'unslop'; + return null; +} export function buildReport(ctx, checks, screenshots = []) { const summary = { errors: 0, warns: 0, skipped: 0, passed: 0 }; for (const check of checks) { if (check.status === 'pass') summary.passed += 1; - else if (check.status === 'skip' || check.status === 'skipped-static' || check.status === 'llm-required' || check.status === 'transcript') summary.skipped += 1; + else if (check.status === 'skip' || check.status === 'skipped-static' || check.status === 'llm-required' || check.status === 'transcript' || check.status === 'delegated-candidate') summary.skipped += 1; else if (check.status === 'unimplemented') summary.warns += 1; else if (check.status === 'warn') summary.warns += 1; else if (check.status === 'fail' && check.severity === 'error') summary.errors += 1; else if (check.status === 'fail') summary.warns += 1; } + const llmPasses = llmPassesFor(ctx, checks); return { file: ctx.filePath, profile: ctx.profile, @@ -18,7 +73,8 @@ export function buildReport(ctx, checks, screenshots = []) { summary, checks, screenshots, - llm_passes_required: llmPassesFor(ctx, checks), + llm_passes_required: llmPasses, + llm_dispatch_plan: buildLlmDispatchPlan(llmPasses), }; } @@ -37,17 +93,43 @@ export function printHumanReport(report, { quiet = false } = {}) { console.log(`${check.status.toUpperCase().padEnd(13)} ${check.severity.padEnd(5)} ${check.id}${where}${evidence}`); } if (failing.length > 80) console.log(`... ${failing.length - 80} more findings`); + const blocked = (report.llm_dispatch_plan || []).filter( + (entry) => entry.owner === 'artifacture' && entry.status !== 'ready', + ); + if (blocked.length > 0) { + console.log(`LLM ROUTES ${blocked.length} Artifacture pass(es) skipped: no eval-qualified model`); + } } function llmPassesFor(ctx, checks) { const required = new Set(); - if (checks.some((check) => check.stage === 'llm-pass' && check.status === 'llm-required')) { + if ( + checks.some( + (check) => + check.stage === 'llm-pass' && + check.status === 'llm-required' && + !DELEGATED_CANDIDATE_IDS.has(check.id), + ) + ) { const aesthetic = ctx.preset === 'custom' ? ctx.presetHint || 'custom' : ctx.preset; required.add('hierarchy'); - required.add(`aesthetic-${aesthetic}`); + if (aesthetic !== 'custom') required.add(`aesthetic-${aesthetic}`); required.add('completeness'); - required.add('copy'); - if (['page', 'slides', 'magazine'].includes(ctx.profile)) required.add('visual-tells'); + } + if (checks.some((check) => check.id.startsWith('diagram-') && check.status === 'llm-required')) { + required.add('diagram'); + } + if (checks.some((check) => check.id.startsWith('poster-') && check.status === 'llm-required')) { + required.add('poster'); + } + if (shouldDelegate(ctx, checks, IMPECCABLE_CRITIQUE_CANDIDATE_IDS, ['impeccable', 'impeccable:critique'])) { + required.add('impeccable:critique'); + } + if (shouldDelegate(ctx, checks, UNSLOP_CANDIDATE_IDS, ['unslop', 'unslop:cleanup', 'unslop:cleanup-report'])) { + required.add('unslop:cleanup-report'); + } + if (declaresCheck(ctx.html || '', ['artifacture:slop-gap'])) { + required.add('artifacture:slop-gap'); } const hasReviewUnits = ['slides', 'magazine'].includes(ctx.profile) || @@ -65,3 +147,27 @@ function llmPassesFor(ctx, checks) { } return Array.from(required); } + +function shouldDelegate(ctx, checks, candidateIds, explicitTokens) { + if ( + candidateIds === IMPECCABLE_CRITIQUE_CANDIDATE_IDS && + !['page', 'slides', 'magazine', 'poster'].includes(ctx.profile) + ) return false; + if (declaresCheck(ctx.html || '', explicitTokens)) return true; + return checks.some( + (check) => + candidateIds.has(check.id) && + ['warn', 'fail', 'llm-required', 'delegated-candidate'].includes(check.status), + ); +} + +function declaresCheck(html, acceptedTokens) { + for (const match of html.matchAll(/\bdata-ve-checks\s*=\s*["']([^"']+)["']/gi)) { + const tokens = match[1] + .toLowerCase() + .split(/[\s,]+/) + .filter(Boolean); + if (acceptedTokens.some((token) => tokens.includes(token))) return true; + } + return false; +} diff --git a/plugins/visual-explainer/scripts/verify/rubric-criteria.json b/plugins/visual-explainer/scripts/verify/rubric-criteria.json new file mode 100644 index 0000000..b5ab8ea --- /dev/null +++ b/plugins/visual-explainer/scripts/verify/rubric-criteria.json @@ -0,0 +1,25 @@ +{ + "version": 1, + "criteria": [ + { + "id": "layout-repeated-track-symmetry", + "pass": "layout", + "rubric": "rubrics/pass-layout.md" + }, + { + "id": "artifact-slop-false-sequence", + "pass": "artifacture:slop-gap", + "rubric": "rubrics/pass-artifact-slop-gap.md" + }, + { + "id": "artifact-slop-false-state", + "pass": "artifacture:slop-gap", + "rubric": "rubrics/pass-artifact-slop-gap.md" + }, + { + "id": "artifact-slop-false-provenance", + "pass": "artifacture:slop-gap", + "rubric": "rubrics/pass-artifact-slop-gap.md" + } + ] +} diff --git a/plugins/visual-explainer/scripts/verify/rubrics/pass-artifact-slop-gap.md b/plugins/visual-explainer/scripts/verify/rubrics/pass-artifact-slop-gap.md new file mode 100644 index 0000000..c3ed908 --- /dev/null +++ b/plugins/visual-explainer/scripts/verify/rubrics/pass-artifact-slop-gap.md @@ -0,0 +1,70 @@ +# Artifacture slop-gap specialist + +This is a narrow semantic-integrity pass for explanatory artifacts. It does not +judge general visual taste or prose style. + +## Run when + +Run only when `llm_passes_required` includes `artifacture:slop-gap`. + +The orchestrator must name the candidate state or region. Do not scan the whole +artifact for generic AI aesthetics. + +## Ownership boundary + +Route elsewhere and stay silent here when the issue is: + +- generic visual sameness, typography, color, cards, glow, glass, ornament, or + design specificity: call Impeccable; +- AI-written phrases, cadence, em-dashes, fragments, buzzwords, or voice: call + Unslop `cleanup --report`; +- clipping, symmetry, crowding, dead space, arrows, or component geometry: use + Artifacture's layout and mechanical checks. + +This pass owns only decoration that communicates a false artifact semantic. + +## Inputs + +- One screenshot or tight crop of each named candidate region. +- Visible text and accessibility labels from that region. +- The smallest source/brief excerpt needed to determine whether the implied + sequence, state, confidence, or provenance is real. + +Do not read unrelated screenshots or the full source document. + +## Curated criteria + +These verdict IDs are registered in `rubric-criteria.json` under the opt-in +`artifacture:slop-gap` pass. They are not always-on catalog scans. + +- `[artifact-slop-false-sequence]` Decorative numbering, arrows, step labels, or + progress marks imply an order, dependency, or progression that the content + does not actually have. +- `[artifact-slop-false-state]` Status dots, badges, terminal/system labels, + meters, confidence marks, or loading/progress treatments imply measured or + live state when no such state exists. +- `[artifact-slop-false-provenance]` Citation-like numbers, source tags, + timestamps, evidence labels, or reference markers imply a traceable source + that the artifact cannot supply. + +## Judgment rules + +1. Require a false semantic implication, not merely unnecessary decoration. +2. Name the exact implication a reasonable viewer would infer. +3. Stay silent when the source, interaction, navigation, or data supports it. +4. Stay silent on aesthetic costume alone; that belongs to Impeccable. +5. Stay silent on writing rhythm alone; that belongs to Unslop. +6. One finding covers one named region and one false semantic. + +## Verdict JSON schema + +`{"pass":true,"findings":[{"check_id":"artifact-slop-false-sequence|artifact-slop-false-state|artifact-slop-false-provenance","evidence":"","fix":""}]}` + +Set `pass` to `false` when `findings` is non-empty. + +## Silence examples + +- Section `02` is linked from the agenda and marks a real reading order. +- A confidence badge displays a calculated value with an accessible label. +- A footnote marker resolves to a supplied source. +- A decorative dot is visibly ornamental and does not resemble status. diff --git a/plugins/visual-explainer/scripts/verify/rubrics/pass-copy.md b/plugins/visual-explainer/scripts/verify/rubrics/pass-copy.md deleted file mode 100644 index a808c74..0000000 --- a/plugins/visual-explainer/scripts/verify/rubrics/pass-copy.md +++ /dev/null @@ -1,42 +0,0 @@ -Inputs: -- Excluded-filtered prose text only. -- Exclude code, identifiers, filenames, table headers, labels, counters, timestamps, status strings, Mermaid labels, and square-bracket system messages. - -Questions: -- [title-claim-function] For each narrative slide title or section heading, does it name the concrete test, mechanism, result, decision, action, or completion criterion the unit earns? Flag calendar/program containers delivering outcomes, tool/data/score/package personification, abstract uplift transformations, journey/paradigm/pillars framing, generic topic handles when the body contains a stronger claim, and counting titles that describe deck structure instead of the point. -- [unslop-prose-style] Beyond fixed phrases, does this prose show AI-slop stylistic tells such as manufactured emphasis, predictable three-item list rhythm, hollow however/moreover transitions, or empty summary sentences that `/unslop` would rewrite? -- [copy-redundancy] Does any section heading merely rephrase its own first sentence with no added fact, number, or mechanism, or is the same explanatory point repeated across two or more sections? -- [jargon-undefined] For a broad or onboarding audience, does the copy rely on two or more acronyms or internal system names that are load-bearing yet never expanded on first use, glossed, or otherwise defined? - -Verdict JSON schema: -`{"pass":true,"findings":[{"check_id":"","evidence":"paragraph beginning ...","fix":"Run /unslop on the prose block and replace the rendered copy."}]}` - -Fail examples: -- A capstone slide says "Week 1 ends with a calibrated eval" instead of naming the evidence that completes the capstone. -- A bias slide says "Your judge is lying to someone" instead of naming order, verbosity, and self-preference as the mechanisms. -- A handoff slide says "The package outlives the cohort" instead of naming the routes or owner decision. -- A substantive slide is titled only "Architecture", "Proof plan", "Three pillars", or "What the budget buys" when its body contains a specific result or decision. -- "In today's fast-paced landscape" opens a technical explainer. -- Three adjacent cards all use the same "X, Y, and Z" rhythm. -- A section ends with an empty recap sentence that adds no information. -- A heading says "Faster deploys" and the first sentence only says "Deploys are now faster." -- An onboarding explainer uses CRDT, WAL, and HLC without expanding or defining them. - -Pass examples: -- "Examples calibrate the judge to your bar" describes a causal mechanism. -- "Week 1 ends on Friday" is a literal temporal fact. -- "Can one packet reconstruct both products?" states the decision test. -- A cover, divider, agenda, or exact technical reference label may use a concise literal title. - -Title review is semantic, not a blanket subject-verb ban. The title is part of -the explanatory model: it tells the viewer what to inspect. - -## De-Slop Rubric Addendum - -Instruction hierarchy: if `/unslop` is available, invoke it on drafted prose before writing HTML. This rubric is the fallback when `/unslop` is unavailable and the verification layer for rendered copy. Deterministic literals live in `copy-slop-phrases`; this pass covers judgment. - -Scope only prose surfaces: headlines, leads, card/tile descriptions, callouts, captions. Exempt code, labels, numbers, table cells, Mermaid labels, bracketed system messages, identifiers, filenames, timestamps, and literal quotes. - -Cut on sight: throat-clearing openers, emphasis crutches, chatbot artifacts, significance inflation, promotional language, vague attribution, generic conclusions, binary-contrast drama, false concession, and punctuation tells such as repeated colon reveals or excessive em dashes. - -Rewrite rule for findings: cut everything before the claim, replace inflation with the specific fact, preserve every number/name/date/version/quote, and avoid weakening technical terms. diff --git a/plugins/visual-explainer/scripts/verify/rubrics/pass-layout.md b/plugins/visual-explainer/scripts/verify/rubrics/pass-layout.md index 5c1cb85..26f4532 100644 --- a/plugins/visual-explainer/scripts/verify/rubrics/pass-layout.md +++ b/plugins/visual-explainer/scripts/verify/rubrics/pass-layout.md @@ -1,7 +1,7 @@ Inputs: - `report.json`. - Standard screenshots: 1440x900 light, 1440x900 dark, 390x844 light, 390x844 dark. -- Candidate element lists named in the report for table-like div grids, clipping, fixed chrome, slide screenshots, composition strips, sparse diagram slides, and demo frames when present. +- Candidate element lists named in the report for table-like div grids, clipping, fixed chrome, repeated-track layouts, slide screenshots, composition strips, sparse diagram slides, and demo frames when present. Questions: - [real-table-for-tabular-data] Does this div-grid present row/column tabular data such as a comparison, audit, or status matrix that should be a semantic ``, rather than a card grid, KPI row, or feature grid? @@ -9,6 +9,7 @@ Questions: - [text-visibly-clipped] In the 390x844 light and dark screenshots, is any real content text visibly cut off by a container edge, with characters chopped or a sentence truncated without intentional ellipsis? - [fixed-ui-obscures-content] In the 390x844 screenshots, does fixed-position chrome such as a theme toggle, nav, or counter hide readable text or block a clickable target? - [slide-single-focal-point] For each applicable slide screenshot, does the slide present one clear focal element, or do three or more elements compete for attention? +- [layout-repeated-track-symmetry] Only when a composition visibly uses repeated columns or rows: do equivalent tracks have balanced widths, aligned dividers/baselines, and intentional whitespace? Flag unexplained drift that makes a peer region look missing or subordinate. Stay silent when the asymmetry is visibly intentional and supported by content hierarchy. This is a registered rubric criterion under the `layout` pass, not an always-on catalog scan. - [composition-variety-visual] In the flagged consecutive slide screenshot strip, do the slides look spatially repetitive, with the same alignment and whitespace balance, rather than genuinely varied? - [sparse-diagram-slide] In the flagged slide screenshot, does the slide read as sparse or empty, such as a tiny diagram floating in a large viewport with no supporting content? @@ -19,3 +20,4 @@ Fail examples: - A mobile heading loses its final word at the right edge and there is no ellipsis. - A floating theme toggle covers a paragraph link at 390px width. - Four consecutive slides reuse the same centered title, centered chart, and bottom caption layout. +- A three-column input/router/output slide gives the middle track a different width and breaks row alignment even though all three regions are peers. diff --git a/plugins/visual-explainer/scripts/verify/rubrics/pass-visual-tells.md b/plugins/visual-explainer/scripts/verify/rubrics/pass-visual-tells.md deleted file mode 100644 index 3760537..0000000 --- a/plugins/visual-explainer/scripts/verify/rubrics/pass-visual-tells.md +++ /dev/null @@ -1,15 +0,0 @@ -Inputs: -- `report.json` with deterministic visual-tell findings. -- Light/dark screenshots for page, slides, or magazine profiles. -- Candidate extracts named by deterministic checks. - -Questions: -- [font-pairing-same-classification] Do heading and body fonts use two different non-code families in the same typographic classification with no real contrast axis? YES = warn. -- [default-accent-reflex] Is the only saturated accent a reflexive default indigo/blue or warm orange with no brand, subject, or data rationale? YES = warn. - -Verdict JSON schema: -`{"pass":true,"findings":[{"check_id":"","evidence":"screenshot/candidate ...","fix":"..."}]}` - -Fail examples: -- Heading in Poppins and body in Montserrat, both geometric sans, presented as a deliberate pairing. -- A custom explainer uses only #6366f1 as accent color with no brand or subject reason. diff --git a/scripts/check-manifests.mjs b/scripts/check-manifests.mjs index bea0cca..7328b1f 100644 --- a/scripts/check-manifests.mjs +++ b/scripts/check-manifests.mjs @@ -26,6 +26,7 @@ function readSkillFrontmatterVersion(relPath) { const pkg = readJson('package.json'); const pluginManifest = readJson('plugins/visual-explainer/.claude-plugin/plugin.json'); const marketplace = readJson('.claude-plugin/marketplace.json'); +const rubricCriteria = readJson('plugins/visual-explainer/scripts/verify/rubric-criteria.json'); const marketplaceEntry = marketplace.plugins.find((plugin) => plugin.name === 'visual-explainer'); const sources = [ @@ -43,6 +44,20 @@ const repository = marketplaceEntry?.repository; const uniqueVersions = new Set(sources.map((source) => source.value)); const versionsOk = uniqueVersions.size === 1; const repositoryOk = repository === EXPECTED_REPOSITORY; +const criterionIds = rubricCriteria.criteria.map((criterion) => criterion.id); +const criteriaOk = + rubricCriteria.version === 1 && + criterionIds.length === new Set(criterionIds).size && + rubricCriteria.criteria.every( + (criterion) => + criterion.id && + criterion.pass && + criterion.rubric && + readFileSync( + path.join(REPO_ROOT, 'plugins/visual-explainer/scripts/verify', criterion.rubric), + 'utf8', + ).includes(criterion.id), + ); function printTable() { const rows = [ @@ -55,12 +70,13 @@ function printTable() { } } -if (!versionsOk || !repositoryOk) { +if (!versionsOk || !repositoryOk || !criteriaOk) { console.error('Manifest consistency check FAILED\n'); printTable(); console.error(''); if (!versionsOk) console.error(`Version mismatch across sources: ${[...uniqueVersions].join(', ')}`); if (!repositoryOk) console.error(`Repository mismatch: expected "${EXPECTED_REPOSITORY}", got "${repository}"`); + if (!criteriaOk) console.error('Rubric criterion registry is malformed or out of sync with its rubric files'); process.exit(1); } From 55fb8bf6449de568b6122c23d0a7e64ed30ad660 Mon Sep 17 00:00:00 2001 From: Clayton Kim Date: Sat, 25 Jul 2026 09:01:34 -0700 Subject: [PATCH 2/4] Add empirical visual model eval harness --- .gitignore | 2 + evals/visual-model-policy/README.md | 275 +++- evals/visual-model-policy/adapters/README.md | 49 + .../adapters/fixture-all-clean.mjs | 26 + evals/visual-model-policy/aggregate.mjs | 640 ++++++++++ evals/visual-model-policy/aggregate.test.mjs | 395 ++++++ evals/visual-model-policy/corpus.json | 1112 +++++++++++++++++ evals/visual-model-policy/corpus.mjs | 188 +++ evals/visual-model-policy/corpus.test.mjs | 73 ++ .../example-measurements.json | 29 +- .../experiment.template.json | 43 + evals/visual-model-policy/merge.mjs | 230 ++++ evals/visual-model-policy/merge.test.mjs | 108 ++ evals/visual-model-policy/render-corpus.mjs | 393 ++++++ evals/visual-model-policy/run.mjs | 766 ++++++++++++ evals/visual-model-policy/run.test.mjs | 275 ++++ evals/visual-model-policy/select.mjs | 92 +- evals/visual-model-policy/select.test.mjs | 55 + evals/visual-model-policy/telemetry.mjs | 35 + package.json | 8 +- 20 files changed, 4737 insertions(+), 57 deletions(-) create mode 100644 evals/visual-model-policy/adapters/README.md create mode 100644 evals/visual-model-policy/adapters/fixture-all-clean.mjs create mode 100644 evals/visual-model-policy/aggregate.mjs create mode 100644 evals/visual-model-policy/aggregate.test.mjs create mode 100644 evals/visual-model-policy/corpus.json create mode 100644 evals/visual-model-policy/corpus.mjs create mode 100644 evals/visual-model-policy/corpus.test.mjs create mode 100644 evals/visual-model-policy/experiment.template.json create mode 100644 evals/visual-model-policy/merge.mjs create mode 100644 evals/visual-model-policy/merge.test.mjs create mode 100644 evals/visual-model-policy/render-corpus.mjs create mode 100644 evals/visual-model-policy/run.mjs create mode 100644 evals/visual-model-policy/run.test.mjs create mode 100644 evals/visual-model-policy/telemetry.mjs diff --git a/.gitignore b/.gitignore index 08f0eaf..3a66e56 100644 --- a/.gitignore +++ b/.gitignore @@ -29,6 +29,8 @@ skills-lock.json .codex-briefs/ evals/model-matrix/out/ evals/visual-model-policy/runs/ +evals/visual-model-policy/corpus/rendered/ evals/visual-model-policy/policy.generated.json +evals/visual-model-policy/experiment.local.json docs/goals/ artifacts/ diff --git a/evals/visual-model-policy/README.md b/evals/visual-model-policy/README.md index 9c26b4a..0cda263 100644 --- a/evals/visual-model-policy/README.md +++ b/evals/visual-model-policy/README.md @@ -2,53 +2,110 @@ This harness answers one operational question: -> What is the smallest model, and the cheapest safe screenshot batch size, that -> can run each routed visual check family? +> What is the smallest model, and the largest cost-effective screenshot batch +> that remains safe, for each Artifacture-owned visual check family? It does not benchmark artifact generation. `evals/model-matrix/` owns that separate problem. ## Policy -Artifacture's main thread is an orchestrator. It gathers deterministic evidence, -dispatches visual judgment, merges findings, and reports uncertainty. It does -not inspect every screenshot with the host model. +Artifacture's main thread is an orchestrator. It gathers deterministic +evidence, dispatches visual judgment, merges findings, and reports uncertainty. +It does not inspect every screenshot with the host model. -Model choice is per pass, not global. A small model may qualify for clipping and -symmetry while a larger model remains necessary for operating-model fidelity. -Batch size is also per pass. Four screenshots in one request are allowed only -when the measured grounding and silence gates remain green. +Model choice is per family, not global. Batch qualification is also per family +and per model. A model qualified at one screenshot is not qualified at two or +four, and a layout qualification never applies to diagrams, preset fidelity, +operating-model fidelity, or the opt-in Artifacture semantic gap. -The selector always: +The selector: -1. filters out configurations that miss a quality or sample-size gate; +1. rejects cells that miss quality, unique-evidence, adjudication, or telemetry + gates; 2. chooses the lowest-ranked (smallest) qualified model; -3. chooses the lowest-cost qualified batch size within that model; and -4. records larger qualified models only as runtime escalation fallbacks. +3. chooses the lowest measured cost per case among that model's qualified batch + sizes, preferring the larger batch only on a cost tie; and +4. records larger qualified models only as runtime escalation candidates. -A cheaper larger model does not displace a qualified smaller model. Change the -candidate `rank` ordering when deployment policy defines “smallest” differently. +A cheaper larger model does not displace a qualified smaller model. Candidate +`rank` is the deployment definition of “smallest.” -## Required eval data +## Corpus -Use human-reviewed fire/clean cases for each pass. Include hard negatives: -intentional asymmetry, authored dead space, dense expert surfaces, decorative -but truthful states, and screenshots containing several unrelated regions. +`corpus.json` defines 112 label-blind states: at least 10 proposed fire cases +and 10 proposed clean hard negatives for each owned family: -For every `model × pass × batch_size` cell, record: +- layout, including clipping, focal crowding, authored versus accidental dead + space, and repeated-track symmetry; +- diagram fidelity; +- aesthetic and preset fidelity; +- operating-model fidelity; and +- `artifact-slop-gap`, Artifacture's explicit opt-in semantic gap. -- at least 3 independent runs; -- at least 10 positive and 10 negative cases; -- TP, FP, TN, and FN; -- correct image-and-region grounding; -- JSON validity and explicit abstentions; -- p95 latency and actual provider cost; and -- the exact model, image detail, and stable prefix identity used. +Every state has a stable opaque case/state/image ID, named regions, visible +text, the smallest truth excerpt needed for judgment, an initial label, and +adjudication notes. Impeccable and Unslop criteria are deliberately absent. +Every criterion has at least four states, so batch size 4 can exercise every +criterion. Rendering fails if any two case IDs produce identical PNG bytes; +aggregation keys unique evidence to the recorded PNG hash, not merely the case +ID. -The default graduation gates are: +The checked-in labels are proposed from fixture intent, not represented as +human review. Render and inspect all pairs before a live run: + +```bash +npm run ve:render-visual-model-corpus +``` + +After a human reviews every rendered pair, update `corpus.json`: + +```json +{ + "label_review": { + "status": "human-reviewed", + "reviewer": "reviewer identity", + "reviewed_at": "2026-07-25T12:00:00Z", + "notes": "What was reviewed and any relabeling performed." + } +} +``` + +The live runner refuses pending labels. Synthetic adapters may exercise the +contract, but their records are permanently marked `synthetic` and the selector +rejects them. + +Rendered PNGs and raw runs are ignored because they are reproducible or +provider-local artifacts. `render-manifest.json` records the exact rendered +viewport and paths for the local corpus instance. + +## Required measurements + +For every `model × family × configured_batch_size` cell, record: + +- at least 3 independent replicates; +- at least 10 unique fire and 10 unique clean cases; +- coverage of every criterion represented by the family corpus; +- TP, FP, TN, and FN over valid non-abstaining observations; +- correct image-and-region grounding for positive cases; +- JSON validity and explicit abstentions over all responses; +- p95 request latency and actual provider cost; and +- exact provider, model, image detail, prefix identity, cache tokens, token + usage, and time to first token. + +Repeated observations do not count as independent cases. Aggregation records +both observation totals and unique screenshot totals, and the selector gates +both. A cell is also blocked unless every request in the deterministic +experiment matrix is present and bound to the exact experiment/corpus, +rendered-image, shared-prefix, criterion-suffix, and randomization hashes. + +Default graduation gates: | Metric | Gate | |---|---:| +| Replicates | ≥ 3 | +| Unique fire / clean cases | ≥ 10 / ≥ 10 | +| Criterion fire / clean coverage | ≥ 1 / ≥ 1 each | | Precision | ≥ 0.90 | | Recall | ≥ 0.90 | | Silence accuracy | ≥ 0.95 | @@ -56,27 +113,148 @@ The default graduation gates are: | JSON validity | ≥ 0.99 | | Abstention rate | ≤ 0.05 | | p95 latency | ≤ 30 s | -| Cost per case | ≤ $0.01 | +| Cost per classified case | ≤ $0.01 | + +These are floors. Higher-consequence families may set stricter thresholds in +the experiment. + +## Reproducible workflow + +Copy the template to an ignored or otherwise local experiment file. List the +ordered model ladder in `candidates`, but set `run_candidate` to only the +smallest untested candidate. The runner enforces that ordering and never +schedules every configured model at once. Use the provider's exact direct-API +model ID; do not use a marketing alias whose implementation can drift. + +```bash +cp evals/visual-model-policy/experiment.template.json \ + evals/visual-model-policy/experiment.local.json +``` + +Edit the experiment ID, candidate ladder, `run_candidate`, provider, and any family-specific +thresholds. Preview the exact matrix without calling a provider: + +```bash +npm run ve:run-visual-model-eval -- \ + --experiment evals/visual-model-policy/experiment.local.json \ + --dry-run +``` -These are floors, not universal truth. High-consequence families can override -them in the measurement file. +A provider adapter must implement the contract in `adapters/README.md`. Run the +measured ladder: -## Generate a policy +```bash +npm run ve:run-visual-model-eval -- \ + --experiment evals/visual-model-policy/experiment.local.json \ + --adapter /absolute/path/to/provider-adapter.mjs +``` + +The runner writes one append-only JSONL record per request under +`runs//records.jsonl`. It never truncates prior records. Each +request has an immutable image prefix followed by exactly one criterion suffix. +Human labels and adjudication notes are never sent to the adapter. Every record +also captures the deterministic randomization key and exact image SHA-256 used +for independent-evidence accounting. + +Aggregate raw records into the selector schema: + +```bash +npm run ve:aggregate-visual-model-eval -- \ + --records evals/visual-model-policy/runs//records.jsonl \ + --experiment evals/visual-model-policy/experiment.local.json \ + --out evals/visual-model-policy/runs//measurements.json +``` + +Aggregation exits non-zero when any disagreement lacks human adjudication or +when records are synthetic. It still writes the diagnostic measurements so the +review queue is visible. Append adjudications to the same JSONL file rather +than editing request records: + +```json +{"schema_version":1,"record_type":"visual-eval-adjudication","observation_id":"","decision":"confirm-corpus-label","adjudicator":"reviewer identity","notes":"Why the corpus label is correct.","recorded_at":"2026-07-25T12:00:00Z"} +``` + +Allowed decisions are `confirm-corpus-label`, `relabel-corpus` with a new +`human_label`, and `exclude`. Re-aggregate to a new output path after review; +the aggregator refuses to overwrite an existing measurement file. A +`relabel-corpus` decision applies to every replicate and wrapped observation of +that case; conflicting case-level relabels are rejected. + +`example-measurements.json` is synthetic schema documentation. Its provenance +flags deliberately prevent it from producing a route. -Copy `example-measurements.json`, replace the placeholder candidates with the -actual direct-API models under test, and populate results from the eval run: +Aggregate each candidate run against its own experiment file. After multiple +candidate ranks are complete, merge their reviewed measurements. The merge +requires identical corpus, thresholds, and candidate ladder contracts and +rejects duplicate cells, incomplete request matrices, skipped lower ranks, or +prior-candidate evidence hashes that do not match the exact included files: + +```bash +npm run ve:merge-visual-model-measurements -- \ + --input evals/visual-model-policy/runs//measurements.reviewed.json \ + --input evals/visual-model-policy/runs//measurements.reviewed.json \ + --out evals/visual-model-policy/runs/ladder.measurements.json +``` + +Generate a policy only from the non-synthetic, fully adjudicated merged +measurements: ```bash npm run ve:select-visual-model-policy -- \ - --input evals/visual-model-policy/measurements.json \ + --input evals/visual-model-policy/runs/ladder.measurements.json \ --out ~/.artifacture/visual-model-policy.json ``` -The command exits `1` if any measured pass has no qualified route. Do not turn -that into an automatic frontier-model fallback. Add cases, test the next model, -or disclose `no-eval-qualified-model`. +The selector exits `1` if any measured family has no qualified route. Do not +turn that into an implicit frontier fallback. Add evidence, test the next model +rank, or disclose `no-eval-qualified-model`. + +## Ladder discipline + +Run batch sizes `1, 2, 4` for the smallest candidate first. A larger batch is +safe only if its own grounding, silence, schema, latency, and cost measurements +graduate. Cases are deterministically shuffled per replicate. A criterion tail +wraps to already observed cases only to keep the request at the exact configured +batch size; wrapped observations never increase unique evidence. Missing +criterion coverage still blocks graduation. -Set `ARTIFACTURE_VISUAL_MODEL_POLICY` when the policy lives elsewhere. +After aggregation and selection, add the completed candidate to +`ladder_state.completed_candidates` with a `qualified` or `disqualified` +status and reviewed-measurement SHA-256 for each measured family. +The next run must name the next smallest untested candidate for every family +listed in `passes`; use a pass subset when family ladders are at different +ranks. Once one family has a selected model and a larger-model escalation +candidate, the runner refuses a still-larger candidate for that family. + +```json +{ + "ladder_state": { + "completed_candidates": [{ + "id": "exact-provider-model-id", + "pass_evidence": { + "layout": { + "status": "qualified", + "evidence_sha256": "" + }, + "diagram": { + "status": "disqualified", + "evidence_sha256": "" + } + } + }] + } +} +``` + +No production measurements or model choices are checked in. The external inputs +still required are: + +1. human sign-off on the rendered corpus labels; +2. exact ordered provider model candidates; +3. a live adapter with credentials supplied through its environment; and +4. provider-reported token/cache telemetry and actual cost; and +5. a controlled provider cache experiment if routing economics will rely on + prefix-cache savings. ## Runtime escalation @@ -90,12 +268,19 @@ to the next eval-qualified model only for: - an explicit out-of-distribution signal. Do not escalate merely because the main thread prefers a more articulate -critique. The model's job is to find and ground violations, not to write the -final narrative. +critique. The model finds and grounds violations; the main thread owns the final +narrative. ## Prefix caching -Use the immutable evidence-prefix contract in `evals/visual-cache/`. Cache -measurements and model qualification are independent gates: a cache hit does not -make an inaccurate model acceptable, and a qualified model does not justify an -unmeasured batch size. +Qualification records provider-reported cache read/write tokens, total latency, +time to first token, prefix identity, and actual cost. Cache measurements and +model qualification remain independent: a cache hit does not make an inaccurate +model acceptable, and a qualified model does not justify an unmeasured batch. + +This runner does not claim to implement the controlled cold/warm/suffix-count +matrix in `docs/plans/visual-eval-prefix-caching.md`, and it does not fold a +primer cost into model qualification. That separate experiment still needs +provider-specific cache-key controls, ordered primer/warm sequencing, suffix +counts `1, 2, 4, 8`, and its larger replicate counts. Only non-zero provider +cache-read telemetry establishes a cache hit. diff --git a/evals/visual-model-policy/adapters/README.md b/evals/visual-model-policy/adapters/README.md new file mode 100644 index 0000000..d5d36e2 --- /dev/null +++ b/evals/visual-model-policy/adapters/README.md @@ -0,0 +1,49 @@ +# Provider adapter contract + +`run.mjs` is provider-independent. A live adapter is a local ESM module that +exports: + +```js +export async function invoke(request, context) { + return { + provider_response_id: "provider request id", + raw_text: "{\"verdicts\":[...]}", + telemetry: { + complete: true, + latency_ms: 1200, + time_to_first_token_ms: 400, + input_tokens: 1800, + cache_read_tokens: 1200, + cache_write_tokens: 0, + output_tokens: 220, + actual_cost_usd: 0.0014 + } + }; +} +``` + +The adapter translates the provider-neutral `request.prefix` and +`request.suffix` fields into one direct API call: + +1. fixed `tools_json` definitions; +2. fixed system/verdict contract; +3. fixed shared instructions; +4. optional fixed design-system excerpt; +5. images in the manifest's stable order and exact detail setting; +6. provider cache breakpoint, when supported; and +7. exactly one criterion suffix. + +The response must report actual provider telemetry. Use `0` only when the +provider explicitly reports zero; do not estimate tokens, cache reads, total +latency, time to first token, or cost. `actual_cost_usd` may be calculated from +provider-reported token counts only when the adapter records the exact dated +price source beside its code. Set `complete: true` only when every required +telemetry field is available. Provider failures are recorded with incomplete +telemetry and can never qualify a policy cell. + +`context` contains request identity and adapter options only. It never contains +human labels or adjudication notes. Credentials remain in environment variables +owned by the adapter; the runner neither reads nor stores them. + +`fixture-all-clean.mjs` exists only to exercise the plumbing. It exports +`synthetic = true`; those records cannot generate a production policy. diff --git a/evals/visual-model-policy/adapters/fixture-all-clean.mjs b/evals/visual-model-policy/adapters/fixture-all-clean.mjs new file mode 100644 index 0000000..3ca6af5 --- /dev/null +++ b/evals/visual-model-policy/adapters/fixture-all-clean.mjs @@ -0,0 +1,26 @@ +// Contract fixture only. Records produced through this adapter are marked +// synthetic by run.mjs and are rejected by select.mjs. +export const synthetic = true; + +export async function invoke(request) { + return { + provider_response_id: `fixture:${request.prefix_manifest.prefix_id.slice(0, 12)}`, + raw_text: JSON.stringify({ + verdicts: request.suffix.cases.map((entry) => ({ + state_id: entry.state_id, + pass: true, + findings: [], + })), + }), + telemetry: { + complete: true, + latency_ms: 1, + time_to_first_token_ms: 1, + input_tokens: 0, + cache_read_tokens: 0, + cache_write_tokens: 0, + output_tokens: 0, + actual_cost_usd: 0, + }, + }; +} diff --git a/evals/visual-model-policy/aggregate.mjs b/evals/visual-model-policy/aggregate.mjs new file mode 100644 index 0000000..a9c297b --- /dev/null +++ b/evals/visual-model-policy/aggregate.mjs @@ -0,0 +1,640 @@ +#!/usr/bin/env node +import fs from 'node:fs/promises'; +import path from 'node:path'; +import process from 'node:process'; +import { pathToFileURL } from 'node:url'; +import { + expandCorpus, + loadCorpus, + validateCorpus, +} from './corpus.mjs'; +import { + buildExperimentPlan, + corpusContractId, + experimentContractId, + prepareExperimentPlan, + requestMatrixId, + requestIdFor, +} from './run.mjs'; +import { normalizeTelemetry } from './telemetry.mjs'; + +export function aggregateRecords(records, { + candidates, + thresholds, + sourceFiles = [], + requiredCriteriaByPass = {}, + experiment = null, + corpus = null, + preparedPlan = null, +} = {}) { + if (!Array.isArray(records)) throw new Error('records[] is required'); + if (!Array.isArray(candidates) || candidates.length === 0) { + throw new Error('candidates[] is required'); + } + const candidateById = new Map(candidates.map((candidate) => [candidate.id, candidate])); + const requestIds = new Set(); + const adjudicationByObservation = new Map(); + const requestRecords = []; + let expectedByRequest = null; + let expectedContractId = null; + let expectedMatrixId = null; + if (experiment || corpus) { + if (!experiment || !corpus) throw new Error('experiment and corpus must be supplied together'); + if (!preparedPlan) { + throw new Error('preparedPlan is required for experiment-bound aggregation'); + } + const cases = expandCorpus(corpus, { corpusPath: experiment.corpus_path }); + const selectedPasses = new Set(experiment.passes || cases.map((entry) => entry.family)); + const plan = preparedPlan; + if (plan.length === 0) throw new Error('expected experiment request matrix cannot be empty'); + expectedByRequest = new Map(plan.map((cell) => [ + requestIdFor(experiment.id, cell), + cell, + ])); + expectedMatrixId = preparedPlan ? requestMatrixId(preparedPlan) : null; + expectedContractId = experimentContractId(experiment, corpus, expectedMatrixId); + } + + for (const record of records) { + if (record?.schema_version !== 1) throw new Error('every record requires schema_version=1'); + if (record.record_type === 'visual-eval-adjudication') { + validateAdjudication(record); + if (adjudicationByObservation.has(record.observation_id)) { + throw new Error(`duplicate adjudication for ${record.observation_id}`); + } + adjudicationByObservation.set(record.observation_id, record); + continue; + } + validateRequestRecord(record); + if (requestIds.has(record.request_id)) throw new Error(`duplicate request_id ${record.request_id}`); + requestIds.add(record.request_id); + if (expectedByRequest) { + const expected = expectedByRequest.get(record.request_id); + if (!expected) throw new Error(`unexpected request_id ${record.request_id}`); + if ( + record.experiment_id !== experiment.id + || record.experiment_contract_id !== expectedContractId + ) { + throw new Error(`request ${record.request_id} does not match the experiment contract`); + } + const expectedFields = { + provider: expected.provider, + model: expected.model, + model_rank: expected.model_rank, + model_class: expected.model_class, + pass: expected.pass, + criterion_id: expected.criterion_id, + configured_batch_size: expected.configured_batch_size, + actual_batch_size: expected.actual_batch_size, + image_detail: expected.image_detail, + replicate: expected.replicate, + randomization_key: expected.randomization_key, + }; + for (const [field, value] of Object.entries(expectedFields)) { + if (record[field] !== value) { + throw new Error(`request ${record.request_id} does not match planned ${field}`); + } + } + const expectedCaseIds = expected.cases.map((entry) => entry.case_id); + const expectedImageIds = expected.cases.map((entry) => entry.image.id); + if ( + JSON.stringify(record.case_ids) !== JSON.stringify(expectedCaseIds) + || JSON.stringify(record.image_ids) !== JSON.stringify(expectedImageIds) + ) { + throw new Error(`request ${record.request_id} evidence does not match the expected matrix`); + } + if (expected.request) { + if ( + record.request_matrix_sha256 !== expectedMatrixId + || record.prefix_id !== expected.request.prefix_manifest.prefix_id + || record.suffix_sha256 !== expected.request.suffix.suffix_sha256 + ) { + throw new Error(`request ${record.request_id} prompt identity does not match the expected matrix`); + } + const expectedHashByImage = new Map( + expected.request.prefix_manifest.images.map((image) => [image.id, image.sha256]), + ); + const expectedCaseById = new Map( + expected.cases.map((evalCase) => [evalCase.case_id, evalCase]), + ); + for (const observation of record.observations) { + const evalCase = expectedCaseById.get(observation.case_id); + if ( + !evalCase + || observation.image_id !== evalCase.image.id + || observation.criterion_id !== evalCase.criterion_id + || observation.human_label !== evalCase.human_label + || observation.image_sha256 !== expectedHashByImage.get(evalCase.image.id) + ) { + throw new Error( + `observation ${observation.observation_id} does not match expected corpus evidence`, + ); + } + } + } + } + if (!candidateById.has(record.model)) { + throw new Error(`record references unknown candidate ${record.model}`); + } + const candidate = candidateById.get(record.model); + if ( + Number(record.model_rank) !== Number(candidate.rank) + || record.provider !== candidate.provider + ) { + throw new Error(`record candidate metadata does not match ${record.model}`); + } + requestRecords.push(record); + } + + const experimentIds = new Set(requestRecords.map((record) => record.experiment_id)); + const contractIds = new Set(requestRecords.map((record) => record.experiment_contract_id)); + if (experimentIds.size > 1 || contractIds.size > 1) { + throw new Error('records from different experiment contracts cannot be aggregated together'); + } + const experimentComplete = expectedByRequest !== null + && requestIds.size === expectedByRequest.size + && [...expectedByRequest.keys()].every((requestId) => requestIds.has(requestId)); + const observationById = new Map(); + for (const record of requestRecords) { + for (const observation of record.observations) { + if (observationById.has(observation.observation_id)) { + throw new Error(`duplicate observation_id ${observation.observation_id}`); + } + observationById.set(observation.observation_id, observation); + } + } + const relabelByCase = new Map(); + for (const adjudication of adjudicationByObservation.values()) { + const observation = observationById.get(adjudication.observation_id); + if (!observation) { + throw new Error(`adjudication references unknown observation ${adjudication.observation_id}`); + } + if (adjudication.decision !== 'relabel-corpus') continue; + const prior = relabelByCase.get(observation.case_id); + if (prior && prior !== adjudication.human_label) { + throw new Error(`conflicting corpus relabels for case ${observation.case_id}`); + } + relabelByCase.set(observation.case_id, adjudication.human_label); + } + for (const adjudication of adjudicationByObservation.values()) { + const observation = observationById.get(adjudication.observation_id); + if ( + adjudication.decision === 'confirm-corpus-label' + && relabelByCase.has(observation.case_id) + ) { + throw new Error(`conflicting confirm and relabel decisions for case ${observation.case_id}`); + } + } + + const grouped = new Map(); + for (const record of requestRecords) { + const key = [ + record.provider, + record.model, + record.pass, + record.configured_batch_size, + record.image_detail, + ].join('\u0000'); + if (!grouped.has(key)) grouped.set(key, []); + grouped.get(key).push(record); + } + + const pendingAdjudication = new Set(); + const results = [...grouped.values()].map((cellRecords) => { + const observations = []; + for (const record of cellRecords) { + for (const observation of record.observations) { + const adjudication = adjudicationByObservation.get(observation.observation_id); + const corpusRelabel = relabelByCase.get(observation.case_id) || null; + const effective = applyAdjudication(observation, adjudication, corpusRelabel); + if (effective.disagreement && !adjudication && !corpusRelabel) { + pendingAdjudication.add(observation.observation_id); + } + if (!effective.excluded) observations.push(effective); + } + } + const classified = observations.filter( + (entry) => entry.json_valid && !entry.abstained && entry.predicted_label, + ); + const positives = classified.filter((entry) => entry.human_label === 'fire'); + const negatives = classified.filter((entry) => entry.human_label === 'clean'); + const uniqueImages = new Map(); + const evidenceByCase = new Map(); + for (const observation of classified) { + const priorEvidence = evidenceByCase.get(observation.case_id); + if ( + priorEvidence + && ( + priorEvidence.image_sha256 !== observation.image_sha256 + || priorEvidence.human_label !== observation.human_label + ) + ) { + throw new Error(`case ${observation.case_id} has inconsistent evidence identity`); + } + evidenceByCase.set(observation.case_id, { + image_sha256: observation.image_sha256, + human_label: observation.human_label, + }); + if (!uniqueImages.has(observation.image_sha256)) { + uniqueImages.set(observation.image_sha256, observation.human_label); + } else if (uniqueImages.get(observation.image_sha256) !== observation.human_label) { + throw new Error( + `rendered image ${observation.image_sha256} has inconsistent human labels`, + ); + } + } + const first = cellRecords[0]; + const criterionMetrics = aggregateCriteria( + classified, + requiredCriteriaByPass[first.pass] || [], + ); + const telemetryComplete = cellRecords.every( + (record) => record.telemetry.complete === true, + ); + const totalCost = telemetryComplete + ? sum(cellRecords.map((record) => record.telemetry.actual_cost_usd)) + : null; + const jsonValid = observations.filter((entry) => entry.json_valid).length; + const abstentions = observations.filter((entry) => entry.abstained).length; + const groundingTotal = positives.length; + const grounded = positives.filter( + (entry) => entry.predicted_label === 'fire' && entry.grounded, + ).length; + const cellPending = observations + .filter((entry) => ( + entry.disagreement + && !adjudicationByObservation.has(entry.observation_id) + && !relabelByCase.has(entry.case_id) + )) + .map((entry) => entry.observation_id); + return { + pass: first.pass, + model: first.model, + batch_size: first.configured_batch_size, + image_detail: first.image_detail, + runs: new Set(cellRecords.map( + (record) => `${record.experiment_id}:${record.replicate}`, + )).size, + requests: cellRecords.length, + cases: classified.length, + positives: positives.length, + negatives: negatives.length, + unique_cases: uniqueImages.size, + unique_positives: [...uniqueImages.values()].filter((label) => label === 'fire').length, + unique_negatives: [...uniqueImages.values()].filter((label) => label === 'clean').length, + tp: positives.filter((entry) => entry.predicted_label === 'fire').length, + fp: negatives.filter((entry) => entry.predicted_label === 'fire').length, + tn: negatives.filter((entry) => entry.predicted_label === 'clean').length, + fn: positives.filter((entry) => entry.predicted_label === 'clean').length, + grounded, + grounding_total: groundingTotal, + json_valid: jsonValid, + responses: observations.length, + abstentions, + total_cost_usd: totalCost === null ? null : roundMoney(totalCost), + telemetry_complete: telemetryComplete, + experiment_complete: experimentComplete, + p95_latency_ms: telemetryComplete + ? percentile(cellRecords.map((record) => record.telemetry.latency_ms), 0.95) + : null, + p95_time_to_first_token_ms: telemetryComplete + ? percentile( + cellRecords.map((record) => record.telemetry.time_to_first_token_ms), + 0.95, + ) + : null, + telemetry: { + input_tokens: sumPresent(cellRecords.map((record) => record.telemetry.input_tokens)), + cache_read_tokens: sumPresent(cellRecords.map((record) => record.telemetry.cache_read_tokens)), + cache_write_tokens: sumPresent(cellRecords.map((record) => record.telemetry.cache_write_tokens)), + output_tokens: sumPresent(cellRecords.map((record) => record.telemetry.output_tokens)), + prefix_ids: [...new Set(cellRecords.map((record) => record.prefix_id))].sort(), + }, + criteria: criterionMetrics, + adjudication_complete: cellPending.length === 0, + synthetic: cellRecords.some((record) => record.synthetic === true), + }; + }).sort(compareResults); + + return { + schema_version: 1, + generated_at: new Date().toISOString(), + source: { + record_files: [...sourceFiles], + request_records: requestRecords.length, + adjudication_records: adjudicationByObservation.size, + expected_request_records: expectedByRequest?.size ?? null, + experiment_complete: experimentComplete, + corpus_id: corpus?.corpus_id ?? null, + corpus_sha256: corpus ? corpusContractId(corpus) : null, + experiment_contract_id: expectedContractId, + ladder_state: experiment?.ladder_state || null, + }, + thresholds, + candidates, + results, + pending_adjudication: [...pendingAdjudication].sort(), + }; +} + +function aggregateCriteria(observations, requiredCriteria) { + const groups = new Map(requiredCriteria.map((criterionId) => [criterionId, []])); + for (const observation of observations) { + if (!groups.has(observation.criterion_id)) groups.set(observation.criterion_id, []); + groups.get(observation.criterion_id).push(observation); + } + return [...groups.entries()].sort(([a], [b]) => a.localeCompare(b)).map(([criterionId, rows]) => { + const positives = rows.filter((entry) => entry.human_label === 'fire'); + const negatives = rows.filter((entry) => entry.human_label === 'clean'); + return { + id: criterionId, + cases: rows.length, + unique_positives: new Set(positives.map((entry) => entry.image_sha256)).size, + unique_negatives: new Set(negatives.map((entry) => entry.image_sha256)).size, + tp: positives.filter((entry) => entry.predicted_label === 'fire').length, + fp: negatives.filter((entry) => entry.predicted_label === 'fire').length, + tn: negatives.filter((entry) => entry.predicted_label === 'clean').length, + fn: positives.filter((entry) => entry.predicted_label === 'clean').length, + }; + }); +} + +function applyAdjudication(observation, adjudication, corpusRelabel) { + const normalized = { + ...observation, + human_label: corpusRelabel || observation.human_label, + disagreement: ( + observation.predicted_label !== null + && observation.predicted_label !== (corpusRelabel || observation.human_label) + ), + }; + if (!adjudication) return { ...normalized, excluded: false }; + if (adjudication.decision === 'confirm-corpus-label') { + return { ...normalized, excluded: false, adjudication }; + } + if (adjudication.decision === 'relabel-corpus') { + return { ...normalized, excluded: false, adjudication }; + } + return { ...normalized, excluded: true, adjudication }; +} + +function validateRequestRecord(record) { + for (const field of [ + 'request_id', + 'experiment_id', + 'experiment_contract_id', + 'request_matrix_sha256', + 'replicate', + 'provider', + 'model', + 'pass', + 'criterion_id', + 'configured_batch_size', + 'actual_batch_size', + 'image_detail', + 'prefix_id', + 'suffix_sha256', + 'randomization_key', + 'case_ids', + 'image_ids', + 'response', + 'telemetry', + 'observations', + 'synthetic', + ]) { + if (record?.[field] === undefined || record[field] === null) { + throw new Error(`request record requires ${field}`); + } + } + if (record.record_type !== 'visual-eval-request') { + throw new Error(`unsupported record_type ${record.record_type}`); + } + if (!Array.isArray(record.observations) || record.observations.length === 0) { + throw new Error(`request ${record.request_id} requires observations[]`); + } + if (typeof record.synthetic !== 'boolean') { + throw new Error(`request ${record.request_id} synthetic must be a boolean`); + } + if (Number(record.actual_batch_size) !== Number(record.configured_batch_size)) { + throw new Error( + `request ${record.request_id} actual_batch_size must equal configured_batch_size`, + ); + } + const batchSize = Number(record.configured_batch_size); + if ( + !Array.isArray(record.case_ids) + || !Array.isArray(record.image_ids) + || record.case_ids.length !== batchSize + || record.image_ids.length !== batchSize + || record.observations.length !== batchSize + ) { + throw new Error(`request ${record.request_id} evidence arrays must match configured_batch_size`); + } + if ( + new Set(record.case_ids).size !== batchSize + || new Set(record.image_ids).size !== batchSize + ) { + throw new Error(`request ${record.request_id} evidence identities must be unique within a batch`); + } + for (const field of [ + 'experiment_contract_id', + 'request_matrix_sha256', + 'prefix_id', + 'suffix_sha256', + 'randomization_key', + ]) { + if (!/^[a-f0-9]{64}$/.test(record[field])) { + throw new Error(`request ${record.request_id} has invalid ${field}`); + } + } + for (const observation of record.observations) { + for (const field of [ + 'observation_id', + 'case_id', + 'image_id', + 'image_sha256', + 'human_label', + 'criterion_id', + 'json_valid', + 'abstained', + 'grounded', + ]) { + if (observation[field] === undefined || observation[field] === null) { + throw new Error(`observation in ${record.request_id} requires ${field}`); + } + } + if (!/^[a-f0-9]{64}$/.test(observation.image_sha256)) { + throw new Error(`observation ${observation.observation_id} has invalid image_sha256`); + } + const caseIndex = record.case_ids.indexOf(observation.case_id); + if ( + caseIndex < 0 + || record.image_ids[caseIndex] !== observation.image_id + || observation.criterion_id !== record.criterion_id + ) { + throw new Error( + `observation ${observation.observation_id} does not match its request evidence`, + ); + } + if (!['fire', 'clean'].includes(observation.human_label)) { + throw new Error(`invalid human_label ${observation.human_label}`); + } + if ( + observation.predicted_label !== null + && !['fire', 'clean'].includes(observation.predicted_label) + ) { + throw new Error(`invalid predicted_label ${observation.predicted_label}`); + } + } + if ( + new Set(record.observations.map((entry) => entry.observation_id)).size !== batchSize + || new Set(record.observations.map((entry) => entry.case_id)).size !== batchSize + || new Set(record.observations.map((entry) => entry.image_id)).size !== batchSize + ) { + throw new Error(`request ${record.request_id} observations must be unique within a batch`); + } + record.telemetry = normalizeTelemetry(record.telemetry, { + allowIncomplete: true, + context: `request ${record.request_id} telemetry`, + }); +} + +function validateAdjudication(record) { + for (const field of ['observation_id', 'decision', 'adjudicator', 'notes', 'recorded_at']) { + if (!record?.[field]) throw new Error(`adjudication requires ${field}`); + } + if (!['confirm-corpus-label', 'relabel-corpus', 'exclude'].includes(record.decision)) { + throw new Error(`unsupported adjudication decision ${record.decision}`); + } + if ( + record.decision === 'relabel-corpus' + && !['fire', 'clean'].includes(record.human_label) + ) { + throw new Error('relabel-corpus adjudication requires human_label fire or clean'); + } +} + +function percentile(values, quantile) { + if (values.length === 0) throw new Error('cannot compute percentile of empty values'); + const sorted = [...values].map(Number).sort((a, b) => a - b); + return sorted[Math.max(0, Math.ceil(sorted.length * quantile) - 1)]; +} + +function sum(values) { + return values.reduce((total, value) => total + Number(value), 0); +} + +function sumPresent(values) { + return values.some((value) => value === null) ? null : sum(values); +} + +function roundMoney(value) { + return Number(value.toFixed(12)); +} + +function compareResults(a, b) { + return ( + String(a.pass).localeCompare(String(b.pass)) + || String(a.model).localeCompare(String(b.model)) + || Number(a.batch_size) - Number(b.batch_size) + || String(a.image_detail).localeCompare(String(b.image_detail)) + ); +} + +export async function readJsonlFiles(files) { + const records = []; + for (const file of files) { + const source = await fs.readFile(file, 'utf8'); + for (const [index, rawLine] of source.split('\n').entries()) { + const line = rawLine.trim(); + if (!line) continue; + try { + records.push(JSON.parse(line)); + } catch (error) { + throw new Error(`${file}:${index + 1}: ${error.message}`); + } + } + } + return records; +} + +function parseArgs(argv) { + const args = { records: [], experiment: null, output: null }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === '--records') args.records.push(argv[++index]); + else if (arg === '--experiment') args.experiment = argv[++index]; + else if (arg === '--out') args.output = argv[++index]; + else throw new Error(`unknown argument ${arg}`); + } + if (args.records.length === 0 || !args.experiment) { + throw new Error('usage: aggregate.mjs --records records.jsonl [--records more.jsonl] --experiment experiment.json [--out measurements.json]'); + } + return args; +} + +async function main() { + const args = parseArgs(process.argv.slice(2)); + const experiment = JSON.parse(await fs.readFile(args.experiment, 'utf8')); + const files = args.records.map((file) => path.resolve(file)); + const records = await readJsonlFiles(files); + const experimentPath = path.resolve(args.experiment); + const corpusPath = path.resolve( + path.dirname(experimentPath), + experiment.corpus_path, + ); + const corpus = await loadCorpus(corpusPath); + validateCorpus(corpus, { corpusPath }); + const resolvedExperiment = { ...experiment, corpus_path: corpusPath }; + const requiredCriteriaByPass = {}; + const expandedCases = expandCorpus(corpus, { corpusPath }); + for (const evalCase of expandedCases) { + if (!requiredCriteriaByPass[evalCase.family]) requiredCriteriaByPass[evalCase.family] = new Set(); + requiredCriteriaByPass[evalCase.family].add(evalCase.criterion_id); + } + const selectedPasses = new Set(experiment.passes || expandedCases.map((entry) => entry.family)); + const plan = buildExperimentPlan( + resolvedExperiment, + expandedCases.filter((entry) => selectedPasses.has(entry.family)), + ); + if (plan.length === 0) throw new Error('experiment produced an empty request plan'); + const preparedPlan = await prepareExperimentPlan(resolvedExperiment, plan); + const measurements = aggregateRecords(records, { + candidates: experiment.candidates, + thresholds: experiment.thresholds, + sourceFiles: files, + requiredCriteriaByPass: Object.fromEntries( + Object.entries(requiredCriteriaByPass).map(([family, criteria]) => [ + family, + [...criteria].sort(), + ]), + ), + experiment: resolvedExperiment, + corpus, + preparedPlan, + }); + const rendered = `${JSON.stringify(measurements, null, 2)}\n`; + if (args.output) { + const output = path.resolve(args.output); + await fs.mkdir(path.dirname(output), { recursive: true }); + await fs.writeFile(output, rendered, { flag: 'wx' }); + process.stdout.write(`${output}\n`); + } else { + process.stdout.write(rendered); + } + if ( + measurements.pending_adjudication.length > 0 + || measurements.results.some((row) => ( + row.synthetic || !row.experiment_complete || !row.telemetry_complete + )) + ) { + process.exitCode = 1; + } +} + +if (import.meta.url === pathToFileURL(process.argv[1] || '').href) { + main().catch((error) => { + console.error(error.stack || error.message || error); + process.exit(2); + }); +} diff --git a/evals/visual-model-policy/aggregate.test.mjs b/evals/visual-model-policy/aggregate.test.mjs new file mode 100644 index 0000000..74709c5 --- /dev/null +++ b/evals/visual-model-policy/aggregate.test.mjs @@ -0,0 +1,395 @@ +import assert from 'node:assert/strict'; +import crypto from 'node:crypto'; +import { fileURLToPath } from 'node:url'; +import test from 'node:test'; +import { aggregateRecords } from './aggregate.mjs'; +import { expandCorpus, loadCorpus } from './corpus.mjs'; +import { + buildExperimentPlan, + experimentContractId, + requestMatrixId, + requestIdFor, +} from './run.mjs'; + +function record({ + requestId, + replicate, + caseId, + humanLabel, + predictedLabel, + grounded = true, + latencyMs = 100, + cost = 0.001, +}) { + return { + schema_version: 1, + record_type: 'visual-eval-request', + request_id: requestId, + experiment_id: 'experiment-1', + experiment_contract_id: 'd'.repeat(64), + request_matrix_sha256: 'e'.repeat(64), + replicate, + provider: 'fixture', + model: 'tiny', + model_rank: 1, + model_class: 'small', + pass: 'layout', + criterion_id: 'text-visibly-clipped', + configured_batch_size: 1, + actual_batch_size: 1, + image_detail: 'high', + prefix_id: 'a'.repeat(64), + suffix_sha256: 'b'.repeat(64), + randomization_key: 'c'.repeat(64), + case_ids: [caseId], + image_ids: [`image:${caseId}`], + response: { json_valid: true, abstained: false }, + telemetry: { + latency_ms: latencyMs, + time_to_first_token_ms: latencyMs / 2, + input_tokens: 100, + cache_read_tokens: 50, + cache_write_tokens: 0, + output_tokens: 20, + actual_cost_usd: cost, + }, + observations: [{ + observation_id: `${requestId}:${caseId}`, + case_id: caseId, + image_id: `image:${caseId}`, + image_sha256: crypto.createHash('sha256').update(caseId).digest('hex'), + criterion_id: 'text-visibly-clipped', + human_label: humanLabel, + predicted_label: predictedLabel, + grounded, + json_valid: true, + abstained: false, + }], + synthetic: false, + }; +} + +test('aggregation emits selector counts, unique evidence, telemetry, and p95', () => { + const records = [ + record({ requestId: 'r1-fire', replicate: 1, caseId: 'fire-a', humanLabel: 'fire', predictedLabel: 'fire', latencyMs: 100 }), + record({ requestId: 'r1-clean', replicate: 1, caseId: 'clean-a', humanLabel: 'clean', predictedLabel: 'clean', latencyMs: 200 }), + record({ requestId: 'r2-fire', replicate: 2, caseId: 'fire-a', humanLabel: 'fire', predictedLabel: 'fire', latencyMs: 300 }), + record({ requestId: 'r2-clean', replicate: 2, caseId: 'clean-a', humanLabel: 'clean', predictedLabel: 'clean', latencyMs: 400 }), + ]; + const measurements = aggregateRecords(records, { + candidates: [{ id: 'tiny', rank: 1, class: 'small', provider: 'fixture' }], + }); + const [result] = measurements.results; + + assert.equal(result.runs, 2); + assert.equal(result.cases, 4); + assert.equal(result.positives, 2); + assert.equal(result.negatives, 2); + assert.equal(result.unique_cases, 2); + assert.equal(result.unique_positives, 1); + assert.equal(result.unique_negatives, 1); + assert.equal(result.tp, 2); + assert.equal(result.tn, 2); + assert.equal(result.grounded, 2); + assert.equal(result.grounding_total, 2); + assert.equal(result.total_cost_usd, 0.004); + assert.equal(result.p95_latency_ms, 400); + assert.equal(result.adjudication_complete, true); + assert.equal(result.synthetic, false); + assert.equal(result.telemetry_complete, true); + assert.equal(result.experiment_complete, false); +}); + +test('unique evidence is keyed by rendered pixels rather than case id', () => { + const first = record({ + requestId: 'pixel-copy-a', + replicate: 1, + caseId: 'case-a', + humanLabel: 'fire', + predictedLabel: 'fire', + }); + const second = record({ + requestId: 'pixel-copy-b', + replicate: 2, + caseId: 'case-b', + humanLabel: 'fire', + predictedLabel: 'fire', + }); + second.observations[0].image_sha256 = first.observations[0].image_sha256; + const measurements = aggregateRecords([first, second], { + candidates: [{ id: 'tiny', rank: 1, class: 'small', provider: 'fixture' }], + }); + assert.equal(measurements.results[0].cases, 2); + assert.equal(measurements.results[0].unique_cases, 1); + assert.equal(measurements.results[0].unique_positives, 1); +}); + +test('unadjudicated disagreements remain visible and cannot become policy evidence', () => { + const records = [ + record({ + requestId: 'false-positive', + replicate: 1, + caseId: 'clean-a', + humanLabel: 'clean', + predictedLabel: 'fire', + }), + ]; + const measurements = aggregateRecords(records, { + candidates: [{ id: 'tiny', rank: 1, class: 'small', provider: 'fixture' }], + }); + assert.equal(measurements.results[0].fp, 1); + assert.equal(measurements.results[0].adjudication_complete, false); + assert.deepEqual(measurements.pending_adjudication, ['false-positive:clean-a']); +}); + +test('append-only adjudication records can confirm a disagreement', () => { + const request = record({ + requestId: 'false-positive', + replicate: 1, + caseId: 'clean-a', + humanLabel: 'clean', + predictedLabel: 'fire', + }); + const adjudication = { + schema_version: 1, + record_type: 'visual-eval-adjudication', + observation_id: 'false-positive:clean-a', + decision: 'confirm-corpus-label', + adjudicator: 'human-reviewer', + notes: 'The authored asymmetry is intentional and the model finding is false.', + recorded_at: '2026-07-25T00:00:00.000Z', + }; + const measurements = aggregateRecords([request, adjudication], { + candidates: [{ id: 'tiny', rank: 1, class: 'small', provider: 'fixture' }], + }); + assert.equal(measurements.results[0].adjudication_complete, true); + assert.deepEqual(measurements.pending_adjudication, []); +}); + +test('a corpus relabel applies consistently to every replicate of the case', () => { + const first = record({ + requestId: 'relabel-r1', + replicate: 1, + caseId: 'case-a', + humanLabel: 'fire', + predictedLabel: 'clean', + }); + const second = record({ + requestId: 'relabel-r2', + replicate: 2, + caseId: 'case-a', + humanLabel: 'fire', + predictedLabel: 'clean', + }); + const adjudication = { + schema_version: 1, + record_type: 'visual-eval-adjudication', + observation_id: 'relabel-r1:case-a', + decision: 'relabel-corpus', + human_label: 'clean', + adjudicator: 'human-reviewer', + notes: 'The reviewed state is an intentional clean control.', + recorded_at: '2026-07-25T00:00:00.000Z', + }; + const measurements = aggregateRecords([first, second, adjudication], { + candidates: [{ id: 'tiny', rank: 1, class: 'small', provider: 'fixture' }], + }); + assert.equal(measurements.results[0].tn, 2); + assert.equal(measurements.results[0].adjudication_complete, true); + assert.deepEqual(measurements.pending_adjudication, []); +}); + +test('synthetic adapter records are marked non-policy evidence', () => { + const synthetic = { + ...record({ + requestId: 'fixture-only', + replicate: 1, + caseId: 'fire-a', + humanLabel: 'fire', + predictedLabel: 'fire', + }), + synthetic: true, + }; + const measurements = aggregateRecords([synthetic], { + candidates: [{ id: 'tiny', rank: 1, class: 'small', provider: 'fixture' }], + }); + assert.equal(measurements.results[0].synthetic, true); +}); + +test('aggregation rejects undersized requests in a configured batch cell', () => { + const undersized = record({ + requestId: 'undersized', + replicate: 1, + caseId: 'fire-a', + humanLabel: 'fire', + predictedLabel: 'fire', + }); + undersized.configured_batch_size = 4; + assert.throws( + () => aggregateRecords([undersized], { + candidates: [{ id: 'tiny', rank: 1, class: 'small', provider: 'fixture' }], + }), + /actual_batch_size must equal configured_batch_size/, + ); +}); + +test('aggregation fails closed on malformed batch cardinality and missing provenance', () => { + const malformed = record({ + requestId: 'malformed-batch', + replicate: 1, + caseId: 'fire-a', + humanLabel: 'fire', + predictedLabel: 'fire', + }); + malformed.configured_batch_size = 2; + malformed.actual_batch_size = 2; + assert.throws( + () => aggregateRecords([malformed], { + candidates: [{ id: 'tiny', rank: 1, class: 'small', provider: 'fixture' }], + }), + /evidence arrays must match configured_batch_size/, + ); + + const missingProvenance = record({ + requestId: 'missing-provenance', + replicate: 1, + caseId: 'clean-a', + humanLabel: 'clean', + predictedLabel: 'clean', + }); + delete missingProvenance.synthetic; + assert.throws( + () => aggregateRecords([missingProvenance], { + candidates: [{ id: 'tiny', rank: 1, class: 'small', provider: 'fixture' }], + }), + /requires synthetic/, + ); +}); + +test('incomplete telemetry remains null rather than becoming a zero measurement', () => { + const incomplete = record({ + requestId: 'incomplete-telemetry', + replicate: 1, + caseId: 'fire-a', + humanLabel: 'fire', + predictedLabel: 'fire', + }); + incomplete.telemetry = { + complete: false, + latency_ms: 100, + time_to_first_token_ms: null, + input_tokens: null, + cache_read_tokens: null, + cache_write_tokens: null, + output_tokens: null, + actual_cost_usd: null, + }; + const measurements = aggregateRecords([incomplete], { + candidates: [{ id: 'tiny', rank: 1, class: 'small', provider: 'fixture' }], + }); + assert.equal(measurements.results[0].total_cost_usd, null); + assert.equal(measurements.results[0].p95_latency_ms, null); + assert.equal(measurements.results[0].p95_time_to_first_token_ms, null); +}); + +test('aggregation binds qualification to the complete expected request matrix', async () => { + const corpusPath = fileURLToPath(new URL('./corpus.json', import.meta.url)); + const corpus = await loadCorpus(corpusPath); + const candidate = { id: 'tiny', rank: 1, class: 'small', provider: 'fixture' }; + const experiment = { + id: 'matrix-contract', + corpus_path: corpusPath, + run_candidate: candidate.id, + candidates: [candidate], + image_detail: 'high', + batch_sizes: [4], + replicates: 1, + passes: ['layout'], + thresholds: {}, + ladder_state: { completed_candidates: [] }, + }; + const cases = expandCorpus(corpus, { corpusPath }) + .filter((entry) => entry.family === 'layout'); + const plan = buildExperimentPlan(experiment, cases); + const preparedPlan = plan.map((cell) => { + const requestId = requestIdFor(experiment.id, cell); + return { + ...cell, + experiment_id: experiment.id, + request: { + prefix_manifest: { + prefix_id: crypto.createHash('sha256').update(`prefix:${requestId}`).digest('hex'), + images: cell.cases.map((entry) => ({ + id: entry.image.id, + sha256: crypto.createHash('sha256').update(entry.case_id).digest('hex'), + })), + }, + suffix: { + suffix_sha256: crypto.createHash('sha256').update(`suffix:${requestId}`).digest('hex'), + }, + }, + }; + }); + const matrixId = requestMatrixId(preparedPlan); + const contractId = experimentContractId(experiment, corpus, matrixId); + const records = preparedPlan.map((cell) => { + const requestId = requestIdFor(experiment.id, cell); + return { + schema_version: 1, + record_type: 'visual-eval-request', + request_id: requestId, + experiment_id: experiment.id, + experiment_contract_id: contractId, + request_matrix_sha256: matrixId, + replicate: cell.replicate, + provider: candidate.provider, + model: candidate.id, + model_rank: candidate.rank, + model_class: candidate.class, + pass: cell.pass, + criterion_id: cell.criterion_id, + configured_batch_size: cell.configured_batch_size, + actual_batch_size: cell.actual_batch_size, + image_detail: cell.image_detail, + prefix_id: cell.request.prefix_manifest.prefix_id, + suffix_sha256: cell.request.suffix.suffix_sha256, + randomization_key: cell.randomization_key, + case_ids: cell.cases.map((entry) => entry.case_id), + image_ids: cell.cases.map((entry) => entry.image.id), + response: { json_valid: true, abstained: false }, + telemetry: { + complete: true, + latency_ms: 100, + time_to_first_token_ms: 50, + input_tokens: 100, + cache_read_tokens: 0, + cache_write_tokens: 0, + output_tokens: 20, + actual_cost_usd: 0.001, + }, + observations: cell.cases.map((entry) => ({ + observation_id: `${requestId}:${entry.case_id}`, + case_id: entry.case_id, + image_id: entry.image.id, + image_sha256: crypto.createHash('sha256').update(entry.case_id).digest('hex'), + criterion_id: entry.criterion_id, + human_label: entry.human_label, + predicted_label: entry.human_label, + grounded: entry.human_label === 'fire', + json_valid: true, + abstained: false, + })), + synthetic: false, + }; + }); + const options = { + candidates: [candidate], + experiment, + corpus, + preparedPlan, + }; + const complete = aggregateRecords(records, options); + assert.equal(complete.results.every((entry) => entry.experiment_complete), true); + const partial = aggregateRecords(records.slice(1), options); + assert.equal(partial.results.every((entry) => !entry.experiment_complete), true); +}); diff --git a/evals/visual-model-policy/corpus.json b/evals/visual-model-policy/corpus.json new file mode 100644 index 0000000..3915f6f --- /dev/null +++ b/evals/visual-model-policy/corpus.json @@ -0,0 +1,1112 @@ +{ + "schema_version": 1, + "corpus_id": "artifacture-owned-visual-v1", + "rendered_root": "corpus/rendered", + "label_review": { + "status": "pending-human-review", + "reviewer": null, + "reviewed_at": null, + "notes": "Proposed labels follow the authored pair intent. A human reviewer must inspect every rendered pair and sign off before live measurements." + }, + "families": [ + { + "id": "layout", + "route_key": "layout", + "rubric": "../../plugins/visual-explainer/scripts/verify/rubrics/pass-layout.md", + "pairs": [ + { + "id": "mobile-heading-edge", + "criterion_id": "text-visibly-clipped", + "region": { "id": "hero-heading", "label": "Mobile hero heading" }, + "fire": { + "title": "Heading clipped at the viewport edge", + "visible_text": "Operational evidence architecture", + "truth_excerpt": "The full heading is authored as “Operational evidence architecture.”", + "adjudication_notes": "The final word is cut mid-character at the right edge; no ellipsis or disclosure control is present.", + "render": { "template": "text", "variant": "clipped-right" } + }, + "clean": { + "title": "Long heading wraps intentionally", + "visible_text": "Operational evidence architecture", + "truth_excerpt": "The heading may wrap to preserve every authored word.", + "adjudication_notes": "The two-line wrap is intentional and every character remains visible; line wrapping is not clipping.", + "render": { "template": "text", "variant": "wrapped" } + } + }, + { + "id": "dense-table-label", + "criterion_id": "text-visibly-clipped", + "region": { "id": "audit-table", "label": "Dense audit table" }, + "fire": { + "title": "Audit label chopped inside a fixed cell", + "visible_text": "Representative calibration queue", + "truth_excerpt": "The complete queue name is required to distinguish it from the frontier queue.", + "adjudication_notes": "The label is visibly chopped inside an overflow-hidden cell and cannot be recovered from adjacent content.", + "render": { "template": "table", "variant": "clipped-label" } + }, + "clean": { + "title": "Dense expert table scrolls without truncating labels", + "visible_text": "Representative calibration queue", + "truth_excerpt": "Dense expert surfaces are allowed when their labels remain available.", + "adjudication_notes": "The table is intentionally dense, but the label is complete and the horizontal overflow affordance is visible.", + "render": { "template": "table", "variant": "dense-scroll" } + } + }, + { + "id": "title-diagram-kpis", + "criterion_id": "slide-single-focal-point", + "region": { "id": "slide-canvas", "label": "Presentation slide canvas" }, + "fire": { + "title": "Three primary treatments compete", + "visible_text": "Policy router · 93% coverage · Review topology", + "truth_excerpt": "The slide’s narrative job is to explain the routing topology.", + "adjudication_notes": "A huge title, saturated diagram, and equally large KPI row all demand first attention, leaving no single focal point.", + "render": { "template": "slide", "variant": "competing-foci" } + }, + "clean": { + "title": "Dense evidence supports one dominant diagram", + "visible_text": "Policy router · 93% coverage · Review topology", + "truth_excerpt": "The router is the claim; supporting metrics may remain subordinate.", + "adjudication_notes": "The surface contains many expert details, but scale and contrast clearly make the router diagram the single focal element.", + "render": { "template": "slide", "variant": "dense-one-focus" } + } + }, + { + "id": "comparison-dashboard", + "criterion_id": "slide-single-focal-point", + "region": { "id": "dashboard-main", "label": "Comparison dashboard" }, + "fire": { + "title": "Every dashboard tile is visually primary", + "visible_text": "Latency · Cost · Recall · Coverage", + "truth_excerpt": "The decision is whether recall remains above the graduation gate.", + "adjudication_notes": "Four equally loud metric tiles and two charts compete despite one decision-driving measure.", + "render": { "template": "dashboard", "variant": "uniform-loud" } + }, + "clean": { + "title": "Expert dashboard emphasizes the decision metric", + "visible_text": "Latency · Cost · Recall · Coverage", + "truth_excerpt": "Recall is the decision-driving measure; other metrics are supporting context.", + "adjudication_notes": "The dashboard remains information-dense but recall is clearly dominant through placement and type scale.", + "render": { "template": "dashboard", "variant": "hierarchical-dense" } + } + }, + { + "id": "tiny-diagram-ocean", + "criterion_id": "sparse-diagram-slide", + "region": { "id": "diagram-frame", "label": "Diagram and surrounding canvas" }, + "fire": { + "title": "Tiny diagram floats in unexplained space", + "visible_text": "Input → Review → Output", + "truth_excerpt": "The three-stage route is the full content of the slide.", + "adjudication_notes": "The diagram occupies a small corner while most of the canvas is unexplained blank space, making the content feel missing.", + "render": { "template": "diagram-space", "variant": "tiny-unbalanced" } + }, + "clean": { + "title": "Authored breathing room frames a short claim", + "visible_text": "Input → Review → Output", + "truth_excerpt": "The brief calls for a quiet transition slide with one compact route.", + "adjudication_notes": "The blank area is deliberately balanced around a centered claim and does not resemble a collapsed or missing region.", + "render": { "template": "diagram-space", "variant": "intentional-breathing" } + } + }, + { + "id": "collapsed-peer-column", + "criterion_id": "sparse-diagram-slide", + "region": { "id": "peer-map", "label": "Three-column peer map" }, + "fire": { + "title": "Collapsed third column leaves accidental dead space", + "visible_text": "Signals · Policy · Actions", + "truth_excerpt": "All three peer stages have content in the source brief.", + "adjudication_notes": "The third stage collapses to a tiny block and leaves a large blank region that reads as missing content.", + "render": { "template": "diagram-space", "variant": "collapsed-column" } + }, + "clean": { + "title": "Evidence rail intentionally leaves reading space", + "visible_text": "Narrative · Evidence", + "truth_excerpt": "The composition is an editorial 70/30 narrative and evidence split.", + "adjudication_notes": "The narrow evidence rail and open lower area are authored hierarchy, not a missing peer column.", + "render": { "template": "diagram-space", "variant": "editorial-space" } + } + }, + { + "id": "three-peer-widths", + "criterion_id": "layout-repeated-track-symmetry", + "region": { "id": "peer-tracks", "label": "Input, router, and output tracks" }, + "fire": { + "title": "Equivalent peer tracks drift in width", + "visible_text": "Input · Router · Output", + "truth_excerpt": "The three stages are declared as equivalent review tracks.", + "adjudication_notes": "The middle track is inexplicably narrower and divider positions do not align across otherwise equivalent regions.", + "render": { "template": "tracks", "variant": "unequal-peers" } + }, + "clean": { + "title": "Intentional 60/40 editorial split", + "visible_text": "Narrative · Evidence", + "truth_excerpt": "The narrative is primary and the evidence rail is explicitly supporting.", + "adjudication_notes": "The unequal widths express different content roles and therefore should not be treated as failed peer symmetry.", + "render": { "template": "tracks", "variant": "editorial-60-40" } + } + }, + { + "id": "row-baseline-drift", + "criterion_id": "layout-repeated-track-symmetry", + "region": { "id": "comparison-rows", "label": "Before and after comparison rows" }, + "fire": { + "title": "Peer row baselines do not align", + "visible_text": "Before · After", + "truth_excerpt": "Rows represent direct before/after counterparts.", + "adjudication_notes": "Corresponding row dividers drift vertically, undermining the declared one-to-one comparison.", + "render": { "template": "tracks", "variant": "baseline-drift" } + }, + "clean": { + "title": "Content-driven masonry is not a peer table", + "visible_text": "Field notes · Evidence fragments", + "truth_excerpt": "Items are independent editorial fragments with deliberately varied lengths.", + "adjudication_notes": "The staggered baselines belong to an editorial masonry composition, not equivalent repeated rows.", + "render": { "template": "tracks", "variant": "masonry" } + } + }, + { + "id": "missing-track-weight", + "criterion_id": "layout-repeated-track-symmetry", + "region": { "id": "quadrant-grid", "label": "Four peer quadrants" }, + "fire": { + "title": "One peer quadrant appears absent", + "visible_text": "Detect · Route · Review · Learn", + "truth_excerpt": "The operating loop defines four peer responsibilities.", + "adjudication_notes": "One quadrant receives a fraction of the area and large accidental whitespace makes it appear unimplemented.", + "render": { "template": "tracks", "variant": "missing-weight" } + }, + "clean": { + "title": "One dominant result spans two grid columns", + "visible_text": "Inputs · Method · Result", + "truth_excerpt": "The result is explicitly the conclusion and should dominate supporting inputs.", + "adjudication_notes": "The spanning result is intentional hierarchy rather than an equivalent peer rendered at the wrong size.", + "render": { "template": "tracks", "variant": "hero-span" } + } + }, + { + "id": "mobile-control-crowding", + "criterion_id": "slide-single-focal-point", + "region": { "id": "mobile-header", "label": "Mobile header and primary content" }, + "fire": { + "title": "Controls crowd the primary claim", + "visible_text": "Model policy · Theme · Filter · Export", + "truth_excerpt": "The page title and selected policy are the primary content.", + "adjudication_notes": "Large controls surround and visually overpower the title, producing several competing focal targets.", + "render": { "template": "mobile", "variant": "crowded-controls" } + }, + "clean": { + "title": "Compact expert controls remain subordinate", + "visible_text": "Model policy · Theme · Filter · Export", + "truth_excerpt": "Frequent operators need all controls available without losing the page claim.", + "adjudication_notes": "The surface is dense by design, but compact chrome remains subordinate to one clear title and active policy.", + "render": { "template": "mobile", "variant": "compact-controls" } + } + } + ] + }, + { + "id": "diagram", + "route_key": "diagram", + "rubric": "../../plugins/visual-explainer/scripts/verify/rubrics/pass-diagram.md", + "pairs": [ + { + "id": "mixed-grammar", + "criterion_id": "diagram-type-coherent", + "region": { "id": "figure", "label": "Architecture figure" }, + "fire": { + "title": "Swimlanes and containment conflict", + "visible_text": "Client · Router · Cache · Store", + "truth_excerpt": "The content describes one request flow.", + "adjudication_notes": "Half the figure uses lanes while the other half uses nested containment for the same relation.", + "render": { "template": "diagram", "variant": "mixed-grammar" } + }, + "clean": { + "title": "One flow grammar with a highlighted boundary", + "visible_text": "Client · Router · Cache · Store", + "truth_excerpt": "The boundary denotes deployment ownership inside one request flow.", + "adjudication_notes": "A boundary annotation does not create a second diagram type; connectors and nodes retain one stable grammar.", + "render": { "template": "diagram", "variant": "coherent-flow" } + } + }, + { + "id": "timeline-scale", + "criterion_id": "diagram-proportional-honesty-visual", + "region": { "id": "timeline", "label": "Delivery timeline" }, + "fire": { + "title": "One-day and six-month gaps look equal", + "visible_text": "Day 1 · Day 2 · Month 6", + "truth_excerpt": "The milestones occur after one day and then roughly six months.", + "adjudication_notes": "Equal visual spacing materially misrepresents the stated time intervals.", + "render": { "template": "diagram", "variant": "dishonest-timeline" } + }, + "clean": { + "title": "Compressed timeline explicitly marks a break", + "visible_text": "Day 1 · Day 2 · // · Month 6", + "truth_excerpt": "The timeline is compressed and must disclose the discontinuity.", + "adjudication_notes": "A visible axis break and label make the non-linear spacing explicit rather than deceptively proportional.", + "render": { "template": "diagram", "variant": "timeline-break" } + } + }, + { + "id": "quantity-bars", + "criterion_id": "diagram-proportional-honesty-visual", + "region": { "id": "bar-figure", "label": "Cost comparison bars" }, + "fire": { + "title": "Near-equal bars represent a fourfold difference", + "visible_text": "$10 · $40", + "truth_excerpt": "The second value is four times the first.", + "adjudication_notes": "The bars differ only slightly in length, visually erasing a load-bearing fourfold ratio.", + "render": { "template": "diagram", "variant": "dishonest-bars" } + }, + "clean": { + "title": "Symbolic comparison avoids a false quantitative scale", + "visible_text": "$10 baseline · $40 candidate", + "truth_excerpt": "The composition is a labeled comparison and does not claim scaled bar length.", + "adjudication_notes": "Equal containers are acceptable because explicit values carry the comparison and no quantitative axis is implied.", + "render": { "template": "diagram", "variant": "labeled-comparison" } + } + }, + { + "id": "legend-orphan", + "criterion_id": "diagram-legend-matches-figure", + "region": { "id": "legend-and-plot", "label": "Legend and plotted categories" }, + "fire": { + "title": "Legend names an absent cache category", + "visible_text": "API · Cache · Store", + "truth_excerpt": "Only API and Store categories are drawn.", + "adjudication_notes": "The legend includes Cache even though no corresponding mark exists in the figure.", + "render": { "template": "diagram", "variant": "orphan-legend" } + }, + "clean": { + "title": "Muted category still appears in the figure", + "visible_text": "API · Cache · Store", + "truth_excerpt": "Cache is present but intentionally de-emphasized because it is inactive.", + "adjudication_notes": "Every legend category has a plotted counterpart; low emphasis is not absence.", + "render": { "template": "diagram", "variant": "complete-legend" } + } + }, + { + "id": "missing-legend-category", + "criterion_id": "diagram-legend-matches-figure", + "region": { "id": "legend-and-nodes", "label": "Node categories and legend" }, + "fire": { + "title": "Decision nodes lack a legend entry", + "visible_text": "Service · Data", + "truth_excerpt": "Diamonds encode decisions and rectangles encode services.", + "adjudication_notes": "A distinct load-bearing diamond category is used repeatedly but the legend never defines it.", + "render": { "template": "diagram", "variant": "missing-legend-entry" } + }, + "clean": { + "title": "Annotation callout does not need a legend category", + "visible_text": "Service · Data", + "truth_excerpt": "The lone dashed note is explanatory annotation, not a data category.", + "adjudication_notes": "A conventional one-off annotation is self-labelled and does not create an unexplained visual category.", + "render": { "template": "diagram", "variant": "annotation-no-legend" } + } + }, + { + "id": "three-box-list", + "criterion_id": "diagram-necessity", + "region": { "id": "figure", "label": "Three-item figure" }, + "fire": { + "title": "Three independent bullets are boxed and arrowed", + "visible_text": "Reliable · Fast · Affordable", + "truth_excerpt": "The brief lists three independent qualities with no sequence or relationship.", + "adjudication_notes": "Boxes and arrows add a false flow while conveying no information beyond a short list.", + "render": { "template": "diagram", "variant": "decorated-list" } + }, + "clean": { + "title": "Three nodes expose a real dependency", + "visible_text": "Capture → Adjudicate → Graduate", + "truth_excerpt": "Graduation requires adjudication, which requires captured observations.", + "adjudication_notes": "The connectors encode necessary dependency and therefore add information beyond the labels.", + "render": { "template": "diagram", "variant": "necessary-sequence" } + } + }, + { + "id": "peer-table-as-graph", + "criterion_id": "diagram-necessity", + "region": { "id": "comparison-figure", "label": "Model comparison figure" }, + "fire": { + "title": "Model comparison is an arbitrary node graph", + "visible_text": "Nano · Mini · Frontier", + "truth_excerpt": "The source contains aligned latency, accuracy, and cost values.", + "adjudication_notes": "A graph obscures exact aligned comparisons that a compact table would communicate more faithfully.", + "render": { "template": "diagram", "variant": "table-as-graph" } + }, + "clean": { + "title": "Network topology needs relational placement", + "visible_text": "Gateway · Workers · Stores", + "truth_excerpt": "The claim depends on fan-out, shared stores, and retry paths.", + "adjudication_notes": "Topology and connector direction are load-bearing; a flat table would discard the relevant relationships.", + "render": { "template": "diagram", "variant": "necessary-topology" } + } + }, + { + "id": "dual-focal", + "criterion_id": "diagram-focal-single-dominant", + "region": { "id": "system-map", "label": "System map" }, + "fire": { + "title": "Two unrelated nodes claim equal dominance", + "visible_text": "Policy Router · Analytics Hub", + "truth_excerpt": "The narrative job is to explain policy routing.", + "adjudication_notes": "Two saturated oversized nodes compete even though only the router is load-bearing to the claim.", + "render": { "template": "diagram", "variant": "dual-focal" } + }, + "clean": { + "title": "Two endpoints frame one dominant transformation", + "visible_text": "Evidence · Policy Router · Decision", + "truth_excerpt": "The policy router is the transformation under explanation.", + "adjudication_notes": "Endpoints remain visible but the central router is unmistakably dominant.", + "render": { "template": "diagram", "variant": "single-focal" } + } + }, + { + "id": "uniform-box-forest", + "criterion_id": "diagram-type-coherent", + "region": { "id": "org-map", "label": "Organization map" }, + "fire": { + "title": "Hierarchy and sequence use identical connectors", + "visible_text": "Team · Stage · Owner", + "truth_excerpt": "The source distinguishes reporting hierarchy from workflow sequence.", + "adjudication_notes": "Identical lines and boxes collapse two different relation types into an incoherent grammar.", + "render": { "template": "diagram", "variant": "ambiguous-relations" } + }, + "clean": { + "title": "Uniform node forest is the requested inventory", + "visible_text": "Service A · Service B · Service C", + "truth_excerpt": "The figure is an inventory grouped only by deployment zone.", + "adjudication_notes": "Uniform boxes are truthful because the nodes are genuine peers and containment is the sole relation.", + "render": { "template": "diagram", "variant": "peer-inventory" } + } + }, + { + "id": "residual-probability", + "criterion_id": "diagram-proportional-honesty-visual", + "region": { "id": "probability-map", "label": "Probability distribution" }, + "fire": { + "title": "Named outcomes consume more than the whole", + "visible_text": "A 55% · B 35% · Other 20%", + "truth_excerpt": "Displayed probabilities must sum to the complete distribution.", + "adjudication_notes": "The visual encodes 110 percent while presenting a complete distribution, a material proportional contradiction.", + "render": { "template": "diagram", "variant": "invalid-probability" } + }, + "clean": { + "title": "Residual probability remains visible", + "visible_text": "A 55% · B 25% · Other 20%", + "truth_excerpt": "Residual probability is material and must remain represented.", + "adjudication_notes": "All proportions sum to one hundred percent and the residual category is explicitly preserved.", + "render": { "template": "diagram", "variant": "valid-probability" } + } + }, + { + "id": "competing-control-planes", + "criterion_id": "diagram-focal-single-dominant", + "region": { "id": "control-plane-map", "label": "Control-plane architecture map" }, + "fire": { + "title": "Two control planes receive equal visual dominance", + "visible_text": "Admission Gate · Policy Engine · Audit Store", + "truth_excerpt": "The policy engine is the decision-making center; admission and audit are supporting boundaries.", + "adjudication_notes": "The admission gate and policy engine are both oversized and saturated, so the diagram gives two competing answers to what controls the system.", + "render": { "template": "diagram", "variant": "dual-focal" } + }, + "clean": { + "title": "One policy engine dominates supporting boundaries", + "visible_text": "Admission Gate · Policy Engine · Audit Store", + "truth_excerpt": "The policy engine is the decision-making center; admission and audit are supporting boundaries.", + "adjudication_notes": "The central policy engine receives the sole dominant treatment while the admission gate and audit store remain legible supporting endpoints.", + "render": { "template": "diagram", "variant": "single-focal" } + } + } + ] + }, + { + "id": "aesthetic", + "route_key": "aesthetic", + "rubric": "../../plugins/visual-explainer/scripts/verify/rubrics/pass-aesthetic.md", + "pairs": [ + { + "id": "light-mode-panel", + "criterion_id": "preset-both-mode-visual", + "region": { "id": "mode-pair", "label": "Light and dark mode pair" }, + "fire": { + "title": "Hard-coded dark panel leaks into light mode", + "visible_text": "Status overview", + "truth_excerpt": "Both screenshots should express the same preset in their respective mode.", + "adjudication_notes": "One panel remains dark only in the light screenshot and its text fails to invert with the rest of the preset.", + "render": { "template": "preset", "variant": "wrong-mode-panel" } + }, + "clean": { + "title": "Dark data surface is intentional in both modes", + "visible_text": "Terminal trace", + "truth_excerpt": "The embedded terminal is explicitly a dark simulated surface in either page mode.", + "adjudication_notes": "The stable dark terminal is content semantics, not a failed page-mode inversion, and its own text remains readable.", + "render": { "template": "preset", "variant": "intentional-terminal" } + } + }, + { + "id": "mode-icon-fill", + "criterion_id": "preset-both-mode-visual", + "region": { "id": "mode-pair", "label": "Mode comparison with icons" }, + "fire": { + "title": "Icons disappear against the dark surface", + "visible_text": "Inputs · Results", + "truth_excerpt": "Icon and label meaning must remain visible in both modes.", + "adjudication_notes": "The dark screenshot retains a dark icon fill, making load-bearing symbols effectively invisible.", + "render": { "template": "preset", "variant": "invisible-dark-icons" } + }, + "clean": { + "title": "Decorative watermark fades in dark mode", + "visible_text": "Inputs · Results", + "truth_excerpt": "The watermark is non-semantic decoration and may be quiet.", + "adjudication_notes": "All load-bearing icons and text remain visible; only a deliberately non-semantic watermark loses prominence.", + "render": { "template": "preset", "variant": "quiet-watermark" } + } + }, + { + "id": "mono-status-container", + "criterion_id": "mono-status-value-judgment", + "region": { "id": "status-board", "label": "Mono-Industrial status board" }, + "fire": { + "title": "Orange tints an entire status section", + "visible_text": "Review queue", + "truth_excerpt": "Mono-Industrial reserves color for specific values with status meaning.", + "adjudication_notes": "Orange colors a container and heading without representing an actual warning value.", + "render": { "template": "preset", "variant": "decorative-status-color" } + }, + "clean": { + "title": "One warning value carries orange", + "visible_text": "Review queue · 3 delayed", + "truth_excerpt": "Three delayed items are a measured warning state.", + "adjudication_notes": "Color is scoped to the precise delayed count and therefore conveys genuine value semantics.", + "render": { "template": "preset", "variant": "semantic-status-color" } + } + }, + { + "id": "mono-surprise", + "criterion_id": "mono-one-surprise", + "region": { "id": "mono-page", "label": "Mono-Industrial composition" }, + "fire": { + "title": "Several unrelated elements break the grid", + "visible_text": "Policy · Evidence · Result", + "truth_excerpt": "The preset permits one deliberate compositional or typographic surprise.", + "adjudication_notes": "Three rotated and oversized elements break the system, so no single authored surprise remains.", + "render": { "template": "preset", "variant": "many-surprises" } + }, + "clean": { + "title": "One oversized decision breaks the pattern", + "visible_text": "Policy · DECISION · Result", + "truth_excerpt": "The decision is the one intended moment of surprise.", + "adjudication_notes": "Exactly one typographic element breaks the otherwise consistent system and the break supports hierarchy.", + "render": { "template": "preset", "variant": "one-surprise" } + } + }, + { + "id": "mono-three-layers", + "criterion_id": "mono-three-layer-squint", + "region": { "id": "mono-page", "label": "Mono-Industrial hierarchy" }, + "fire": { + "title": "Five equally loud emphasis layers", + "visible_text": "Title · KPI · Section · Label · Note", + "truth_excerpt": "The active preset calls for three legible emphasis layers.", + "adjudication_notes": "Multiple near-equal weights and sizes create at least five competing emphasis levels with no stable hierarchy.", + "render": { "template": "preset", "variant": "too-many-layers" } + }, + "clean": { + "title": "Dense page resolves into three emphasis layers", + "visible_text": "Title · KPI · Section · Label · Note", + "truth_excerpt": "Dense content is compatible with three clearly grouped emphasis levels.", + "adjudication_notes": "Many elements are present, but they resolve into a dominant hero, section level, and quiet instrument text.", + "render": { "template": "preset", "variant": "three-layers" } + } + }, + { + "id": "nothing-red-underline", + "criterion_id": "nothing-accent-red-judgment", + "region": { "id": "nothing-page", "label": "Nothing preset page" }, + "fire": { + "title": "Red accent is a decorative underline", + "visible_text": "Signal architecture", + "truth_excerpt": "Nothing red is reserved for urgent, destructive, or error meaning.", + "adjudication_notes": "The single red use merely decorates a neutral heading and communicates no status.", + "render": { "template": "preset", "variant": "decorative-red" } + }, + "clean": { + "title": "Red marks a destructive reset action", + "visible_text": "Reset policy", + "truth_excerpt": "Resetting the policy is destructive and requires explicit warning.", + "adjudication_notes": "The red accent is scoped to a genuine destructive action rather than used as ornament.", + "render": { "template": "preset", "variant": "destructive-red" } + } + }, + { + "id": "nothing-grid-break", + "criterion_id": "nothing-single-grid-break", + "region": { "id": "nothing-grid", "label": "Nothing preset grid" }, + "fire": { + "title": "Three cards independently escape the grid", + "visible_text": "Capture · Route · Learn", + "truth_excerpt": "The preset allows one meaningful grid break.", + "adjudication_notes": "Several arbitrary offsets break the composition without establishing one deliberate exception.", + "render": { "template": "preset", "variant": "many-grid-breaks" } + }, + "clean": { + "title": "Full-width navigation is ordinary chrome", + "visible_text": "Capture · Route · Learn", + "truth_excerpt": "Full-width navigation is excluded from the single content-grid-break count.", + "adjudication_notes": "One content element breaks the grid; the full-width navigation is legitimate chrome and not a second break.", + "render": { "template": "preset", "variant": "one-grid-break" } + } + }, + { + "id": "demo-frame", + "criterion_id": "demo-aesthetic-match", + "region": { "id": "demo-embed", "label": "Embedded demo and host page" }, + "fire": { + "title": "Glossy rounded demo conflicts with flat host", + "visible_text": "Live policy inspector", + "truth_excerpt": "The host preset uses flat square surfaces and restrained borders.", + "adjudication_notes": "Large radius, glow, and heavy shadow make the demo feel pasted onto a visibly different design system.", + "render": { "template": "preset", "variant": "mismatched-demo" } + }, + "clean": { + "title": "Functional browser chrome remains distinct", + "visible_text": "Live policy inspector", + "truth_excerpt": "The demo needs a minimal frame to communicate that it is interactive.", + "adjudication_notes": "The frame is functionally necessary and still matches the host’s radius, border, color, and caption treatment.", + "render": { "template": "preset", "variant": "matched-demo" } + } + }, + { + "id": "mono-mobile-hero", + "criterion_id": "mono-three-layer-squint", + "region": { "id": "mobile-hero", "label": "Mono-Industrial mobile hero" }, + "fire": { + "title": "Hero collapses into a tiny card on mobile", + "visible_text": "Route with evidence", + "truth_excerpt": "The hero should remain intact and full-width on mobile.", + "adjudication_notes": "The primary claim is shrunk into one small bordered card among peers, destroying the intended hierarchy.", + "render": { "template": "preset", "variant": "mobile-hero-card" } + }, + "clean": { + "title": "Full-width mobile hero with compact supporting cards", + "visible_text": "Route with evidence", + "truth_excerpt": "Supporting items may compact as long as the hero stays intact.", + "adjudication_notes": "The hero remains full-width and dominant while subordinate evidence becomes a denser mobile row.", + "render": { "template": "preset", "variant": "mobile-hero-full" } + } + }, + { + "id": "mode-background", + "criterion_id": "preset-both-mode-visual", + "region": { "id": "mode-pair", "label": "Full-page light and dark screenshots" }, + "fire": { + "title": "Dark mode keeps a light page background", + "visible_text": "Evidence map", + "truth_excerpt": "The named preset supports both light and dark surfaces.", + "adjudication_notes": "The dark-mode screenshot preserves the light canvas while inverting text, causing broken contrast and wrong-mode bleed.", + "render": { "template": "preset", "variant": "wrong-mode-background" } + }, + "clean": { + "title": "Paper inset remains light in both modes", + "visible_text": "Export preview", + "truth_excerpt": "A print-preview inset intentionally represents white paper in either application mode.", + "adjudication_notes": "The light inset is an explicitly labelled paper preview with readable local colors, not a failed global inversion.", + "render": { "template": "preset", "variant": "paper-preview" } + } + }, + { + "id": "mono-neutral-count", + "criterion_id": "mono-status-value-judgment", + "region": { "id": "queue-summary", "label": "Mono-Industrial queue summary" }, + "fire": { + "title": "Neutral queue total receives warning color", + "visible_text": "Calibration total · 12 cases", + "truth_excerpt": "A total case count is neutral unless a measured threshold has been exceeded.", + "adjudication_notes": "The colored status treatment decorates a neutral inventory count and implies a warning that the underlying value does not support.", + "render": { "template": "preset", "variant": "decorative-status-color" } + }, + "clean": { + "title": "Threshold breach receives scoped warning color", + "visible_text": "Calibration total · 12 overdue", + "truth_excerpt": "Twelve overdue cases exceed the documented queue threshold.", + "adjudication_notes": "The warning color is scoped to a measured threshold breach and therefore carries explicit status meaning rather than decoration.", + "render": { "template": "preset", "variant": "semantic-status-color" } + } + }, + { + "id": "mono-offset-summary", + "criterion_id": "mono-one-surprise", + "region": { "id": "summary-grid", "label": "Mono-Industrial summary grid" }, + "fire": { + "title": "Multiple summary metrics independently break the grid", + "visible_text": "Escalation summary · Cost · Recall", + "truth_excerpt": "Only the escalation decision is intended to break the otherwise regular summary grid.", + "adjudication_notes": "Several rotated and enlarged metrics compete as independent exceptions, leaving no single meaningful compositional surprise.", + "render": { "template": "preset", "variant": "many-surprises" } + }, + "clean": { + "title": "One escalation decision breaks the summary grid", + "visible_text": "Escalation summary · Cost · Recall", + "truth_excerpt": "Only the escalation decision is intended to break the otherwise regular summary grid.", + "adjudication_notes": "A single decision treatment departs from the regular grid while the supporting cost and recall measures retain the base system.", + "render": { "template": "preset", "variant": "one-surprise" } + } + }, + { + "id": "nothing-red-badge", + "criterion_id": "nothing-accent-red-judgment", + "region": { "id": "policy-action", "label": "Nothing preset policy action" }, + "fire": { + "title": "Red badge decorates a neutral policy version", + "visible_text": "Policy version 24", + "truth_excerpt": "The policy version is informational and has no urgent, destructive, or error state.", + "adjudication_notes": "Red is applied to a neutral version badge only for visual interest, creating false urgency without corresponding semantics.", + "render": { "template": "preset", "variant": "decorative-red" } + }, + "clean": { + "title": "Red identifies irreversible policy deletion", + "visible_text": "Delete policy version", + "truth_excerpt": "Deleting the policy version is irreversible and requires destructive emphasis.", + "adjudication_notes": "Red is limited to an irreversible deletion action, so the accent communicates the precise destructive meaning reserved by the preset.", + "render": { "template": "preset", "variant": "destructive-red" } + } + }, + { + "id": "nothing-evidence-offsets", + "criterion_id": "nothing-single-grid-break", + "region": { "id": "evidence-grid", "label": "Nothing preset evidence grid" }, + "fire": { + "title": "Several evidence cards use unrelated offsets", + "visible_text": "Evidence ledger · Sources · Decisions", + "truth_excerpt": "The decision ledger is the sole content element intended to cross the grid.", + "adjudication_notes": "Multiple cards rotate and offset independently, turning the permitted single grid break into a collection of arbitrary exceptions.", + "render": { "template": "preset", "variant": "many-grid-breaks" } + }, + "clean": { + "title": "One decision ledger crosses the evidence grid", + "visible_text": "Evidence ledger · Sources · Decisions", + "truth_excerpt": "The decision ledger is the sole content element intended to cross the grid.", + "adjudication_notes": "Only the decision ledger departs from the content grid; the surrounding source cards and ordinary chrome remain aligned.", + "render": { "template": "preset", "variant": "one-grid-break" } + } + }, + { + "id": "demo-inspector-theme", + "criterion_id": "demo-aesthetic-match", + "region": { "id": "inspector-embed", "label": "Embedded inspector and host preset" }, + "fire": { + "title": "Soft gradient inspector conflicts with industrial host", + "visible_text": "Evidence inspector", + "truth_excerpt": "The host uses square panels, flat fills, and instrument-like typography.", + "adjudication_notes": "The embedded inspector introduces pill controls, gradients, glow, and soft shadows that visibly belong to a different design language.", + "render": { "template": "preset", "variant": "mismatched-demo" } + }, + "clean": { + "title": "Inspector inherits the host surface language", + "visible_text": "Evidence inspector", + "truth_excerpt": "The inspector may retain functional controls while matching the host surface language.", + "adjudication_notes": "The embedded inspector retains necessary controls but matches the host’s square geometry, flat color, border weight, and typography.", + "render": { "template": "preset", "variant": "matched-demo" } + } + } + ] + }, + { + "id": "operating-model", + "route_key": "operating-model", + "rubric": "../../plugins/visual-explainer/scripts/verify/rubrics/pass-operating-model.md", + "pairs": [ + { + "id": "review-routing", + "criterion_id": "operating-model-fit", + "region": { "id": "review-routing", "label": "Review routing unit" }, + "fire": { + "title": "Four cards hide the selection policy", + "visible_text": "Mandatory · Representative · Calibration · Frontier", + "truth_excerpt": "A policy router assigns cases to distinct review queues with different entry rules.", + "adjudication_notes": "Equal cards name destinations but omit the router, mandatory path, sampling rule, and deduplication.", + "render": { "template": "operating", "variant": "routing-cards" } + }, + "clean": { + "title": "Router exposes queue selection and overlap handling", + "visible_text": "Mandatory · Representative · Calibration · Frontier", + "truth_excerpt": "A policy router assigns cases to distinct review queues with different entry rules.", + "adjudication_notes": "The composition shows the universal gate, routing rules, separate queues, and deduplication path.", + "render": { "template": "operating", "variant": "routing-model" } + } + }, + { + "id": "causal-loop", + "criterion_id": "operating-model-fit", + "region": { "id": "causal-loop", "label": "Calibration feedback loop" }, + "fire": { + "title": "Feedback loop is drawn as a one-way pipeline", + "visible_text": "Observe → Adjudicate → Calibrate → Deploy", + "truth_excerpt": "Deployment outcomes feed the next calibration cycle.", + "adjudication_notes": "The one-way pipeline omits the load-bearing return path and misstates the process as terminal.", + "render": { "template": "operating", "variant": "one-way-loop" } + }, + "clean": { + "title": "Feedback path returns outcomes to calibration", + "visible_text": "Observe → Adjudicate → Calibrate → Deploy ↩", + "truth_excerpt": "Deployment outcomes feed the next calibration cycle.", + "adjudication_notes": "The return connector and its label make the iterative causal structure reconstructable.", + "render": { "template": "operating", "variant": "closed-loop" } + } + }, + { + "id": "probability-residual", + "criterion_id": "operating-model-fit", + "region": { "id": "diagnostic-state", "label": "Probability-bearing diagnostic state" }, + "fire": { + "title": "Named diagnoses omit residual uncertainty", + "visible_text": "A 60% · B 25%", + "truth_excerpt": "Fifteen percent residual probability remains material to the decision.", + "adjudication_notes": "The state looks complete while hiding residual uncertainty that changes downstream review behavior.", + "render": { "template": "operating", "variant": "missing-residual" } + }, + "clean": { + "title": "Residual uncertainty remains an explicit branch", + "visible_text": "A 60% · B 25% · Other 15%", + "truth_excerpt": "Fifteen percent residual probability remains material to the decision.", + "adjudication_notes": "The full distribution and uncertain branch remain visible and connected to escalation.", + "render": { "template": "operating", "variant": "visible-residual" } + } + }, + { + "id": "provenance-trace", + "criterion_id": "operating-model-fit", + "region": { "id": "evidence-trace", "label": "Evidence provenance trace" }, + "fire": { + "title": "Evidence cards omit source-to-decision paths", + "visible_text": "Study · Finding · Decision", + "truth_excerpt": "The viewer must reconstruct which source supports each decision claim.", + "adjudication_notes": "Grouped cards contain the right nouns but no connectors or identifiers preserve provenance.", + "render": { "template": "operating", "variant": "provenance-cards" } + }, + "clean": { + "title": "Trace IDs connect sources through findings to decisions", + "visible_text": "S1 → F1 → D1", + "truth_excerpt": "The viewer must reconstruct which source supports each decision claim.", + "adjudication_notes": "Stable identifiers and connectors preserve the evidence path without requiring presenter narration.", + "render": { "template": "operating", "variant": "provenance-trace" } + } + }, + { + "id": "resource-flow", + "criterion_id": "operating-model-fit", + "region": { "id": "resource-flow", "label": "Resource-to-output flow" }, + "fire": { + "title": "Resource allocation is shown as independent metrics", + "visible_text": "Reviewers 8 · Queue 120 · SLA 2h", + "truth_excerpt": "Reviewer capacity governs queue throughput and the resulting SLA.", + "adjudication_notes": "Independent metrics conceal the load-bearing resource-to-throughput relationship.", + "render": { "template": "operating", "variant": "resource-metrics" } + }, + "clean": { + "title": "Capacity and queue connect to throughput and SLA", + "visible_text": "8 reviewers → 40/hr → 2h SLA", + "truth_excerpt": "Reviewer capacity governs queue throughput and the resulting SLA.", + "adjudication_notes": "The visual grammar exposes the governing flow and labels each transformation.", + "render": { "template": "operating", "variant": "resource-flow" } + } + }, + { + "id": "state-transition", + "criterion_id": "operating-model-fit", + "region": { "id": "state-machine", "label": "Review state transitions" }, + "fire": { + "title": "Statuses appear without allowed transitions", + "visible_text": "Pending · Reviewed · Escalated", + "truth_excerpt": "Cases can escalate from pending or reviewed, but only reviewed cases can close.", + "adjudication_notes": "Status badges expose no transition rules and cannot communicate the allowed state changes.", + "render": { "template": "operating", "variant": "state-badges" } + }, + "clean": { + "title": "State machine exposes guarded transitions", + "visible_text": "Pending → Reviewed → Closed · Escalated", + "truth_excerpt": "Cases can escalate from pending or reviewed, but only reviewed cases can close.", + "adjudication_notes": "Directed and labelled transitions make the guarded state changes reconstructable.", + "render": { "template": "operating", "variant": "state-machine" } + } + }, + { + "id": "comparison-treatment", + "criterion_id": "operating-model-fit", + "region": { "id": "cost-comparison", "label": "Review cost comparison" }, + "fire": { + "title": "Simple cost comparison becomes an elaborate system map", + "visible_text": "Repeated review · One-time verification", + "truth_excerpt": "The claim is a two-term break-even comparison.", + "adjudication_notes": "A complex network invents dependencies and obscures a claim that only needs aligned arithmetic.", + "render": { "template": "operating", "variant": "overbuilt-comparison" } + }, + "clean": { + "title": "Aligned equation is proportionate to the claim", + "visible_text": "n × review cost > verification cost", + "truth_excerpt": "The claim is a two-term break-even comparison.", + "adjudication_notes": "The relational treatment exposes the exact comparison without forcing an operating-model diagram.", + "render": { "template": "operating", "variant": "relational-equation" } + } + }, + { + "id": "simple-cover", + "criterion_id": "operating-model-fit", + "region": { "id": "cover-unit", "label": "Cover slide" }, + "fire": { + "title": "Cover claim is forced into a node graph", + "visible_text": "Evidence you can operate", + "truth_excerpt": "The unit is a cover with one claim and no relational narrative job.", + "adjudication_notes": "A decorative graph implies a system the cover does not claim and is disproportionate to the unit.", + "render": { "template": "operating", "variant": "cover-graph" } + }, + "clean": { + "title": "Cover remains a single dominant claim", + "visible_text": "Evidence you can operate", + "truth_excerpt": "The unit is a cover with one claim and no relational narrative job.", + "adjudication_notes": "Routing to none is correct; no diagram is required for a single introductory claim.", + "render": { "template": "operating", "variant": "cover-none" } + } + }, + { + "id": "simulated-surface", + "criterion_id": "operating-model-fit", + "region": { "id": "clinical-ide", "label": "Clinical decision workspace" }, + "fire": { + "title": "Interactive workflow is flattened into generic cards", + "visible_text": "Evidence · Hypotheses · Next actions", + "truth_excerpt": "The clinician’s interaction among evidence, hypotheses, and actions is the claim.", + "adjudication_notes": "Three generic cards omit selection, comparison, and action state that define the workflow.", + "render": { "template": "operating", "variant": "workspace-cards" } + }, + "clean": { + "title": "Simulated workspace exposes the interaction", + "visible_text": "Evidence · Hypotheses · Next actions", + "truth_excerpt": "The clinician’s interaction among evidence, hypotheses, and actions is the claim.", + "adjudication_notes": "The simulated surface faithfully exposes selection state, linked evidence, and available next actions.", + "render": { "template": "operating", "variant": "workspace-surface" } + } + }, + { + "id": "dependency-map", + "criterion_id": "operating-model-fit", + "region": { "id": "dependency-map", "label": "Deployment dependency map" }, + "fire": { + "title": "Dependency order is presented as unordered bullets", + "visible_text": "Schema · Migration · Service · Client", + "truth_excerpt": "Deployment requires schema before migration, then service, then client.", + "adjudication_notes": "An unordered list hides the load-bearing prerequisites and deploy order.", + "render": { "template": "operating", "variant": "dependency-list" } + }, + "clean": { + "title": "Dependency map preserves prerequisites", + "visible_text": "Schema → Migration → Service → Client", + "truth_excerpt": "Deployment requires schema before migration, then service, then client.", + "adjudication_notes": "The ordered connectors expose every prerequisite and the final client dependency.", + "render": { "template": "operating", "variant": "dependency-map" } + } + } + ] + }, + { + "id": "artifact-slop-gap", + "route_key": "artifact-slop-gap", + "rubric": "../../plugins/visual-explainer/scripts/verify/rubrics/pass-artifact-slop-gap.md", + "pairs": [ + { + "id": "independent-options", + "criterion_id": "artifact-slop-false-sequence", + "region": { "id": "option-row", "label": "Three independent options" }, + "fire": { + "title": "Independent options are numbered and arrowed", + "visible_text": "01 Explore → 02 Compare → 03 Export", + "truth_excerpt": "Users may choose Explore, Compare, or Export in any order.", + "adjudication_notes": "Numbers and arrows imply a required progression that the source explicitly does not establish.", + "render": { "template": "slop", "variant": "false-sequence" } + }, + "clean": { + "title": "Numbered workflow follows real dependencies", + "visible_text": "01 Capture → 02 Adjudicate → 03 Graduate", + "truth_excerpt": "Adjudication requires captured observations; graduation requires adjudication.", + "adjudication_notes": "Every number and connector corresponds to a documented dependency and real order.", + "render": { "template": "slop", "variant": "true-sequence" } + } + }, + { + "id": "section-number", + "criterion_id": "artifact-slop-false-sequence", + "region": { "id": "section-heading", "label": "Numbered section heading" }, + "fire": { + "title": "Random section number implies missing chapters", + "visible_text": "04 · Architecture", + "truth_excerpt": "The artifact has one standalone page and no ordered sections.", + "adjudication_notes": "The isolated 04 marker implies a larger ordered document that does not exist.", + "render": { "template": "slop", "variant": "orphan-section-number" } + }, + "clean": { + "title": "Section number resolves through the document agenda", + "visible_text": "04 · Architecture", + "truth_excerpt": "The agenda links Architecture as section four of six.", + "adjudication_notes": "The section marker corresponds to real navigation and a genuine reading order.", + "render": { "template": "slop", "variant": "real-section-number" } + } + }, + { + "id": "confidence-meter", + "criterion_id": "artifact-slop-false-state", + "region": { "id": "confidence-meter", "label": "Confidence meter" }, + "fire": { + "title": "Static mockup claims 87 percent confidence", + "visible_text": "Confidence 87%", + "truth_excerpt": "No calculation, measurement, or source supplies a confidence value.", + "adjudication_notes": "The meter looks measured and live even though the artifact has no supporting state.", + "render": { "template": "slop", "variant": "false-confidence" } + }, + "clean": { + "title": "Confidence derives from displayed test results", + "visible_text": "Confidence 87% · 87/100 checks", + "truth_excerpt": "The value is calculated from 87 passing checks out of 100 and links to the method.", + "adjudication_notes": "The measured value and its derivation are both visible, so the state implication is supported.", + "render": { "template": "slop", "variant": "measured-confidence" } + } + }, + { + "id": "live-status-dot", + "criterion_id": "artifact-slop-false-state", + "region": { "id": "status-line", "label": "System status line" }, + "fire": { + "title": "Green dot claims a live connection", + "visible_text": "● System live", + "truth_excerpt": "The artifact is a static exported explainer with no connection telemetry.", + "adjudication_notes": "The conventional green status dot and “live” label imply a measured connection that cannot exist.", + "render": { "template": "slop", "variant": "false-live-status" } + }, + "clean": { + "title": "Decorative dot is visibly non-status", + "visible_text": "• System architecture", + "truth_excerpt": "The dot is a typographic bullet before a neutral heading.", + "adjudication_notes": "No status word, color semantics, or monitoring treatment suggests live state.", + "render": { "template": "slop", "variant": "decorative-dot" } + } + }, + { + "id": "progress-bar", + "criterion_id": "artifact-slop-false-state", + "region": { "id": "progress", "label": "Progress indicator" }, + "fire": { + "title": "Decorative progress bar implies task completion", + "visible_text": "Analysis 72%", + "truth_excerpt": "The page performs no analysis and tracks no progress.", + "adjudication_notes": "A filled meter and percentage imply active measured completion without any underlying task.", + "render": { "template": "slop", "variant": "false-progress" } + }, + "clean": { + "title": "Deck progress reflects actual navigation", + "visible_text": "Slide 7 of 10 · 70%", + "truth_excerpt": "The viewer is currently on slide seven of ten.", + "adjudication_notes": "The progress treatment is directly bound to real navigation state and accurately calculated.", + "render": { "template": "slop", "variant": "real-progress" } + } + }, + { + "id": "citation-marker", + "criterion_id": "artifact-slop-false-provenance", + "region": { "id": "claim", "label": "Claim with citation marker" }, + "fire": { + "title": "Citation marker has no source target", + "visible_text": "Review cost falls 40% [4]", + "truth_excerpt": "The artifact supplies no source list, footnote, or link target.", + "adjudication_notes": "The bracketed marker conventionally implies traceable evidence that the artifact cannot provide.", + "render": { "template": "slop", "variant": "false-citation" } + }, + "clean": { + "title": "Footnote marker resolves to a supplied source", + "visible_text": "Review cost falls 40% [4]", + "truth_excerpt": "Footnote four names the study, publication date, and resolvable URL.", + "adjudication_notes": "The provenance cue is backed by a concrete source target in the artifact.", + "render": { "template": "slop", "variant": "real-citation" } + } + }, + { + "id": "evidence-badge", + "criterion_id": "artifact-slop-false-provenance", + "region": { "id": "evidence-badge", "label": "Evidence validation badge" }, + "fire": { + "title": "Validated badge has no validation process", + "visible_text": "✓ Evidence validated", + "truth_excerpt": "No validator, review event, or source record exists.", + "adjudication_notes": "The checkmark and validation label assert provenance and verification unsupported by the artifact.", + "render": { "template": "slop", "variant": "false-validation" } + }, + "clean": { + "title": "Validation badge links to a signed review event", + "visible_text": "✓ Evidence validated · R-184", + "truth_excerpt": "Review event R-184 names the reviewer, timestamp, and checked sources.", + "adjudication_notes": "The validation claim resolves to a real review record with traceable provenance.", + "render": { "template": "slop", "variant": "real-validation" } + } + }, + { + "id": "source-timestamp", + "criterion_id": "artifact-slop-false-provenance", + "region": { "id": "source-strip", "label": "Source and timestamp strip" }, + "fire": { + "title": "Decorative source strip invents freshness", + "visible_text": "SOURCE · LIVE · 12:42 UTC", + "truth_excerpt": "The content is static prose with no source feed or retrieval time.", + "adjudication_notes": "Source and timestamp chrome imply a live traceable feed that the artifact does not possess.", + "render": { "template": "slop", "variant": "false-source-strip" } + }, + "clean": { + "title": "Snapshot timestamp identifies a real retrieval", + "visible_text": "Snapshot · 12:42 UTC · dataset v18", + "truth_excerpt": "Dataset version 18 was retrieved at 12:42 UTC and is bundled with the report.", + "adjudication_notes": "The timestamp and version accurately identify a supplied snapshot rather than manufacturing freshness.", + "render": { "template": "slop", "variant": "real-source-strip" } + } + }, + { + "id": "terminal-label", + "criterion_id": "artifact-slop-false-state", + "region": { "id": "terminal", "label": "Terminal-style panel" }, + "fire": { + "title": "Fake terminal reports successful deployment", + "visible_text": "$ deploy --prod · SUCCESS", + "truth_excerpt": "The artifact never ran a deployment command and has no execution result.", + "adjudication_notes": "Terminal chrome and SUCCESS output imply an executed command and measured state that never occurred.", + "render": { "template": "slop", "variant": "false-terminal" } + }, + "clean": { + "title": "Terminal is explicitly labelled as an example", + "visible_text": "Example command · $ deploy --dry-run", + "truth_excerpt": "The panel documents a command example and makes no claim that it ran.", + "adjudication_notes": "The example label and dry-run wording prevent a reasonable viewer from inferring live execution state.", + "render": { "template": "slop", "variant": "example-terminal" } + } + }, + { + "id": "choice-arrows", + "criterion_id": "artifact-slop-false-sequence", + "region": { "id": "choice-map", "label": "Choice map" }, + "fire": { + "title": "Parallel choices are connected as a pipeline", + "visible_text": "PDF → Slides → Web", + "truth_excerpt": "PDF, Slides, and Web are independent export formats selected one at a time.", + "adjudication_notes": "Directional arrows falsely imply transformation from one export format into the next.", + "render": { "template": "slop", "variant": "false-choice-flow" } + }, + "clean": { + "title": "Branching connector shows mutually exclusive choices", + "visible_text": "Export → PDF | Slides | Web", + "truth_excerpt": "One export action branches to three mutually exclusive formats.", + "adjudication_notes": "The branch grammar faithfully represents choice and does not imply sequential transformation.", + "render": { "template": "slop", "variant": "real-choice-branch" } + } + } + ] + } + ] +} diff --git a/evals/visual-model-policy/corpus.mjs b/evals/visual-model-policy/corpus.mjs new file mode 100644 index 0000000..86a72d4 --- /dev/null +++ b/evals/visual-model-policy/corpus.mjs @@ -0,0 +1,188 @@ +import crypto from 'node:crypto'; +import fs from 'node:fs/promises'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const ROOT = path.dirname(fileURLToPath(import.meta.url)); +const DEFAULT_CORPUS_PATH = path.join(ROOT, 'corpus.json'); +const DELEGATED_CRITERIA = new Set([ + 'impeccable:critique', + 'unslop:cleanup-report', +]); +const ALLOWED_LABELS = new Set(['fire', 'clean']); +export const RENDER_VARIANTS = Object.freeze(Object.fromEntries( + Object.entries({ + text: ['clipped-right', 'wrapped'], + table: ['clipped-label', 'dense-scroll'], + slide: ['competing-foci', 'dense-one-focus'], + dashboard: ['hierarchical-dense', 'uniform-loud'], + 'diagram-space': ['collapsed-column', 'editorial-space', 'intentional-breathing', 'tiny-unbalanced'], + tracks: ['baseline-drift', 'editorial-60-40', 'hero-span', 'masonry', 'missing-weight', 'unequal-peers'], + mobile: ['compact-controls', 'crowded-controls'], + diagram: [ + 'ambiguous-relations', 'annotation-no-legend', 'coherent-flow', 'complete-legend', + 'decorated-list', 'dishonest-bars', 'dishonest-timeline', 'dual-focal', + 'invalid-probability', 'labeled-comparison', 'missing-legend-entry', + 'mixed-grammar', 'necessary-sequence', 'necessary-topology', 'orphan-legend', + 'peer-inventory', 'single-focal', 'table-as-graph', 'timeline-break', + 'valid-probability', + ], + preset: [ + 'decorative-red', 'decorative-status-color', 'destructive-red', + 'intentional-terminal', 'invisible-dark-icons', 'many-grid-breaks', + 'many-surprises', 'matched-demo', 'mismatched-demo', 'mobile-hero-card', + 'mobile-hero-full', 'one-grid-break', 'one-surprise', 'paper-preview', + 'quiet-watermark', 'semantic-status-color', 'three-layers', 'too-many-layers', + 'wrong-mode-background', 'wrong-mode-panel', + ], + operating: [ + 'closed-loop', 'cover-graph', 'cover-none', 'dependency-list', + 'dependency-map', 'missing-residual', 'one-way-loop', 'overbuilt-comparison', + 'provenance-cards', 'provenance-trace', 'relational-equation', 'resource-flow', + 'resource-metrics', 'routing-cards', 'routing-model', 'state-badges', + 'state-machine', 'visible-residual', 'workspace-cards', 'workspace-surface', + ], + slop: [ + 'decorative-dot', 'example-terminal', 'false-choice-flow', 'false-citation', + 'false-confidence', 'false-live-status', 'false-progress', 'false-sequence', + 'false-source-strip', 'false-terminal', 'false-validation', + 'measured-confidence', 'orphan-section-number', 'real-choice-branch', + 'real-citation', 'real-progress', 'real-section-number', 'real-source-strip', + 'real-validation', 'true-sequence', + ], + }).map(([template, variants]) => [template, Object.freeze(new Set(variants))]), +)); + +export async function loadCorpus(corpusPath = DEFAULT_CORPUS_PATH) { + return JSON.parse(await fs.readFile(corpusPath, 'utf8')); +} + +export function expandCorpus(corpus, { corpusPath = DEFAULT_CORPUS_PATH } = {}) { + const renderedRoot = path.resolve(path.dirname(corpusPath), corpus.rendered_root || 'corpus/rendered'); + const cases = []; + for (const family of corpus.families || []) { + for (const pair of family.pairs || []) { + for (const humanLabel of ['fire', 'clean']) { + const state = pair[humanLabel]; + if (!state) continue; + const caseId = opaqueCaseId(corpus.corpus_id, family.id, pair.id, humanLabel); + const imageId = caseId; + cases.push({ + case_id: caseId, + pair_id: pair.id, + state_id: caseId, + family: family.id, + route_key: family.route_key, + rubric: path.resolve(path.dirname(corpusPath), family.rubric), + criterion_id: pair.criterion_id, + human_label: humanLabel, + hard_negative: humanLabel === 'clean', + title: state.title, + visible_text: state.visible_text, + truth_excerpt: state.truth_excerpt, + adjudication_notes: state.adjudication_notes, + render: state.render, + image: { + id: imageId, + path: path.join(renderedRoot, `${imageId}.png`), + detail: 'high', + }, + regions: pair.regions || [pair.region], + }); + } + } + } + return cases; +} + +export function validateCorpus(corpus, options = {}) { + if (corpus?.schema_version !== 1 || !corpus.corpus_id || !Array.isArray(corpus.families)) { + throw new Error('corpus requires schema_version=1, corpus_id, and families[]'); + } + if (!['pending-human-review', 'human-reviewed'].includes(corpus.label_review?.status)) { + throw new Error('corpus requires label_review.status pending-human-review or human-reviewed'); + } + if ( + corpus.label_review.status === 'human-reviewed' + && (!corpus.label_review.reviewer || !corpus.label_review.reviewed_at) + ) { + throw new Error('human-reviewed corpus requires reviewer and reviewed_at'); + } + const familyIds = new Set(); + const caseIds = new Set(); + const imageIds = new Set(); + const summary = { + corpus_id: corpus.corpus_id, + label_review: corpus.label_review, + cases: 0, + families: {}, + }; + for (const family of corpus.families) { + if (!family.id || !family.route_key || !family.rubric || !Array.isArray(family.pairs)) { + throw new Error('every family requires id, route_key, rubric, and pairs[]'); + } + if (familyIds.has(family.id)) throw new Error(`duplicate family ${family.id}`); + familyIds.add(family.id); + summary.families[family.id] = { fire: 0, clean: 0, hard_negatives: 0, criteria: {} }; + for (const pair of family.pairs) { + if (!pair.id || !pair.criterion_id) { + throw new Error(`family ${family.id} contains a pair without id or criterion_id`); + } + if (DELEGATED_CRITERIA.has(pair.criterion_id)) { + throw new Error( + `delegated criterion ${pair.criterion_id} is outside Artifacture's eval ownership`, + ); + } + const regions = pair.regions || [pair.region]; + if ( + regions.length === 0 + || regions.some((region) => !region?.id || !region?.label) + ) { + throw new Error(`${family.id}:${pair.id} requires named regions`); + } + for (const label of ALLOWED_LABELS) { + const state = pair[label]; + if (!state) throw new Error(`${family.id}:${pair.id} requires ${label} state`); + for (const field of [ + 'title', + 'visible_text', + 'truth_excerpt', + 'adjudication_notes', + 'render', + ]) { + if (!state[field]) throw new Error(`${family.id}:${pair.id}:${label} requires ${field}`); + } + if (state.adjudication_notes.length < 20) { + throw new Error(`${family.id}:${pair.id}:${label} adjudication_notes are too short`); + } + const allowedVariants = RENDER_VARIANTS[state.render.template]; + if (!allowedVariants?.has(state.render.variant)) { + throw new Error( + `${family.id}:${pair.id}:${label} has unsupported render ${state.render.template}/${state.render.variant}`, + ); + } + const caseId = opaqueCaseId(corpus.corpus_id, family.id, pair.id, label); + if (caseIds.has(caseId)) throw new Error(`duplicate case ${caseId}`); + if (imageIds.has(caseId)) throw new Error(`duplicate image ${caseId}`); + caseIds.add(caseId); + imageIds.add(caseId); + summary.cases += 1; + summary.families[family.id][label] += 1; + if (label === 'clean') summary.families[family.id].hard_negatives += 1; + const criterion = summary.families[family.id].criteria[pair.criterion_id] + || { fire: 0, clean: 0 }; + criterion[label] += 1; + summary.families[family.id].criteria[pair.criterion_id] = criterion; + } + } + } + expandCorpus(corpus, options); + return summary; +} + +function opaqueCaseId(corpusId, familyId, pairId, label) { + return `ve:${crypto.createHash('sha256') + .update(`${corpusId}\u0000${familyId}\u0000${pairId}\u0000${label}`) + .digest('hex') + .slice(0, 16)}`; +} diff --git a/evals/visual-model-policy/corpus.test.mjs b/evals/visual-model-policy/corpus.test.mjs new file mode 100644 index 0000000..09b3102 --- /dev/null +++ b/evals/visual-model-policy/corpus.test.mjs @@ -0,0 +1,73 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { + expandCorpus, + loadCorpus, + validateCorpus, +} from './corpus.mjs'; + +const REQUIRED_FAMILIES = [ + 'layout', + 'diagram', + 'aesthetic', + 'operating-model', + 'artifact-slop-gap', +]; + +test('the visual corpus has independent graduation evidence for every owned family', async () => { + const corpus = await loadCorpus(); + const summary = validateCorpus(corpus); + + assert.deepEqual(Object.keys(summary.families).sort(), [...REQUIRED_FAMILIES].sort()); + assert.equal(summary.label_review.status, 'pending-human-review'); + for (const family of REQUIRED_FAMILIES) { + assert.ok(summary.families[family].fire >= 10, `${family} fire cases`); + assert.ok(summary.families[family].clean >= 10, `${family} clean cases`); + assert.equal( + summary.families[family].hard_negatives, + summary.families[family].clean, + `${family} hard negatives`, + ); + for (const [criterionId, counts] of Object.entries(summary.families[family].criteria)) { + assert.ok(counts.fire >= 2, `${family}:${criterionId} fire cases support batch 4`); + assert.ok(counts.clean >= 2, `${family}:${criterionId} clean cases support batch 4`); + } + } +}); + +test('layout corpus covers clipping, crowding, dead space, and symmetry', async () => { + const cases = expandCorpus(await loadCorpus()).filter((entry) => entry.family === 'layout'); + assert.deepEqual( + [...new Set(cases.map((entry) => entry.criterion_id))].sort(), + [ + 'layout-repeated-track-symmetry', + 'slide-single-focal-point', + 'sparse-diagram-slide', + 'text-visibly-clipped', + ], + ); +}); + +test('every case exposes stable evidence identity, a named region, and adjudication notes', async () => { + const cases = expandCorpus(await loadCorpus()); + assert.equal(cases.length, 112); + for (const entry of cases) { + assert.match(entry.case_id, /^[a-z0-9][a-z0-9:-]+$/); + assert.match(entry.state_id, /^[a-z0-9][a-z0-9:-]+$/); + assert.match(entry.image.id, /^[a-z0-9][a-z0-9:-]+$/); + assert.ok(entry.image.path.endsWith(`${entry.image.id}.png`)); + assert.ok(entry.regions.length > 0); + assert.ok(entry.regions.every((region) => region.id && region.label)); + assert.ok(['fire', 'clean'].includes(entry.human_label)); + assert.ok(entry.adjudication_notes.length >= 20); + } +}); + +test('the owned corpus cannot absorb delegated skill criteria', async () => { + const corpus = await loadCorpus(); + corpus.families[0].pairs[0].criterion_id = 'impeccable:critique'; + assert.throws( + () => validateCorpus(corpus), + /delegated criterion impeccable:critique is outside Artifacture's eval ownership/, + ); +}); diff --git a/evals/visual-model-policy/example-measurements.json b/evals/visual-model-policy/example-measurements.json index f259fa1..99878d5 100644 --- a/evals/visual-model-policy/example-measurements.json +++ b/evals/visual-model-policy/example-measurements.json @@ -4,6 +4,8 @@ "min_runs": 3, "min_positives": 10, "min_negatives": 10, + "min_criterion_positives": 1, + "min_criterion_negatives": 1, "precision": 0.9, "recall": 0.9, "silence_accuracy": 0.95, @@ -46,7 +48,14 @@ "responses": 100, "abstentions": 1, "total_cost_usd": 0.1, - "p95_latency_ms": 5000 + "p95_latency_ms": 5000, + "unique_cases": 20, + "unique_positives": 10, + "unique_negatives": 10, + "adjudication_complete": true, + "synthetic": true, + "telemetry_complete": false, + "experiment_complete": false }, { "pass": "layout", @@ -66,7 +75,14 @@ "responses": 100, "abstentions": 2, "total_cost_usd": 0.06, - "p95_latency_ms": 6500 + "p95_latency_ms": 6500, + "unique_cases": 20, + "unique_positives": 10, + "unique_negatives": 10, + "adjudication_complete": true, + "synthetic": true, + "telemetry_complete": false, + "experiment_complete": false }, { "pass": "layout", @@ -86,7 +102,14 @@ "responses": 100, "abstentions": 0, "total_cost_usd": 0.4, - "p95_latency_ms": 9000 + "p95_latency_ms": 9000, + "unique_cases": 20, + "unique_positives": 10, + "unique_negatives": 10, + "adjudication_complete": true, + "synthetic": true, + "telemetry_complete": false, + "experiment_complete": false } ] } diff --git a/evals/visual-model-policy/experiment.template.json b/evals/visual-model-policy/experiment.template.json new file mode 100644 index 0000000..35a25fe --- /dev/null +++ b/evals/visual-model-policy/experiment.template.json @@ -0,0 +1,43 @@ +{ + "schema_version": 1, + "id": "replace-with-dated-experiment-id", + "corpus_path": "corpus.json", + "image_detail": "high", + "replicates": 3, + "batch_sizes": [1, 2, 4], + "passes": [ + "layout", + "diagram", + "aesthetic", + "operating-model", + "artifact-slop-gap" + ], + "thresholds": { + "min_runs": 3, + "min_positives": 10, + "min_negatives": 10, + "min_criterion_positives": 1, + "min_criterion_negatives": 1, + "precision": 0.9, + "recall": 0.9, + "silence_accuracy": 0.95, + "grounding_accuracy": 0.95, + "json_validity": 0.99, + "max_abstention_rate": 0.05, + "max_p95_latency_ms": 30000, + "max_cost_per_case_usd": 0.01 + }, + "run_candidate": "REPLACE_WITH_SMALLEST_PROVIDER_MODEL_ID", + "ladder_state": { + "completed_candidates": [] + }, + "candidates": [ + { + "id": "REPLACE_WITH_SMALLEST_PROVIDER_MODEL_ID", + "rank": 1, + "class": "small", + "provider": "REPLACE_WITH_PROVIDER" + } + ], + "adapter_options": {} +} diff --git a/evals/visual-model-policy/merge.mjs b/evals/visual-model-policy/merge.mjs new file mode 100644 index 0000000..bdcc354 --- /dev/null +++ b/evals/visual-model-policy/merge.mjs @@ -0,0 +1,230 @@ +#!/usr/bin/env node +import crypto from 'node:crypto'; +import fs from 'node:fs/promises'; +import path from 'node:path'; +import process from 'node:process'; +import { pathToFileURL } from 'node:url'; +import { canonicalJson } from '../lib/canonical-json.mjs'; +import { + DEFAULT_THRESHOLDS, + qualificationFor, +} from './select.mjs'; + +export function mergeMeasurements(measurements, { + sourceFiles = [], + sourceSha256 = [], +} = {}) { + if (!Array.isArray(measurements) || measurements.length === 0) { + throw new Error('measurements[] is required'); + } + const first = measurements[0]; + validateMeasurementSet(first); + const candidatesIdentity = canonicalJson(first.candidates); + const thresholdsIdentity = canonicalJson(first.thresholds || {}); + const corpusSha256 = first.source.corpus_sha256; + const results = []; + const resultKeys = new Set(); + const pendingAdjudication = new Set(); + const experimentContracts = new Set(); + const artifactByModelPass = new Map(); + const measurementByModelPass = new Map(); + + if ( + sourceSha256.length !== measurements.length + || sourceSha256.some((hash) => !/^[a-f0-9]{64}$/.test(hash)) + ) { + throw new Error('sourceSha256 must provide one exact artifact hash per measurement set'); + } + + for (const [measurementIndex, measurement] of measurements.entries()) { + validateMeasurementSet(measurement); + if ( + canonicalJson(measurement.candidates) !== candidatesIdentity + || canonicalJson(measurement.thresholds || {}) !== thresholdsIdentity + ) { + throw new Error('measurement sets must use identical candidates and thresholds'); + } + if (measurement.source.corpus_sha256 !== corpusSha256) { + throw new Error('measurement sets must use the identical corpus contract'); + } + experimentContracts.add(measurement.source.experiment_contract_id); + const measuredModels = new Set(measurement.results.map((result) => result.model)); + if (measuredModels.size !== 1) { + throw new Error('each measurement set must contain exactly one candidate model'); + } + const measuredModel = [...measuredModels][0]; + for (const pass of new Set(measurement.results.map((result) => result.pass))) { + const key = `${measuredModel}\u0000${pass}`; + if (artifactByModelPass.has(key)) { + throw new Error(`candidate ${measuredModel} pass ${pass} spans multiple measurement sets`); + } + artifactByModelPass.set(key, sourceSha256[measurementIndex]); + measurementByModelPass.set(key, measurement); + } + for (const result of measurement.results) { + const key = [ + result.pass, + result.model, + result.batch_size, + result.image_detail, + ].join('\u0000'); + if (resultKeys.has(key)) throw new Error(`duplicate measurement cell ${key}`); + resultKeys.add(key); + results.push(result); + } + for (const observationId of measurement.pending_adjudication || []) { + pendingAdjudication.add(observationId); + } + } + + validateLadderCoverage({ + candidates: first.candidates, + thresholds: { ...DEFAULT_THRESHOLDS, ...(first.thresholds || {}) }, + results, + artifactByModelPass, + measurementByModelPass, + }); + + return { + schema_version: 1, + generated_at: new Date().toISOString(), + source: { + measurement_files: [...sourceFiles], + measurement_sha256: [...sourceSha256], + measurement_sets: measurements.length, + corpus_id: first.source.corpus_id, + corpus_sha256: corpusSha256, + experiment_contract_ids: [...experimentContracts].sort(), + }, + thresholds: first.thresholds, + candidates: first.candidates, + results: results.sort(compareResults), + pending_adjudication: [...pendingAdjudication].sort(), + }; +} + +function validateLadderCoverage({ + candidates, + thresholds, + results, + artifactByModelPass, + measurementByModelPass, +}) { + const ordered = [...candidates].sort((a, b) => Number(a.rank) - Number(b.rank)); + const passes = [...new Set(results.map((result) => result.pass))]; + for (const pass of passes) { + const measured = ordered.filter((candidate) => ( + results.some((result) => result.pass === pass && result.model === candidate.id) + )); + const expectedPrefix = ordered.slice(0, measured.length); + if ( + measured.length === 0 + || expectedPrefix.some((candidate, index) => candidate.id !== measured[index]?.id) + ) { + throw new Error(`measurements for ${pass} must form a contiguous smallest-first candidate prefix`); + } + for (let index = 1; index < measured.length; index += 1) { + const current = measured[index]; + const measurement = measurementByModelPass.get(`${current.id}\u0000${pass}`); + const completed = measurement.source.ladder_state?.completed_candidates || []; + for (let lowerIndex = 0; lowerIndex < index; lowerIndex += 1) { + const lower = measured[lowerIndex]; + const evidence = completed + .find((entry) => entry.id === lower.id) + ?.pass_evidence?.[pass]; + const lowerResults = results.filter( + (result) => result.pass === pass && result.model === lower.id, + ); + const expectedStatus = lowerResults.some( + (result) => qualificationFor(result, thresholds).qualified, + ) ? 'qualified' : 'disqualified'; + const expectedHash = artifactByModelPass.get(`${lower.id}\u0000${pass}`); + if ( + evidence?.status !== expectedStatus + || evidence?.evidence_sha256 !== expectedHash + ) { + throw new Error( + `${current.id}:${pass} does not bind verified evidence for prior candidate ${lower.id}`, + ); + } + } + } + } +} + +function validateMeasurementSet(measurement) { + if ( + measurement?.schema_version !== 1 + || !measurement.source?.corpus_id + || !/^[a-f0-9]{64}$/.test(measurement.source.corpus_sha256 || '') + || !/^[a-f0-9]{64}$/.test(measurement.source.experiment_contract_id || '') + || measurement.source.experiment_complete !== true + || !Array.isArray(measurement.candidates) + || !Array.isArray(measurement.results) + ) { + throw new Error('each measurement set requires a complete experiment and corpus contract'); + } + if (measurement.results.some((result) => result.experiment_complete !== true)) { + throw new Error('measurement results must come from a complete experiment matrix'); + } +} + +function compareResults(a, b) { + return ( + String(a.pass).localeCompare(String(b.pass)) + || String(a.model).localeCompare(String(b.model)) + || Number(a.batch_size) - Number(b.batch_size) + || String(a.image_detail).localeCompare(String(b.image_detail)) + ); +} + +function parseArgs(argv) { + const args = { inputs: [], output: null }; + for (let index = 0; index < argv.length; index += 1) { + if (argv[index] === '--input') args.inputs.push(argv[++index]); + else if (argv[index] === '--out') args.output = argv[++index]; + else throw new Error(`unknown argument ${argv[index]}`); + } + if (args.inputs.length === 0) { + throw new Error('usage: merge.mjs --input measurements.json [--input more.json] [--out merged.json]'); + } + return args; +} + +async function main() { + const args = parseArgs(process.argv.slice(2)); + const files = args.inputs.map((file) => path.resolve(file)); + const sources = await Promise.all(files.map((file) => fs.readFile(file, 'utf8'))); + const measurements = sources.map((source) => JSON.parse(source)); + const sourceSha256 = sources.map((source) => ( + crypto.createHash('sha256').update(source).digest('hex') + )); + const merged = mergeMeasurements(measurements, { + sourceFiles: files, + sourceSha256, + }); + const rendered = `${JSON.stringify(merged, null, 2)}\n`; + if (args.output) { + const output = path.resolve(args.output); + await fs.mkdir(path.dirname(output), { recursive: true }); + await fs.writeFile(output, rendered, { flag: 'wx' }); + process.stdout.write(`${output}\n`); + } else { + process.stdout.write(rendered); + } + if ( + merged.pending_adjudication.length > 0 + || merged.results.some((result) => ( + result.synthetic || !result.telemetry_complete || !result.adjudication_complete + )) + ) { + process.exitCode = 1; + } +} + +if (import.meta.url === pathToFileURL(process.argv[1] || '').href) { + main().catch((error) => { + console.error(error.stack || error.message || error); + process.exit(2); + }); +} diff --git a/evals/visual-model-policy/merge.test.mjs b/evals/visual-model-policy/merge.test.mjs new file mode 100644 index 0000000..87488cf --- /dev/null +++ b/evals/visual-model-policy/merge.test.mjs @@ -0,0 +1,108 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { mergeMeasurements } from './merge.mjs'; + +function measurement(model, contract, completedCandidates = []) { + return { + schema_version: 1, + source: { + corpus_id: 'corpus', + corpus_sha256: 'a'.repeat(64), + experiment_contract_id: contract, + experiment_complete: true, + ladder_state: { completed_candidates: completedCandidates }, + }, + thresholds: { recall: 0.9 }, + candidates: [ + { id: 'nano', provider: 'fixture', rank: 1 }, + { id: 'mini', provider: 'fixture', rank: 2 }, + ], + results: [{ + pass: 'layout', + model, + batch_size: 1, + image_detail: 'high', + runs: 3, + cases: 20, + positives: 10, + negatives: 10, + unique_cases: 20, + unique_positives: 10, + unique_negatives: 10, + tp: 10, + fp: 0, + tn: 10, + fn: 0, + grounded: 10, + grounding_total: 10, + json_valid: 20, + responses: 20, + abstentions: 0, + total_cost_usd: 0.01, + p95_latency_ms: 100, + adjudication_complete: true, + synthetic: false, + telemetry_complete: true, + experiment_complete: true, + }], + pending_adjudication: [], + }; +} + +test('merges independently complete candidate measurements for selection', () => { + const nanoHash = 'd'.repeat(64); + const miniHash = 'e'.repeat(64); + const merged = mergeMeasurements([ + measurement('nano', 'b'.repeat(64)), + measurement('mini', 'c'.repeat(64), [{ + id: 'nano', + pass_evidence: { + layout: { status: 'qualified', evidence_sha256: nanoHash }, + }, + }]), + ], { sourceSha256: [nanoHash, miniHash] }); + assert.deepEqual(merged.results.map((entry) => entry.model), ['mini', 'nano']); + assert.equal(merged.source.measurement_sets, 2); + assert.deepEqual(merged.source.experiment_contract_ids, [ + 'b'.repeat(64), + 'c'.repeat(64), + ]); +}); + +test('refuses corpus drift and incomplete request matrices', () => { + const drifted = measurement('mini', 'c'.repeat(64)); + drifted.source.corpus_sha256 = 'd'.repeat(64); + assert.throws( + () => mergeMeasurements( + [measurement('nano', 'b'.repeat(64)), drifted], + { sourceSha256: ['d'.repeat(64), 'e'.repeat(64)] }, + ), + /identical corpus contract/, + ); + const incomplete = measurement('nano', 'b'.repeat(64)); + incomplete.source.experiment_complete = false; + assert.throws( + () => mergeMeasurements([incomplete], { sourceSha256: ['d'.repeat(64)] }), + /complete experiment and corpus contract/, + ); +}); + +test('refuses a higher candidate without contiguous bound lower-rank evidence', () => { + assert.throws( + () => mergeMeasurements( + [measurement('mini', 'c'.repeat(64))], + { sourceSha256: ['e'.repeat(64)] }, + ), + /contiguous smallest-first candidate prefix/, + ); + assert.throws( + () => mergeMeasurements( + [ + measurement('nano', 'b'.repeat(64)), + measurement('mini', 'c'.repeat(64)), + ], + { sourceSha256: ['d'.repeat(64), 'e'.repeat(64)] }, + ), + /does not bind verified evidence/, + ); +}); diff --git a/evals/visual-model-policy/render-corpus.mjs b/evals/visual-model-policy/render-corpus.mjs new file mode 100644 index 0000000..3a58a13 --- /dev/null +++ b/evals/visual-model-policy/render-corpus.mjs @@ -0,0 +1,393 @@ +#!/usr/bin/env node +import crypto from 'node:crypto'; +import fs from 'node:fs/promises'; +import path from 'node:path'; +import process from 'node:process'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import { chromium } from 'playwright'; +import { + expandCorpus, + loadCorpus, + validateCorpus, +} from './corpus.mjs'; + +const VIEWPORT = Object.freeze({ width: 960, height: 600 }); + +export function renderCaseHtml(evalCase) { + const { template, variant } = evalCase.render; + const scene = renderTemplate(template, variant, evalCase); + return ` + + + + + + + +
+
${escapeHtml(evalCase.family)} / ${escapeHtml(evalCase.criterion_id)}
+ ${scene} +
+ +`; +} + +export async function renderCorpus({ + corpusPath, + outputRoot, + caseIds = null, +} = {}) { + const resolvedCorpusPath = path.resolve(corpusPath); + const corpus = await loadCorpus(resolvedCorpusPath); + validateCorpus(corpus, { corpusPath: resolvedCorpusPath }); + let cases = expandCorpus(corpus, { corpusPath: resolvedCorpusPath }); + if (caseIds) { + const wanted = new Set(caseIds); + cases = cases.filter((entry) => wanted.has(entry.case_id)); + const missing = [...wanted].filter((id) => !cases.some((entry) => entry.case_id === id)); + if (missing.length > 0) throw new Error(`unknown corpus case(s): ${missing.join(', ')}`); + } + const root = outputRoot + ? path.resolve(outputRoot) + : path.dirname(cases[0]?.image.path || path.resolve('corpus/rendered/placeholder.png')); + await fs.mkdir(root, { recursive: true }); + const browser = await chromium.launch({ headless: true }); + const renderHashes = new Map(); + const caseByHash = new Map(); + const renderedImages = []; + try { + const page = await browser.newPage({ + viewport: VIEWPORT, + deviceScaleFactor: 1, + colorScheme: 'light', + }); + for (const evalCase of cases) { + await page.setContent(renderCaseHtml(evalCase), { waitUntil: 'load' }); + const file = path.join(root, `${evalCase.image.id}.png`); + const bytes = await page.screenshot({ path: file, type: 'png' }); + const hash = crypto.createHash('sha256').update(bytes).digest('hex'); + const duplicate = caseByHash.get(hash); + if (duplicate) { + throw new Error( + `duplicate rendered pixels for ${duplicate.case_id} and ${evalCase.case_id}`, + ); + } + caseByHash.set(hash, evalCase); + const pairKey = `${evalCase.family}\u0000${evalCase.pair_id}`; + const prior = renderHashes.get(pairKey); + if (prior?.hash === hash && prior.human_label !== evalCase.human_label) { + throw new Error( + `opposite labels rendered identical pixels for ${evalCase.family}:${evalCase.pair_id}`, + ); + } + renderHashes.set(pairKey, { hash, human_label: evalCase.human_label }); + renderedImages.push({ + case_id: evalCase.case_id, + image_id: evalCase.image.id, + path: file, + sha256: hash, + }); + } + } finally { + await browser.close(); + } + const manifest = { + schema_version: 1, + corpus_id: corpus.corpus_id, + rendered_at: new Date().toISOString(), + viewport: VIEWPORT, + images: renderedImages, + }; + await fs.writeFile( + path.join(root, 'render-manifest.json'), + `${JSON.stringify(manifest, null, 2)}\n`, + ); + return manifest; +} + +function renderTemplate(template, variant, evalCase) { + const regionId = evalCase.regions[0].id; + const text = escapeHtml(evalCase.visible_text); + if (template === 'text') { + const clipped = variant === 'clipped-right'; + const width = clipped ? 410 : 640; + const wrap = clipped ? 'white-space:nowrap; overflow:hidden;' : 'white-space:normal;'; + return `

${sceneHeading(template)}

+
+
${text}
+
`; + } + if (template === 'table') { + const clipped = variant === 'clipped-label'; + return `

${sceneHeading(template)}

+
+ ${['Queue', 'Owner', 'Latency'].map((label, index) => ` +
+ ${index === 1 ? text : label} + Measured evidence ${index + 1} +
`).join('')} +
`; + } + if (template === 'slide' || template === 'dashboard') { + const equal = variant === 'competing-foci' || variant === 'uniform-loud'; + return `

${sceneHeading(template)}

+
+ ${['93%', 'ROUTER', '2.4s'].map((label, index) => ` +
+
${label}
+

${text}

+
`).join('')} +
`; + } + if (template === 'diagram-space') { + const collapsed = variant === 'collapsed-column'; + const tiny = variant === 'tiny-unbalanced' || collapsed; + const labels = evalCase.visible_text.split(/\s*(?:→|·)\s*/).slice(0, 3); + return `

${sceneHeading(template)}

+
+
+ ${labels.map((label, index) => `${index ? '
→
' : ''}
${escapeHtml(label)}
`).join('')} +
+
`; + } + if (template === 'tracks') { + const uneven = ['unequal-peers', 'baseline-drift', 'missing-weight'].includes(variant); + const editorial = variant === 'editorial-60-40'; + const columns = uneven ? '1.2fr .55fr 1.35fr' : (editorial ? '1.5fr 1fr' : '1fr 1fr 1fr'); + const labels = editorial ? ['Narrative', 'Evidence'] : ['Input', 'Router', 'Output']; + return `

${sceneHeading(template)}

+
+ ${labels.map((label, index) => `
+

${label}

${text}

+
`).join('')} +
`; + } + if (template === 'mobile') { + const crowded = variant === 'crowded-controls'; + return `

${sceneHeading(template)}

+
+
+

${text}

+ ${['Theme', 'Filter', 'Export'].map((label) => `${label}`).join('')} +
+
Policy 02
+
`; + } + if (template === 'diagram') return renderDiagram(variant, evalCase); + if (template === 'preset') return renderPreset(variant, evalCase); + if (template === 'operating') return renderOperating(variant, evalCase); + if (template === 'slop') return renderSlop(variant, evalCase); + throw new Error(`unsupported render template ${template}`); +} + +function renderDiagram(variant, evalCase) { + const regionId = evalCase.regions[0].id; + const timeline = ['dishonest-timeline', 'timeline-break'].includes(variant); + const bars = ['dishonest-bars', 'labeled-comparison', 'invalid-probability', 'valid-probability'].includes(variant); + if (timeline) { + return `

${sceneHeading('diagram')}

+
+
+ ${['Day 1', 'Day 2', variant === 'timeline-break' ? '// Month 6' : 'Month 6'].map((label) => `
${label}
`).join('')} +
+
`; + } + if (bars) { + const dishonestScale = variant === 'dishonest-bars' || variant === 'invalid-probability'; + const widths = dishonestScale ? [48, 58, 70] : [30, 62, 86]; + const labels = variant.includes('probability') + ? ['A 55%', variant === 'invalid-probability' ? 'B 35%' : 'B 25%', 'Other 20%'] + : ['A $10', 'B $40', 'Other $20']; + return `

${sceneHeading('diagram')}

+ ${widths.map((width, index) => `
${labels[index]}
`).join('')} +
`; + } + const nodes = ['Capture', 'Policy', 'Decision']; + const connectors = ['→', '→']; + const mixedRelations = variant === 'mixed-grammar' || variant === 'ambiguous-relations'; + const distortedMiddle = [ + 'mixed-grammar', + 'orphan-legend', + 'missing-legend-entry', + 'decorated-list', + 'table-as-graph', + 'ambiguous-relations', + ].includes(variant); + const dualFocal = variant === 'dual-focal'; + const singleFocal = variant === 'single-focal'; + const wrongLegend = variant === 'orphan-legend' || variant === 'missing-legend-entry'; + return `

${sceneHeading('diagram')}

+
+ ${nodes.map((node, index) => { + const focal = (dualFocal && index < 2) || (singleFocal && index === 1); + const style = [ + distortedMiddle && index === 1 ? 'border-radius:50%;transform:translateY(50px)' : '', + focal ? 'background:#22231f;color:#f6f3eb;transform:scale(1.08)' : '', + ].filter(Boolean).join(';'); + return `${index ? `${connectors[index - 1]}` : ''}
${node}${escapeHtml(evalCase.visible_text)}
`; + }).join('')} +
+
API${wrongLegend ? 'CACHE' : 'POLICY'}STORE
+
`; +} + +function renderPreset(variant, evalCase) { + const regionId = evalCase.regions[0].id; + const wrongMode = ['wrong-mode-panel', 'wrong-mode-background', 'invisible-dark-icons'].includes(variant); + const mismatchedDemo = variant === 'mismatched-demo'; + const mobileHeroCard = variant === 'mobile-hero-card'; + const multipleSurprises = variant === 'many-surprises' || variant === 'many-grid-breaks'; + const tooManyLayers = variant === 'too-many-layers'; + const statusAccent = [ + 'semantic-status-color', + 'destructive-red', + ].includes(variant); + const decorativeStatus = variant === 'decorative-status-color'; + const decorativeRed = variant === 'decorative-red'; + return `

${sceneHeading('preset')}

+ ${['LIGHT', 'DARK'].map((mode, index) => `
+
${mode}
+
+
${escapeHtml(evalCase.visible_text.split(' · ')[0])}
+

Measured route and supporting evidence.

+
+
STATUSEVIDENCE
+
`).join('')} +
`; +} + +function renderOperating(variant, evalCase) { + const regionId = evalCase.regions[0].id; + const labels = evalCase.visible_text.split(' · ').slice(0, 4); + const independentCards = [ + 'routing-cards', + 'one-way-loop', + 'missing-residual', + 'provenance-cards', + 'resource-metrics', + 'state-badges', + 'overbuilt-comparison', + 'cover-graph', + 'workspace-cards', + 'dependency-list', + ].includes(variant); + const cards = independentCards || variant === 'relational-equation' || variant === 'cover-none'; + return `

${sceneHeading('operating')}

+
+ ${labels.map((label, index) => `${!cards && index ? `${variant === 'closed-loop' && index === labels.length - 1 ? '↩' : '→'}` : ''}
${escapeHtml(label)}${independentCards ? 'Independent item' : `Rule ${index + 1}`}
`).join('')} +
+ ${!cards ? `
guards, dependencies, and provenance remain explicit
` : ''} +
`; +} + +function renderSlop(variant, evalCase) { + const regionId = evalCase.regions[0].id; + const parts = evalCase.visible_text.split(' · '); + const arrows = variant.includes('sequence') || variant.includes('flow'); + const support = { + 'real-section-number': 'Agenda · 01 Context · 02 Evidence · 03 Method · 04 Architecture', + 'real-citation': '[4] Evaluation Systems Review · 2026 · https://example.org/source', + }[variant] || ''; + return `

${sceneHeading('slop')}

+
+ ${parts.map((part, index) => `${index && arrows ? '→' : ''}
${escapeHtml(part)}
`).join('')} +
+
+
${escapeHtml(evalCase.visible_text)}
+ ${support ? `
${escapeHtml(support)}
` : ''} +
`; +} + +function escapeHtml(value) { + return String(value) + .replaceAll('&', '&') + .replaceAll('<', '<') + .replaceAll('>', '>') + .replaceAll('"', '"') + .replaceAll("'", '''); +} + +function sceneHeading(template) { + return escapeHtml({ + text: 'Mobile report', + table: 'Review inventory', + slide: 'Model policy', + dashboard: 'Evaluation dashboard', + 'diagram-space': 'Routing overview', + tracks: 'System map', + mobile: 'Model policy', + diagram: 'Evidence architecture', + preset: 'Preset specimen', + operating: 'Review operating model', + slop: 'Artifact excerpt', + }[template] || 'Artifact state'); +} + +function parseArgs(argv) { + const args = { + corpus: path.resolve(path.dirname(fileURLToPath(import.meta.url)), 'corpus.json'), + output: null, + cases: null, + }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === '--corpus') args.corpus = argv[++index]; + else if (arg === '--out') args.output = argv[++index]; + else if (arg === '--cases') args.cases = argv[++index].split(',').map((value) => value.trim()).filter(Boolean); + else throw new Error(`unknown argument ${arg}`); + } + return args; +} + +async function main() { + const args = parseArgs(process.argv.slice(2)); + const manifest = await renderCorpus({ + corpusPath: args.corpus, + outputRoot: args.output, + caseIds: args.cases, + }); + process.stdout.write(`Rendered ${manifest.images.length} corpus images\n`); + process.stdout.write(`${path.dirname(manifest.images[0]?.path || args.output)}\n`); +} + +if (import.meta.url === pathToFileURL(process.argv[1] || '').href) { + main().catch((error) => { + console.error(error.stack || error.message || error); + process.exit(1); + }); +} diff --git a/evals/visual-model-policy/run.mjs b/evals/visual-model-policy/run.mjs new file mode 100644 index 0000000..d36346b --- /dev/null +++ b/evals/visual-model-policy/run.mjs @@ -0,0 +1,766 @@ +#!/usr/bin/env node +import crypto from 'node:crypto'; +import fs from 'node:fs/promises'; +import path from 'node:path'; +import process from 'node:process'; +import { pathToFileURL } from 'node:url'; +import { canonicalJson } from '../lib/canonical-json.mjs'; +import { + attachCriterionSuffix, + buildPrefixManifest, +} from '../visual-cache/prefix.mjs'; +import { + expandCorpus, + loadCorpus, + validateCorpus, +} from './corpus.mjs'; +import { normalizeTelemetry } from './telemetry.mjs'; + +export const VERDICT_SYSTEM_TEXT = `Return JSON only. The top-level object must be {"verdicts":[...]}. Each supplied state_id must have exactly one verdict. A verdict is either {"state_id":"...","abstain":true,"reason":"..."} or {"state_id":"...","pass":true|false,"findings":[...]}. Every finding requires check_id, image_id, region_id, evidence, and fix.`; + +export const SHARED_INSTRUCTIONS = `Inspect only the supplied evidence. Judge only the selected criterion. Never infer a finding from another image or another criterion. Use the exact state_id, image_id, and region_id supplied for grounding. Stay silent when the criterion does not fire.`; + +export function corpusContractId(corpus) { + return crypto.createHash('sha256').update(canonicalJson(corpus)).digest('hex'); +} + +export function experimentContractId(experiment, corpus, requestMatrixSha256 = null) { + return crypto.createHash('sha256').update(canonicalJson({ + schema_version: 1, + experiment_id: experiment.id, + corpus_sha256: corpusContractId(corpus), + provider_contract: { + candidate: experiment.candidates.find((entry) => entry.id === experiment.run_candidate), + image_detail: experiment.image_detail, + batch_sizes: experiment.batch_sizes, + replicates: experiment.replicates, + passes: experiment.passes, + thresholds: experiment.thresholds, + adapter_options: experiment.adapter_options || {}, + ladder_state: experiment.ladder_state || { completed_candidates: [] }, + request_matrix_sha256: requestMatrixSha256, + }, + })).digest('hex'); +} + +export function requestIdFor(experimentId, cell) { + return `${experimentId}:${cell.model}:${cell.pass}:${cell.criterion_id}:b${cell.configured_batch_size}:r${cell.replicate}:${cell.batch_index}`; +} + +export function requestMatrixId(preparedPlan) { + return crypto.createHash('sha256').update(canonicalJson( + preparedPlan.map(({ request, ...cell }) => ({ + request_id: requestIdFor(cell.experiment_id, cell), + provider: cell.provider, + model: cell.model, + model_rank: cell.model_rank, + pass: cell.pass, + criterion_id: cell.criterion_id, + configured_batch_size: cell.configured_batch_size, + image_detail: cell.image_detail, + replicate: cell.replicate, + batch_index: cell.batch_index, + randomization_key: cell.randomization_key, + case_ids: cell.cases.map((entry) => entry.case_id), + image_ids: cell.cases.map((entry) => entry.image.id), + image_sha256s: request.prefix_manifest.images.map((entry) => entry.sha256), + prefix_id: request.prefix_manifest.prefix_id, + suffix_sha256: request.suffix.suffix_sha256, + })), + )).digest('hex'); +} + +export async function buildBatchRequest({ + provider, + model, + imageDetail = 'high', + pass, + criterionId, + criterionPrompt, + cases, + designSystemText = '', + toolsJson = '[]', +}) { + if (!Array.isArray(cases) || cases.length === 0) throw new Error('cases[] is required'); + if (cases.some((entry) => entry.family !== pass && entry.route_key !== pass)) { + throw new Error(`all cases must belong to pass ${pass}`); + } + if (cases.some((entry) => entry.criterion_id !== criterionId)) { + throw new Error(`all cases must use criterion ${criterionId}`); + } + const images = cases.map((entry) => ({ + id: entry.image.id, + path: entry.image.path, + detail: imageDetail, + })); + const prefixManifest = await buildPrefixManifest({ + provider, + model, + systemText: VERDICT_SYSTEM_TEXT, + sharedInstructions: SHARED_INSTRUCTIONS, + designSystemText, + toolsJson, + images, + }); + const suffixCases = cases.map((entry) => ({ + case_id: entry.case_id, + state_id: entry.state_id, + image_id: entry.image.id, + regions: entry.regions.map(({ id, label }) => ({ id, label })), + visible_text: entry.visible_text, + truth_excerpt: entry.truth_excerpt, + })); + const suffixPayload = { + pass, + criteria: [criterionId], + prompt: criterionPrompt, + cases: suffixCases, + }; + const suffixIdentity = attachCriterionSuffix(prefixManifest, { + pass, + criteria: [criterionId], + prompt: canonicalJson(suffixPayload), + }); + return { + schema_version: 1, + prefix_manifest: prefixManifest, + prefix: { + tools_json: toolsJson, + system_text: VERDICT_SYSTEM_TEXT, + shared_instructions: SHARED_INSTRUCTIONS, + design_system_text: designSystemText, + images: prefixManifest.images.map((image) => ({ + id: image.id, + detail: image.detail, + sha256: image.sha256, + path: cases.find((entry) => entry.image.id === image.id).image.path, + })), + }, + suffix: { + ...suffixPayload, + suffix_sha256: suffixIdentity.suffix_sha256, + }, + }; +} + +export function evaluateProviderResponse({ response, cases, criterionId }) { + const caseByState = new Map(cases.map((entry) => [entry.state_id, entry])); + let parsed; + try { + parsed = JSON.parse(response.raw_text); + } catch { + return invalidBatch(response, cases, 'response is not valid JSON'); + } + if (!parsed || !Array.isArray(parsed.verdicts)) { + return invalidBatch(response, cases, 'response requires verdicts[]'); + } + const foreignState = parsed.verdicts.find( + (verdict) => verdict?.state_id && !caseByState.has(verdict.state_id), + ); + if (foreignState) { + return invalidBatch( + response, + cases, + `response contains unknown state_id ${foreignState.state_id}`, + ); + } + + const verdictsByState = new Map(); + const duplicateStates = new Set(); + for (const verdict of parsed.verdicts) { + if (!verdict?.state_id || verdictsByState.has(verdict.state_id)) { + if (verdict?.state_id) duplicateStates.add(verdict.state_id); + continue; + } + verdictsByState.set(verdict.state_id, verdict); + } + const observations = cases.map((evalCase) => { + const verdict = verdictsByState.get(evalCase.state_id); + const validationError = validateVerdict({ + verdict, + evalCase, + criterionId, + duplicate: duplicateStates.has(evalCase.state_id), + }); + if (validationError) { + return observationFor(evalCase, { + predictedLabel: null, + jsonValid: false, + abstained: false, + grounded: false, + validationError, + verdict: verdict || null, + }); + } + if (verdict.abstain === true) { + return observationFor(evalCase, { + predictedLabel: null, + jsonValid: true, + abstained: true, + grounded: false, + validationError: null, + verdict, + }); + } + const matchingFindings = verdict.findings.filter( + (finding) => finding.check_id === criterionId, + ); + const predictedLabel = matchingFindings.length > 0 ? 'fire' : 'clean'; + const grounded = matchingFindings.length > 0 && matchingFindings.every((finding) => ( + finding.image_id === evalCase.image.id + && evalCase.regions.some((region) => region.id === finding.region_id) + )); + return observationFor(evalCase, { + predictedLabel, + jsonValid: true, + abstained: false, + grounded, + validationError: null, + verdict, + }); + }); + return { + raw_text: response.raw_text, + provider_response_id: response.provider_response_id || null, + json_valid: observations.every((entry) => entry.json_valid), + abstained: observations.some((entry) => entry.abstained), + observations, + telemetry: normalizeTelemetry(response.telemetry, { + allowIncomplete: true, + context: 'provider telemetry', + }), + }; +} + +export async function appendRawRecord(recordsPath, record) { + const output = path.resolve(recordsPath); + await fs.mkdir(path.dirname(output), { recursive: true }); + await fs.appendFile(output, `${JSON.stringify(record)}\n`, { + encoding: 'utf8', + flag: 'a', + }); + return output; +} + +export function criterionPromptFor(rubricText, criterionId) { + const marker = `[${criterionId}]`; + const markerCount = [...rubricText.matchAll(/\[[a-z0-9:-]+\]/g)].length; + if (!rubricText.includes(marker)) { + throw new Error(`rubric does not define criterion ${criterionId}`); + } + if (markerCount === 1) return rubricText.trim(); + const lines = rubricText.split('\n'); + const start = lines.findIndex((line) => line.includes(marker)); + const selected = [lines[start]]; + for (let index = start + 1; index < lines.length; index += 1) { + if (/^\s{2,}\S/.test(lines[index])) selected.push(lines[index]); + else break; + } + return `Judge only this registered criterion:\n\n${selected.join('\n').trim()}`; +} + +export async function runExperiment({ + experiment, + corpus, + adapter, + recordsPath, + dryRun = false, + now = () => new Date(), +}) { + validateExperiment(experiment); + if ( + !dryRun + && experiment.candidates.some((candidate) => ( + /replace/i.test(candidate.id) || /replace/i.test(candidate.provider) + )) + ) { + throw new Error('replace all placeholder candidate and provider values before a live run'); + } + validateCorpus(corpus); + const allCases = expandCorpus(corpus, { corpusPath: experiment.corpus_path }); + const selectedPasses = new Set(experiment.passes || allCases.map((entry) => entry.family)); + const cases = allCases.filter((entry) => selectedPasses.has(entry.family)); + if (selectedPasses.size === 0 || cases.length === 0) { + throw new Error('experiment passes must select at least one known corpus family'); + } + const unknownPasses = [...selectedPasses].filter( + (pass) => !allCases.some((entry) => entry.family === pass), + ); + if (unknownPasses.length > 0) { + throw new Error(`experiment references unknown pass(es): ${unknownPasses.join(', ')}`); + } + await assertImagesExist(cases); + const plan = buildExperimentPlan(experiment, cases); + if (plan.length === 0) throw new Error('experiment produced an empty request plan'); + const preparedPlan = await prepareExperimentPlan(experiment, plan); + const matrixId = requestMatrixId(preparedPlan); + const contractId = experimentContractId(experiment, corpus, matrixId); + if (dryRun) { + return { + experiment_id: experiment.id, + dry_run: true, + requests: preparedPlan.map(({ request, ...cell }) => cell), + }; + } + if (!adapter || typeof adapter.invoke !== 'function') { + throw new Error('adapter must export invoke(request, context)'); + } + if ( + adapter.synthetic !== true + && corpus.label_review?.status !== 'human-reviewed' + ) { + throw new Error( + 'live measurements require corpus.label_review.status=human-reviewed with reviewer and reviewed_at', + ); + } + + let written = 0; + for (const { request, ...cell } of preparedPlan) { + const requestId = requestIdFor(experiment.id, cell); + const startedAt = now(); + let adapterResponse; + try { + adapterResponse = await adapter.invoke(request, { + request_id: requestId, + experiment_id: experiment.id, + replicate: cell.replicate, + configured_batch_size: cell.configured_batch_size, + adapter_options: experiment.adapter_options || {}, + }); + } catch (error) { + adapterResponse = { + raw_text: '', + error: error.message || String(error), + telemetry: { + complete: false, + source: 'runner-local-clock', + latency_ms: now().getTime() - startedAt.getTime(), + time_to_first_token_ms: null, + input_tokens: null, + cache_read_tokens: null, + cache_write_tokens: null, + output_tokens: null, + actual_cost_usd: null, + }, + }; + } + const evaluated = evaluateProviderResponse({ + response: adapterResponse, + cases: cell.cases, + criterionId: cell.criterion_id, + }); + const imageHashById = new Map( + request.prefix_manifest.images.map((image) => [image.id, image.sha256]), + ); + const record = { + schema_version: 1, + record_type: 'visual-eval-request', + request_id: requestId, + experiment_id: experiment.id, + experiment_contract_id: contractId, + request_matrix_sha256: matrixId, + recorded_at: now().toISOString(), + replicate: cell.replicate, + provider: cell.provider, + model: cell.model, + model_rank: cell.model_rank, + model_class: cell.model_class, + pass: cell.pass, + criterion_id: cell.criterion_id, + configured_batch_size: cell.configured_batch_size, + actual_batch_size: cell.cases.length, + image_detail: cell.image_detail, + prefix_id: request.prefix_manifest.prefix_id, + suffix_sha256: request.suffix.suffix_sha256, + randomization_key: cell.randomization_key, + case_ids: cell.cases.map((entry) => entry.case_id), + image_ids: cell.cases.map((entry) => entry.image.id), + response: { + provider_response_id: evaluated.provider_response_id, + raw_text: evaluated.raw_text, + json_valid: evaluated.json_valid, + abstained: evaluated.abstained, + error: adapterResponse.error || null, + }, + telemetry: evaluated.telemetry, + observations: evaluated.observations.map((observation) => ({ + ...observation, + image_sha256: imageHashById.get(observation.image_id), + observation_id: `${requestId}:${observation.case_id}`, + })), + synthetic: adapter.synthetic === true, + }; + await appendRawRecord(recordsPath, record); + written += 1; + } + return { experiment_id: experiment.id, dry_run: false, requests_written: written, records_path: recordsPath }; +} + +export async function prepareExperimentPlan(experiment, plan) { + const prepared = []; + for (const cell of plan) { + const rubricText = await fs.readFile( + path.resolve(path.dirname(experiment.corpus_path), cell.rubric), + 'utf8', + ); + const criterionPrompt = criterionPromptFor(rubricText, cell.criterion_id); + const request = await buildBatchRequest({ + provider: cell.provider, + model: cell.model, + imageDetail: cell.image_detail, + pass: cell.pass, + criterionId: cell.criterion_id, + criterionPrompt, + cases: cell.cases, + }); + prepared.push({ + ...cell, + experiment_id: experiment.id, + request, + }); + } + return prepared; +} + +export function buildExperimentPlan(experiment, cases) { + validateLadderProgress( + experiment, + experiment.passes || [...new Set(cases.map((entry) => entry.family))], + ); + const plan = []; + const candidate = experiment.candidates.find( + (entry) => entry.id === experiment.run_candidate, + ); + for (const selectedCandidate of [candidate]) { + for (const batchSize of experiment.batch_sizes) { + for (let replicate = 1; replicate <= experiment.replicates; replicate += 1) { + const groups = groupByCriterion(cases); + for (const [key, criterionCases] of groups) { + const [pass, criterionId] = key.split('\u0000'); + if (criterionCases.length < batchSize) { + throw new Error( + `${pass}:${criterionId} has ${criterionCases.length} states and cannot measure batch ${batchSize}`, + ); + } + const randomizationKey = [ + experiment.id, + selectedCandidate.id, + pass, + criterionId, + batchSize, + replicate, + ].join('\u0000'); + const randomized = deterministicOrder(criterionCases, randomizationKey); + const batchCount = Math.ceil(randomized.length / batchSize); + for (let batchIndex = 0; batchIndex < batchCount; batchIndex += 1) { + const batch = Array.from( + { length: batchSize }, + (_, index) => randomized[(batchIndex * batchSize + index) % randomized.length], + ); + plan.push({ + provider: selectedCandidate.provider, + model: selectedCandidate.id, + model_rank: selectedCandidate.rank, + model_class: selectedCandidate.class || null, + pass, + criterion_id: criterionId, + rubric: batch[0].rubric, + image_detail: experiment.image_detail, + configured_batch_size: batchSize, + actual_batch_size: batch.length, + replicate, + batch_index: batchIndex + 1, + randomization_key: crypto + .createHash('sha256') + .update(randomizationKey) + .digest('hex'), + cases: batch, + }); + } + } + } + } + } + return plan; +} + +function deterministicOrder(cases, key) { + return [...cases].sort((a, b) => { + const score = (entry) => crypto + .createHash('sha256') + .update(`${key}\u0000${entry.case_id}`) + .digest('hex'); + return score(a).localeCompare(score(b)) || a.case_id.localeCompare(b.case_id); + }); +} + +function observationFor(evalCase, { + predictedLabel, + jsonValid, + abstained, + grounded, + validationError, + verdict, +}) { + return { + case_id: evalCase.case_id, + state_id: evalCase.state_id, + image_id: evalCase.image.id, + criterion_id: evalCase.criterion_id, + human_label: evalCase.human_label, + predicted_label: predictedLabel, + disagreement: predictedLabel !== null && predictedLabel !== evalCase.human_label, + json_valid: jsonValid, + abstained, + grounded, + validation_error: validationError, + verdict, + }; +} + +function validateVerdict({ verdict, evalCase, criterionId, duplicate }) { + if (duplicate) return `duplicate verdict for ${evalCase.state_id}`; + if (!verdict) return `missing verdict for ${evalCase.state_id}`; + if (verdict.abstain === true) { + if (!verdict.reason) return 'abstention requires reason'; + return null; + } + if (typeof verdict.pass !== 'boolean' || !Array.isArray(verdict.findings)) { + return 'verdict requires pass boolean and findings[]'; + } + for (const finding of verdict.findings) { + for (const field of ['check_id', 'image_id', 'region_id', 'evidence', 'fix']) { + if (!finding?.[field]) return `finding requires ${field}`; + } + if (finding.check_id !== criterionId) return `finding uses unexpected check_id ${finding.check_id}`; + } + if (verdict.pass !== (verdict.findings.length === 0)) { + return 'pass must be true exactly when findings is empty'; + } + return null; +} + +function invalidBatch(response, cases, validationError) { + return { + raw_text: response.raw_text || '', + provider_response_id: response.provider_response_id || null, + json_valid: false, + abstained: false, + observations: cases.map((entry) => observationFor(entry, { + predictedLabel: null, + jsonValid: false, + abstained: false, + grounded: false, + validationError, + verdict: null, + })), + telemetry: normalizeTelemetry(response.telemetry, { + allowIncomplete: true, + context: 'provider telemetry', + }), + }; +} + +function groupByCriterion(cases) { + const groups = new Map(); + for (const entry of cases) { + const key = `${entry.family}\u0000${entry.criterion_id}`; + if (!groups.has(key)) groups.set(key, []); + groups.get(key).push(entry); + } + return [...groups.entries()].sort(([a], [b]) => a.localeCompare(b)); +} + +function validateExperiment(experiment) { + if ( + !experiment?.id + || !experiment.corpus_path + || !experiment.run_candidate + || !Array.isArray(experiment.candidates) + || experiment.candidates.length === 0 + || !Array.isArray(experiment.batch_sizes) + || experiment.batch_sizes.length === 0 + ) { + throw new Error( + 'experiment requires id, corpus_path, run_candidate, candidates[], and batch_sizes[]', + ); + } + if (!Number.isInteger(experiment.replicates) || experiment.replicates < 1) { + throw new Error('experiment replicates must be a positive integer'); + } + if (!['low', 'high', 'original'].includes(experiment.image_detail)) { + throw new Error('experiment image_detail must be low, high, or original'); + } + if ( + experiment.batch_sizes.some((value) => !Number.isInteger(value) || value < 1) + || new Set(experiment.batch_sizes).size !== experiment.batch_sizes.length + ) { + throw new Error('experiment batch_sizes must be unique positive integers'); + } + const ranks = new Set(); + const candidateIds = new Set(); + for (const candidate of experiment.candidates) { + if (!candidate.id || !candidate.provider || !Number.isInteger(candidate.rank) || candidate.rank < 1) { + throw new Error('each candidate requires id, provider, and positive integer rank'); + } + if (ranks.has(candidate.rank)) throw new Error(`duplicate candidate rank ${candidate.rank}`); + if (candidateIds.has(candidate.id)) throw new Error(`duplicate candidate id ${candidate.id}`); + ranks.add(candidate.rank); + candidateIds.add(candidate.id); + } + if (!experiment.candidates.some((candidate) => candidate.id === experiment.run_candidate)) { + throw new Error(`run_candidate ${experiment.run_candidate} is not present in candidates[]`); + } +} + +function validateLadderProgress(experiment, selectedPasses) { + const candidates = [...(experiment.candidates || [])].sort( + (a, b) => Number(a.rank) - Number(b.rank), + ); + const completed = experiment.ladder_state?.completed_candidates || []; + if (!Array.isArray(completed)) { + throw new Error('ladder_state.completed_candidates must be an array'); + } + const candidateById = new Map(candidates.map((candidate) => [candidate.id, candidate])); + const completedById = new Map(); + for (const entry of completed) { + if ( + !candidateById.has(entry?.id) + || !entry.pass_evidence + || typeof entry.pass_evidence !== 'object' + ) { + throw new Error( + 'each completed candidate requires a configured id and pass_evidence', + ); + } + for (const [pass, evidence] of Object.entries(entry.pass_evidence)) { + if ( + !pass + || !['qualified', 'disqualified'].includes(evidence?.status) + || !/^[a-f0-9]{64}$/.test(evidence?.evidence_sha256 || '') + ) { + throw new Error(`invalid ladder evidence for ${pass}`); + } + } + if (Object.keys(entry.pass_evidence).length === 0) { + throw new Error(`completed candidate ${entry.id} requires at least one pass evidence entry`); + } + if (completedById.has(entry.id)) throw new Error(`duplicate completed candidate ${entry.id}`); + completedById.set(entry.id, entry); + } + for (const pass of selectedPasses) { + const completedForPass = candidates.filter( + (candidate) => completedById.get(candidate.id)?.pass_evidence?.[pass], + ); + const completedPrefix = candidates.slice(0, completedForPass.length); + if (completedPrefix.some((candidate) => !completedForPass.includes(candidate))) { + throw new Error( + `completed candidates for ${pass} must form a contiguous smallest-first ladder prefix`, + ); + } + const qualified = completedForPass.filter( + (candidate) => ( + completedById.get(candidate.id).pass_evidence[pass].status === 'qualified' + ), + ); + if (qualified.length >= 2) { + throw new Error(`${pass} already has a qualified model and escalation candidate; stop its ladder`); + } + const nextCandidate = candidates.find((candidate) => !completedForPass.includes(candidate)); + if (!nextCandidate) throw new Error(`all configured candidates are already completed for ${pass}`); + if (experiment.run_candidate !== nextCandidate.id) { + throw new Error( + `run_candidate must be next smallest untested candidate ${nextCandidate.id} for ${pass}`, + ); + } + } +} + +async function assertImagesExist(cases) { + const missing = []; + for (const entry of cases) { + try { + await fs.access(entry.image.path); + } catch { + missing.push(entry.image.path); + } + } + if (missing.length > 0) { + throw new Error( + `corpus images are missing (${missing.length}); run npm run ve:render-visual-model-corpus`, + ); + } +} + +async function loadAdapter(adapterPath) { + const module = await import(pathToFileURL(path.resolve(adapterPath)).href); + if (typeof module.invoke !== 'function') { + throw new Error(`adapter ${adapterPath} must export invoke(request, context)`); + } + return module; +} + +function parseArgs(argv) { + const args = { experiment: null, adapter: null, records: null, dryRun: false }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === '--experiment') args.experiment = argv[++index]; + else if (arg === '--adapter') args.adapter = argv[++index]; + else if (arg === '--records') args.records = argv[++index]; + else if (arg === '--dry-run') args.dryRun = true; + else throw new Error(`unknown argument ${arg}`); + } + if (!args.experiment) { + throw new Error('usage: run.mjs --experiment experiment.json [--adapter adapter.mjs] [--records records.jsonl] [--dry-run]'); + } + if (!args.dryRun && !args.adapter) throw new Error('--adapter is required unless --dry-run is used'); + return args; +} + +async function main() { + const args = parseArgs(process.argv.slice(2)); + const experimentPath = path.resolve(args.experiment); + const experiment = JSON.parse(await fs.readFile(experimentPath, 'utf8')); + experiment.corpus_path = path.resolve( + path.dirname(experimentPath), + experiment.corpus_path, + ); + const corpus = await loadCorpus(experiment.corpus_path); + const adapter = args.dryRun ? null : await loadAdapter(args.adapter); + const recordsPath = path.resolve( + args.records + || path.join( + path.dirname(experimentPath), + 'runs', + experiment.id, + 'records.jsonl', + ), + ); + const outcome = await runExperiment({ + experiment, + corpus, + adapter, + recordsPath, + dryRun: args.dryRun, + }); + if (args.dryRun) { + const summary = { + experiment_id: outcome.experiment_id, + dry_run: true, + requests: outcome.requests.length, + cells: [...new Set(outcome.requests.map((entry) => ( + `${entry.model}/${entry.pass}/b${entry.configured_batch_size}` + )))], + }; + process.stdout.write(`${JSON.stringify(summary, null, 2)}\n`); + } else { + process.stdout.write(`${JSON.stringify(outcome, null, 2)}\n`); + } +} + +if (import.meta.url === pathToFileURL(process.argv[1] || '').href) { + main().catch((error) => { + console.error(error.stack || error.message || error); + process.exit(1); + }); +} diff --git a/evals/visual-model-policy/run.test.mjs b/evals/visual-model-policy/run.test.mjs new file mode 100644 index 0000000..532966f --- /dev/null +++ b/evals/visual-model-policy/run.test.mjs @@ -0,0 +1,275 @@ +import assert from 'node:assert/strict'; +import fs from 'node:fs/promises'; +import os from 'node:os'; +import path from 'node:path'; +import test from 'node:test'; +import { + appendRawRecord, + buildBatchRequest, + buildExperimentPlan, + criterionPromptFor, + evaluateProviderResponse, + runExperiment, +} from './run.mjs'; + +async function fixture(t) { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'artifacture-visual-run-')); + t.after(() => fs.rm(dir, { recursive: true, force: true })); + const imagePath = path.join(dir, 'layout:clip:fire.png'); + await fs.writeFile(imagePath, Buffer.from('not-a-real-png')); + return { + dir, + case: { + case_id: 'layout:clip:fire', + state_id: 'layout:clip:fire', + family: 'layout', + criterion_id: 'text-visibly-clipped', + human_label: 'fire', + hard_negative: false, + image: { id: 'layout:clip:fire', path: imagePath, detail: 'high' }, + regions: [{ id: 'mobile-heading', label: 'Mobile heading' }], + visible_text: 'A heading is visibly chopped', + truth_excerpt: 'The full heading must remain readable.', + adjudication_notes: 'Characters are cut off without an intentional ellipsis.', + }, + }; +} + +test('batch requests put immutable evidence before one criterion suffix and hide labels', async (t) => { + const { case: evalCase } = await fixture(t); + const request = await buildBatchRequest({ + provider: 'test-provider', + model: 'test-small', + imageDetail: 'high', + pass: 'layout', + criterionId: 'text-visibly-clipped', + criterionPrompt: 'Judge only visible text clipping.', + cases: [evalCase], + }); + + assert.equal(request.prefix_manifest.images[0].id, evalCase.image.id); + assert.deepEqual(Object.keys(request.prefix), [ + 'tools_json', + 'system_text', + 'shared_instructions', + 'design_system_text', + 'images', + ]); + assert.equal(request.suffix.pass, 'layout'); + assert.deepEqual(request.suffix.criteria, ['text-visibly-clipped']); + assert.equal(request.suffix.cases[0].state_id, evalCase.state_id); + assert.equal(JSON.stringify(request).includes('human_label'), false); + assert.equal(JSON.stringify(request).includes('adjudication_notes'), false); +}); + +test('the committed corpus uses label-blind evidence ids', async () => { + const { expandCorpus, loadCorpus } = await import('./corpus.mjs'); + const cases = expandCorpus(await loadCorpus()); + assert.equal(cases.some((entry) => /fire|clean/.test(entry.case_id)), false); + assert.equal(cases.some((entry) => /fire|clean/.test(entry.image.id)), false); +}); + +test('provider verdicts are scored only when they use stable image and region ids', async (t) => { + const { case: evalCase } = await fixture(t); + const response = { + raw_text: JSON.stringify({ + verdicts: [{ + state_id: evalCase.state_id, + pass: false, + findings: [{ + check_id: 'text-visibly-clipped', + image_id: evalCase.image.id, + region_id: 'mobile-heading', + evidence: 'The final characters are cut by the right edge.', + fix: 'Allow wrapping.', + }], + }], + }), + telemetry: { + latency_ms: 120, + time_to_first_token_ms: 40, + input_tokens: 200, + cache_read_tokens: 150, + cache_write_tokens: 0, + output_tokens: 40, + actual_cost_usd: 0.0002, + }, + }; + + const evaluated = evaluateProviderResponse({ + response, + cases: [evalCase], + criterionId: 'text-visibly-clipped', + }); + assert.equal(evaluated.json_valid, true); + assert.equal(evaluated.observations[0].predicted_label, 'fire'); + assert.equal(evaluated.observations[0].grounded, true); + + const ungrounded = evaluateProviderResponse({ + response: { + ...response, + raw_text: response.raw_text.replace('mobile-heading', 'unknown-region'), + }, + cases: [evalCase], + criterionId: 'text-visibly-clipped', + }); + assert.equal(ungrounded.observations[0].grounded, false); + + const foreignState = evaluateProviderResponse({ + response: { + ...response, + raw_text: JSON.stringify({ + verdicts: [ + ...JSON.parse(response.raw_text).verdicts, + { + state_id: 'foreign-state', + pass: true, + findings: [], + }, + ], + }), + }, + cases: [evalCase], + criterionId: 'text-visibly-clipped', + }); + assert.equal(foreignState.json_valid, false); + assert.match(foreignState.observations[0].validation_error, /unknown state_id/); +}); + +test('raw record writes append and never replace earlier measurements', async (t) => { + const { dir } = await fixture(t); + const recordsPath = path.join(dir, 'records.jsonl'); + await appendRawRecord(recordsPath, { schema_version: 1, request_id: 'request-1' }); + await appendRawRecord(recordsPath, { schema_version: 1, request_id: 'request-2' }); + const lines = (await fs.readFile(recordsPath, 'utf8')).trim().split('\n').map(JSON.parse); + assert.deepEqual(lines.map((line) => line.request_id), ['request-1', 'request-2']); +}); + +test('criterion suffix extraction excludes sibling family questions', () => { + const rubric = `Questions: +- [text-visibly-clipped] Is text cut off? +- [layout-repeated-track-symmetry] Do peer tracks drift? + +Verdict JSON schema: {...}`; + const prompt = criterionPromptFor(rubric, 'text-visibly-clipped'); + assert.match(prompt, /Is text cut off/); + assert.doesNotMatch(prompt, /peer tracks drift/); +}); + +test('the staged ladder runs one candidate and wraps randomized tails into full batches', () => { + const cases = ['a', 'b', 'c', 'd', 'e'].map((id) => ({ + case_id: id, + family: 'layout', + criterion_id: 'text-visibly-clipped', + rubric: '/rubric.md', + })); + const plan = buildExperimentPlan({ + run_candidate: 'nano', + candidates: [ + { id: 'nano', provider: 'test', rank: 1, class: 'small' }, + { id: 'frontier', provider: 'test', rank: 2, class: 'large' }, + ], + batch_sizes: [4], + replicates: 1, + image_detail: 'high', + }, cases); + assert.equal(plan.length, 2); + assert.equal(plan[0].model, 'nano'); + assert.equal(plan.every((entry) => entry.actual_batch_size === 4), true); + assert.equal(plan.every((entry) => new Set(entry.cases).size === 4), true); + assert.deepEqual( + [...new Set(plan.flatMap((entry) => entry.cases.map((item) => item.case_id)))].sort(), + ['a', 'b', 'c', 'd', 'e'], + ); +}); + +test('the candidate ladder enforces smallest-first progress and stops after escalation', () => { + const cases = ['fire-a', 'clean-a'].map((id) => ({ + case_id: id, + family: 'layout', + criterion_id: 'text-visibly-clipped', + rubric: '/rubric.md', + })); + const candidates = [ + { id: 'nano', provider: 'test', rank: 1, class: 'small' }, + { id: 'mini', provider: 'test', rank: 2, class: 'medium' }, + { id: 'frontier', provider: 'test', rank: 3, class: 'large' }, + ]; + assert.throws( + () => buildExperimentPlan({ + id: 'ladder', + run_candidate: 'mini', + candidates, + batch_sizes: [1], + replicates: 1, + image_detail: 'high', + }, cases), + /next smallest untested candidate nano/, + ); + assert.doesNotThrow(() => buildExperimentPlan({ + id: 'ladder', + run_candidate: 'mini', + candidates, + ladder_state: { + completed_candidates: [{ + id: 'nano', + pass_evidence: { + layout: { status: 'qualified', evidence_sha256: 'a'.repeat(64) }, + }, + }], + }, + batch_sizes: [1], + replicates: 1, + image_detail: 'high', + }, cases)); + assert.throws( + () => buildExperimentPlan({ + id: 'ladder', + run_candidate: 'frontier', + candidates, + ladder_state: { + completed_candidates: [ + { + id: 'nano', + pass_evidence: { + layout: { status: 'qualified', evidence_sha256: 'a'.repeat(64) }, + }, + }, + { + id: 'mini', + pass_evidence: { + layout: { status: 'qualified', evidence_sha256: 'b'.repeat(64) }, + }, + }, + ], + }, + batch_sizes: [1], + replicates: 1, + image_detail: 'high', + }, cases), + /stop its ladder/, + ); +}); + +test('an unknown pass cannot become an empty successful experiment', async () => { + const { loadCorpus } = await import('./corpus.mjs'); + const corpus = await loadCorpus(); + await assert.rejects( + () => runExperiment({ + experiment: { + id: 'unknown-pass', + corpus_path: new URL('./corpus.json', import.meta.url).pathname, + run_candidate: 'nano', + candidates: [{ id: 'nano', provider: 'fixture', rank: 1 }], + batch_sizes: [1], + replicates: 1, + image_detail: 'high', + passes: ['not-a-family'], + }, + corpus, + dryRun: true, + recordsPath: '/tmp/unused-visual-records.jsonl', + }), + /known corpus family|unknown pass/, + ); +}); diff --git a/evals/visual-model-policy/select.mjs b/evals/visual-model-policy/select.mjs index c71234c..2a0e87c 100644 --- a/evals/visual-model-policy/select.mjs +++ b/evals/visual-model-policy/select.mjs @@ -10,6 +10,8 @@ export const DEFAULT_THRESHOLDS = Object.freeze({ min_runs: 3, min_positives: 10, min_negatives: 10, + min_criterion_positives: 1, + min_criterion_negatives: 1, precision: 0.9, recall: 0.9, silence_accuracy: 0.95, @@ -38,8 +40,12 @@ export function metricsFor(result) { grounding_accuracy: ratio(result.grounded, result.grounding_total, 0), json_validity: ratio(result.json_valid, responses, 0), abstention_rate: ratio(result.abstentions, responses, 0), - cost_per_case_usd: Number(result.total_cost_usd || 0) / cases, - p95_latency_ms: Number(result.p95_latency_ms), + cost_per_case_usd: result.total_cost_usd === null + ? null + : Number(result.total_cost_usd) / cases, + p95_latency_ms: result.p95_latency_ms === null + ? null + : Number(result.p95_latency_ms), }; } @@ -50,7 +56,7 @@ export function qualificationFor(result, thresholds = DEFAULT_THRESHOLDS) { if (Number(actual) < Number(minimum)) reasons.push(`${field} ${actual} < ${minimum}`); }; const requireAtMost = (field, actual, maximum) => { - if (!Number.isFinite(Number(actual)) || Number(actual) > Number(maximum)) { + if (actual === null || !Number.isFinite(Number(actual)) || Number(actual) > Number(maximum)) { reasons.push(`${field} ${actual} > ${maximum}`); } }; @@ -58,6 +64,36 @@ export function qualificationFor(result, thresholds = DEFAULT_THRESHOLDS) { requireAtLeast('runs', result.runs, thresholds.min_runs); requireAtLeast('positives', result.positives, thresholds.min_positives); requireAtLeast('negatives', result.negatives, thresholds.min_negatives); + if (result.unique_positives !== undefined || result.unique_negatives !== undefined) { + requireAtLeast('unique_positives', result.unique_positives, thresholds.min_positives); + requireAtLeast('unique_negatives', result.unique_negatives, thresholds.min_negatives); + } + if (result.adjudication_complete !== true) { + reasons.push('adjudication_complete must be true'); + } + if (result.synthetic !== false) { + reasons.push('synthetic measurements are not policy evidence'); + } + if (result.telemetry_complete !== true) { + reasons.push('telemetry_complete must be true'); + } + if (result.experiment_complete !== true) { + reasons.push('experiment_complete must be true'); + } + if (Array.isArray(result.criteria)) { + for (const criterion of result.criteria) { + requireAtLeast( + `criterion ${criterion.id} unique_positives`, + criterion.unique_positives, + thresholds.min_criterion_positives, + ); + requireAtLeast( + `criterion ${criterion.id} unique_negatives`, + criterion.unique_negatives, + thresholds.min_criterion_negatives, + ); + } + } requireAtLeast('precision', metrics.precision, thresholds.precision); requireAtLeast('recall', metrics.recall, thresholds.recall); requireAtLeast('silence_accuracy', metrics.silence_accuracy, thresholds.silence_accuracy); @@ -120,6 +156,11 @@ export function selectVisualModelPolicy(input) { cases: selected.cases, positives: selected.positives, negatives: selected.negatives, + ...(selected.unique_cases === undefined ? {} : { + unique_cases: selected.unique_cases, + unique_positives: selected.unique_positives, + unique_negatives: selected.unique_negatives, + }), }, escalation_chain: ladder.slice(1).map((row) => ({ model: row.model, @@ -147,6 +188,9 @@ function validateInput(input) { if (!input || !Array.isArray(input.candidates) || !Array.isArray(input.results)) { throw new Error('input requires candidates[] and results[]'); } + if (input.candidates.length === 0 || input.results.length === 0) { + throw new Error('input requires non-empty candidates[] and results[]'); + } const ids = new Set(); const ranks = new Set(); for (const candidate of input.candidates) { @@ -163,8 +207,19 @@ function validateInput(input) { 'pass', 'model', 'batch_size', 'runs', 'cases', 'positives', 'negatives', 'tp', 'fp', 'tn', 'fn', 'grounded', 'grounding_total', 'json_valid', 'responses', 'abstentions', 'total_cost_usd', 'p95_latency_ms', + 'unique_cases', 'unique_positives', 'unique_negatives', + 'adjudication_complete', 'synthetic', 'telemetry_complete', + 'experiment_complete', ]) { - if (result[field] === undefined || result[field] === null || result[field] === '') { + const nullableIncompleteTelemetry = ( + ['total_cost_usd', 'p95_latency_ms'].includes(field) + && result.telemetry_complete === false + ); + if ( + result[field] === undefined + || result[field] === '' + || (result[field] === null && !nullableIncompleteTelemetry) + ) { throw new Error(`result requires ${field}`); } } @@ -176,9 +231,24 @@ function validateInput(input) { 'positives', 'negatives', 'tp', 'fp', 'tn', 'fn', 'grounded', 'json_valid', 'abstentions', ]) nonNegativeInt(result[field], field); + for (const field of ['unique_cases', 'unique_positives', 'unique_negatives']) { + nonNegativeInt(result[field], field); + } + for (const field of [ + 'adjudication_complete', + 'synthetic', + 'telemetry_complete', + 'experiment_complete', + ]) { + if (typeof result[field] !== 'boolean') throw new Error(`${field} must be a boolean`); + } positiveInt(result.grounding_total, 'grounding_total'); - nonNegativeFinite(result.total_cost_usd, 'total_cost_usd'); - nonNegativeFinite(result.p95_latency_ms, 'p95_latency_ms'); + if (result.total_cost_usd !== null) { + nonNegativeFinite(result.total_cost_usd, 'total_cost_usd'); + } + if (result.p95_latency_ms !== null) { + nonNegativeFinite(result.p95_latency_ms, 'p95_latency_ms'); + } if (Number(result.positives) + Number(result.negatives) !== Number(result.cases)) { throw new Error('positives + negatives must equal cases'); } @@ -197,6 +267,12 @@ function validateInput(input) { if (Number(result.grounded) > Number(result.grounding_total)) { throw new Error('grounded cannot exceed grounding_total'); } + if ( + result.unique_cases !== undefined + && Number(result.unique_positives) + Number(result.unique_negatives) !== Number(result.unique_cases) + ) { + throw new Error('unique_positives + unique_negatives must equal unique_cases'); + } } } @@ -232,7 +308,9 @@ function roundMetrics(metrics) { return Object.fromEntries( Object.entries(metrics).map(([key, value]) => [ key, - Number.isFinite(value) ? Number(value.toFixed(key === 'cost_per_case_usd' ? 8 : 4)) : value, + Number.isFinite(value) + ? Number(value.toFixed(key === 'cost_per_case_usd' ? 8 : 4)) + : value, ]), ); } diff --git a/evals/visual-model-policy/select.test.mjs b/evals/visual-model-policy/select.test.mjs index 50ca0b7..d41d335 100644 --- a/evals/visual-model-policy/select.test.mjs +++ b/evals/visual-model-policy/select.test.mjs @@ -33,6 +33,13 @@ function result(overrides = {}) { abstentions: 1, total_cost_usd: 0.1, p95_latency_ms: 5000, + unique_cases: 20, + unique_positives: 10, + unique_negatives: 10, + adjudication_complete: true, + synthetic: false, + telemetry_complete: true, + experiment_complete: true, ...overrides, }; } @@ -168,3 +175,51 @@ test('requires grounding observations', () => { /grounding_total must be a positive integer/, ); }); + +test('repeated observations cannot substitute for independent labeled cases', () => { + const policy = selectVisualModelPolicy({ + candidates, + results: [result({ + unique_cases: 8, + unique_positives: 4, + unique_negatives: 4, + })], + }); + assert.equal(policy.routes.layout, undefined); + assert.match(policy.blocked.layout.evaluated[0].reasons.join('\n'), /unique_positives/); +}); + +test('blocks measurements with pending human adjudication', () => { + const policy = selectVisualModelPolicy({ + candidates, + results: [result({ adjudication_complete: false })], + }); + assert.equal(policy.routes.layout, undefined); + assert.match(policy.blocked.layout.evaluated[0].reasons.join('\n'), /adjudication_complete/); +}); + +test('blocks synthetic dry-run evidence', () => { + const policy = selectVisualModelPolicy({ + candidates, + results: [result({ synthetic: true })], + }); + assert.equal(policy.routes.layout, undefined); + assert.match(policy.blocked.layout.evaluated[0].reasons.join('\n'), /synthetic/); +}); + +test('blocks incomplete provider telemetry', () => { + const policy = selectVisualModelPolicy({ + candidates, + results: [result({ telemetry_complete: false })], + }); + assert.equal(policy.routes.layout, undefined); + assert.match(policy.blocked.layout.evaluated[0].reasons.join('\n'), /telemetry_complete/); +}); + +test('blocks an incomplete experiment request matrix', () => { + const policy = selectVisualModelPolicy({ + candidates, + results: [result({ experiment_complete: false })], + }); + assert.match(policy.blocked.layout.evaluated[0].reasons.join('\n'), /experiment_complete/); +}); diff --git a/evals/visual-model-policy/telemetry.mjs b/evals/visual-model-policy/telemetry.mjs new file mode 100644 index 0000000..024f9d8 --- /dev/null +++ b/evals/visual-model-policy/telemetry.mjs @@ -0,0 +1,35 @@ +export const TELEMETRY_FIELDS = Object.freeze([ + 'latency_ms', + 'time_to_first_token_ms', + 'input_tokens', + 'cache_read_tokens', + 'cache_write_tokens', + 'output_tokens', + 'actual_cost_usd', +]); + +export function normalizeTelemetry(telemetry = {}, { + allowIncomplete = false, + context = 'provider telemetry', +} = {}) { + const complete = telemetry.complete !== false; + if (!complete && !allowIncomplete) throw new Error(`${context} is incomplete`); + const output = { complete }; + for (const field of TELEMETRY_FIELDS) { + const value = telemetry[field]; + if (!complete && (value === undefined || value === null)) { + output[field] = null; + continue; + } + if (value === undefined || value === null) { + throw new Error(`${context} requires non-negative ${field}`); + } + const number = Number(value); + if (!Number.isFinite(number) || number < 0) { + throw new Error(`${context} requires non-negative ${field}`); + } + output[field] = number; + } + if (telemetry.source) output.source = String(telemetry.source); + return output; +} diff --git a/package.json b/package.json index 9dfc80b..25b8a83 100644 --- a/package.json +++ b/package.json @@ -19,11 +19,15 @@ "ve:eval": "node evals/run.mjs && node evals/design-systems/run.mjs", "ve:eval-presentation": "node evals/run-presentation.mjs", "ve:eval-rubrics": "node evals/run-rubrics.mjs", - "ve:eval-visual-model-policy": "node --test evals/visual-model-policy/select.test.mjs evals/visual-cache/prefix.test.mjs plugins/visual-explainer/scripts/verify/lib/model-policy.test.mjs", + "ve:eval-visual-model-policy": "node --test evals/visual-model-policy/corpus.test.mjs evals/visual-model-policy/run.test.mjs evals/visual-model-policy/aggregate.test.mjs evals/visual-model-policy/merge.test.mjs evals/visual-model-policy/select.test.mjs evals/visual-cache/prefix.test.mjs plugins/visual-explainer/scripts/verify/lib/model-policy.test.mjs", + "ve:render-visual-model-corpus": "node evals/visual-model-policy/render-corpus.mjs", + "ve:run-visual-model-eval": "node evals/visual-model-policy/run.mjs", + "ve:aggregate-visual-model-eval": "node evals/visual-model-policy/aggregate.mjs", + "ve:merge-visual-model-measurements": "node evals/visual-model-policy/merge.mjs", "ve:select-visual-model-policy": "node evals/visual-model-policy/select.mjs", "check:manifests": "node scripts/check-manifests.mjs", "ve:learn": "node scripts/ve-mdx/learn.mjs", - "test": "node --test visual-explainer-mdx/diagram-layout.test.mjs visual-explainer-mdx/presentation-core.test.mjs scripts/ve-mdx/design-systems.test.mjs scripts/ve-mdx/export-react-singleton.test.mjs scripts/ve-mdx/learn-extractors.test.mjs evals/visual-model-policy/select.test.mjs evals/visual-cache/prefix.test.mjs plugins/visual-explainer/scripts/verify/lib/model-policy.test.mjs" + "test": "node --test visual-explainer-mdx/diagram-layout.test.mjs visual-explainer-mdx/presentation-core.test.mjs scripts/ve-mdx/design-systems.test.mjs scripts/ve-mdx/export-react-singleton.test.mjs scripts/ve-mdx/learn-extractors.test.mjs evals/visual-model-policy/corpus.test.mjs evals/visual-model-policy/run.test.mjs evals/visual-model-policy/aggregate.test.mjs evals/visual-model-policy/merge.test.mjs evals/visual-model-policy/select.test.mjs evals/visual-cache/prefix.test.mjs plugins/visual-explainer/scripts/verify/lib/model-policy.test.mjs" }, "dependencies": { "@mdx-js/rollup": "^3.1.1", From 5885fd792fbb910a790ba9c58871df0b503e45b6 Mon Sep 17 00:00:00 2001 From: Clayton Kim Date: Sat, 25 Jul 2026 11:44:33 -0700 Subject: [PATCH 3/4] Curate visual evaluation seed corpus --- evals/visual-model-policy/README.md | 77 +- evals/visual-model-policy/aggregate.mjs | 44 +- evals/visual-model-policy/aggregate.test.mjs | 98 +- evals/visual-model-policy/corpus.json | 1047 ++++++++--------- evals/visual-model-policy/corpus.mjs | 122 +- evals/visual-model-policy/corpus.test.mjs | 42 +- .../example-measurements.json | 9 + .../experiment.template.json | 2 +- evals/visual-model-policy/merge.test.mjs | 3 + evals/visual-model-policy/render-corpus.mjs | 188 ++- evals/visual-model-policy/run.mjs | 55 +- evals/visual-model-policy/run.test.mjs | 28 + evals/visual-model-policy/select.mjs | 39 +- evals/visual-model-policy/select.test.mjs | 18 + 14 files changed, 1140 insertions(+), 632 deletions(-) diff --git a/evals/visual-model-policy/README.md b/evals/visual-model-policy/README.md index 0cda263..54284e1 100644 --- a/evals/visual-model-policy/README.md +++ b/evals/visual-model-policy/README.md @@ -15,8 +15,8 @@ evidence, dispatches visual judgment, merges findings, and reports uncertainty. It does not inspect every screenshot with the host model. Model choice is per family, not global. Batch qualification is also per family -and per model. A model qualified at one screenshot is not qualified at two or -four, and a layout qualification never applies to diagrams, preset fidelity, +and per model. A model qualified at one screenshot is not qualified at two, and +a layout qualification never applies to diagrams, preset fidelity, operating-model fidelity, or the opt-in Artifacture semantic gap. The selector: @@ -33,8 +33,8 @@ A cheaper larger model does not displace a qualified smaller model. Candidate ## Corpus -`corpus.json` defines 112 label-blind states: at least 10 proposed fire cases -and 10 proposed clean hard negatives for each owned family: +`corpus.json` defines a deliberately compact seed corpus: 36 paired scenarios +and 72 label-blind states across the owned families: - layout, including clipping, focal crowding, authored versus accidental dead space, and repeated-track symmetry; @@ -46,13 +46,24 @@ and 10 proposed clean hard negatives for each owned family: Every state has a stable opaque case/state/image ID, named regions, visible text, the smallest truth excerpt needed for judgment, an initial label, and adjudication notes. Impeccable and Unslop criteria are deliberately absent. -Every criterion has at least four states, so batch size 4 can exercise every -criterion. Rendering fails if any two case IDs produce identical PNG bytes; -aggregation keys unique evidence to the recorded PNG hash, not merely the case -ID. - -The checked-in labels are proposed from fixture intent, not represented as -human review. Render and inspect all pairs before a live run: +Every criterion has enough cases to exercise the seed's declared batch sizes +of 1 and 2. + +Rendering rejects duplicate PNG bytes unless both cases explicitly declare +that their label depends on different source evidence. Aggregation therefore +keys unique evidence to the SHA-256 of the rendered PNG plus its visible text +and truth excerpt, not merely the case ID or screenshot bytes. These +source-conditioned controls test whether a verifier actually uses supplied +evidence when identical pixels can be either valid or invalid. + +This is a seed corpus, not a graduation corpus. Its checked-in labels are +proposed from fixture intent, not represented as human review, and each family +currently has fewer than the default 10 positive and 10 negative unique- +evidence cases. The runner refuses live qualification until the corpus is +explicitly marked `ready`; the selector's default evidence floors provide a +second independent block. Replace synthetic fixtures with representative +product captures, add genuinely distinct cases, and inspect all pairs before a +live run: ```bash npm run ve:render-visual-model-corpus @@ -71,9 +82,15 @@ After a human reviews every rendered pair, update `corpus.json`: } ``` -The live runner refuses pending labels. Synthetic adapters may exercise the -contract, but their records are permanently marked `synthetic` and the selector -rejects them. +Human review alone does not make this seed graduation-ready. Only set +`graduation_status` to `ready` after the corpus also meets the configured +unique-evidence floors with representative captures. Graduation requires +`capture_provenance.status=representative-artifacture-captures`, a source- +artifact SHA-256 for every state, and at least 10 distinct rendered images and +10 distinct source artifacts per label in every measured family. The live +runner refuses pending labels or a non-ready corpus. Synthetic adapters may +exercise the contract, but their records are permanently marked `synthetic` +and the selector rejects them. Rendered PNGs and raw runs are ignored because they are reproducible or provider-local artifacts. `render-manifest.json` records the exact rendered @@ -85,6 +102,8 @@ For every `model × family × configured_batch_size` cell, record: - at least 3 independent replicates; - at least 10 unique fire and 10 unique clean cases; +- at least 10 distinct fire and 10 distinct clean rendered images and source + artifacts; - coverage of every criterion represented by the family corpus; - TP, FP, TN, and FN over valid non-abstaining observations; - correct image-and-region grounding for positive cases; @@ -93,11 +112,15 @@ For every `model × family × configured_batch_size` cell, record: - exact provider, model, image detail, prefix identity, cache tokens, token usage, and time to first token. -Repeated observations do not count as independent cases. Aggregation records -both observation totals and unique screenshot totals, and the selector gates -both. A cell is also blocked unless every request in the deterministic +Repeated observations do not count as independent evidence. Aggregation +records both observation totals and unique evidence-bundle totals, and the +selector gates both evidence bundles and distinct rendered images. The legacy +`unique_cases` fields count evidence bundles; explicit `unique_images` fields +preserve pixel diversity so changed source text cannot inflate visual +coverage. A cell is also blocked unless every request in the deterministic experiment matrix is present and bound to the exact experiment/corpus, -rendered-image, shared-prefix, criterion-suffix, and randomization hashes. +rendered-image, source evidence, shared-prefix, criterion-suffix, and +randomization hashes. Default graduation gates: @@ -211,12 +234,18 @@ rank, or disclose `no-eval-qualified-model`. ## Ladder discipline -Run batch sizes `1, 2, 4` for the smallest candidate first. A larger batch is -safe only if its own grounding, silence, schema, latency, and cost measurements -graduate. Cases are deterministically shuffled per replicate. A criterion tail -wraps to already observed cases only to keep the request at the exact configured -batch size; wrapped observations never increase unique evidence. Missing -criterion coverage still blocks graduation. +Run the corpus-declared batch sizes `1, 2` for the smallest candidate first. A +larger batch is safe only if the corpus supports it and its own grounding, +silence, schema, latency, and cost measurements graduate. Cases are +deterministically shuffled per replicate. A criterion tail wraps to already +observed cases only to keep the request at the exact configured batch size; +wrapped observations never increase unique evidence. Missing criterion +coverage still blocks graduation. + +This seed can compare batch 1 with batch 2 only. Before evaluating batch 4 or +larger, add enough non-duplicate criterion coverage and declare the expanded +batch size in both the corpus and experiment. Absence of a batch-4 result is not +evidence that batch 2 is globally maximal. After aggregation and selection, add the completed candidate to `ladder_state.completed_candidates` with a `qualified` or `disqualified` diff --git a/evals/visual-model-policy/aggregate.mjs b/evals/visual-model-policy/aggregate.mjs index a9c297b..e37b01a 100644 --- a/evals/visual-model-policy/aggregate.mjs +++ b/evals/visual-model-policy/aggregate.mjs @@ -4,6 +4,7 @@ import path from 'node:path'; import process from 'node:process'; import { pathToFileURL } from 'node:url'; import { + evidenceSha256, expandCorpus, loadCorpus, validateCorpus, @@ -125,6 +126,10 @@ export function aggregateRecords(records, { || observation.criterion_id !== evalCase.criterion_id || observation.human_label !== evalCase.human_label || observation.image_sha256 !== expectedHashByImage.get(evalCase.image.id) + || observation.evidence_sha256 !== evidenceSha256( + expectedHashByImage.get(evalCase.image.id), + evalCase, + ) ) { throw new Error( `observation ${observation.observation_id} does not match expected corpus evidence`, @@ -218,7 +223,10 @@ export function aggregateRecords(records, { ); const positives = classified.filter((entry) => entry.human_label === 'fire'); const negatives = classified.filter((entry) => entry.human_label === 'clean'); - const uniqueImages = new Map(); + const uniqueEvidence = new Map(); + const uniqueImages = new Set(); + const uniquePositiveImages = new Set(); + const uniqueNegativeImages = new Set(); const evidenceByCase = new Map(); for (const observation of classified) { const priorEvidence = evidenceByCase.get(observation.case_id); @@ -226,6 +234,7 @@ export function aggregateRecords(records, { priorEvidence && ( priorEvidence.image_sha256 !== observation.image_sha256 + || priorEvidence.evidence_sha256 !== observation.evidence_sha256 || priorEvidence.human_label !== observation.human_label ) ) { @@ -233,15 +242,21 @@ export function aggregateRecords(records, { } evidenceByCase.set(observation.case_id, { image_sha256: observation.image_sha256, + evidence_sha256: observation.evidence_sha256, human_label: observation.human_label, }); - if (!uniqueImages.has(observation.image_sha256)) { - uniqueImages.set(observation.image_sha256, observation.human_label); - } else if (uniqueImages.get(observation.image_sha256) !== observation.human_label) { + if (!uniqueEvidence.has(observation.evidence_sha256)) { + uniqueEvidence.set(observation.evidence_sha256, observation.human_label); + } else if (uniqueEvidence.get(observation.evidence_sha256) !== observation.human_label) { throw new Error( - `rendered image ${observation.image_sha256} has inconsistent human labels`, + `evidence bundle ${observation.evidence_sha256} has inconsistent human labels`, ); } + uniqueImages.add(observation.image_sha256); + (observation.human_label === 'fire' + ? uniquePositiveImages + : uniqueNegativeImages + ).add(observation.image_sha256); } const first = cellRecords[0]; const criterionMetrics = aggregateCriteria( @@ -279,9 +294,12 @@ export function aggregateRecords(records, { cases: classified.length, positives: positives.length, negatives: negatives.length, - unique_cases: uniqueImages.size, - unique_positives: [...uniqueImages.values()].filter((label) => label === 'fire').length, - unique_negatives: [...uniqueImages.values()].filter((label) => label === 'clean').length, + unique_cases: uniqueEvidence.size, + unique_positives: [...uniqueEvidence.values()].filter((label) => label === 'fire').length, + unique_negatives: [...uniqueEvidence.values()].filter((label) => label === 'clean').length, + unique_images: uniqueImages.size, + unique_image_positives: uniquePositiveImages.size, + unique_image_negatives: uniqueNegativeImages.size, tp: positives.filter((entry) => entry.predicted_label === 'fire').length, fp: negatives.filter((entry) => entry.predicted_label === 'fire').length, tn: negatives.filter((entry) => entry.predicted_label === 'clean').length, @@ -349,8 +367,10 @@ function aggregateCriteria(observations, requiredCriteria) { return { id: criterionId, cases: rows.length, - unique_positives: new Set(positives.map((entry) => entry.image_sha256)).size, - unique_negatives: new Set(negatives.map((entry) => entry.image_sha256)).size, + unique_positives: new Set(positives.map((entry) => entry.evidence_sha256)).size, + unique_negatives: new Set(negatives.map((entry) => entry.evidence_sha256)).size, + unique_image_positives: new Set(positives.map((entry) => entry.image_sha256)).size, + unique_image_negatives: new Set(negatives.map((entry) => entry.image_sha256)).size, tp: positives.filter((entry) => entry.predicted_label === 'fire').length, fp: negatives.filter((entry) => entry.predicted_label === 'fire').length, tn: negatives.filter((entry) => entry.predicted_label === 'clean').length, @@ -453,6 +473,7 @@ function validateRequestRecord(record) { 'case_id', 'image_id', 'image_sha256', + 'evidence_sha256', 'human_label', 'criterion_id', 'json_valid', @@ -466,6 +487,9 @@ function validateRequestRecord(record) { if (!/^[a-f0-9]{64}$/.test(observation.image_sha256)) { throw new Error(`observation ${observation.observation_id} has invalid image_sha256`); } + if (!/^[a-f0-9]{64}$/.test(observation.evidence_sha256)) { + throw new Error(`observation ${observation.observation_id} has invalid evidence_sha256`); + } const caseIndex = record.case_ids.indexOf(observation.case_id); if ( caseIndex < 0 diff --git a/evals/visual-model-policy/aggregate.test.mjs b/evals/visual-model-policy/aggregate.test.mjs index 74709c5..f79c82a 100644 --- a/evals/visual-model-policy/aggregate.test.mjs +++ b/evals/visual-model-policy/aggregate.test.mjs @@ -3,7 +3,7 @@ import crypto from 'node:crypto'; import { fileURLToPath } from 'node:url'; import test from 'node:test'; import { aggregateRecords } from './aggregate.mjs'; -import { expandCorpus, loadCorpus } from './corpus.mjs'; +import { evidenceSha256, expandCorpus, loadCorpus } from './corpus.mjs'; import { buildExperimentPlan, experimentContractId, @@ -20,7 +20,16 @@ function record({ grounded = true, latencyMs = 100, cost = 0.001, + imageSha256 = null, + visibleText = caseId, + truthExcerpt = caseId, }) { + const renderedSha256 = imageSha256 + || crypto.createHash('sha256').update(caseId).digest('hex'); + const evidenceIdentity = evidenceSha256(renderedSha256, { + visible_text: visibleText, + truth_excerpt: truthExcerpt, + }); return { schema_version: 1, record_type: 'visual-eval-request', @@ -57,7 +66,8 @@ function record({ observation_id: `${requestId}:${caseId}`, case_id: caseId, image_id: `image:${caseId}`, - image_sha256: crypto.createHash('sha256').update(caseId).digest('hex'), + image_sha256: renderedSha256, + evidence_sha256: evidenceIdentity, criterion_id: 'text-visibly-clipped', human_label: humanLabel, predicted_label: predictedLabel, @@ -88,6 +98,9 @@ test('aggregation emits selector counts, unique evidence, telemetry, and p95', ( assert.equal(result.unique_cases, 2); assert.equal(result.unique_positives, 1); assert.equal(result.unique_negatives, 1); + assert.equal(result.unique_images, 2); + assert.equal(result.unique_image_positives, 1); + assert.equal(result.unique_image_negatives, 1); assert.equal(result.tp, 2); assert.equal(result.tn, 2); assert.equal(result.grounded, 2); @@ -100,13 +113,17 @@ test('aggregation emits selector counts, unique evidence, telemetry, and p95', ( assert.equal(result.experiment_complete, false); }); -test('unique evidence is keyed by rendered pixels rather than case id', () => { +test('unique evidence is keyed by the image-and-source evidence bundle', () => { + const sharedImageSha256 = crypto.createHash('sha256').update('shared-pixels').digest('hex'); const first = record({ requestId: 'pixel-copy-a', replicate: 1, caseId: 'case-a', humanLabel: 'fire', predictedLabel: 'fire', + imageSha256: sharedImageSha256, + visibleText: 'Same visible evidence', + truthExcerpt: 'Same source evidence', }); const second = record({ requestId: 'pixel-copy-b', @@ -114,14 +131,59 @@ test('unique evidence is keyed by rendered pixels rather than case id', () => { caseId: 'case-b', humanLabel: 'fire', predictedLabel: 'fire', + imageSha256: sharedImageSha256, + visibleText: 'Same visible evidence', + truthExcerpt: 'Same source evidence', }); - second.observations[0].image_sha256 = first.observations[0].image_sha256; const measurements = aggregateRecords([first, second], { candidates: [{ id: 'tiny', rank: 1, class: 'small', provider: 'fixture' }], }); assert.equal(measurements.results[0].cases, 2); assert.equal(measurements.results[0].unique_cases, 1); assert.equal(measurements.results[0].unique_positives, 1); + assert.equal(measurements.results[0].unique_images, 1); +}); + +test('source-conditioned labels remain independent when rendered pixels match', () => { + const sharedImageSha256 = crypto.createHash('sha256').update('confidence-meter').digest('hex'); + const fire = record({ + requestId: 'source-conditioned-fire', + replicate: 1, + caseId: 'unsupported-confidence', + humanLabel: 'fire', + predictedLabel: 'fire', + imageSha256: sharedImageSha256, + visibleText: 'Confidence 87%', + truthExcerpt: 'No measurement supports the displayed confidence.', + }); + const clean = record({ + requestId: 'source-conditioned-clean', + replicate: 1, + caseId: 'measured-confidence', + humanLabel: 'clean', + predictedLabel: 'clean', + imageSha256: sharedImageSha256, + visibleText: 'Confidence 87%', + truthExcerpt: '87 of 100 checks passed in the attached run.', + }); + assert.equal( + clean.observations[0].image_sha256, + fire.observations[0].image_sha256, + ); + assert.notEqual( + clean.observations[0].evidence_sha256, + fire.observations[0].evidence_sha256, + ); + + const measurements = aggregateRecords([fire, clean], { + candidates: [{ id: 'tiny', rank: 1, class: 'small', provider: 'fixture' }], + }); + assert.equal(measurements.results[0].unique_cases, 2); + assert.equal(measurements.results[0].unique_positives, 1); + assert.equal(measurements.results[0].unique_negatives, 1); + assert.equal(measurements.results[0].unique_images, 1); + assert.equal(measurements.results[0].unique_image_positives, 1); + assert.equal(measurements.results[0].unique_image_negatives, 1); }); test('unadjudicated disagreements remain visible and cannot become policy evidence', () => { @@ -367,18 +429,22 @@ test('aggregation binds qualification to the complete expected request matrix', output_tokens: 20, actual_cost_usd: 0.001, }, - observations: cell.cases.map((entry) => ({ - observation_id: `${requestId}:${entry.case_id}`, - case_id: entry.case_id, - image_id: entry.image.id, - image_sha256: crypto.createHash('sha256').update(entry.case_id).digest('hex'), - criterion_id: entry.criterion_id, - human_label: entry.human_label, - predicted_label: entry.human_label, - grounded: entry.human_label === 'fire', - json_valid: true, - abstained: false, - })), + observations: cell.cases.map((entry) => { + const imageSha256 = crypto.createHash('sha256').update(entry.case_id).digest('hex'); + return { + observation_id: `${requestId}:${entry.case_id}`, + case_id: entry.case_id, + image_id: entry.image.id, + image_sha256: imageSha256, + evidence_sha256: evidenceSha256(imageSha256, entry), + criterion_id: entry.criterion_id, + human_label: entry.human_label, + predicted_label: entry.human_label, + grounded: entry.human_label === 'fire', + json_valid: true, + abstained: false, + }; + }), synthetic: false, }; }); diff --git a/evals/visual-model-policy/corpus.json b/evals/visual-model-policy/corpus.json index 3915f6f..46c1167 100644 --- a/evals/visual-model-policy/corpus.json +++ b/evals/visual-model-policy/corpus.json @@ -1,12 +1,12 @@ { "schema_version": 1, - "corpus_id": "artifacture-owned-visual-v1", + "corpus_id": "artifacture-owned-visual-v2-seed", "rendered_root": "corpus/rendered", "label_review": { "status": "pending-human-review", "reviewer": null, "reviewed_at": null, - "notes": "Proposed labels follow the authored pair intent. A human reviewer must inspect every rendered pair and sign off before live measurements." + "notes": "Curated seed labels only. The corpus intentionally remains below the default ten-positive/ten-negative graduation floor while real Artifacture captures and human adjudication are added." }, "families": [ { @@ -17,194 +17,240 @@ { "id": "mobile-heading-edge", "criterion_id": "text-visibly-clipped", - "region": { "id": "hero-heading", "label": "Mobile hero heading" }, + "region": { + "id": "hero-heading", + "label": "Mobile hero heading" + }, "fire": { "title": "Heading clipped at the viewport edge", "visible_text": "Operational evidence architecture", "truth_excerpt": "The full heading is authored as “Operational evidence architecture.”", "adjudication_notes": "The final word is cut mid-character at the right edge; no ellipsis or disclosure control is present.", - "render": { "template": "text", "variant": "clipped-right" } + "render": { + "template": "text", + "variant": "clipped-right" + } }, "clean": { "title": "Long heading wraps intentionally", "visible_text": "Operational evidence architecture", "truth_excerpt": "The heading may wrap to preserve every authored word.", "adjudication_notes": "The two-line wrap is intentional and every character remains visible; line wrapping is not clipping.", - "render": { "template": "text", "variant": "wrapped" } + "render": { + "template": "text", + "variant": "wrapped" + } + }, + "viewport": { + "width": 390, + "height": 844 } }, { "id": "dense-table-label", "criterion_id": "text-visibly-clipped", - "region": { "id": "audit-table", "label": "Dense audit table" }, + "region": { + "id": "audit-table", + "label": "Dense audit table" + }, "fire": { "title": "Audit label chopped inside a fixed cell", "visible_text": "Representative calibration queue", "truth_excerpt": "The complete queue name is required to distinguish it from the frontier queue.", "adjudication_notes": "The label is visibly chopped inside an overflow-hidden cell and cannot be recovered from adjacent content.", - "render": { "template": "table", "variant": "clipped-label" } + "render": { + "template": "table", + "variant": "clipped-label" + } }, "clean": { "title": "Dense expert table scrolls without truncating labels", "visible_text": "Representative calibration queue", "truth_excerpt": "Dense expert surfaces are allowed when their labels remain available.", "adjudication_notes": "The table is intentionally dense, but the label is complete and the horizontal overflow affordance is visible.", - "render": { "template": "table", "variant": "dense-scroll" } + "render": { + "template": "table", + "variant": "dense-scroll" + } } }, { "id": "title-diagram-kpis", "criterion_id": "slide-single-focal-point", - "region": { "id": "slide-canvas", "label": "Presentation slide canvas" }, + "region": { + "id": "slide-canvas", + "label": "Presentation slide canvas" + }, "fire": { "title": "Three primary treatments compete", "visible_text": "Policy router · 93% coverage · Review topology", "truth_excerpt": "The slide’s narrative job is to explain the routing topology.", "adjudication_notes": "A huge title, saturated diagram, and equally large KPI row all demand first attention, leaving no single focal point.", - "render": { "template": "slide", "variant": "competing-foci" } + "render": { + "template": "slide", + "variant": "competing-foci" + } }, "clean": { "title": "Dense evidence supports one dominant diagram", "visible_text": "Policy router · 93% coverage · Review topology", "truth_excerpt": "The router is the claim; supporting metrics may remain subordinate.", "adjudication_notes": "The surface contains many expert details, but scale and contrast clearly make the router diagram the single focal element.", - "render": { "template": "slide", "variant": "dense-one-focus" } - } - }, - { - "id": "comparison-dashboard", - "criterion_id": "slide-single-focal-point", - "region": { "id": "dashboard-main", "label": "Comparison dashboard" }, - "fire": { - "title": "Every dashboard tile is visually primary", - "visible_text": "Latency · Cost · Recall · Coverage", - "truth_excerpt": "The decision is whether recall remains above the graduation gate.", - "adjudication_notes": "Four equally loud metric tiles and two charts compete despite one decision-driving measure.", - "render": { "template": "dashboard", "variant": "uniform-loud" } - }, - "clean": { - "title": "Expert dashboard emphasizes the decision metric", - "visible_text": "Latency · Cost · Recall · Coverage", - "truth_excerpt": "Recall is the decision-driving measure; other metrics are supporting context.", - "adjudication_notes": "The dashboard remains information-dense but recall is clearly dominant through placement and type scale.", - "render": { "template": "dashboard", "variant": "hierarchical-dense" } + "render": { + "template": "slide", + "variant": "dense-one-focus" + } } }, { "id": "tiny-diagram-ocean", "criterion_id": "sparse-diagram-slide", - "region": { "id": "diagram-frame", "label": "Diagram and surrounding canvas" }, + "region": { + "id": "diagram-frame", + "label": "Diagram and surrounding canvas" + }, "fire": { "title": "Tiny diagram floats in unexplained space", "visible_text": "Input → Review → Output", "truth_excerpt": "The three-stage route is the full content of the slide.", "adjudication_notes": "The diagram occupies a small corner while most of the canvas is unexplained blank space, making the content feel missing.", - "render": { "template": "diagram-space", "variant": "tiny-unbalanced" } + "render": { + "template": "diagram-space", + "variant": "tiny-unbalanced" + } }, "clean": { "title": "Authored breathing room frames a short claim", "visible_text": "Input → Review → Output", "truth_excerpt": "The brief calls for a quiet transition slide with one compact route.", "adjudication_notes": "The blank area is deliberately balanced around a centered claim and does not resemble a collapsed or missing region.", - "render": { "template": "diagram-space", "variant": "intentional-breathing" } + "render": { + "template": "diagram-space", + "variant": "intentional-breathing" + } } }, { "id": "collapsed-peer-column", "criterion_id": "sparse-diagram-slide", - "region": { "id": "peer-map", "label": "Three-column peer map" }, + "region": { + "id": "peer-map", + "label": "Three-column peer map" + }, "fire": { "title": "Collapsed third column leaves accidental dead space", "visible_text": "Signals · Policy · Actions", "truth_excerpt": "All three peer stages have content in the source brief.", "adjudication_notes": "The third stage collapses to a tiny block and leaves a large blank region that reads as missing content.", - "render": { "template": "diagram-space", "variant": "collapsed-column" } + "render": { + "template": "diagram-space", + "variant": "collapsed-column" + } }, "clean": { "title": "Evidence rail intentionally leaves reading space", "visible_text": "Narrative · Evidence", "truth_excerpt": "The composition is an editorial 70/30 narrative and evidence split.", "adjudication_notes": "The narrow evidence rail and open lower area are authored hierarchy, not a missing peer column.", - "render": { "template": "diagram-space", "variant": "editorial-space" } + "render": { + "template": "diagram-space", + "variant": "editorial-space" + } } }, { "id": "three-peer-widths", "criterion_id": "layout-repeated-track-symmetry", - "region": { "id": "peer-tracks", "label": "Input, router, and output tracks" }, + "region": { + "id": "peer-tracks", + "label": "Input, router, and output tracks" + }, "fire": { "title": "Equivalent peer tracks drift in width", "visible_text": "Input · Router · Output", "truth_excerpt": "The three stages are declared as equivalent review tracks.", "adjudication_notes": "The middle track is inexplicably narrower and divider positions do not align across otherwise equivalent regions.", - "render": { "template": "tracks", "variant": "unequal-peers" } + "render": { + "template": "tracks", + "variant": "unequal-peers" + } }, "clean": { - "title": "Intentional 60/40 editorial split", - "visible_text": "Narrative · Evidence", - "truth_excerpt": "The narrative is primary and the evidence rail is explicitly supporting.", - "adjudication_notes": "The unequal widths express different content roles and therefore should not be treated as failed peer symmetry.", - "render": { "template": "tracks", "variant": "editorial-60-40" } + "title": "Equivalent peer tracks stay balanced", + "visible_text": "Input · Router · Output", + "truth_excerpt": "The three stages are declared as equivalent review tracks.", + "adjudication_notes": "All three peer tracks use equal widths and aligned dividers, providing the direct clean counterpart for the drift case.", + "render": { + "template": "tracks", + "variant": "equal-peers" + } } }, { "id": "row-baseline-drift", "criterion_id": "layout-repeated-track-symmetry", - "region": { "id": "comparison-rows", "label": "Before and after comparison rows" }, + "region": { + "id": "comparison-rows", + "label": "Before and after comparison rows" + }, "fire": { "title": "Peer row baselines do not align", "visible_text": "Before · After", "truth_excerpt": "Rows represent direct before/after counterparts.", "adjudication_notes": "Corresponding row dividers drift vertically, undermining the declared one-to-one comparison.", - "render": { "template": "tracks", "variant": "baseline-drift" } + "render": { + "template": "tracks", + "variant": "baseline-drift" + } }, "clean": { - "title": "Content-driven masonry is not a peer table", - "visible_text": "Field notes · Evidence fragments", - "truth_excerpt": "Items are independent editorial fragments with deliberately varied lengths.", - "adjudication_notes": "The staggered baselines belong to an editorial masonry composition, not equivalent repeated rows.", - "render": { "template": "tracks", "variant": "masonry" } - } - }, - { - "id": "missing-track-weight", - "criterion_id": "layout-repeated-track-symmetry", - "region": { "id": "quadrant-grid", "label": "Four peer quadrants" }, - "fire": { - "title": "One peer quadrant appears absent", - "visible_text": "Detect · Route · Review · Learn", - "truth_excerpt": "The operating loop defines four peer responsibilities.", - "adjudication_notes": "One quadrant receives a fraction of the area and large accidental whitespace makes it appear unimplemented.", - "render": { "template": "tracks", "variant": "missing-weight" } - }, - "clean": { - "title": "One dominant result spans two grid columns", - "visible_text": "Inputs · Method · Result", - "truth_excerpt": "The result is explicitly the conclusion and should dominate supporting inputs.", - "adjudication_notes": "The spanning result is intentional hierarchy rather than an equivalent peer rendered at the wrong size.", - "render": { "template": "tracks", "variant": "hero-span" } + "title": "Peer row baselines align", + "visible_text": "Before · After", + "truth_excerpt": "Rows represent direct before/after counterparts.", + "adjudication_notes": "Corresponding dividers and content baselines align across all three peer tracks.", + "render": { + "template": "tracks", + "variant": "equal-peers" + } } }, { "id": "mobile-control-crowding", "criterion_id": "slide-single-focal-point", - "region": { "id": "mobile-header", "label": "Mobile header and primary content" }, + "region": { + "id": "mobile-header", + "label": "Mobile header and primary content" + }, "fire": { "title": "Controls crowd the primary claim", "visible_text": "Model policy · Theme · Filter · Export", "truth_excerpt": "The page title and selected policy are the primary content.", "adjudication_notes": "Large controls surround and visually overpower the title, producing several competing focal targets.", - "render": { "template": "mobile", "variant": "crowded-controls" } + "render": { + "template": "mobile", + "variant": "crowded-controls" + } }, "clean": { "title": "Compact expert controls remain subordinate", "visible_text": "Model policy · Theme · Filter · Export", "truth_excerpt": "Frequent operators need all controls available without losing the page claim.", "adjudication_notes": "The surface is dense by design, but compact chrome remains subordinate to one clear title and active policy.", - "render": { "template": "mobile", "variant": "compact-controls" } + "render": { + "template": "mobile", + "variant": "compact-controls" + } + }, + "viewport": { + "width": 390, + "height": 844 } } - ] + ], + "viewport": { + "width": 1280, + "height": 720 + } }, { "id": "diagram", @@ -214,213 +260,204 @@ { "id": "mixed-grammar", "criterion_id": "diagram-type-coherent", - "region": { "id": "figure", "label": "Architecture figure" }, + "region": { + "id": "figure", + "label": "Architecture figure" + }, "fire": { "title": "Swimlanes and containment conflict", "visible_text": "Client · Router · Cache · Store", "truth_excerpt": "The content describes one request flow.", "adjudication_notes": "Half the figure uses lanes while the other half uses nested containment for the same relation.", - "render": { "template": "diagram", "variant": "mixed-grammar" } + "render": { + "template": "diagram", + "variant": "mixed-grammar" + } }, "clean": { "title": "One flow grammar with a highlighted boundary", "visible_text": "Client · Router · Cache · Store", "truth_excerpt": "The boundary denotes deployment ownership inside one request flow.", "adjudication_notes": "A boundary annotation does not create a second diagram type; connectors and nodes retain one stable grammar.", - "render": { "template": "diagram", "variant": "coherent-flow" } + "render": { + "template": "diagram", + "variant": "coherent-flow" + } } }, { "id": "timeline-scale", "criterion_id": "diagram-proportional-honesty-visual", - "region": { "id": "timeline", "label": "Delivery timeline" }, + "region": { + "id": "timeline", + "label": "Delivery timeline" + }, "fire": { "title": "One-day and six-month gaps look equal", "visible_text": "Day 1 · Day 2 · Month 6", "truth_excerpt": "The milestones occur after one day and then roughly six months.", "adjudication_notes": "Equal visual spacing materially misrepresents the stated time intervals.", - "render": { "template": "diagram", "variant": "dishonest-timeline" } + "render": { + "template": "diagram", + "variant": "dishonest-timeline" + } }, "clean": { "title": "Compressed timeline explicitly marks a break", "visible_text": "Day 1 · Day 2 · // · Month 6", "truth_excerpt": "The timeline is compressed and must disclose the discontinuity.", "adjudication_notes": "A visible axis break and label make the non-linear spacing explicit rather than deceptively proportional.", - "render": { "template": "diagram", "variant": "timeline-break" } + "render": { + "template": "diagram", + "variant": "timeline-break" + } } }, { "id": "quantity-bars", "criterion_id": "diagram-proportional-honesty-visual", - "region": { "id": "bar-figure", "label": "Cost comparison bars" }, + "region": { + "id": "bar-figure", + "label": "Cost comparison bars" + }, "fire": { "title": "Near-equal bars represent a fourfold difference", "visible_text": "$10 · $40", "truth_excerpt": "The second value is four times the first.", "adjudication_notes": "The bars differ only slightly in length, visually erasing a load-bearing fourfold ratio.", - "render": { "template": "diagram", "variant": "dishonest-bars" } + "render": { + "template": "diagram", + "variant": "dishonest-bars" + } }, "clean": { - "title": "Symbolic comparison avoids a false quantitative scale", + "title": "Scaled bars preserve the fourfold relation", "visible_text": "$10 baseline · $40 candidate", - "truth_excerpt": "The composition is a labeled comparison and does not claim scaled bar length.", - "adjudication_notes": "Equal containers are acceptable because explicit values carry the comparison and no quantitative axis is implied.", - "render": { "template": "diagram", "variant": "labeled-comparison" } + "truth_excerpt": "The second value is four times the first.", + "adjudication_notes": "The bar lengths preserve the load-bearing ratio and agree with their explicit value labels.", + "render": { + "template": "diagram", + "variant": "proportional-bars" + } } }, { "id": "legend-orphan", "criterion_id": "diagram-legend-matches-figure", - "region": { "id": "legend-and-plot", "label": "Legend and plotted categories" }, + "region": { + "id": "legend-and-plot", + "label": "Legend and plotted categories" + }, "fire": { "title": "Legend names an absent cache category", "visible_text": "API · Cache · Store", "truth_excerpt": "Only API and Store categories are drawn.", "adjudication_notes": "The legend includes Cache even though no corresponding mark exists in the figure.", - "render": { "template": "diagram", "variant": "orphan-legend" } + "render": { + "template": "diagram", + "variant": "orphan-legend" + } }, "clean": { "title": "Muted category still appears in the figure", "visible_text": "API · Cache · Store", "truth_excerpt": "Cache is present but intentionally de-emphasized because it is inactive.", "adjudication_notes": "Every legend category has a plotted counterpart; low emphasis is not absence.", - "render": { "template": "diagram", "variant": "complete-legend" } + "render": { + "template": "diagram", + "variant": "complete-legend" + } } }, { "id": "missing-legend-category", "criterion_id": "diagram-legend-matches-figure", - "region": { "id": "legend-and-nodes", "label": "Node categories and legend" }, + "region": { + "id": "legend-and-nodes", + "label": "Node categories and legend" + }, "fire": { "title": "Decision nodes lack a legend entry", "visible_text": "Service · Data", "truth_excerpt": "Diamonds encode decisions and rectangles encode services.", "adjudication_notes": "A distinct load-bearing diamond category is used repeatedly but the legend never defines it.", - "render": { "template": "diagram", "variant": "missing-legend-entry" } + "render": { + "template": "diagram", + "variant": "missing-legend-entry" + } }, "clean": { "title": "Annotation callout does not need a legend category", "visible_text": "Service · Data", "truth_excerpt": "The lone dashed note is explanatory annotation, not a data category.", "adjudication_notes": "A conventional one-off annotation is self-labelled and does not create an unexplained visual category.", - "render": { "template": "diagram", "variant": "annotation-no-legend" } + "render": { + "template": "diagram", + "variant": "annotation-no-legend" + } } }, { "id": "three-box-list", "criterion_id": "diagram-necessity", - "region": { "id": "figure", "label": "Three-item figure" }, + "region": { + "id": "figure", + "label": "Three-item figure" + }, "fire": { "title": "Three independent bullets are boxed and arrowed", "visible_text": "Reliable · Fast · Affordable", "truth_excerpt": "The brief lists three independent qualities with no sequence or relationship.", "adjudication_notes": "Boxes and arrows add a false flow while conveying no information beyond a short list.", - "render": { "template": "diagram", "variant": "decorated-list" } + "render": { + "template": "diagram", + "variant": "decorated-list" + } }, "clean": { "title": "Three nodes expose a real dependency", "visible_text": "Capture → Adjudicate → Graduate", "truth_excerpt": "Graduation requires adjudication, which requires captured observations.", "adjudication_notes": "The connectors encode necessary dependency and therefore add information beyond the labels.", - "render": { "template": "diagram", "variant": "necessary-sequence" } - } - }, - { - "id": "peer-table-as-graph", - "criterion_id": "diagram-necessity", - "region": { "id": "comparison-figure", "label": "Model comparison figure" }, - "fire": { - "title": "Model comparison is an arbitrary node graph", - "visible_text": "Nano · Mini · Frontier", - "truth_excerpt": "The source contains aligned latency, accuracy, and cost values.", - "adjudication_notes": "A graph obscures exact aligned comparisons that a compact table would communicate more faithfully.", - "render": { "template": "diagram", "variant": "table-as-graph" } - }, - "clean": { - "title": "Network topology needs relational placement", - "visible_text": "Gateway · Workers · Stores", - "truth_excerpt": "The claim depends on fan-out, shared stores, and retry paths.", - "adjudication_notes": "Topology and connector direction are load-bearing; a flat table would discard the relevant relationships.", - "render": { "template": "diagram", "variant": "necessary-topology" } + "render": { + "template": "diagram", + "variant": "necessary-sequence" + } } }, { "id": "dual-focal", "criterion_id": "diagram-focal-single-dominant", - "region": { "id": "system-map", "label": "System map" }, + "region": { + "id": "system-map", + "label": "System map" + }, "fire": { "title": "Two unrelated nodes claim equal dominance", "visible_text": "Policy Router · Analytics Hub", "truth_excerpt": "The narrative job is to explain policy routing.", "adjudication_notes": "Two saturated oversized nodes compete even though only the router is load-bearing to the claim.", - "render": { "template": "diagram", "variant": "dual-focal" } + "render": { + "template": "diagram", + "variant": "dual-focal" + } }, "clean": { "title": "Two endpoints frame one dominant transformation", "visible_text": "Evidence · Policy Router · Decision", "truth_excerpt": "The policy router is the transformation under explanation.", "adjudication_notes": "Endpoints remain visible but the central router is unmistakably dominant.", - "render": { "template": "diagram", "variant": "single-focal" } - } - }, - { - "id": "uniform-box-forest", - "criterion_id": "diagram-type-coherent", - "region": { "id": "org-map", "label": "Organization map" }, - "fire": { - "title": "Hierarchy and sequence use identical connectors", - "visible_text": "Team · Stage · Owner", - "truth_excerpt": "The source distinguishes reporting hierarchy from workflow sequence.", - "adjudication_notes": "Identical lines and boxes collapse two different relation types into an incoherent grammar.", - "render": { "template": "diagram", "variant": "ambiguous-relations" } - }, - "clean": { - "title": "Uniform node forest is the requested inventory", - "visible_text": "Service A · Service B · Service C", - "truth_excerpt": "The figure is an inventory grouped only by deployment zone.", - "adjudication_notes": "Uniform boxes are truthful because the nodes are genuine peers and containment is the sole relation.", - "render": { "template": "diagram", "variant": "peer-inventory" } - } - }, - { - "id": "residual-probability", - "criterion_id": "diagram-proportional-honesty-visual", - "region": { "id": "probability-map", "label": "Probability distribution" }, - "fire": { - "title": "Named outcomes consume more than the whole", - "visible_text": "A 55% · B 35% · Other 20%", - "truth_excerpt": "Displayed probabilities must sum to the complete distribution.", - "adjudication_notes": "The visual encodes 110 percent while presenting a complete distribution, a material proportional contradiction.", - "render": { "template": "diagram", "variant": "invalid-probability" } - }, - "clean": { - "title": "Residual probability remains visible", - "visible_text": "A 55% · B 25% · Other 20%", - "truth_excerpt": "Residual probability is material and must remain represented.", - "adjudication_notes": "All proportions sum to one hundred percent and the residual category is explicitly preserved.", - "render": { "template": "diagram", "variant": "valid-probability" } - } - }, - { - "id": "competing-control-planes", - "criterion_id": "diagram-focal-single-dominant", - "region": { "id": "control-plane-map", "label": "Control-plane architecture map" }, - "fire": { - "title": "Two control planes receive equal visual dominance", - "visible_text": "Admission Gate · Policy Engine · Audit Store", - "truth_excerpt": "The policy engine is the decision-making center; admission and audit are supporting boundaries.", - "adjudication_notes": "The admission gate and policy engine are both oversized and saturated, so the diagram gives two competing answers to what controls the system.", - "render": { "template": "diagram", "variant": "dual-focal" } - }, - "clean": { - "title": "One policy engine dominates supporting boundaries", - "visible_text": "Admission Gate · Policy Engine · Audit Store", - "truth_excerpt": "The policy engine is the decision-making center; admission and audit are supporting boundaries.", - "adjudication_notes": "The central policy engine receives the sole dominant treatment while the admission gate and audit store remain legible supporting endpoints.", - "render": { "template": "diagram", "variant": "single-focal" } + "render": { + "template": "diagram", + "variant": "single-focal" + } } } - ] + ], + "viewport": { + "width": 1024, + "height": 768 + } }, { "id": "aesthetic", @@ -430,289 +467,232 @@ { "id": "light-mode-panel", "criterion_id": "preset-both-mode-visual", - "region": { "id": "mode-pair", "label": "Light and dark mode pair" }, + "region": { + "id": "mode-pair", + "label": "Light and dark mode pair" + }, "fire": { "title": "Hard-coded dark panel leaks into light mode", "visible_text": "Status overview", "truth_excerpt": "Both screenshots should express the same preset in their respective mode.", "adjudication_notes": "One panel remains dark only in the light screenshot and its text fails to invert with the rest of the preset.", - "render": { "template": "preset", "variant": "wrong-mode-panel" } + "render": { + "template": "preset", + "variant": "wrong-mode-panel" + } }, "clean": { "title": "Dark data surface is intentional in both modes", "visible_text": "Terminal trace", "truth_excerpt": "The embedded terminal is explicitly a dark simulated surface in either page mode.", "adjudication_notes": "The stable dark terminal is content semantics, not a failed page-mode inversion, and its own text remains readable.", - "render": { "template": "preset", "variant": "intentional-terminal" } + "render": { + "template": "preset", + "variant": "intentional-terminal" + } } }, { "id": "mode-icon-fill", "criterion_id": "preset-both-mode-visual", - "region": { "id": "mode-pair", "label": "Mode comparison with icons" }, + "region": { + "id": "mode-pair", + "label": "Mode comparison with icons" + }, "fire": { "title": "Icons disappear against the dark surface", "visible_text": "Inputs · Results", "truth_excerpt": "Icon and label meaning must remain visible in both modes.", "adjudication_notes": "The dark screenshot retains a dark icon fill, making load-bearing symbols effectively invisible.", - "render": { "template": "preset", "variant": "invisible-dark-icons" } + "render": { + "template": "preset", + "variant": "invisible-dark-icons" + } }, "clean": { "title": "Decorative watermark fades in dark mode", "visible_text": "Inputs · Results", "truth_excerpt": "The watermark is non-semantic decoration and may be quiet.", "adjudication_notes": "All load-bearing icons and text remain visible; only a deliberately non-semantic watermark loses prominence.", - "render": { "template": "preset", "variant": "quiet-watermark" } + "render": { + "template": "preset", + "variant": "quiet-watermark" + } } }, { "id": "mono-status-container", "criterion_id": "mono-status-value-judgment", - "region": { "id": "status-board", "label": "Mono-Industrial status board" }, + "region": { + "id": "status-board", + "label": "Mono-Industrial status board" + }, "fire": { "title": "Orange tints an entire status section", "visible_text": "Review queue", "truth_excerpt": "Mono-Industrial reserves color for specific values with status meaning.", "adjudication_notes": "Orange colors a container and heading without representing an actual warning value.", - "render": { "template": "preset", "variant": "decorative-status-color" } + "render": { + "template": "preset", + "variant": "decorative-status-color" + } }, "clean": { "title": "One warning value carries orange", "visible_text": "Review queue · 3 delayed", "truth_excerpt": "Three delayed items are a measured warning state.", "adjudication_notes": "Color is scoped to the precise delayed count and therefore conveys genuine value semantics.", - "render": { "template": "preset", "variant": "semantic-status-color" } + "render": { + "template": "preset", + "variant": "semantic-status-color" + } } }, { "id": "mono-surprise", "criterion_id": "mono-one-surprise", - "region": { "id": "mono-page", "label": "Mono-Industrial composition" }, + "region": { + "id": "mono-page", + "label": "Mono-Industrial composition" + }, "fire": { "title": "Several unrelated elements break the grid", "visible_text": "Policy · Evidence · Result", "truth_excerpt": "The preset permits one deliberate compositional or typographic surprise.", "adjudication_notes": "Three rotated and oversized elements break the system, so no single authored surprise remains.", - "render": { "template": "preset", "variant": "many-surprises" } + "render": { + "template": "preset", + "variant": "many-surprises" + } }, "clean": { "title": "One oversized decision breaks the pattern", "visible_text": "Policy · DECISION · Result", "truth_excerpt": "The decision is the one intended moment of surprise.", "adjudication_notes": "Exactly one typographic element breaks the otherwise consistent system and the break supports hierarchy.", - "render": { "template": "preset", "variant": "one-surprise" } + "render": { + "template": "preset", + "variant": "one-surprise" + } } }, { "id": "mono-three-layers", "criterion_id": "mono-three-layer-squint", - "region": { "id": "mono-page", "label": "Mono-Industrial hierarchy" }, + "region": { + "id": "mono-page", + "label": "Mono-Industrial hierarchy" + }, "fire": { "title": "Five equally loud emphasis layers", "visible_text": "Title · KPI · Section · Label · Note", "truth_excerpt": "The active preset calls for three legible emphasis layers.", "adjudication_notes": "Multiple near-equal weights and sizes create at least five competing emphasis levels with no stable hierarchy.", - "render": { "template": "preset", "variant": "too-many-layers" } + "render": { + "template": "preset", + "variant": "too-many-layers" + } }, "clean": { "title": "Dense page resolves into three emphasis layers", "visible_text": "Title · KPI · Section · Label · Note", "truth_excerpt": "Dense content is compatible with three clearly grouped emphasis levels.", "adjudication_notes": "Many elements are present, but they resolve into a dominant hero, section level, and quiet instrument text.", - "render": { "template": "preset", "variant": "three-layers" } + "render": { + "template": "preset", + "variant": "three-layers" + } } }, { "id": "nothing-red-underline", "criterion_id": "nothing-accent-red-judgment", - "region": { "id": "nothing-page", "label": "Nothing preset page" }, + "region": { + "id": "nothing-page", + "label": "Nothing preset page" + }, "fire": { "title": "Red accent is a decorative underline", "visible_text": "Signal architecture", "truth_excerpt": "Nothing red is reserved for urgent, destructive, or error meaning.", "adjudication_notes": "The single red use merely decorates a neutral heading and communicates no status.", - "render": { "template": "preset", "variant": "decorative-red" } + "render": { + "template": "preset", + "variant": "decorative-red" + } }, "clean": { "title": "Red marks a destructive reset action", "visible_text": "Reset policy", "truth_excerpt": "Resetting the policy is destructive and requires explicit warning.", "adjudication_notes": "The red accent is scoped to a genuine destructive action rather than used as ornament.", - "render": { "template": "preset", "variant": "destructive-red" } + "render": { + "template": "preset", + "variant": "destructive-red" + } } }, { "id": "nothing-grid-break", "criterion_id": "nothing-single-grid-break", - "region": { "id": "nothing-grid", "label": "Nothing preset grid" }, + "region": { + "id": "nothing-grid", + "label": "Nothing preset grid" + }, "fire": { "title": "Three cards independently escape the grid", "visible_text": "Capture · Route · Learn", "truth_excerpt": "The preset allows one meaningful grid break.", "adjudication_notes": "Several arbitrary offsets break the composition without establishing one deliberate exception.", - "render": { "template": "preset", "variant": "many-grid-breaks" } + "render": { + "template": "preset", + "variant": "many-grid-breaks" + } }, "clean": { - "title": "Full-width navigation is ordinary chrome", + "title": "One review card deliberately spans tracks", "visible_text": "Capture · Route · Learn", - "truth_excerpt": "Full-width navigation is excluded from the single content-grid-break count.", - "adjudication_notes": "One content element breaks the grid; the full-width navigation is legitimate chrome and not a second break.", - "render": { "template": "preset", "variant": "one-grid-break" } + "truth_excerpt": "The review card is the composition's one intended grid break.", + "adjudication_notes": "One content element deliberately spans tracks while every other card remains on the established grid.", + "render": { + "template": "preset", + "variant": "one-grid-break" + } } }, { "id": "demo-frame", "criterion_id": "demo-aesthetic-match", - "region": { "id": "demo-embed", "label": "Embedded demo and host page" }, + "region": { + "id": "demo-embed", + "label": "Embedded demo and host page" + }, "fire": { "title": "Glossy rounded demo conflicts with flat host", "visible_text": "Live policy inspector", "truth_excerpt": "The host preset uses flat square surfaces and restrained borders.", "adjudication_notes": "Large radius, glow, and heavy shadow make the demo feel pasted onto a visibly different design system.", - "render": { "template": "preset", "variant": "mismatched-demo" } + "render": { + "template": "preset", + "variant": "mismatched-demo" + } }, "clean": { "title": "Functional browser chrome remains distinct", "visible_text": "Live policy inspector", "truth_excerpt": "The demo needs a minimal frame to communicate that it is interactive.", "adjudication_notes": "The frame is functionally necessary and still matches the host’s radius, border, color, and caption treatment.", - "render": { "template": "preset", "variant": "matched-demo" } - } - }, - { - "id": "mono-mobile-hero", - "criterion_id": "mono-three-layer-squint", - "region": { "id": "mobile-hero", "label": "Mono-Industrial mobile hero" }, - "fire": { - "title": "Hero collapses into a tiny card on mobile", - "visible_text": "Route with evidence", - "truth_excerpt": "The hero should remain intact and full-width on mobile.", - "adjudication_notes": "The primary claim is shrunk into one small bordered card among peers, destroying the intended hierarchy.", - "render": { "template": "preset", "variant": "mobile-hero-card" } - }, - "clean": { - "title": "Full-width mobile hero with compact supporting cards", - "visible_text": "Route with evidence", - "truth_excerpt": "Supporting items may compact as long as the hero stays intact.", - "adjudication_notes": "The hero remains full-width and dominant while subordinate evidence becomes a denser mobile row.", - "render": { "template": "preset", "variant": "mobile-hero-full" } - } - }, - { - "id": "mode-background", - "criterion_id": "preset-both-mode-visual", - "region": { "id": "mode-pair", "label": "Full-page light and dark screenshots" }, - "fire": { - "title": "Dark mode keeps a light page background", - "visible_text": "Evidence map", - "truth_excerpt": "The named preset supports both light and dark surfaces.", - "adjudication_notes": "The dark-mode screenshot preserves the light canvas while inverting text, causing broken contrast and wrong-mode bleed.", - "render": { "template": "preset", "variant": "wrong-mode-background" } - }, - "clean": { - "title": "Paper inset remains light in both modes", - "visible_text": "Export preview", - "truth_excerpt": "A print-preview inset intentionally represents white paper in either application mode.", - "adjudication_notes": "The light inset is an explicitly labelled paper preview with readable local colors, not a failed global inversion.", - "render": { "template": "preset", "variant": "paper-preview" } - } - }, - { - "id": "mono-neutral-count", - "criterion_id": "mono-status-value-judgment", - "region": { "id": "queue-summary", "label": "Mono-Industrial queue summary" }, - "fire": { - "title": "Neutral queue total receives warning color", - "visible_text": "Calibration total · 12 cases", - "truth_excerpt": "A total case count is neutral unless a measured threshold has been exceeded.", - "adjudication_notes": "The colored status treatment decorates a neutral inventory count and implies a warning that the underlying value does not support.", - "render": { "template": "preset", "variant": "decorative-status-color" } - }, - "clean": { - "title": "Threshold breach receives scoped warning color", - "visible_text": "Calibration total · 12 overdue", - "truth_excerpt": "Twelve overdue cases exceed the documented queue threshold.", - "adjudication_notes": "The warning color is scoped to a measured threshold breach and therefore carries explicit status meaning rather than decoration.", - "render": { "template": "preset", "variant": "semantic-status-color" } - } - }, - { - "id": "mono-offset-summary", - "criterion_id": "mono-one-surprise", - "region": { "id": "summary-grid", "label": "Mono-Industrial summary grid" }, - "fire": { - "title": "Multiple summary metrics independently break the grid", - "visible_text": "Escalation summary · Cost · Recall", - "truth_excerpt": "Only the escalation decision is intended to break the otherwise regular summary grid.", - "adjudication_notes": "Several rotated and enlarged metrics compete as independent exceptions, leaving no single meaningful compositional surprise.", - "render": { "template": "preset", "variant": "many-surprises" } - }, - "clean": { - "title": "One escalation decision breaks the summary grid", - "visible_text": "Escalation summary · Cost · Recall", - "truth_excerpt": "Only the escalation decision is intended to break the otherwise regular summary grid.", - "adjudication_notes": "A single decision treatment departs from the regular grid while the supporting cost and recall measures retain the base system.", - "render": { "template": "preset", "variant": "one-surprise" } - } - }, - { - "id": "nothing-red-badge", - "criterion_id": "nothing-accent-red-judgment", - "region": { "id": "policy-action", "label": "Nothing preset policy action" }, - "fire": { - "title": "Red badge decorates a neutral policy version", - "visible_text": "Policy version 24", - "truth_excerpt": "The policy version is informational and has no urgent, destructive, or error state.", - "adjudication_notes": "Red is applied to a neutral version badge only for visual interest, creating false urgency without corresponding semantics.", - "render": { "template": "preset", "variant": "decorative-red" } - }, - "clean": { - "title": "Red identifies irreversible policy deletion", - "visible_text": "Delete policy version", - "truth_excerpt": "Deleting the policy version is irreversible and requires destructive emphasis.", - "adjudication_notes": "Red is limited to an irreversible deletion action, so the accent communicates the precise destructive meaning reserved by the preset.", - "render": { "template": "preset", "variant": "destructive-red" } - } - }, - { - "id": "nothing-evidence-offsets", - "criterion_id": "nothing-single-grid-break", - "region": { "id": "evidence-grid", "label": "Nothing preset evidence grid" }, - "fire": { - "title": "Several evidence cards use unrelated offsets", - "visible_text": "Evidence ledger · Sources · Decisions", - "truth_excerpt": "The decision ledger is the sole content element intended to cross the grid.", - "adjudication_notes": "Multiple cards rotate and offset independently, turning the permitted single grid break into a collection of arbitrary exceptions.", - "render": { "template": "preset", "variant": "many-grid-breaks" } - }, - "clean": { - "title": "One decision ledger crosses the evidence grid", - "visible_text": "Evidence ledger · Sources · Decisions", - "truth_excerpt": "The decision ledger is the sole content element intended to cross the grid.", - "adjudication_notes": "Only the decision ledger departs from the content grid; the surrounding source cards and ordinary chrome remain aligned.", - "render": { "template": "preset", "variant": "one-grid-break" } - } - }, - { - "id": "demo-inspector-theme", - "criterion_id": "demo-aesthetic-match", - "region": { "id": "inspector-embed", "label": "Embedded inspector and host preset" }, - "fire": { - "title": "Soft gradient inspector conflicts with industrial host", - "visible_text": "Evidence inspector", - "truth_excerpt": "The host uses square panels, flat fills, and instrument-like typography.", - "adjudication_notes": "The embedded inspector introduces pill controls, gradients, glow, and soft shadows that visibly belong to a different design language.", - "render": { "template": "preset", "variant": "mismatched-demo" } - }, - "clean": { - "title": "Inspector inherits the host surface language", - "visible_text": "Evidence inspector", - "truth_excerpt": "The inspector may retain functional controls while matching the host surface language.", - "adjudication_notes": "The embedded inspector retains necessary controls but matches the host’s square geometry, flat color, border weight, and typography.", - "render": { "template": "preset", "variant": "matched-demo" } + "render": { + "template": "preset", + "variant": "matched-demo" + } } } - ] + ], + "viewport": { + "width": 1180, + "height": 720 + } }, { "id": "operating-model", @@ -722,194 +702,178 @@ { "id": "review-routing", "criterion_id": "operating-model-fit", - "region": { "id": "review-routing", "label": "Review routing unit" }, + "region": { + "id": "review-routing", + "label": "Review routing unit" + }, "fire": { "title": "Four cards hide the selection policy", "visible_text": "Mandatory · Representative · Calibration · Frontier", "truth_excerpt": "A policy router assigns cases to distinct review queues with different entry rules.", "adjudication_notes": "Equal cards name destinations but omit the router, mandatory path, sampling rule, and deduplication.", - "render": { "template": "operating", "variant": "routing-cards" } + "render": { + "template": "operating", + "variant": "routing-cards" + } }, "clean": { "title": "Router exposes queue selection and overlap handling", "visible_text": "Mandatory · Representative · Calibration · Frontier", "truth_excerpt": "A policy router assigns cases to distinct review queues with different entry rules.", "adjudication_notes": "The composition shows the universal gate, routing rules, separate queues, and deduplication path.", - "render": { "template": "operating", "variant": "routing-model" } + "render": { + "template": "operating", + "variant": "routing-model" + } } }, { "id": "causal-loop", "criterion_id": "operating-model-fit", - "region": { "id": "causal-loop", "label": "Calibration feedback loop" }, + "region": { + "id": "causal-loop", + "label": "Calibration feedback loop" + }, "fire": { "title": "Feedback loop is drawn as a one-way pipeline", "visible_text": "Observe → Adjudicate → Calibrate → Deploy", "truth_excerpt": "Deployment outcomes feed the next calibration cycle.", "adjudication_notes": "The one-way pipeline omits the load-bearing return path and misstates the process as terminal.", - "render": { "template": "operating", "variant": "one-way-loop" } + "render": { + "template": "operating", + "variant": "one-way-loop" + } }, "clean": { "title": "Feedback path returns outcomes to calibration", "visible_text": "Observe → Adjudicate → Calibrate → Deploy ↩", "truth_excerpt": "Deployment outcomes feed the next calibration cycle.", "adjudication_notes": "The return connector and its label make the iterative causal structure reconstructable.", - "render": { "template": "operating", "variant": "closed-loop" } - } - }, - { - "id": "probability-residual", - "criterion_id": "operating-model-fit", - "region": { "id": "diagnostic-state", "label": "Probability-bearing diagnostic state" }, - "fire": { - "title": "Named diagnoses omit residual uncertainty", - "visible_text": "A 60% · B 25%", - "truth_excerpt": "Fifteen percent residual probability remains material to the decision.", - "adjudication_notes": "The state looks complete while hiding residual uncertainty that changes downstream review behavior.", - "render": { "template": "operating", "variant": "missing-residual" } - }, - "clean": { - "title": "Residual uncertainty remains an explicit branch", - "visible_text": "A 60% · B 25% · Other 15%", - "truth_excerpt": "Fifteen percent residual probability remains material to the decision.", - "adjudication_notes": "The full distribution and uncertain branch remain visible and connected to escalation.", - "render": { "template": "operating", "variant": "visible-residual" } + "render": { + "template": "operating", + "variant": "closed-loop" + } } }, { "id": "provenance-trace", "criterion_id": "operating-model-fit", - "region": { "id": "evidence-trace", "label": "Evidence provenance trace" }, + "region": { + "id": "evidence-trace", + "label": "Evidence provenance trace" + }, "fire": { "title": "Evidence cards omit source-to-decision paths", "visible_text": "Study · Finding · Decision", "truth_excerpt": "The viewer must reconstruct which source supports each decision claim.", "adjudication_notes": "Grouped cards contain the right nouns but no connectors or identifiers preserve provenance.", - "render": { "template": "operating", "variant": "provenance-cards" } + "render": { + "template": "operating", + "variant": "provenance-cards" + } }, "clean": { "title": "Trace IDs connect sources through findings to decisions", "visible_text": "S1 → F1 → D1", "truth_excerpt": "The viewer must reconstruct which source supports each decision claim.", "adjudication_notes": "Stable identifiers and connectors preserve the evidence path without requiring presenter narration.", - "render": { "template": "operating", "variant": "provenance-trace" } - } - }, - { - "id": "resource-flow", - "criterion_id": "operating-model-fit", - "region": { "id": "resource-flow", "label": "Resource-to-output flow" }, - "fire": { - "title": "Resource allocation is shown as independent metrics", - "visible_text": "Reviewers 8 · Queue 120 · SLA 2h", - "truth_excerpt": "Reviewer capacity governs queue throughput and the resulting SLA.", - "adjudication_notes": "Independent metrics conceal the load-bearing resource-to-throughput relationship.", - "render": { "template": "operating", "variant": "resource-metrics" } - }, - "clean": { - "title": "Capacity and queue connect to throughput and SLA", - "visible_text": "8 reviewers → 40/hr → 2h SLA", - "truth_excerpt": "Reviewer capacity governs queue throughput and the resulting SLA.", - "adjudication_notes": "The visual grammar exposes the governing flow and labels each transformation.", - "render": { "template": "operating", "variant": "resource-flow" } + "render": { + "template": "operating", + "variant": "provenance-trace" + } } }, { "id": "state-transition", "criterion_id": "operating-model-fit", - "region": { "id": "state-machine", "label": "Review state transitions" }, + "region": { + "id": "state-machine", + "label": "Review state transitions" + }, "fire": { "title": "Statuses appear without allowed transitions", "visible_text": "Pending · Reviewed · Escalated", "truth_excerpt": "Cases can escalate from pending or reviewed, but only reviewed cases can close.", "adjudication_notes": "Status badges expose no transition rules and cannot communicate the allowed state changes.", - "render": { "template": "operating", "variant": "state-badges" } + "render": { + "template": "operating", + "variant": "state-badges" + } }, "clean": { "title": "State machine exposes guarded transitions", "visible_text": "Pending → Reviewed → Closed · Escalated", "truth_excerpt": "Cases can escalate from pending or reviewed, but only reviewed cases can close.", "adjudication_notes": "Directed and labelled transitions make the guarded state changes reconstructable.", - "render": { "template": "operating", "variant": "state-machine" } + "render": { + "template": "operating", + "variant": "state-machine" + } } }, { "id": "comparison-treatment", "criterion_id": "operating-model-fit", - "region": { "id": "cost-comparison", "label": "Review cost comparison" }, - "fire": { - "title": "Simple cost comparison becomes an elaborate system map", - "visible_text": "Repeated review · One-time verification", - "truth_excerpt": "The claim is a two-term break-even comparison.", - "adjudication_notes": "A complex network invents dependencies and obscures a claim that only needs aligned arithmetic.", - "render": { "template": "operating", "variant": "overbuilt-comparison" } + "region": { + "id": "cost-comparison", + "label": "Review cost comparison" }, - "clean": { - "title": "Aligned equation is proportionate to the claim", - "visible_text": "n × review cost > verification cost", - "truth_excerpt": "The claim is a two-term break-even comparison.", - "adjudication_notes": "The relational treatment exposes the exact comparison without forcing an operating-model diagram.", - "render": { "template": "operating", "variant": "relational-equation" } - } - }, - { - "id": "simple-cover", - "criterion_id": "operating-model-fit", - "region": { "id": "cover-unit", "label": "Cover slide" }, "fire": { - "title": "Cover claim is forced into a node graph", - "visible_text": "Evidence you can operate", - "truth_excerpt": "The unit is a cover with one claim and no relational narrative job.", - "adjudication_notes": "A decorative graph implies a system the cover does not claim and is disproportionate to the unit.", - "render": { "template": "operating", "variant": "cover-graph" } + "title": "Equation overstates a qualitative comparison", + "visible_text": "n × review cost > verification cost", + "truth_excerpt": "The source offers only a qualitative preference and supplies no costs or multiplicative relationship.", + "adjudication_notes": "The equation asserts a measured quantitative relationship that the source does not establish.", + "render": { + "template": "operating", + "variant": "relational-equation" + }, + "source_conditioned": true }, "clean": { - "title": "Cover remains a single dominant claim", - "visible_text": "Evidence you can operate", - "truth_excerpt": "The unit is a cover with one claim and no relational narrative job.", - "adjudication_notes": "Routing to none is correct; no diagram is required for a single introductory claim.", - "render": { "template": "operating", "variant": "cover-none" } + "title": "Equation faithfully expresses a measured comparison", + "visible_text": "n × review cost > verification cost", + "truth_excerpt": "The source defines review count n and measured review and verification costs using this exact inequality.", + "adjudication_notes": "The same equation is legitimate because every term and relationship is supported by the supplied source evidence.", + "render": { + "template": "operating", + "variant": "relational-equation" + }, + "source_conditioned": true } }, { "id": "simulated-surface", "criterion_id": "operating-model-fit", - "region": { "id": "clinical-ide", "label": "Clinical decision workspace" }, + "region": { + "id": "clinical-ide", + "label": "Clinical decision workspace" + }, "fire": { "title": "Interactive workflow is flattened into generic cards", "visible_text": "Evidence · Hypotheses · Next actions", "truth_excerpt": "The clinician’s interaction among evidence, hypotheses, and actions is the claim.", "adjudication_notes": "Three generic cards omit selection, comparison, and action state that define the workflow.", - "render": { "template": "operating", "variant": "workspace-cards" } + "render": { + "template": "operating", + "variant": "workspace-cards" + } }, "clean": { "title": "Simulated workspace exposes the interaction", "visible_text": "Evidence · Hypotheses · Next actions", "truth_excerpt": "The clinician’s interaction among evidence, hypotheses, and actions is the claim.", "adjudication_notes": "The simulated surface faithfully exposes selection state, linked evidence, and available next actions.", - "render": { "template": "operating", "variant": "workspace-surface" } - } - }, - { - "id": "dependency-map", - "criterion_id": "operating-model-fit", - "region": { "id": "dependency-map", "label": "Deployment dependency map" }, - "fire": { - "title": "Dependency order is presented as unordered bullets", - "visible_text": "Schema · Migration · Service · Client", - "truth_excerpt": "Deployment requires schema before migration, then service, then client.", - "adjudication_notes": "An unordered list hides the load-bearing prerequisites and deploy order.", - "render": { "template": "operating", "variant": "dependency-list" } - }, - "clean": { - "title": "Dependency map preserves prerequisites", - "visible_text": "Schema → Migration → Service → Client", - "truth_excerpt": "Deployment requires schema before migration, then service, then client.", - "adjudication_notes": "The ordered connectors expose every prerequisite and the final client dependency.", - "render": { "template": "operating", "variant": "dependency-map" } + "render": { + "template": "operating", + "variant": "workspace-surface" + } } } - ] + ], + "viewport": { + "width": 1280, + "height": 720 + } }, { "id": "artifact-slop-gap", @@ -919,194 +883,221 @@ { "id": "independent-options", "criterion_id": "artifact-slop-false-sequence", - "region": { "id": "option-row", "label": "Three independent options" }, + "region": { + "id": "option-row", + "label": "Three independent options" + }, "fire": { "title": "Independent options are numbered and arrowed", "visible_text": "01 Explore → 02 Compare → 03 Export", "truth_excerpt": "Users may choose Explore, Compare, or Export in any order.", "adjudication_notes": "Numbers and arrows imply a required progression that the source explicitly does not establish.", - "render": { "template": "slop", "variant": "false-sequence" } + "render": { + "template": "slop", + "variant": "false-sequence" + } }, "clean": { "title": "Numbered workflow follows real dependencies", "visible_text": "01 Capture → 02 Adjudicate → 03 Graduate", "truth_excerpt": "Adjudication requires captured observations; graduation requires adjudication.", "adjudication_notes": "Every number and connector corresponds to a documented dependency and real order.", - "render": { "template": "slop", "variant": "true-sequence" } - } - }, - { - "id": "section-number", - "criterion_id": "artifact-slop-false-sequence", - "region": { "id": "section-heading", "label": "Numbered section heading" }, - "fire": { - "title": "Random section number implies missing chapters", - "visible_text": "04 · Architecture", - "truth_excerpt": "The artifact has one standalone page and no ordered sections.", - "adjudication_notes": "The isolated 04 marker implies a larger ordered document that does not exist.", - "render": { "template": "slop", "variant": "orphan-section-number" } - }, - "clean": { - "title": "Section number resolves through the document agenda", - "visible_text": "04 · Architecture", - "truth_excerpt": "The agenda links Architecture as section four of six.", - "adjudication_notes": "The section marker corresponds to real navigation and a genuine reading order.", - "render": { "template": "slop", "variant": "real-section-number" } + "render": { + "template": "slop", + "variant": "true-sequence" + } } }, { "id": "confidence-meter", "criterion_id": "artifact-slop-false-state", - "region": { "id": "confidence-meter", "label": "Confidence meter" }, + "region": { + "id": "confidence-meter", + "label": "Confidence meter" + }, "fire": { "title": "Static mockup claims 87 percent confidence", "visible_text": "Confidence 87%", "truth_excerpt": "No calculation, measurement, or source supplies a confidence value.", "adjudication_notes": "The meter looks measured and live even though the artifact has no supporting state.", - "render": { "template": "slop", "variant": "false-confidence" } + "render": { + "template": "slop", + "variant": "false-confidence" + }, + "source_conditioned": true }, "clean": { "title": "Confidence derives from displayed test results", - "visible_text": "Confidence 87% · 87/100 checks", + "visible_text": "Confidence 87%", "truth_excerpt": "The value is calculated from 87 passing checks out of 100 and links to the method.", "adjudication_notes": "The measured value and its derivation are both visible, so the state implication is supported.", - "render": { "template": "slop", "variant": "measured-confidence" } - } - }, - { - "id": "live-status-dot", - "criterion_id": "artifact-slop-false-state", - "region": { "id": "status-line", "label": "System status line" }, - "fire": { - "title": "Green dot claims a live connection", - "visible_text": "● System live", - "truth_excerpt": "The artifact is a static exported explainer with no connection telemetry.", - "adjudication_notes": "The conventional green status dot and “live” label imply a measured connection that cannot exist.", - "render": { "template": "slop", "variant": "false-live-status" } - }, - "clean": { - "title": "Decorative dot is visibly non-status", - "visible_text": "• System architecture", - "truth_excerpt": "The dot is a typographic bullet before a neutral heading.", - "adjudication_notes": "No status word, color semantics, or monitoring treatment suggests live state.", - "render": { "template": "slop", "variant": "decorative-dot" } + "render": { + "template": "slop", + "variant": "false-confidence" + }, + "source_conditioned": true } }, { "id": "progress-bar", "criterion_id": "artifact-slop-false-state", - "region": { "id": "progress", "label": "Progress indicator" }, + "region": { + "id": "progress", + "label": "Progress indicator" + }, "fire": { "title": "Decorative progress bar implies task completion", "visible_text": "Analysis 72%", "truth_excerpt": "The page performs no analysis and tracks no progress.", "adjudication_notes": "A filled meter and percentage imply active measured completion without any underlying task.", - "render": { "template": "slop", "variant": "false-progress" } + "render": { + "template": "slop", + "variant": "false-progress" + } }, "clean": { "title": "Deck progress reflects actual navigation", "visible_text": "Slide 7 of 10 · 70%", "truth_excerpt": "The viewer is currently on slide seven of ten.", "adjudication_notes": "The progress treatment is directly bound to real navigation state and accurately calculated.", - "render": { "template": "slop", "variant": "real-progress" } + "render": { + "template": "slop", + "variant": "real-progress" + } } }, { "id": "citation-marker", "criterion_id": "artifact-slop-false-provenance", - "region": { "id": "claim", "label": "Claim with citation marker" }, + "region": { + "id": "claim", + "label": "Claim with citation marker" + }, "fire": { "title": "Citation marker has no source target", - "visible_text": "Review cost falls 40% [4]", + "visible_text": "Claim [7]", "truth_excerpt": "The artifact supplies no source list, footnote, or link target.", "adjudication_notes": "The bracketed marker conventionally implies traceable evidence that the artifact cannot provide.", - "render": { "template": "slop", "variant": "false-citation" } + "render": { + "template": "slop", + "variant": "false-citation" + }, + "source_conditioned": true }, "clean": { "title": "Footnote marker resolves to a supplied source", - "visible_text": "Review cost falls 40% [4]", - "truth_excerpt": "Footnote four names the study, publication date, and resolvable URL.", + "visible_text": "Claim [7]", + "truth_excerpt": "Footnote seven names the study, publication date, and resolvable URL.", "adjudication_notes": "The provenance cue is backed by a concrete source target in the artifact.", - "render": { "template": "slop", "variant": "real-citation" } - } - }, - { - "id": "evidence-badge", - "criterion_id": "artifact-slop-false-provenance", - "region": { "id": "evidence-badge", "label": "Evidence validation badge" }, - "fire": { - "title": "Validated badge has no validation process", - "visible_text": "✓ Evidence validated", - "truth_excerpt": "No validator, review event, or source record exists.", - "adjudication_notes": "The checkmark and validation label assert provenance and verification unsupported by the artifact.", - "render": { "template": "slop", "variant": "false-validation" } - }, - "clean": { - "title": "Validation badge links to a signed review event", - "visible_text": "✓ Evidence validated · R-184", - "truth_excerpt": "Review event R-184 names the reviewer, timestamp, and checked sources.", - "adjudication_notes": "The validation claim resolves to a real review record with traceable provenance.", - "render": { "template": "slop", "variant": "real-validation" } + "render": { + "template": "slop", + "variant": "false-citation" + }, + "source_conditioned": true } }, { "id": "source-timestamp", "criterion_id": "artifact-slop-false-provenance", - "region": { "id": "source-strip", "label": "Source and timestamp strip" }, + "region": { + "id": "source-strip", + "label": "Source and timestamp strip" + }, "fire": { "title": "Decorative source strip invents freshness", - "visible_text": "SOURCE · LIVE · 12:42 UTC", + "visible_text": "Snapshot · 2026-07-25 10:30Z", "truth_excerpt": "The content is static prose with no source feed or retrieval time.", "adjudication_notes": "Source and timestamp chrome imply a live traceable feed that the artifact does not possess.", - "render": { "template": "slop", "variant": "false-source-strip" } + "render": { + "template": "slop", + "variant": "false-source-strip" + }, + "source_conditioned": true }, "clean": { "title": "Snapshot timestamp identifies a real retrieval", - "visible_text": "Snapshot · 12:42 UTC · dataset v18", - "truth_excerpt": "Dataset version 18 was retrieved at 12:42 UTC and is bundled with the report.", + "visible_text": "Snapshot · 2026-07-25 10:30Z", + "truth_excerpt": "Dataset version 18 was retrieved at 2026-07-25 10:30Z and is bundled with the report.", "adjudication_notes": "The timestamp and version accurately identify a supplied snapshot rather than manufacturing freshness.", - "render": { "template": "slop", "variant": "real-source-strip" } + "render": { + "template": "slop", + "variant": "false-source-strip" + }, + "source_conditioned": true } }, { "id": "terminal-label", "criterion_id": "artifact-slop-false-state", - "region": { "id": "terminal", "label": "Terminal-style panel" }, + "region": { + "id": "terminal", + "label": "Terminal-style panel" + }, "fire": { "title": "Fake terminal reports successful deployment", "visible_text": "$ deploy --prod · SUCCESS", "truth_excerpt": "The artifact never ran a deployment command and has no execution result.", "adjudication_notes": "Terminal chrome and SUCCESS output imply an executed command and measured state that never occurred.", - "render": { "template": "slop", "variant": "false-terminal" } + "render": { + "template": "slop", + "variant": "false-terminal" + } }, "clean": { "title": "Terminal is explicitly labelled as an example", "visible_text": "Example command · $ deploy --dry-run", "truth_excerpt": "The panel documents a command example and makes no claim that it ran.", "adjudication_notes": "The example label and dry-run wording prevent a reasonable viewer from inferring live execution state.", - "render": { "template": "slop", "variant": "example-terminal" } + "render": { + "template": "slop", + "variant": "example-terminal" + } } }, { "id": "choice-arrows", "criterion_id": "artifact-slop-false-sequence", - "region": { "id": "choice-map", "label": "Choice map" }, + "region": { + "id": "choice-map", + "label": "Choice map" + }, "fire": { "title": "Parallel choices are connected as a pipeline", - "visible_text": "PDF → Slides → Web", + "visible_text": "Self-hosted · Managed · Hybrid", "truth_excerpt": "PDF, Slides, and Web are independent export formats selected one at a time.", "adjudication_notes": "Directional arrows falsely imply transformation from one export format into the next.", - "render": { "template": "slop", "variant": "false-choice-flow" } + "render": { + "template": "slop", + "variant": "false-choice-flow" + }, + "source_conditioned": true }, "clean": { - "title": "Branching connector shows mutually exclusive choices", - "visible_text": "Export → PDF | Slides | Web", - "truth_excerpt": "One export action branches to three mutually exclusive formats.", - "adjudication_notes": "The branch grammar faithfully represents choice and does not imply sequential transformation.", - "render": { "template": "slop", "variant": "real-choice-branch" } + "title": "Arrows reflect a real deployment sequence", + "visible_text": "Self-hosted · Managed · Hybrid", + "truth_excerpt": "The documented migration moves from self-hosted to managed operation and then to a hybrid steady state.", + "adjudication_notes": "The directional sequence is supported by the supplied migration plan rather than invented from parallel choices.", + "render": { + "template": "slop", + "variant": "false-choice-flow" + }, + "source_conditioned": true } } - ] + ], + "viewport": { + "width": 1024, + "height": 768 + } } + ], + "graduation_status": "seed", + "capture_provenance": { + "status": "synthetic-fixtures", + "notes": "Renderer-authored seed cases. Replace with representative Artifacture captures before graduation." + }, + "target_batch_sizes": [ + 1, + 2 ] } diff --git a/evals/visual-model-policy/corpus.mjs b/evals/visual-model-policy/corpus.mjs index 86a72d4..c67768c 100644 --- a/evals/visual-model-policy/corpus.mjs +++ b/evals/visual-model-policy/corpus.mjs @@ -5,6 +5,7 @@ import { fileURLToPath } from 'node:url'; const ROOT = path.dirname(fileURLToPath(import.meta.url)); const DEFAULT_CORPUS_PATH = path.join(ROOT, 'corpus.json'); +export const DEFAULT_VIEWPORT = Object.freeze({ width: 960, height: 600 }); const DELEGATED_CRITERIA = new Set([ 'impeccable:critique', 'unslop:cleanup-report', @@ -17,12 +18,12 @@ export const RENDER_VARIANTS = Object.freeze(Object.fromEntries( slide: ['competing-foci', 'dense-one-focus'], dashboard: ['hierarchical-dense', 'uniform-loud'], 'diagram-space': ['collapsed-column', 'editorial-space', 'intentional-breathing', 'tiny-unbalanced'], - tracks: ['baseline-drift', 'editorial-60-40', 'hero-span', 'masonry', 'missing-weight', 'unequal-peers'], + tracks: ['baseline-drift', 'editorial-60-40', 'equal-peers', 'hero-span', 'masonry', 'missing-weight', 'unequal-peers'], mobile: ['compact-controls', 'crowded-controls'], diagram: [ 'ambiguous-relations', 'annotation-no-legend', 'coherent-flow', 'complete-legend', 'decorated-list', 'dishonest-bars', 'dishonest-timeline', 'dual-focal', - 'invalid-probability', 'labeled-comparison', 'missing-legend-entry', + 'invalid-probability', 'labeled-comparison', 'missing-legend-entry', 'proportional-bars', 'mixed-grammar', 'necessary-sequence', 'necessary-topology', 'orphan-legend', 'peer-inventory', 'single-focal', 'table-as-graph', 'timeline-break', 'valid-probability', @@ -57,6 +58,18 @@ export async function loadCorpus(corpusPath = DEFAULT_CORPUS_PATH) { return JSON.parse(await fs.readFile(corpusPath, 'utf8')); } +export function evidenceSha256(imageSha256, evalCase) { + if (!/^[a-f0-9]{64}$/.test(imageSha256 || '')) { + throw new Error('evidence identity requires an image SHA-256'); + } + return crypto.createHash('sha256').update([ + imageSha256, + evalCase.visible_text || '', + evalCase.truth_excerpt || '', + evalCase.source_artifact_sha256 || '', + ].join('\u0000')).digest('hex'); +} + export function expandCorpus(corpus, { corpusPath = DEFAULT_CORPUS_PATH } = {}) { const renderedRoot = path.resolve(path.dirname(corpusPath), corpus.rendered_root || 'corpus/rendered'); const cases = []; @@ -81,7 +94,15 @@ export function expandCorpus(corpus, { corpusPath = DEFAULT_CORPUS_PATH } = {}) visible_text: state.visible_text, truth_excerpt: state.truth_excerpt, adjudication_notes: state.adjudication_notes, + source_conditioned: state.source_conditioned === true, + source_artifact_sha256: state.source_artifact_sha256 || null, render: state.render, + viewport: { + ...DEFAULT_VIEWPORT, + ...(family.viewport || {}), + ...(pair.viewport || {}), + ...(state.viewport || {}), + }, image: { id: imageId, path: path.join(renderedRoot, `${imageId}.png`), @@ -102,6 +123,25 @@ export function validateCorpus(corpus, options = {}) { if (!['pending-human-review', 'human-reviewed'].includes(corpus.label_review?.status)) { throw new Error('corpus requires label_review.status pending-human-review or human-reviewed'); } + if (!['seed', 'ready'].includes(corpus.graduation_status)) { + throw new Error('corpus requires graduation_status seed or ready'); + } + if ( + !['synthetic-fixtures', 'representative-artifacture-captures'] + .includes(corpus.capture_provenance?.status) + ) { + throw new Error( + 'corpus requires capture_provenance.status synthetic-fixtures or representative-artifacture-captures', + ); + } + if ( + !Array.isArray(corpus.target_batch_sizes) + || corpus.target_batch_sizes.length === 0 + || corpus.target_batch_sizes.some((value) => !Number.isInteger(value) || value < 1) + || new Set(corpus.target_batch_sizes).size !== corpus.target_batch_sizes.length + ) { + throw new Error('corpus requires unique positive target_batch_sizes'); + } if ( corpus.label_review.status === 'human-reviewed' && (!corpus.label_review.reviewer || !corpus.label_review.reviewed_at) @@ -114,6 +154,9 @@ export function validateCorpus(corpus, options = {}) { const summary = { corpus_id: corpus.corpus_id, label_review: corpus.label_review, + graduation_status: corpus.graduation_status, + capture_provenance: corpus.capture_provenance, + target_batch_sizes: [...corpus.target_batch_sizes], cases: 0, families: {}, }; @@ -140,6 +183,21 @@ export function validateCorpus(corpus, options = {}) { ) { throw new Error(`${family.id}:${pair.id} requires named regions`); } + const sourceConditioned = pair.fire?.source_conditioned || pair.clean?.source_conditioned; + if ( + sourceConditioned + && ( + pair.fire?.source_conditioned !== true + || pair.clean?.source_conditioned !== true + || pair.fire.visible_text !== pair.clean.visible_text + || JSON.stringify(pair.fire.render) !== JSON.stringify(pair.clean.render) + || pair.fire.truth_excerpt === pair.clean.truth_excerpt + ) + ) { + throw new Error( + `${family.id}:${pair.id} source-conditioned states require identical visible evidence and distinct truth excerpts`, + ); + } for (const label of ALLOWED_LABELS) { const state = pair[label]; if (!state) throw new Error(`${family.id}:${pair.id} requires ${label} state`); @@ -155,6 +213,34 @@ export function validateCorpus(corpus, options = {}) { if (state.adjudication_notes.length < 20) { throw new Error(`${family.id}:${pair.id}:${label} adjudication_notes are too short`); } + if ( + state.source_artifact_sha256 !== undefined + && !/^[a-f0-9]{64}$/.test(state.source_artifact_sha256) + ) { + throw new Error(`${family.id}:${pair.id}:${label} has invalid source_artifact_sha256`); + } + if ( + corpus.graduation_status === 'ready' + && !/^[a-f0-9]{64}$/.test(state.source_artifact_sha256 || '') + ) { + throw new Error( + `${family.id}:${pair.id}:${label} graduation evidence requires source_artifact_sha256`, + ); + } + const viewport = { + ...DEFAULT_VIEWPORT, + ...(family.viewport || {}), + ...(pair.viewport || {}), + ...(state.viewport || {}), + }; + if ( + !Number.isInteger(viewport.width) + || viewport.width < 320 + || !Number.isInteger(viewport.height) + || viewport.height < 320 + ) { + throw new Error(`${family.id}:${pair.id}:${label} has invalid viewport`); + } const allowedVariants = RENDER_VARIANTS[state.render.template]; if (!allowedVariants?.has(state.render.variant)) { throw new Error( @@ -176,6 +262,38 @@ export function validateCorpus(corpus, options = {}) { } } } + const maxTargetBatchSize = Math.max(...corpus.target_batch_sizes); + for (const [familyId, family] of Object.entries(summary.families)) { + for (const [criterionId, counts] of Object.entries(family.criteria)) { + if (counts.fire + counts.clean < maxTargetBatchSize) { + throw new Error( + `${familyId}:${criterionId} does not support target batch ${maxTargetBatchSize}`, + ); + } + } + if ( + corpus.graduation_status === 'ready' + && (family.fire < 10 || family.clean < 10) + ) { + throw new Error( + `${familyId} requires at least ten fire and ten clean cases for graduation`, + ); + } + } + if ( + corpus.graduation_status === 'ready' + && corpus.label_review.status !== 'human-reviewed' + ) { + throw new Error('graduation-ready corpus requires human-reviewed labels'); + } + if ( + corpus.graduation_status === 'ready' + && corpus.capture_provenance.status !== 'representative-artifacture-captures' + ) { + throw new Error( + 'graduation-ready corpus requires representative Artifacture capture provenance', + ); + } expandCorpus(corpus, options); return summary; } diff --git a/evals/visual-model-policy/corpus.test.mjs b/evals/visual-model-policy/corpus.test.mjs index 09b3102..c09cd3b 100644 --- a/evals/visual-model-policy/corpus.test.mjs +++ b/evals/visual-model-policy/corpus.test.mjs @@ -1,6 +1,7 @@ import assert from 'node:assert/strict'; import test from 'node:test'; import { + evidenceSha256, expandCorpus, loadCorpus, validateCorpus, @@ -14,23 +15,30 @@ const REQUIRED_FAMILIES = [ 'artifact-slop-gap', ]; -test('the visual corpus has independent graduation evidence for every owned family', async () => { +test('the visual seed corpus covers every owned family without claiming graduation readiness', async () => { const corpus = await loadCorpus(); const summary = validateCorpus(corpus); assert.deepEqual(Object.keys(summary.families).sort(), [...REQUIRED_FAMILIES].sort()); assert.equal(summary.label_review.status, 'pending-human-review'); + assert.equal(summary.graduation_status, 'seed'); + assert.equal(summary.capture_provenance.status, 'synthetic-fixtures'); + assert.deepEqual(summary.target_batch_sizes, [1, 2]); for (const family of REQUIRED_FAMILIES) { - assert.ok(summary.families[family].fire >= 10, `${family} fire cases`); - assert.ok(summary.families[family].clean >= 10, `${family} clean cases`); + assert.ok(summary.families[family].fire >= 6, `${family} fire cases`); + assert.ok(summary.families[family].clean >= 6, `${family} clean cases`); assert.equal( summary.families[family].hard_negatives, summary.families[family].clean, `${family} hard negatives`, ); for (const [criterionId, counts] of Object.entries(summary.families[family].criteria)) { - assert.ok(counts.fire >= 2, `${family}:${criterionId} fire cases support batch 4`); - assert.ok(counts.clean >= 2, `${family}:${criterionId} clean cases support batch 4`); + assert.ok(counts.fire >= 1, `${family}:${criterionId} fire evidence`); + assert.ok(counts.clean >= 1, `${family}:${criterionId} clean evidence`); + assert.ok( + counts.fire + counts.clean >= 2, + `${family}:${criterionId} supports the target batch size`, + ); } } }); @@ -50,7 +58,7 @@ test('layout corpus covers clipping, crowding, dead space, and symmetry', async test('every case exposes stable evidence identity, a named region, and adjudication notes', async () => { const cases = expandCorpus(await loadCorpus()); - assert.equal(cases.length, 112); + assert.equal(cases.length, 72); for (const entry of cases) { assert.match(entry.case_id, /^[a-z0-9][a-z0-9:-]+$/); assert.match(entry.state_id, /^[a-z0-9][a-z0-9:-]+$/); @@ -60,9 +68,31 @@ test('every case exposes stable evidence identity, a named region, and adjudicat assert.ok(entry.regions.every((region) => region.id && region.label)); assert.ok(['fire', 'clean'].includes(entry.human_label)); assert.ok(entry.adjudication_notes.length >= 20); + assert.ok(entry.viewport.width > 0 && entry.viewport.height > 0); } }); +test('source-conditioned cases distinguish evidence context even when pixels match', () => { + const imageSha256 = 'a'.repeat(64); + const unsupported = evidenceSha256(imageSha256, { + visible_text: 'Confidence 87%', + truth_excerpt: 'No measurement exists for the displayed confidence.', + }); + const supported = evidenceSha256(imageSha256, { + visible_text: 'Confidence 87%', + truth_excerpt: '87 of 100 checks passed in the attached evaluation run.', + }); + + assert.notEqual(unsupported, supported); + assert.equal( + unsupported, + evidenceSha256(imageSha256, { + visible_text: 'Confidence 87%', + truth_excerpt: 'No measurement exists for the displayed confidence.', + }), + ); +}); + test('the owned corpus cannot absorb delegated skill criteria', async () => { const corpus = await loadCorpus(); corpus.families[0].pairs[0].criterion_id = 'impeccable:critique'; diff --git a/evals/visual-model-policy/example-measurements.json b/evals/visual-model-policy/example-measurements.json index 99878d5..a124198 100644 --- a/evals/visual-model-policy/example-measurements.json +++ b/evals/visual-model-policy/example-measurements.json @@ -52,6 +52,9 @@ "unique_cases": 20, "unique_positives": 10, "unique_negatives": 10, + "unique_images": 20, + "unique_image_positives": 10, + "unique_image_negatives": 10, "adjudication_complete": true, "synthetic": true, "telemetry_complete": false, @@ -79,6 +82,9 @@ "unique_cases": 20, "unique_positives": 10, "unique_negatives": 10, + "unique_images": 20, + "unique_image_positives": 10, + "unique_image_negatives": 10, "adjudication_complete": true, "synthetic": true, "telemetry_complete": false, @@ -106,6 +112,9 @@ "unique_cases": 20, "unique_positives": 10, "unique_negatives": 10, + "unique_images": 20, + "unique_image_positives": 10, + "unique_image_negatives": 10, "adjudication_complete": true, "synthetic": true, "telemetry_complete": false, diff --git a/evals/visual-model-policy/experiment.template.json b/evals/visual-model-policy/experiment.template.json index 35a25fe..660c984 100644 --- a/evals/visual-model-policy/experiment.template.json +++ b/evals/visual-model-policy/experiment.template.json @@ -4,7 +4,7 @@ "corpus_path": "corpus.json", "image_detail": "high", "replicates": 3, - "batch_sizes": [1, 2, 4], + "batch_sizes": [1, 2], "passes": [ "layout", "diagram", diff --git a/evals/visual-model-policy/merge.test.mjs b/evals/visual-model-policy/merge.test.mjs index 87488cf..86aea2f 100644 --- a/evals/visual-model-policy/merge.test.mjs +++ b/evals/visual-model-policy/merge.test.mjs @@ -29,6 +29,9 @@ function measurement(model, contract, completedCandidates = []) { unique_cases: 20, unique_positives: 10, unique_negatives: 10, + unique_images: 20, + unique_image_positives: 10, + unique_image_negatives: 10, tp: 10, fp: 0, tn: 10, diff --git a/evals/visual-model-policy/render-corpus.mjs b/evals/visual-model-policy/render-corpus.mjs index 3a58a13..b28f95b 100644 --- a/evals/visual-model-policy/render-corpus.mjs +++ b/evals/visual-model-policy/render-corpus.mjs @@ -6,13 +6,12 @@ import process from 'node:process'; import { fileURLToPath, pathToFileURL } from 'node:url'; import { chromium } from 'playwright'; import { + DEFAULT_VIEWPORT, expandCorpus, loadCorpus, validateCorpus, } from './corpus.mjs'; -const VIEWPORT = Object.freeze({ width: 960, height: 600 }); - export function renderCaseHtml(evalCase) { const { template, variant } = evalCase.render; const scene = renderTemplate(template, variant, evalCase); @@ -26,21 +25,25 @@ export function renderCaseHtml(evalCase) { html, body { width: 100%; height: 100%; margin: 0; } body { overflow: hidden; - background: #e9e6df; + background: #d9dde2; color: #171714; font-family: Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; } .frame { - width: 960px; - height: 600px; - padding: 44px 48px; + width: 100vw; + height: 100vh; + padding: clamp(24px, 4vw, 48px); background: linear-gradient(90deg, rgba(23,23,20,.035) 1px, transparent 1px) 0 0 / 32px 32px, #f7f5ef; position: relative; } + .family-diagram { background-color:#edf2f4; background-image:linear-gradient(rgba(41,72,84,.055) 1px,transparent 1px),linear-gradient(90deg,rgba(41,72,84,.055) 1px,transparent 1px); background-size:24px 24px; } + .family-aesthetic { background:#f3f0e9; } + .family-operating-model { background:#f0f3ee; } + .family-artifact-slop-gap { background:#fbf7ed; } .eyebrow { font: 700 12px/1.2 ui-monospace, SFMono-Regular, Menlo, monospace; letter-spacing: .12em; text-transform: uppercase; color: #626159; } - h1 { margin: 12px 0 28px; font-size: 38px; line-height: 1.04; letter-spacing: -.04em; max-width: 720px; } + h1 { margin: 0 0 28px; font-size: clamp(30px, 4vw, 42px); line-height: 1.04; letter-spacing: -.04em; max-width: 720px; } h2, h3, p { margin: 0; } .muted { color: #6c6a61; } .panel { border: 1px solid #aaa79e; background: rgba(255,255,255,.62); } @@ -49,6 +52,7 @@ export function renderCaseHtml(evalCase) { .grid { display: grid; gap: 14px; } .row { display: flex; align-items: center; gap: 14px; } .node { min-width: 132px; padding: 18px; border: 1px solid #7e7b73; background: #fffdf8; } + .node.ink { background: #22231f; color: #f6f3eb; } .node strong { display: block; margin-bottom: 6px; } .arrow { font: 700 22px/1 ui-monospace, monospace; color: #69675f; } .chip { border: 1px solid #8f8c82; padding: 7px 10px; font: 700 11px/1 ui-monospace, monospace; text-transform: uppercase; letter-spacing: .06em; } @@ -59,8 +63,7 @@ export function renderCaseHtml(evalCase) { -
-
${escapeHtml(evalCase.family)} / ${escapeHtml(evalCase.criterion_id)}
+
${scene}
@@ -92,17 +95,18 @@ export async function renderCorpus({ const renderedImages = []; try { const page = await browser.newPage({ - viewport: VIEWPORT, + viewport: DEFAULT_VIEWPORT, deviceScaleFactor: 1, colorScheme: 'light', }); for (const evalCase of cases) { + await page.setViewportSize(evalCase.viewport || DEFAULT_VIEWPORT); await page.setContent(renderCaseHtml(evalCase), { waitUntil: 'load' }); const file = path.join(root, `${evalCase.image.id}.png`); const bytes = await page.screenshot({ path: file, type: 'png' }); const hash = crypto.createHash('sha256').update(bytes).digest('hex'); const duplicate = caseByHash.get(hash); - if (duplicate) { + if (duplicate && !(duplicate.source_conditioned && evalCase.source_conditioned)) { throw new Error( `duplicate rendered pixels for ${duplicate.case_id} and ${evalCase.case_id}`, ); @@ -110,17 +114,26 @@ export async function renderCorpus({ caseByHash.set(hash, evalCase); const pairKey = `${evalCase.family}\u0000${evalCase.pair_id}`; const prior = renderHashes.get(pairKey); - if (prior?.hash === hash && prior.human_label !== evalCase.human_label) { + if ( + prior?.hash === hash + && prior.human_label !== evalCase.human_label + && !(prior.source_conditioned && evalCase.source_conditioned) + ) { throw new Error( `opposite labels rendered identical pixels for ${evalCase.family}:${evalCase.pair_id}`, ); } - renderHashes.set(pairKey, { hash, human_label: evalCase.human_label }); + renderHashes.set(pairKey, { + hash, + human_label: evalCase.human_label, + source_conditioned: evalCase.source_conditioned, + }); renderedImages.push({ case_id: evalCase.case_id, image_id: evalCase.image.id, path: file, sha256: hash, + viewport: evalCase.viewport, }); } } finally { @@ -130,7 +143,7 @@ export async function renderCorpus({ schema_version: 1, corpus_id: corpus.corpus_id, rendered_at: new Date().toISOString(), - viewport: VIEWPORT, + default_viewport: DEFAULT_VIEWPORT, images: renderedImages, }; await fs.writeFile( @@ -145,11 +158,10 @@ function renderTemplate(template, variant, evalCase) { const text = escapeHtml(evalCase.visible_text); if (template === 'text') { const clipped = variant === 'clipped-right'; - const width = clipped ? 410 : 640; const wrap = clipped ? 'white-space:nowrap; overflow:hidden;' : 'white-space:normal;'; return `

${sceneHeading(template)}

-
-
${text}
+
+
${text}
`; } if (template === 'table') { @@ -186,14 +198,15 @@ function renderTemplate(template, variant, evalCase) {
`; } if (template === 'tracks') { - const uneven = ['unequal-peers', 'baseline-drift', 'missing-weight'].includes(variant); + const uneven = ['unequal-peers', 'missing-weight'].includes(variant); + const baselineDrift = variant === 'baseline-drift'; const editorial = variant === 'editorial-60-40'; const columns = uneven ? '1.2fr .55fr 1.35fr' : (editorial ? '1.5fr 1fr' : '1fr 1fr 1fr'); const labels = editorial ? ['Narrative', 'Evidence'] : ['Input', 'Router', 'Output']; return `

${sceneHeading(template)}

- ${labels.map((label, index) => `
-

${label}

${text}

+ ${labels.map((label, index) => `
+

${label}

${text}

`).join('')}
`; } @@ -218,7 +231,7 @@ function renderTemplate(template, variant, evalCase) { function renderDiagram(variant, evalCase) { const regionId = evalCase.regions[0].id; const timeline = ['dishonest-timeline', 'timeline-break'].includes(variant); - const bars = ['dishonest-bars', 'labeled-comparison', 'invalid-probability', 'valid-probability'].includes(variant); + const bars = ['dishonest-bars', 'labeled-comparison', 'proportional-bars', 'invalid-probability', 'valid-probability'].includes(variant); if (timeline) { return `

${sceneHeading('diagram')}

@@ -229,7 +242,9 @@ function renderDiagram(variant, evalCase) { } if (bars) { const dishonestScale = variant === 'dishonest-bars' || variant === 'invalid-probability'; - const widths = dishonestScale ? [48, 58, 70] : [30, 62, 86]; + const widths = variant.includes('probability') + ? (dishonestScale ? [48, 58, 70] : [72, 33, 26]) + : (dishonestScale ? [62, 69, 66] : [24, 82, 43]); const labels = variant.includes('probability') ? ['A 55%', variant === 'invalid-probability' ? 'B 35%' : 'B 25%', 'Other 20%'] : ['A $10', 'B $40', 'Other $20']; @@ -237,6 +252,24 @@ function renderDiagram(variant, evalCase) { ${widths.map((width, index) => `
${labels[index]}
`).join('')}
`; } + if (['orphan-legend', 'complete-legend', 'missing-legend-entry', 'annotation-no-legend'].includes(variant)) { + const orphan = variant === 'orphan-legend'; + const missing = variant === 'missing-legend-entry'; + const annotation = variant === 'annotation-no-legend'; + const figureKinds = missing ? ['SERVICE', 'DECISION', 'DATA'] : ['SERVICE', 'DATA']; + if (variant === 'complete-legend') figureKinds.push('CACHE'); + const legendKinds = orphan || variant === 'complete-legend' + ? ['SERVICE', 'DATA', 'CACHE'] + : ['SERVICE', 'DATA']; + return `

${sceneHeading('diagram')}

+
+ ${figureKinds.map((kind, index) => `${index ? '→' : ''}
${kind}${index + 1}
`).join('')} + ${annotation ? '' : ''} +
+
+
${legendKinds.map((kind) => `${kind}`).join('')}
+
`; + } const nodes = ['Capture', 'Policy', 'Decision']; const connectors = ['→', '→']; const mixedRelations = variant === 'mixed-grammar' || variant === 'ambiguous-relations'; @@ -268,7 +301,41 @@ function renderDiagram(variant, evalCase) { function renderPreset(variant, evalCase) { const regionId = evalCase.regions[0].id; - const wrongMode = ['wrong-mode-panel', 'wrong-mode-background', 'invisible-dark-icons'].includes(variant); + if (variant === 'invisible-dark-icons' || variant === 'quiet-watermark') { + const broken = variant === 'invisible-dark-icons'; + return `

${sceneHeading('preset')}

+ ${['LIGHT', 'DARK'].map((mode, index) => { + const dark = index === 1; + const foreground = dark ? '#f6f3eb' : '#22231f'; + const iconColor = broken && dark ? '#22231f' : foreground; + return `
+
${mode}
+
+ ◆ +
Inputs

Route evidence

+
+
+ +
Results

Grounded finding

+
+
`; + }).join('')} +
`; + } + if (['many-surprises', 'one-surprise', 'many-grid-breaks', 'one-grid-break'].includes(variant)) { + const many = variant.startsWith('many-'); + const gridBreak = variant.endsWith('grid-breaks') || variant === 'one-grid-break'; + return `

${sceneHeading('preset')}

+
+ ${['Capture', 'Review', 'Decision'].map((label, index) => `
${label}${escapeHtml(evalCase.visible_text)}
`).join('')} +
+
${many ? 'Three unrelated elements leave the established grid.' : 'One decision deliberately breaks the established pattern.'}
+
`; + } + const wrongMode = ['wrong-mode-panel', 'wrong-mode-background'].includes(variant); const mismatchedDemo = variant === 'mismatched-demo'; const mobileHeroCard = variant === 'mobile-hero-card'; const multipleSurprises = variant === 'many-surprises' || variant === 'many-grid-breaks'; @@ -294,24 +361,63 @@ function renderPreset(variant, evalCase) { function renderOperating(variant, evalCase) { const regionId = evalCase.regions[0].id; const labels = evalCase.visible_text.split(' · ').slice(0, 4); - const independentCards = [ - 'routing-cards', - 'one-way-loop', - 'missing-residual', - 'provenance-cards', - 'resource-metrics', - 'state-badges', - 'overbuilt-comparison', - 'cover-graph', - 'workspace-cards', - 'dependency-list', - ].includes(variant); - const cards = independentCards || variant === 'relational-equation' || variant === 'cover-none'; - return `

${sceneHeading('operating')}

-
- ${labels.map((label, index) => `${!cards && index ? `${variant === 'closed-loop' && index === labels.length - 1 ? '↩' : '→'}` : ''}
${escapeHtml(label)}${independentCards ? 'Independent item' : `Rule ${index + 1}`}
`).join('')} -
- ${!cards ? `
guards, dependencies, and provenance remain explicit
` : ''} + if (variant === 'relational-equation') { + return `

${sceneHeading('operating')}

+
${escapeHtml(evalCase.visible_text)}
+
`; + } + if (variant === 'routing-model') { + return `

${sceneHeading('operating')}

+
Incoming reviewall eligible items
→
Policy routerpriority + deduplication
+
${['Mandatory', 'Representative', 'Calibration', 'Frontier'].map((label) => `
${label}explicit entry rule
`).join('')}
+
`; + } + if (variant === 'closed-loop' || variant === 'one-way-loop') { + const closed = variant === 'closed-loop'; + return `

${sceneHeading('operating')}

+
${labels.map((label, index) => `${index ? '→' : ''}
${escapeHtml(label)}stage ${index + 1}
`).join('')}
+ ${closed ? '
outcomes ↩ calibration
' : '
No outcome path returns to calibration.
'} +
`; + } + if (variant === 'provenance-trace' || variant === 'provenance-cards') { + const trace = variant === 'provenance-trace'; + return `

${sceneHeading('operating')}

+
${['Source', 'Finding', 'Decision'].map((label, index) => `${trace && index ? '→' : ''}
${label}${trace ? 'trace R-184' : 'unlinked evidence'}
`).join('')}
+
${trace ? 'R-184 binds every transformation.' : 'No identifier connects the cards.'}
+
`; + } + if (variant === 'state-machine' || variant === 'state-badges') { + const machine = variant === 'state-machine'; + if (machine) { + return `

${sceneHeading('operating')}

+
+
Pendingstate 1
+ →review complete +
Reviewedstate 2
+ →reviewed only +
Closedterminal
+
+
+ Pending ↘ +
Escalatedfrom pending or reviewed
+ ↙ Reviewed +
+
`; + } + return `

${sceneHeading('operating')}

+
${['Pending', 'Reviewed', 'Escalated'].map((label) => `
${label}status only
`).join('')}
+
`; + } + if (variant === 'workspace-surface' || variant === 'workspace-cards') { + const surface = variant === 'workspace-surface'; + return `

${sceneHeading('operating')}

+
+ ${['Evidence', 'Hypotheses', 'Next actions'].map((label, index) => `
${label}

${surface ? ['filterable queue', 'active investigation canvas', 'decision inspector'][index] : 'independent summary card'}

`).join('')} +
+
`; + } + return `

${sceneHeading('operating')}

+
${labels.map((label) => `
${escapeHtml(label)}Independent item
`).join('')}
`; } diff --git a/evals/visual-model-policy/run.mjs b/evals/visual-model-policy/run.mjs index d36346b..27d3b30 100644 --- a/evals/visual-model-policy/run.mjs +++ b/evals/visual-model-policy/run.mjs @@ -10,6 +10,7 @@ import { buildPrefixManifest, } from '../visual-cache/prefix.mjs'; import { + evidenceSha256, expandCorpus, loadCorpus, validateCorpus, @@ -277,6 +278,14 @@ export async function runExperiment({ throw new Error('replace all placeholder candidate and provider values before a live run'); } validateCorpus(corpus); + const unsupportedBatchSizes = experiment.batch_sizes.filter( + (batchSize) => !corpus.target_batch_sizes.includes(batchSize), + ); + if (unsupportedBatchSizes.length > 0) { + throw new Error( + `experiment requests unsupported corpus batch size(s): ${unsupportedBatchSizes.join(', ')}`, + ); + } const allCases = expandCorpus(corpus, { corpusPath: experiment.corpus_path }); const selectedPasses = new Set(experiment.passes || allCases.map((entry) => entry.family)); const cases = allCases.filter((entry) => selectedPasses.has(entry.family)); @@ -289,7 +298,7 @@ export async function runExperiment({ if (unknownPasses.length > 0) { throw new Error(`experiment references unknown pass(es): ${unknownPasses.join(', ')}`); } - await assertImagesExist(cases); + const imageHashesByCase = await readCorpusImageHashes(cases); const plan = buildExperimentPlan(experiment, cases); if (plan.length === 0) throw new Error('experiment produced an empty request plan'); const preparedPlan = await prepareExperimentPlan(experiment, plan); @@ -313,6 +322,12 @@ export async function runExperiment({ 'live measurements require corpus.label_review.status=human-reviewed with reviewer and reviewed_at', ); } + if (adapter.synthetic !== true && corpus.graduation_status !== 'ready') { + throw new Error('live measurements require corpus.graduation_status=ready'); + } + if (adapter.synthetic !== true) { + assertGraduationImageDiversity(cases, imageHashesByCase); + } let written = 0; for (const { request, ...cell } of preparedPlan) { @@ -386,6 +401,10 @@ export async function runExperiment({ observations: evaluated.observations.map((observation) => ({ ...observation, image_sha256: imageHashById.get(observation.image_id), + evidence_sha256: evidenceSha256( + imageHashById.get(observation.image_id), + cell.cases.find((entry) => entry.case_id === observation.case_id), + ), observation_id: `${requestId}:${observation.case_id}`, })), synthetic: adapter.synthetic === true, @@ -676,11 +695,16 @@ function validateLadderProgress(experiment, selectedPasses) { } } -async function assertImagesExist(cases) { +async function readCorpusImageHashes(cases) { const missing = []; + const hashes = new Map(); for (const entry of cases) { try { - await fs.access(entry.image.path); + const bytes = await fs.readFile(entry.image.path); + hashes.set( + entry.case_id, + crypto.createHash('sha256').update(bytes).digest('hex'), + ); } catch { missing.push(entry.image.path); } @@ -690,6 +714,31 @@ async function assertImagesExist(cases) { `corpus images are missing (${missing.length}); run npm run ve:render-visual-model-corpus`, ); } + return hashes; +} + +export function assertGraduationImageDiversity(cases, imageHashesByCase) { + const groups = new Map(); + for (const entry of cases) { + const key = `${entry.family}\u0000${entry.human_label}`; + const group = groups.get(key) || { images: new Set(), sources: new Set() }; + group.images.add(imageHashesByCase.get(entry.case_id)); + group.sources.add(entry.source_artifact_sha256); + groups.set(key, group); + } + for (const [key, group] of groups) { + const [family, label] = key.split('\u0000'); + if (group.images.has(undefined) || group.images.size < 10) { + throw new Error( + `${family}:${label} requires at least ten distinct rendered images for live graduation`, + ); + } + if (group.sources.has(null) || group.sources.has(undefined) || group.sources.size < 10) { + throw new Error( + `${family}:${label} requires at least ten distinct source artifacts for live graduation`, + ); + } + } } async function loadAdapter(adapterPath) { diff --git a/evals/visual-model-policy/run.test.mjs b/evals/visual-model-policy/run.test.mjs index 532966f..e7a2807 100644 --- a/evals/visual-model-policy/run.test.mjs +++ b/evals/visual-model-policy/run.test.mjs @@ -5,6 +5,7 @@ import path from 'node:path'; import test from 'node:test'; import { appendRawRecord, + assertGraduationImageDiversity, buildBatchRequest, buildExperimentPlan, criterionPromptFor, @@ -12,6 +13,33 @@ import { runExperiment, } from './run.mjs'; +test('live graduation requires distinct rendered images and source artifacts per label', () => { + const cases = ['fire', 'clean'].flatMap((humanLabel) => ( + Array.from({ length: 10 }, (_, index) => ({ + case_id: `${humanLabel}-${index}`, + family: 'layout', + human_label: humanLabel, + source_artifact_sha256: `${humanLabel === 'fire' ? 'a' : 'b'}${String(index).padStart(63, '0')}`, + })) + )); + const repeatedPixels = new Map(cases.map((entry) => [ + entry.case_id, + entry.human_label === 'fire' ? 'c'.repeat(64) : 'd'.repeat(64), + ])); + assert.throws( + () => assertGraduationImageDiversity(cases, repeatedPixels), + /ten distinct rendered images/, + ); + + const distinctPixels = new Map(cases.map((entry, index) => [ + entry.case_id, + String(index).padStart(64, '0'), + ])); + assert.doesNotThrow( + () => assertGraduationImageDiversity(cases, distinctPixels), + ); +}); + async function fixture(t) { const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'artifacture-visual-run-')); t.after(() => fs.rm(dir, { recursive: true, force: true })); diff --git a/evals/visual-model-policy/select.mjs b/evals/visual-model-policy/select.mjs index 2a0e87c..05f8b4c 100644 --- a/evals/visual-model-policy/select.mjs +++ b/evals/visual-model-policy/select.mjs @@ -68,6 +68,16 @@ export function qualificationFor(result, thresholds = DEFAULT_THRESHOLDS) { requireAtLeast('unique_positives', result.unique_positives, thresholds.min_positives); requireAtLeast('unique_negatives', result.unique_negatives, thresholds.min_negatives); } + requireAtLeast( + 'unique_image_positives', + result.unique_image_positives, + thresholds.min_positives, + ); + requireAtLeast( + 'unique_image_negatives', + result.unique_image_negatives, + thresholds.min_negatives, + ); if (result.adjudication_complete !== true) { reasons.push('adjudication_complete must be true'); } @@ -92,6 +102,16 @@ export function qualificationFor(result, thresholds = DEFAULT_THRESHOLDS) { criterion.unique_negatives, thresholds.min_criterion_negatives, ); + requireAtLeast( + `criterion ${criterion.id} unique_image_positives`, + criterion.unique_image_positives, + thresholds.min_criterion_positives, + ); + requireAtLeast( + `criterion ${criterion.id} unique_image_negatives`, + criterion.unique_image_negatives, + thresholds.min_criterion_negatives, + ); } } requireAtLeast('precision', metrics.precision, thresholds.precision); @@ -160,6 +180,9 @@ export function selectVisualModelPolicy(input) { unique_cases: selected.unique_cases, unique_positives: selected.unique_positives, unique_negatives: selected.unique_negatives, + unique_images: selected.unique_images, + unique_image_positives: selected.unique_image_positives, + unique_image_negatives: selected.unique_image_negatives, }), }, escalation_chain: ladder.slice(1).map((row) => ({ @@ -208,6 +231,7 @@ function validateInput(input) { 'tp', 'fp', 'tn', 'fn', 'grounded', 'grounding_total', 'json_valid', 'responses', 'abstentions', 'total_cost_usd', 'p95_latency_ms', 'unique_cases', 'unique_positives', 'unique_negatives', + 'unique_images', 'unique_image_positives', 'unique_image_negatives', 'adjudication_complete', 'synthetic', 'telemetry_complete', 'experiment_complete', ]) { @@ -231,7 +255,10 @@ function validateInput(input) { 'positives', 'negatives', 'tp', 'fp', 'tn', 'fn', 'grounded', 'json_valid', 'abstentions', ]) nonNegativeInt(result[field], field); - for (const field of ['unique_cases', 'unique_positives', 'unique_negatives']) { + for (const field of [ + 'unique_cases', 'unique_positives', 'unique_negatives', + 'unique_images', 'unique_image_positives', 'unique_image_negatives', + ]) { nonNegativeInt(result[field], field); } for (const field of [ @@ -273,6 +300,16 @@ function validateInput(input) { ) { throw new Error('unique_positives + unique_negatives must equal unique_cases'); } + if ( + Number(result.unique_images) > Number(result.unique_image_positives) + + Number(result.unique_image_negatives) + || Number(result.unique_images) < Math.max( + Number(result.unique_image_positives), + Number(result.unique_image_negatives), + ) + ) { + throw new Error('unique image totals are internally inconsistent'); + } } } diff --git a/evals/visual-model-policy/select.test.mjs b/evals/visual-model-policy/select.test.mjs index d41d335..1b1ad8f 100644 --- a/evals/visual-model-policy/select.test.mjs +++ b/evals/visual-model-policy/select.test.mjs @@ -36,6 +36,9 @@ function result(overrides = {}) { unique_cases: 20, unique_positives: 10, unique_negatives: 10, + unique_images: 20, + unique_image_positives: 10, + unique_image_negatives: 10, adjudication_complete: true, synthetic: false, telemetry_complete: true, @@ -112,6 +115,21 @@ test('blocks a pass when no model has enough positive and negative evidence', () assert.match(policy.blocked.layout.evaluated[0].reasons.join('\n'), /positives/); }); +test('source-conditioned evidence cannot replace distinct rendered-image coverage', () => { + const outcome = qualificationFor(result({ + unique_cases: 20, + unique_positives: 10, + unique_negatives: 10, + unique_images: 2, + unique_image_positives: 1, + unique_image_negatives: 1, + })); + + assert.equal(outcome.qualified, false); + assert.match(outcome.reasons.join('\n'), /unique_image_positives/); + assert.match(outcome.reasons.join('\n'), /unique_image_negatives/); +}); + test('allows stricter per-run thresholds without changing the selector', () => { const policy = selectVisualModelPolicy({ candidates, From 2312b6bd8ace4fd34b6ac8178f2c024dd8e7de5b Mon Sep 17 00:00:00 2001 From: Clayton Kim Date: Sat, 25 Jul 2026 15:28:28 -0700 Subject: [PATCH 4/4] Tune visual verifier prompts from smoke results --- evals/visual-model-policy/README.md | 23 +++++++++++++++++++ evals/visual-model-policy/run.mjs | 4 ++-- evals/visual-model-policy/run.test.mjs | 10 ++++++++ .../scripts/verify/rubrics/pass-aesthetic.md | 6 ++--- .../scripts/verify/rubrics/pass-diagram.md | 4 ++-- .../scripts/verify/rubrics/pass-layout.md | 6 ++--- .../verify/rubrics/pass-operating-model.md | 1 + 7 files changed, 44 insertions(+), 10 deletions(-) diff --git a/evals/visual-model-policy/README.md b/evals/visual-model-policy/README.md index 54284e1..93e8ffb 100644 --- a/evals/visual-model-policy/README.md +++ b/evals/visual-model-policy/README.md @@ -56,6 +56,29 @@ and truth excerpt, not merely the case ID or screenshot bytes. These source-conditioned controls test whether a verifier actually uses supplied evidence when identical pixels can be either valid or invalid. +### Prompt hill-climb smoke (non-policy evidence) + +On 2026-07-25, the shared verdict contract and family rubrics were tuned +against one label-blind smoke of all 72 seed states using three parallel +delegated `gpt-5.6-terra` agents (24 disjoint states per agent). This is a +prompt-development check, not a provider measurement: it has no provider +latency, token-cache, cost, or independent replicate telemetry, and it does +not qualify a route. It reuses the seed corpus rather than a held-out split, +so the improvement is not a generalization estimate. + +| Prompt | Correct | TP | TN | FP | FN | Precision | Recall | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| Baseline | 67/72 | 36 | 31 | 5 | 0 | 87.8% | 100% | +| Hill-climbed | 72/72 | 36 | 36 | 0 | 0 | 100% | 100% | + +The hill-climbed output was emitted as label-free JSON with grounded regions; +the manifests and raw outputs were retained externally at +`/tmp/artifacture-luna-blind-{0,1,2}.json` and +`/tmp/artifacture-smoke-results-v2.json`. Because those files are delegated +agent artifacts rather than adapter records, they are intentionally excluded +from policy aggregation. Re-run the smoke only as a development signal, and +use the provider-independent runner below for any qualification claim. + This is a seed corpus, not a graduation corpus. Its checked-in labels are proposed from fixture intent, not represented as human review, and each family currently has fewer than the default 10 positive and 10 negative unique- diff --git a/evals/visual-model-policy/run.mjs b/evals/visual-model-policy/run.mjs index 27d3b30..a5ecceb 100644 --- a/evals/visual-model-policy/run.mjs +++ b/evals/visual-model-policy/run.mjs @@ -17,9 +17,9 @@ import { } from './corpus.mjs'; import { normalizeTelemetry } from './telemetry.mjs'; -export const VERDICT_SYSTEM_TEXT = `Return JSON only. The top-level object must be {"verdicts":[...]}. Each supplied state_id must have exactly one verdict. A verdict is either {"state_id":"...","abstain":true,"reason":"..."} or {"state_id":"...","pass":true|false,"findings":[...]}. Every finding requires check_id, image_id, region_id, evidence, and fix.`; +export const VERDICT_SYSTEM_TEXT = `Return JSON only. The top-level object must be {"verdicts":[...]}. Each supplied state_id must have exactly one verdict. A verdict is either {"state_id":"...","abstain":true,"reason":"..."} or {"state_id":"...","pass":true|false,"findings":[...]}. pass=true means the selected criterion does not fire; use pass=false only when the supplied image and evidence visibly support a violation. Do not invent a finding from a title, truth excerpt, or unusual-but-intentional design alone. Every finding requires check_id, image_id, region_id, evidence, and fix.`; -export const SHARED_INSTRUCTIONS = `Inspect only the supplied evidence. Judge only the selected criterion. Never infer a finding from another image or another criterion. Use the exact state_id, image_id, and region_id supplied for grounding. Stay silent when the criterion does not fire.`; +export const SHARED_INSTRUCTIONS = `Inspect only the supplied evidence. Judge only the selected criterion. Never infer a finding from another image or another criterion. Use the exact state_id, image_id, and region_id supplied for grounding. Use the truth excerpt to decide whether a visible claim is supported; do not use it alone to invent a violation when the image does not show one. If the selected criterion is not visibly demonstrated, or the supplied evidence supports the composition as acceptable for that criterion, return pass=true with no findings. Stay silent when the criterion does not fire.`; export function corpusContractId(corpus) { return crypto.createHash('sha256').update(canonicalJson(corpus)).digest('hex'); diff --git a/evals/visual-model-policy/run.test.mjs b/evals/visual-model-policy/run.test.mjs index e7a2807..63b81db 100644 --- a/evals/visual-model-policy/run.test.mjs +++ b/evals/visual-model-policy/run.test.mjs @@ -11,6 +11,8 @@ import { criterionPromptFor, evaluateProviderResponse, runExperiment, + SHARED_INSTRUCTIONS, + VERDICT_SYSTEM_TEXT, } from './run.mjs'; test('live graduation requires distinct rendered images and source artifacts per label', () => { @@ -184,6 +186,14 @@ Verdict JSON schema: {...}`; assert.doesNotMatch(prompt, /peer tracks drift/); }); +test('verdict prompt defines pass semantics and conservative evidence grounding', () => { + assert.match(VERDICT_SYSTEM_TEXT, /pass=true means the selected criterion does not fire/); + assert.match(VERDICT_SYSTEM_TEXT, /Do not invent a finding/); + assert.match(SHARED_INSTRUCTIONS, /truth excerpt to decide whether a visible claim is supported/); + assert.match(SHARED_INSTRUCTIONS, /supplied evidence supports the composition as acceptable/); + assert.doesNotMatch(SHARED_INSTRUCTIONS, /asymmetry is intentional/); +}); + test('the staged ladder runs one candidate and wraps randomized tails into full batches', () => { const cases = ['a', 'b', 'c', 'd', 'e'].map((id) => ({ case_id: id, diff --git a/plugins/visual-explainer/scripts/verify/rubrics/pass-aesthetic.md b/plugins/visual-explainer/scripts/verify/rubrics/pass-aesthetic.md index 0723438..cff96be 100644 --- a/plugins/visual-explainer/scripts/verify/rubrics/pass-aesthetic.md +++ b/plugins/visual-explainer/scripts/verify/rubrics/pass-aesthetic.md @@ -5,10 +5,10 @@ Inputs: - Load only the section for the active preset plus `Any named preset`. Questions: -- Any named preset: [preset-both-mode-visual] Do both modes render correctly with no invisible text, no wrong-mode background or fill bleeding through, and no element that clearly failed to invert? -- Mono-Industrial: [mono-status-value-judgment] Does every colored element mark a specific value or datum with genuine ok/warn/error meaning, rather than tinting a container, row, label, or decoration? +- Any named preset: [preset-both-mode-visual] Do both modes render correctly with no invisible text, no wrong-mode background or fill bleeding through, and no element that clearly failed to invert? A load-bearing icon disappearing into a dark surface is a violation even when its adjacent label remains readable; a non-semantic watermark may fade. +- Mono-Industrial: [mono-status-value-judgment] Does every colored element mark a specific value or datum with genuine ok/warn/error meaning, rather than tinting a container, row, label, or decoration? A single colored warning/error value with a neutral surrounding container is a pass. - Mono-Industrial: [mono-one-surprise] Does exactly one element deliberately break the pattern, and is that break typographic or compositional rather than color, icon, or gradient? -- Mono-Industrial: [mono-three-layer-squint] When squinted at, do exactly three legible levels of emphasis exist, and does the hero stay intact and full-width on mobile rather than shrinking into a card? +- Mono-Industrial: [mono-three-layer-squint] When squinted at, do exactly three legible levels of emphasis exist, and does the hero stay intact and full-width on mobile rather than shrinking into a card? Count major hierarchy levels, not every ordinary card, label, or supporting sentence; five equally loud hero/emphasis layers is a violation, while three clearly resolved levels is a pass. - Nothing: [nothing-accent-red-judgment] Is the single red-accent use an urgent, destructive, or error signal rather than decorative? - Nothing: [nothing-single-grid-break] Does exactly one element deliberately break the grid, excluding legitimate full-width chrome such as nav, footer, or section dividers? - Demo embed: [demo-aesthetic-match] Does the embedded demo frame match the page preset in corner radius, border, shadow, and caption chrome, and feel embedded rather than glued on? diff --git a/plugins/visual-explainer/scripts/verify/rubrics/pass-diagram.md b/plugins/visual-explainer/scripts/verify/rubrics/pass-diagram.md index 12f6e83..7f4426d 100644 --- a/plugins/visual-explainer/scripts/verify/rubrics/pass-diagram.md +++ b/plugins/visual-explainer/scripts/verify/rubrics/pass-diagram.md @@ -5,9 +5,9 @@ Inputs: - One-line content brief per figure. Questions: -- [diagram-legend-matches-figure] Does every legend entry correspond to something drawn in the figure, and does every distinct visual category in the figure appear in the legend? +- [diagram-legend-matches-figure] Does every legend entry correspond to something drawn in the figure, and does every distinct visual category in the figure appear in the legend? Compare category identity by the visible labels and symbols; do not require equal styling, equal emphasis, or a special muted treatment when every category is present in both the figure and legend. - [diagram-focal-single-dominant] Is there exactly one clearly dominant focal element, with no missing focal point and no two-or-more elements competing equally for primary attention? -- [diagram-proportional-honesty-visual] Do relative sizes and spacings visually match stated quantities, percentages, counts, or dates? +- [diagram-proportional-honesty-visual] Do relative sizes and spacings visually match stated quantities, percentages, counts, or dates? For an explicitly scaled comparison, verify the visible geometry against the stated ratio. Correctly proportional bars are a pass even without a drawn axis; do not flag them merely because the containers are unequal or because the chart is sparse. - [diagram-type-coherent] Does the figure read as one coherent diagram type matching its content, and if Mermaid was used, was it warranted by 15+ nodes, a uniform-shape forest, or an explicit request? - [diagram-removal-simplicity] Does every node, label, and line carry information, with no removable padding, generic equal cards, or all-identical boxes that erase hierarchy? - [diagram-necessity] Could the content be conveyed just as well by a short 3-column table, bulleted list, or single sentence? diff --git a/plugins/visual-explainer/scripts/verify/rubrics/pass-layout.md b/plugins/visual-explainer/scripts/verify/rubrics/pass-layout.md index 26f4532..ef078fc 100644 --- a/plugins/visual-explainer/scripts/verify/rubrics/pass-layout.md +++ b/plugins/visual-explainer/scripts/verify/rubrics/pass-layout.md @@ -6,12 +6,12 @@ Inputs: Questions: - [real-table-for-tabular-data] Does this div-grid present row/column tabular data such as a comparison, audit, or status matrix that should be a semantic `
`, rather than a card grid, KPI row, or feature grid? - [hierarchy-squint-test] In the blurred 1440x900 light view, is there a clear dominant focal region with quieter subordinate regions, or does the page read uniformly flat with no visual hierarchy? -- [text-visibly-clipped] In the 390x844 light and dark screenshots, is any real content text visibly cut off by a container edge, with characters chopped or a sentence truncated without intentional ellipsis? +- [text-visibly-clipped] In the 390x844 light and dark screenshots, is any real content text visibly cut off by a container edge, with characters chopped or a sentence truncated without intentional ellipsis? Do not flag a narrow but fully readable label, a scrollable table whose content is not visibly chopped, or ordinary responsive wrapping. - [fixed-ui-obscures-content] In the 390x844 screenshots, does fixed-position chrome such as a theme toggle, nav, or counter hide readable text or block a clickable target? - [slide-single-focal-point] For each applicable slide screenshot, does the slide present one clear focal element, or do three or more elements compete for attention? -- [layout-repeated-track-symmetry] Only when a composition visibly uses repeated columns or rows: do equivalent tracks have balanced widths, aligned dividers/baselines, and intentional whitespace? Flag unexplained drift that makes a peer region look missing or subordinate. Stay silent when the asymmetry is visibly intentional and supported by content hierarchy. This is a registered rubric criterion under the `layout` pass, not an always-on catalog scan. +- [layout-repeated-track-symmetry] Only when a composition visibly uses repeated columns or rows: do equivalent tracks have balanced widths, aligned dividers/baselines, and intentional whitespace? When the supplied truth says rows are direct/equivalent counterparts, flag visible baseline or divider drift even if the cards look otherwise intentional. Stay silent when the truth identifies an evidence rail, compact control cluster, independent editorial items, or another supported hierarchy; do not treat those content-driven widths as a symmetry violation merely because peers are unequal. This is a registered rubric criterion under the `layout` pass, not an always-on catalog scan. - [composition-variety-visual] In the flagged consecutive slide screenshot strip, do the slides look spatially repetitive, with the same alignment and whitespace balance, rather than genuinely varied? -- [sparse-diagram-slide] In the flagged slide screenshot, does the slide read as sparse or empty, such as a tiny diagram floating in a large viewport with no supporting content? +- [sparse-diagram-slide] In the flagged slide screenshot, does the slide read as sparse or empty, such as a tiny diagram floating in a large viewport with no supporting content? Do not flag intentional reading space when the supplied evidence identifies an evidence rail or subordinate control area. Verdict JSON schema: `{"pass":true,"findings":[{"check_id":"","evidence":"390-dark screenshot: ...","fix":"..."}]}` diff --git a/plugins/visual-explainer/scripts/verify/rubrics/pass-operating-model.md b/plugins/visual-explainer/scripts/verify/rubrics/pass-operating-model.md index 596e0aa..9d0c85d 100644 --- a/plugins/visual-explainer/scripts/verify/rubrics/pass-operating-model.md +++ b/plugins/visual-explainer/scripts/verify/rubrics/pass-operating-model.md @@ -23,6 +23,7 @@ Routing rules: Questions for units routed `relational`, `operating-model`, or `simulated-surface`: - [operating-model-fit] Does the composition faithfully expose the relationships the viewer needs for the unit's narrative job? +- For source-conditioned claims such as equations, comparisons, or provenance, the supplied source brief is decisive context: if it explicitly defines the visible relationship, pass it; if it says the relationship is unsupported, flag the visible claim even when the pixels are identical. - Does the visual grammar make a truthful claim? Cards imply independent peers; a pipeline implies ordered transformation; a funnel implies attrition; a tree implies hierarchy or prerequisites; a graph implies meaningful relationships; a control plane implies policy routing; a trace implies sequence and provenance. - Can the viewer reconstruct the relevant comparison, sequence, routing rule, state change, evidence path, or dependency without presenter narration? - Does each load-bearing position, connector, line style, size, color, or animation have one stable meaning?