From d4d6f005c5a7bba087ec8240415f6ea8fd62aaef Mon Sep 17 00:00:00 2001 From: Mandeep Singh Date: Mon, 20 Jul 2026 20:22:17 +0530 Subject: [PATCH] fix(stardust): migrate eval criteria.json to tessl weighted_checklist schema The Release Skills workflow's `tessl plugin publish` step began enforcing the new criteria.json schema between 2026-07-04 and 2026-07-07, failing the stardust plugin publish with: Invalid criteria.json (in migrate-self-contained-bundle): context: expected string, received undefined type: expected "weighted_checklist" checklist: expected array, received undefined All 9 stardust eval criteria.json still used the legacy {name, total, criteria:[{id, weight, description}]} shape. Convert them to the {context, type: "weighted_checklist", checklist:[{name, max_score, description}]} shape the other Adobe-skills plugins already use. - id -> name (identifier preserved, lossless) - weight -> max_score (sums to 100 per eval, unchanged) - description kept verbatim - context authored per eval from the README coverage table - README Format/Running sections updated to describe the new shape Item counts and weight sums verified unchanged for all 9 evals. Co-Authored-By: Claude Opus 4.8 (1M context) --- plugins/stardust/evals/README.md | 21 +-- .../evals/direct-from-phrase/criteria.json | 78 +++++++-- .../evals/extract-multipage/criteria.json | 84 ++++++++-- .../intent-reasoning-style/criteria.json | 66 ++++++-- .../evals/migrate-incremental/criteria.json | 96 ++++++++--- .../migrate-multi-template/criteria.json | 150 ++++++++++++++---- .../criteria.json | 144 ++++++++++++++--- .../prototype-before-after/criteria.json | 78 +++++++-- .../replica-source-fidelity/criteria.json | 67 ++++++-- .../reskin-content-fidelity/criteria.json | 67 ++++++-- 10 files changed, 685 insertions(+), 166 deletions(-) diff --git a/plugins/stardust/evals/README.md b/plugins/stardust/evals/README.md index 6f6ca390..8a45c34f 100644 --- a/plugins/stardust/evals/README.md +++ b/plugins/stardust/evals/README.md @@ -46,14 +46,15 @@ Each eval lives in its own directory and contains exactly two files: - `task.md` — Setup, User prompt, Expected behavior. Human-readable scenario specification. -- `criteria.json` — Weighted scoring rubric. Each criterion has an - `id`, a `weight`, and a `description`. `total` should equal the - sum of weights. Used by the eval runner to score the agent's - output. +- `criteria.json` — Weighted scoring rubric in the tessl + `weighted_checklist` shape. Top level is `{ "context", "type": + "weighted_checklist", "checklist" }`; each checklist item has a + `name`, a `max_score`, and a `description`. The `max_score` values + sum to 100 per eval. Used by the eval runner (and enforced by + `tessl plugin publish`) to score the agent's output. -This format mirrors v1's structure (and the format other Adobe-skills -plugins use), so the eval runner that worked for v1 should work for -v2 evals without modification. +This format matches the one the other Adobe-skills plugins use, so the +shared eval runner scores these evals without modification. ## Evals in this suite @@ -100,9 +101,9 @@ in adobe/skills). Each eval is self-describing: a runner reads `task.md` for the scenario, executes it against a clean stardust project, and scores the output against `criteria.json`. -A criterion passes if its `description` is satisfied as judged by -the runner. Per-criterion verdicts are combined as a weighted sum -out of `total` (100 per eval). +A checklist item passes if its `description` is satisfied as judged by +the runner. Per-item verdicts are combined as a weighted sum of +`max_score` (100 per eval). ## What stardust v2 evals deliberately do NOT test diff --git a/plugins/stardust/evals/direct-from-phrase/criteria.json b/plugins/stardust/evals/direct-from-phrase/criteria.json index e31df9dc..763197c8 100644 --- a/plugins/stardust/evals/direct-from-phrase/criteria.json +++ b/plugins/stardust/evals/direct-from-phrase/criteria.json @@ -1,18 +1,66 @@ { - "name": "direct-from-phrase", - "total": 100, - "criteria": [ - { "id": "activated", "weight": 5, "description": "The stardust:direct skill was invoked with the freeform phrase as input." }, - { "id": "dimensional_restatement", "weight": 10, "description": "The agent restated the phrase in stardust's dimensional vocabulary (register / expressive axis / tone / density / distinctiveness / audience / constraints) before doing anything else. Each axis is named and either marked as moved (with direction), pinned, or left alone." }, - { "id": "gaps_identified", "weight": 5, "description": "The agent explicitly identified what's underspecified before asking questions. 'Young' as too coarse (or equivalent) was called out." }, - { "id": "question_ceiling", "weight": 10, "description": "At most TWO clarifying questions were asked. Each had concrete options + an 'other' / 'skip' escape hatch and cited which dimension it resolved." }, - { "id": "plan_shown_before_execution","weight": 15, "description": "After answers (or with no questions needed), the agent showed the resolved plan to the user BEFORE running any impeccable command or writing any file. Plan included: one-sentence restatement, assumptions, command sequence with reasoning, pages affected." }, - { "id": "divergence_resolved", "weight": 10, "description": "Divergence inputs resolved: 4-dim seed (decade x craft x register x ground-family), font deck pick, palette resolution. Recorded in DESIGN.json.extensions.divergence per the v2 storage shape." }, - { "id": "product_md_direct", "weight": 10, "description": "PRODUCT.md authored directly at project root using impeccable's teach.md as format spec. The agent did NOT invoke $impeccable teach. Sections present: Register, Users, Product Purpose, Brand Personality, Anti-references, Design Principles, Accessibility & Inclusion." }, - { "id": "design_md_direct", "weight": 10, "description": "DESIGN.md and DESIGN.json authored directly at project root. DESIGN.md uses Stitch frontmatter + 6 canonical sections. DESIGN.json schemaVersion 2 with extensions (divergence, componentStyle, voice) and narrative blocks." }, - { "id": "direction_md_shape", "weight": 10, "description": "stardust/direction.md exists with provenance + YAML frontmatter + # Active direction section containing required sub-sections: Phrase, Restatement, Movements, Divergence inputs, Command sequence (proposed), User confirmation, Pages in scope. Anti-references explicitly written (or '(none)')." }, - { "id": "state_updated", "weight": 5, "description": "stardust/state.json updated: direction.resolvedAt set, direction.phrase verbatim, in-scope pages move from 'extracted' to 'directed' with history entries." }, - { "id": "no_silent_command_mapping", "weight": 5, "description": "The agent did NOT silently map the phrase to a fixed command lookup. Reasoning was visible (dimensional vocabulary used, command choices justified)." }, - { "id": "no_eds_references", "weight": 5, "description": "No mention of EDS, AEM, dev servers, or framework targets." } + "context": "Phase 2 (direct): tests dimensional restatement of a freeform phrase, at most two clarifying questions, plan-before-execution, direct authoring of the target spec, and the direction.md trace.", + "type": "weighted_checklist", + "checklist": [ + { + "name": "activated", + "max_score": 5, + "description": "The stardust:direct skill was invoked with the freeform phrase as input." + }, + { + "name": "dimensional_restatement", + "max_score": 10, + "description": "The agent restated the phrase in stardust's dimensional vocabulary (register / expressive axis / tone / density / distinctiveness / audience / constraints) before doing anything else. Each axis is named and either marked as moved (with direction), pinned, or left alone." + }, + { + "name": "gaps_identified", + "max_score": 5, + "description": "The agent explicitly identified what's underspecified before asking questions. 'Young' as too coarse (or equivalent) was called out." + }, + { + "name": "question_ceiling", + "max_score": 10, + "description": "At most TWO clarifying questions were asked. Each had concrete options + an 'other' / 'skip' escape hatch and cited which dimension it resolved." + }, + { + "name": "plan_shown_before_execution", + "max_score": 15, + "description": "After answers (or with no questions needed), the agent showed the resolved plan to the user BEFORE running any impeccable command or writing any file. Plan included: one-sentence restatement, assumptions, command sequence with reasoning, pages affected." + }, + { + "name": "divergence_resolved", + "max_score": 10, + "description": "Divergence inputs resolved: 4-dim seed (decade x craft x register x ground-family), font deck pick, palette resolution. Recorded in DESIGN.json.extensions.divergence per the v2 storage shape." + }, + { + "name": "product_md_direct", + "max_score": 10, + "description": "PRODUCT.md authored directly at project root using impeccable's teach.md as format spec. The agent did NOT invoke $impeccable teach. Sections present: Register, Users, Product Purpose, Brand Personality, Anti-references, Design Principles, Accessibility & Inclusion." + }, + { + "name": "design_md_direct", + "max_score": 10, + "description": "DESIGN.md and DESIGN.json authored directly at project root. DESIGN.md uses Stitch frontmatter + 6 canonical sections. DESIGN.json schemaVersion 2 with extensions (divergence, componentStyle, voice) and narrative blocks." + }, + { + "name": "direction_md_shape", + "max_score": 10, + "description": "stardust/direction.md exists with provenance + YAML frontmatter + # Active direction section containing required sub-sections: Phrase, Restatement, Movements, Divergence inputs, Command sequence (proposed), User confirmation, Pages in scope. Anti-references explicitly written (or '(none)')." + }, + { + "name": "state_updated", + "max_score": 5, + "description": "stardust/state.json updated: direction.resolvedAt set, direction.phrase verbatim, in-scope pages move from 'extracted' to 'directed' with history entries." + }, + { + "name": "no_silent_command_mapping", + "max_score": 5, + "description": "The agent did NOT silently map the phrase to a fixed command lookup. Reasoning was visible (dimensional vocabulary used, command choices justified)." + }, + { + "name": "no_eds_references", + "max_score": 5, + "description": "No mention of EDS, AEM, dev servers, or framework targets." + } ] } diff --git a/plugins/stardust/evals/extract-multipage/criteria.json b/plugins/stardust/evals/extract-multipage/criteria.json index e139869f..df09bf08 100644 --- a/plugins/stardust/evals/extract-multipage/criteria.json +++ b/plugins/stardust/evals/extract-multipage/criteria.json @@ -1,19 +1,71 @@ { - "name": "extract-multipage", - "total": 100, - "criteria": [ - { "id": "activated", "weight": 5, "description": "The stardust:extract skill was invoked." }, - { "id": "impeccable_dep_check", "weight": 5, "description": "Master skill setup ran first; impeccable was confirmed installed before extract proceeded." }, - { "id": "discovery_before_crawl", "weight": 10, "description": "Discovery (sitemap.xml -> sitemap_index -> robots.txt -> BFS crawl) happened before any per-page rendering. The discovered URL list was visible in the agent's narration." }, - { "id": "page_cap_confirmation", "weight": 10, "description": "The default 25-page cap was applied and the user was shown the kept list AND the cut list before crawling proceeded. Confirmation was solicited (no silent crawling beyond the cap)." }, - { "id": "playwright_over_webfetch", "weight": 10, "description": "Playwright was used for per-page rendering (1440x900 @ 2x DPR, networkidle + 1.5s, scroll-to-bottom pass). WebFetch / curl was NOT used as a substitute." }, - { "id": "per_page_json_shape", "weight": 10, "description": "Each stardust/current/pages/.json contains all required top-level keys: _provenance (first key), slug, url, finalUrl, title, og, themeColor, headings, landmarks, ctas, links, media, forms, widgets, perSectionStyle." }, - { "id": "brand_extraction_shape", "weight": 10, "description": "stardust/current/_brand-extraction.json exists with all sections: logo, palette (with role names + occurrences + sourceSelectors), type (heading + body families with weights / sizes), spacing, motifs (borderRadius / shadows / patterns), componentStyle (v1 fields preserved), voice samples, register guess." }, - { "id": "logo_locator_chain", "weight": 5, "description": "Logo extracted via the priority chain. _brand-extraction.json.logo.source names the method used (inline-svg | img | apple-touch-icon | og-image | favicon | synthesized). File saved under stardust/current/assets/ (NOT icons/)." }, - { "id": "current_product_md_direct", "weight": 10, "description": "stardust/current/PRODUCT.md was authored directly (sections per impeccable's teach.md format spec). The agent did NOT invoke $impeccable teach for the current-state file." }, - { "id": "current_design_md_direct", "weight": 10, "description": "stardust/current/DESIGN.md and DESIGN.json were authored directly. DESIGN.md uses Stitch frontmatter (colors, typography, rounded, spacing, components) and the 6 canonical sections. DESIGN.json schemaVersion 2 with extensions block." }, - { "id": "state_json_shape", "weight": 5, "description": "stardust/state.json exists with _provenance first, site (originUrl/extractedAt/pageCap/totalDiscovered/crawled), direction (resolvedAt null at this stage), and pages[] (one entry per crawled page, status=extracted)." }, - { "id": "provenance_stamped", "weight": 5, "description": "Every artifact carries a stardust:provenance block per artifact-map.md (first child for HTML, first line above frontmatter for markdown, _provenance first key for JSON)." }, - { "id": "no_eds_references", "weight": 5, "description": "No mention of EDS, AEM, localhost:3000, dev servers, or framework targets in outputs or narration." } + "context": "Phase 1 (extract): tests a multi-page crawl with a cap using Playwright (not WebFetch), correct output file shapes, and direct authoring of the current PRODUCT.md / DESIGN.md.", + "type": "weighted_checklist", + "checklist": [ + { + "name": "activated", + "max_score": 5, + "description": "The stardust:extract skill was invoked." + }, + { + "name": "impeccable_dep_check", + "max_score": 5, + "description": "Master skill setup ran first; impeccable was confirmed installed before extract proceeded." + }, + { + "name": "discovery_before_crawl", + "max_score": 10, + "description": "Discovery (sitemap.xml -> sitemap_index -> robots.txt -> BFS crawl) happened before any per-page rendering. The discovered URL list was visible in the agent's narration." + }, + { + "name": "page_cap_confirmation", + "max_score": 10, + "description": "The default 25-page cap was applied and the user was shown the kept list AND the cut list before crawling proceeded. Confirmation was solicited (no silent crawling beyond the cap)." + }, + { + "name": "playwright_over_webfetch", + "max_score": 10, + "description": "Playwright was used for per-page rendering (1440x900 @ 2x DPR, networkidle + 1.5s, scroll-to-bottom pass). WebFetch / curl was NOT used as a substitute." + }, + { + "name": "per_page_json_shape", + "max_score": 10, + "description": "Each stardust/current/pages/.json contains all required top-level keys: _provenance (first key), slug, url, finalUrl, title, og, themeColor, headings, landmarks, ctas, links, media, forms, widgets, perSectionStyle." + }, + { + "name": "brand_extraction_shape", + "max_score": 10, + "description": "stardust/current/_brand-extraction.json exists with all sections: logo, palette (with role names + occurrences + sourceSelectors), type (heading + body families with weights / sizes), spacing, motifs (borderRadius / shadows / patterns), componentStyle (v1 fields preserved), voice samples, register guess." + }, + { + "name": "logo_locator_chain", + "max_score": 5, + "description": "Logo extracted via the priority chain. _brand-extraction.json.logo.source names the method used (inline-svg | img | apple-touch-icon | og-image | favicon | synthesized). File saved under stardust/current/assets/ (NOT icons/)." + }, + { + "name": "current_product_md_direct", + "max_score": 10, + "description": "stardust/current/PRODUCT.md was authored directly (sections per impeccable's teach.md format spec). The agent did NOT invoke $impeccable teach for the current-state file." + }, + { + "name": "current_design_md_direct", + "max_score": 10, + "description": "stardust/current/DESIGN.md and DESIGN.json were authored directly. DESIGN.md uses Stitch frontmatter (colors, typography, rounded, spacing, components) and the 6 canonical sections. DESIGN.json schemaVersion 2 with extensions block." + }, + { + "name": "state_json_shape", + "max_score": 5, + "description": "stardust/state.json exists with _provenance first, site (originUrl/extractedAt/pageCap/totalDiscovered/crawled), direction (resolvedAt null at this stage), and pages[] (one entry per crawled page, status=extracted)." + }, + { + "name": "provenance_stamped", + "max_score": 5, + "description": "Every artifact carries a stardust:provenance block per artifact-map.md (first child for HTML, first line above frontmatter for markdown, _provenance first key for JSON)." + }, + { + "name": "no_eds_references", + "max_score": 5, + "description": "No mention of EDS, AEM, localhost:3000, dev servers, or framework targets in outputs or narration." + } ] } diff --git a/plugins/stardust/evals/intent-reasoning-style/criteria.json b/plugins/stardust/evals/intent-reasoning-style/criteria.json index 752dd821..5088cd65 100644 --- a/plugins/stardust/evals/intent-reasoning-style/criteria.json +++ b/plugins/stardust/evals/intent-reasoning-style/criteria.json @@ -1,16 +1,56 @@ { - "name": "intent-reasoning-style", - "total": 100, - "criteria": [ - { "id": "routed_to_freeform_branch", "weight": 5, "description": "The phrase 'I want it to be amazing' was routed to the master skill's freeform-phrase branch (intent-reasoning.md procedure) rather than directly to a sub-command." }, - { "id": "dimensional_restatement", "weight": 15, "description": "The agent restated the phrase in the dimensional vocabulary (intent-dimensions.md) before doing anything else. The restatement explicitly identified that the phrase moves NO directional axis." }, - { "id": "amazing_vs_better_distinguished","weight": 10, "description": "The agent distinguished 'amazing' from 'better': 'better' has a default meaning (impeccable critique + audit pass), 'amazing' does not. Asking-then-stopping is the correct path for 'amazing'." }, - { "id": "question_ceiling", "weight": 15, "description": "At MOST two clarifying questions asked. Each was high-leverage (asking it changes the plan), each had concrete options + an 'other' / 'skip' escape hatch, each cited which dimension or gap it resolves." }, - { "id": "no_silent_command_mapping", "weight": 15, "description": "The agent did NOT silently pick a command sequence. It did NOT fall back to 'make it better' defaults. It did NOT propose a multi-command sequence ('bolder + colorize + typeset + ...') that 'covers all bases' to avoid asking." }, - { "id": "no_premature_authoring", "weight": 15, "description": "PRODUCT.md, DESIGN.md, DESIGN.json, and a complete direction.md were NOT written. The agent refused to author tokens from incomplete reasoning." }, - { "id": "pending_direction_persisted","weight": 10, "description": "The agent persisted the partial reasoning so resuming is possible: a # Pending direction () section in stardust/direction.md with at least Phrase, Reasoning so far, Open questions. (Optional but expected.)" }, - { "id": "downstream_block_explained","weight": 5, "description": "The agent made explicit (in narration or in the pending direction's open questions) that prototype and migrate refuse to run while direction is pending." }, - { "id": "reasoning_visible", "weight": 5, "description": "The reasoning was VISIBLE in the agent's output. The user can read what dimensions were considered, what was found missing, and why each question was asked. No magic." }, - { "id": "no_eds_references", "weight": 5, "description": "No mention of EDS, AEM, dev servers, framework targets." } + "context": "Master-skill principle (open and reasoned): tests that vague phrases get clarified rather than silently mapped to commands, and that pending direction is persisted.", + "type": "weighted_checklist", + "checklist": [ + { + "name": "routed_to_freeform_branch", + "max_score": 5, + "description": "The phrase 'I want it to be amazing' was routed to the master skill's freeform-phrase branch (intent-reasoning.md procedure) rather than directly to a sub-command." + }, + { + "name": "dimensional_restatement", + "max_score": 15, + "description": "The agent restated the phrase in the dimensional vocabulary (intent-dimensions.md) before doing anything else. The restatement explicitly identified that the phrase moves NO directional axis." + }, + { + "name": "amazing_vs_better_distinguished", + "max_score": 10, + "description": "The agent distinguished 'amazing' from 'better': 'better' has a default meaning (impeccable critique + audit pass), 'amazing' does not. Asking-then-stopping is the correct path for 'amazing'." + }, + { + "name": "question_ceiling", + "max_score": 15, + "description": "At MOST two clarifying questions asked. Each was high-leverage (asking it changes the plan), each had concrete options + an 'other' / 'skip' escape hatch, each cited which dimension or gap it resolves." + }, + { + "name": "no_silent_command_mapping", + "max_score": 15, + "description": "The agent did NOT silently pick a command sequence. It did NOT fall back to 'make it better' defaults. It did NOT propose a multi-command sequence ('bolder + colorize + typeset + ...') that 'covers all bases' to avoid asking." + }, + { + "name": "no_premature_authoring", + "max_score": 15, + "description": "PRODUCT.md, DESIGN.md, DESIGN.json, and a complete direction.md were NOT written. The agent refused to author tokens from incomplete reasoning." + }, + { + "name": "pending_direction_persisted", + "max_score": 10, + "description": "The agent persisted the partial reasoning so resuming is possible: a # Pending direction () section in stardust/direction.md with at least Phrase, Reasoning so far, Open questions. (Optional but expected.)" + }, + { + "name": "downstream_block_explained", + "max_score": 5, + "description": "The agent made explicit (in narration or in the pending direction's open questions) that prototype and migrate refuse to run while direction is pending." + }, + { + "name": "reasoning_visible", + "max_score": 5, + "description": "The reasoning was VISIBLE in the agent's output. The user can read what dimensions were considered, what was found missing, and why each question was asked. No magic." + }, + { + "name": "no_eds_references", + "max_score": 5, + "description": "No mention of EDS, AEM, dev servers, framework targets." + } ] } diff --git a/plugins/stardust/evals/migrate-incremental/criteria.json b/plugins/stardust/evals/migrate-incremental/criteria.json index 557b494f..79cda8c5 100644 --- a/plugins/stardust/evals/migrate-incremental/criteria.json +++ b/plugins/stardust/evals/migrate-incremental/criteria.json @@ -1,21 +1,81 @@ { - "name": "migrate-incremental", - "total": 100, - "criteria": [ - { "id": "activated", "weight": 5, "description": "The stardust:migrate skill was invoked." }, - { "id": "plan_printed", "weight": 5, "description": "The plan was printed before any writes: in-scope count partitioned by render path (A: approved-from-prototype; B: directed-no-prototype), DESIGN.md and DESIGN.json shas, output destination, idempotent-skip note." }, - { "id": "path_a_used_for_approved", "weight": 10, "description": "Approved pages (home, pricing) used render path A: read the proposed file's body verbatim, refreshed only the :root block from latest DESIGN.md. Provenance renderPath = 'approved-from-prototype'." }, - { "id": "path_b_used_for_directed", "weight": 10, "description": "Directed-but-not-prototyped pages (about, features, contact) used render path B: rendered from scratch using current/pages/.json IA + DESIGN.json components. Provenance renderPath = 'directed-no-prototype'." }, - { "id": "output_path_mapping", "weight": 10, "description": "Nested index.html structure: migrated/index.html for home, migrated//index.html for others, multi-segment slugs (e.g. docs__api) nest as docs/api/index.html. Verified for at least the 5 pages in scope." }, - { "id": "root_block_refreshed", "weight": 10, "description": "Every migrated page's :root block matches the LATEST DESIGN.md tokens via the mapping table in token-contract.md." }, - { "id": "data_attributes_preserved", "weight": 5, "description": "Every
has data-section, data-intent, data-layout. Path A preserved them from the proposed file; path B applied them per the landmark purpose heuristic." }, - { "id": "content_preserved", "weight": 10, "description": "Every page's headlines, body copy, CTA labels, nav labels, and link destinations match the source (current/pages/.json or proposed.html). The redesign restyles, it does not rewrite." }, - { "id": "internal_links_rewritten", "weight": 5, "description": "Internal links (same host as state.json.site.originUrl) rewritten to root-relative paths matching the migrated tree. External links and mailto:/tel: pass through unchanged. Missing targets get data-broken-link='true' + a provenance log entry." }, - { "id": "assets_copied", "weight": 5, "description": "stardust/migrated/assets/ contains the logo and the referenced media files (only files referenced by migrated pages). Asset paths in HTML rewritten to /assets/..." }, - { "id": "robots_and_sitemap", "weight": 3, "description": "stardust/migrated/robots.txt and stardust/migrated/sitemap.xml are present and derived from the migrated inventory." }, - { "id": "provenance_complete", "weight": 5, "description": "Each page's provenance includes designMd sha, designJson sha, sourceCurrent sha, sourceProposed sha (path A only), renderPath, againstDirection, divergenceVersion." }, - { "id": "state_marked_migrated", "weight": 4, "description": "state.json: all 5 pages moved to status 'migrated' after run 1, with history entries appended and any stale flags cleared." }, - { "id": "idempotent_skip_run2", "weight": 10, "description": "Run 2 (no changes between runs) produced zero file writes. Every page sha-compared and skipped. Summary: 0 migrated, 5 unchanged. state.json NOT updated for unchanged pages." }, - { "id": "no_eds_references", "weight": 3, "description": "No mention of EDS, AEM, framework component generation, or production CMS payload. Output is platform-agnostic static HTML only." } + "context": "Phase 4 (migrate): tests both render paths (A and B), nested index.html output, content preservation, and the idempotent skip on re-run.", + "type": "weighted_checklist", + "checklist": [ + { + "name": "activated", + "max_score": 5, + "description": "The stardust:migrate skill was invoked." + }, + { + "name": "plan_printed", + "max_score": 5, + "description": "The plan was printed before any writes: in-scope count partitioned by render path (A: approved-from-prototype; B: directed-no-prototype), DESIGN.md and DESIGN.json shas, output destination, idempotent-skip note." + }, + { + "name": "path_a_used_for_approved", + "max_score": 10, + "description": "Approved pages (home, pricing) used render path A: read the proposed file's body verbatim, refreshed only the :root block from latest DESIGN.md. Provenance renderPath = 'approved-from-prototype'." + }, + { + "name": "path_b_used_for_directed", + "max_score": 10, + "description": "Directed-but-not-prototyped pages (about, features, contact) used render path B: rendered from scratch using current/pages/.json IA + DESIGN.json components. Provenance renderPath = 'directed-no-prototype'." + }, + { + "name": "output_path_mapping", + "max_score": 10, + "description": "Nested index.html structure: migrated/index.html for home, migrated//index.html for others, multi-segment slugs (e.g. docs__api) nest as docs/api/index.html. Verified for at least the 5 pages in scope." + }, + { + "name": "root_block_refreshed", + "max_score": 10, + "description": "Every migrated page's :root block matches the LATEST DESIGN.md tokens via the mapping table in token-contract.md." + }, + { + "name": "data_attributes_preserved", + "max_score": 5, + "description": "Every
has data-section, data-intent, data-layout. Path A preserved them from the proposed file; path B applied them per the landmark purpose heuristic." + }, + { + "name": "content_preserved", + "max_score": 10, + "description": "Every page's headlines, body copy, CTA labels, nav labels, and link destinations match the source (current/pages/.json or proposed.html). The redesign restyles, it does not rewrite." + }, + { + "name": "internal_links_rewritten", + "max_score": 5, + "description": "Internal links (same host as state.json.site.originUrl) rewritten to root-relative paths matching the migrated tree. External links and mailto:/tel: pass through unchanged. Missing targets get data-broken-link='true' + a provenance log entry." + }, + { + "name": "assets_copied", + "max_score": 5, + "description": "stardust/migrated/assets/ contains the logo and the referenced media files (only files referenced by migrated pages). Asset paths in HTML rewritten to /assets/..." + }, + { + "name": "robots_and_sitemap", + "max_score": 3, + "description": "stardust/migrated/robots.txt and stardust/migrated/sitemap.xml are present and derived from the migrated inventory." + }, + { + "name": "provenance_complete", + "max_score": 5, + "description": "Each page's provenance includes designMd sha, designJson sha, sourceCurrent sha, sourceProposed sha (path A only), renderPath, againstDirection, divergenceVersion." + }, + { + "name": "state_marked_migrated", + "max_score": 4, + "description": "state.json: all 5 pages moved to status 'migrated' after run 1, with history entries appended and any stale flags cleared." + }, + { + "name": "idempotent_skip_run2", + "max_score": 10, + "description": "Run 2 (no changes between runs) produced zero file writes. Every page sha-compared and skipped. Summary: 0 migrated, 5 unchanged. state.json NOT updated for unchanged pages." + }, + { + "name": "no_eds_references", + "max_score": 3, + "description": "No mention of EDS, AEM, framework component generation, or production CMS payload. Output is platform-agnostic static HTML only." + } ] } diff --git a/plugins/stardust/evals/migrate-multi-template/criteria.json b/plugins/stardust/evals/migrate-multi-template/criteria.json index a6c08031..bb566e7a 100644 --- a/plugins/stardust/evals/migrate-multi-template/criteria.json +++ b/plugins/stardust/evals/migrate-multi-template/criteria.json @@ -1,30 +1,126 @@ { - "name": "migrate-multi-template", - "total": 100, - "criteria": [ - { "id": "activated", "weight": 2, "description": "The stardust:migrate skill was invoked." }, - { "id": "plan_printed_with_branches", "weight": 4, "description": "Plan printed before any writes: in-scope count partitioned by render branch (Path A approved-from-prototype, Path A' template-applied from sibling, Path B unique-render), DESIGN.md / DESIGN.json / canon shas, output destination, idempotent-skip note." }, - { "id": "path_a_render", "weight": 4, "description": "Approved pages (home, news/post-housing-summit, news) used Path A: proposed.html body verbatim, :root refreshed from latest DESIGN.md (canon.pinned values winning for pinned tokens), canon.css injected. Provenance renderBranch = 'A'." }, - { "id": "path_a_prime_render", "weight": 12, "description": "Directed pages whose type matches an approved sibling (18 article, 2 program, 1 form) used Path A': forked the matching archetype's structure from prototypes/-proposed.html, injected this page's typed slots from current/pages/.json § slots, applied canon. Provenance renderBranch = 'A''." }, - { "id": "path_b_unique_render", "weight": 5, "description": "The unique-typed page (404) used Path B: composed from DESIGN.md/json tokens + canon header/footer + module catalog + canon.css. Logged a unique-render decision in migrationDecisions[]. Provenance renderBranch = 'B'." }, - { "id": "data_attributes_full_set", "weight": 4, "description": "Required data-* vocabulary applied: data-template on body/main matching page type; data-section on every
; data-module on each module instance; data-slot on slot containers locally scoped to nearest data-template/data-module parent. v2.1 additive set per skills/stardust/reference/data-attributes.md." }, - { "id": "canon_chrome_applied", "weight": 5, "description": "Header and footer emitted from stardust/canon/header.html and stardust/canon/footer.html verbatim. Container elements carry data-canon flag. Identical bytes across all 25 migrated pages (canon-author home and the 24 templates + unique pages all share the same chrome)." }, - { "id": "canon_css_injected", "weight": 1, "description": "Contents of stardust/canon/canon.css injected as the second block in each page's first