Skip to content

fix(stardust): migrate eval criteria.json to tessl weighted_checklist… #267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
mandeep294 wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

fix(stardust): migrate eval criteria.json to tessl weighted_checklist… #267

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 11 additions & 10 deletions plugins/stardust/evals/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,14 +46,15 @@ Each eval lives in its own directory and contains exactly two files:

- `task.md` — Setup, User prompt, Expected behavior. Human-readable
scenario specification.
- `criteria.json` — Weighted scoring rubric. Each criterion has an
`id`, a `weight`, and a `description`. `total` should equal the
sum of weights. Used by the eval runner to score the agent's
output.
- `criteria.json` — Weighted scoring rubric in the tessl
`weighted_checklist` shape. Top level is `{ "context", "type":
"weighted_checklist", "checklist" }`; each checklist item has a
`name`, a `max_score`, and a `description`. The `max_score` values
sum to 100 per eval. Used by the eval runner (and enforced by
`tessl plugin publish`) to score the agent's output.

This format mirrors v1's structure (and the format other Adobe-skills
plugins use), so the eval runner that worked for v1 should work for
v2 evals without modification.
This format matches the one the other Adobe-skills plugins use, so the
shared eval runner scores these evals without modification.

## Evals in this suite

Expand Down Expand Up @@ -100,9 +101,9 @@ in adobe/skills). Each eval is self-describing: a runner reads
`task.md` for the scenario, executes it against a clean stardust
project, and scores the output against `criteria.json`.

A criterion passes if its `description` is satisfied as judged by
the runner. Per-criterion verdicts are combined as a weighted sum
out of `total` (100 per eval).
A checklist item passes if its `description` is satisfied as judged by
the runner. Per-item verdicts are combined as a weighted sum of
`max_score` (100 per eval).

## What stardust v2 evals deliberately do NOT test

Expand Down
78 changes: 63 additions & 15 deletions plugins/stardust/evals/direct-from-phrase/criteria.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,66 @@
{
"name": "direct-from-phrase",
"total": 100,
"criteria": [
{ "id": "activated", "weight": 5, "description": "The stardust:direct skill was invoked with the freeform phrase as input." },
{ "id": "dimensional_restatement", "weight": 10, "description": "The agent restated the phrase in stardust's dimensional vocabulary (register / expressive axis / tone / density / distinctiveness / audience / constraints) before doing anything else. Each axis is named and either marked as moved (with direction), pinned, or left alone." },
{ "id": "gaps_identified", "weight": 5, "description": "The agent explicitly identified what's underspecified before asking questions. 'Young' as too coarse (or equivalent) was called out." },
{ "id": "question_ceiling", "weight": 10, "description": "At most TWO clarifying questions were asked. Each had concrete options + an 'other' / 'skip' escape hatch and cited which dimension it resolved." },
{ "id": "plan_shown_before_execution","weight": 15, "description": "After answers (or with no questions needed), the agent showed the resolved plan to the user BEFORE running any impeccable command or writing any file. Plan included: one-sentence restatement, assumptions, command sequence with reasoning, pages affected." },
{ "id": "divergence_resolved", "weight": 10, "description": "Divergence inputs resolved: 4-dim seed (decade x craft x register x ground-family), font deck pick, palette resolution. Recorded in DESIGN.json.extensions.divergence per the v2 storage shape." },
{ "id": "product_md_direct", "weight": 10, "description": "PRODUCT.md authored directly at project root using impeccable's teach.md as format spec. The agent did NOT invoke $impeccable teach. Sections present: Register, Users, Product Purpose, Brand Personality, Anti-references, Design Principles, Accessibility & Inclusion." },
{ "id": "design_md_direct", "weight": 10, "description": "DESIGN.md and DESIGN.json authored directly at project root. DESIGN.md uses Stitch frontmatter + 6 canonical sections. DESIGN.json schemaVersion 2 with extensions (divergence, componentStyle, voice) and narrative blocks." },
{ "id": "direction_md_shape", "weight": 10, "description": "stardust/direction.md exists with provenance + YAML frontmatter + # Active direction section containing required sub-sections: Phrase, Restatement, Movements, Divergence inputs, Command sequence (proposed), User confirmation, Pages in scope. Anti-references explicitly written (or '(none)')." },
{ "id": "state_updated", "weight": 5, "description": "stardust/state.json updated: direction.resolvedAt set, direction.phrase verbatim, in-scope pages move from 'extracted' to 'directed' with history entries." },
{ "id": "no_silent_command_mapping", "weight": 5, "description": "The agent did NOT silently map the phrase to a fixed command lookup. Reasoning was visible (dimensional vocabulary used, command choices justified)." },
{ "id": "no_eds_references", "weight": 5, "description": "No mention of EDS, AEM, dev servers, or framework targets." }
"context": "Phase 2 (direct): tests dimensional restatement of a freeform phrase, at most two clarifying questions, plan-before-execution, direct authoring of the target spec, and the direction.md trace.",
"type": "weighted_checklist",
"checklist": [
{
"name": "activated",
"max_score": 5,
"description": "The stardust:direct skill was invoked with the freeform phrase as input."
},
{
"name": "dimensional_restatement",
"max_score": 10,
"description": "The agent restated the phrase in stardust's dimensional vocabulary (register / expressive axis / tone / density / distinctiveness / audience / constraints) before doing anything else. Each axis is named and either marked as moved (with direction), pinned, or left alone."
},
{
"name": "gaps_identified",
"max_score": 5,
"description": "The agent explicitly identified what's underspecified before asking questions. 'Young' as too coarse (or equivalent) was called out."
},
{
"name": "question_ceiling",
"max_score": 10,
"description": "At most TWO clarifying questions were asked. Each had concrete options + an 'other' / 'skip' escape hatch and cited which dimension it resolved."
},
{
"name": "plan_shown_before_execution",
"max_score": 15,
"description": "After answers (or with no questions needed), the agent showed the resolved plan to the user BEFORE running any impeccable command or writing any file. Plan included: one-sentence restatement, assumptions, command sequence with reasoning, pages affected."
},
{
"name": "divergence_resolved",
"max_score": 10,
"description": "Divergence inputs resolved: 4-dim seed (decade x craft x register x ground-family), font deck pick, palette resolution. Recorded in DESIGN.json.extensions.divergence per the v2 storage shape."
},
{
"name": "product_md_direct",
"max_score": 10,
"description": "PRODUCT.md authored directly at project root using impeccable's teach.md as format spec. The agent did NOT invoke $impeccable teach. Sections present: Register, Users, Product Purpose, Brand Personality, Anti-references, Design Principles, Accessibility & Inclusion."
},
{
"name": "design_md_direct",
"max_score": 10,
"description": "DESIGN.md and DESIGN.json authored directly at project root. DESIGN.md uses Stitch frontmatter + 6 canonical sections. DESIGN.json schemaVersion 2 with extensions (divergence, componentStyle, voice) and narrative blocks."
},
{
"name": "direction_md_shape",
"max_score": 10,
"description": "stardust/direction.md exists with provenance + YAML frontmatter + # Active direction section containing required sub-sections: Phrase, Restatement, Movements, Divergence inputs, Command sequence (proposed), User confirmation, Pages in scope. Anti-references explicitly written (or '(none)')."
},
{
"name": "state_updated",
"max_score": 5,
"description": "stardust/state.json updated: direction.resolvedAt set, direction.phrase verbatim, in-scope pages move from 'extracted' to 'directed' with history entries."
},
{
"name": "no_silent_command_mapping",
"max_score": 5,
"description": "The agent did NOT silently map the phrase to a fixed command lookup. Reasoning was visible (dimensional vocabulary used, command choices justified)."
},
{
"name": "no_eds_references",
"max_score": 5,
"description": "No mention of EDS, AEM, dev servers, or framework targets."
}
]
}
84 changes: 68 additions & 16 deletions plugins/stardust/evals/extract-multipage/criteria.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,71 @@
{
"name": "extract-multipage",
"total": 100,
"criteria": [
{ "id": "activated", "weight": 5, "description": "The stardust:extract skill was invoked." },
{ "id": "impeccable_dep_check", "weight": 5, "description": "Master skill setup ran first; impeccable was confirmed installed before extract proceeded." },
{ "id": "discovery_before_crawl", "weight": 10, "description": "Discovery (sitemap.xml -> sitemap_index -> robots.txt -> BFS crawl) happened before any per-page rendering. The discovered URL list was visible in the agent's narration." },
{ "id": "page_cap_confirmation", "weight": 10, "description": "The default 25-page cap was applied and the user was shown the kept list AND the cut list before crawling proceeded. Confirmation was solicited (no silent crawling beyond the cap)." },
{ "id": "playwright_over_webfetch", "weight": 10, "description": "Playwright was used for per-page rendering (1440x900 @ 2x DPR, networkidle + 1.5s, scroll-to-bottom pass). WebFetch / curl was NOT used as a substitute." },
{ "id": "per_page_json_shape", "weight": 10, "description": "Each stardust/current/pages/<slug>.json contains all required top-level keys: _provenance (first key), slug, url, finalUrl, title, og, themeColor, headings, landmarks, ctas, links, media, forms, widgets, perSectionStyle." },
{ "id": "brand_extraction_shape", "weight": 10, "description": "stardust/current/_brand-extraction.json exists with all sections: logo, palette (with role names + occurrences + sourceSelectors), type (heading + body families with weights / sizes), spacing, motifs (borderRadius / shadows / patterns), componentStyle (v1 fields preserved), voice samples, register guess." },
{ "id": "logo_locator_chain", "weight": 5, "description": "Logo extracted via the priority chain. _brand-extraction.json.logo.source names the method used (inline-svg | img | apple-touch-icon | og-image | favicon | synthesized). File saved under stardust/current/assets/ (NOT icons/)." },
{ "id": "current_product_md_direct", "weight": 10, "description": "stardust/current/PRODUCT.md was authored directly (sections per impeccable's teach.md format spec). The agent did NOT invoke $impeccable teach for the current-state file." },
{ "id": "current_design_md_direct", "weight": 10, "description": "stardust/current/DESIGN.md and DESIGN.json were authored directly. DESIGN.md uses Stitch frontmatter (colors, typography, rounded, spacing, components) and the 6 canonical sections. DESIGN.json schemaVersion 2 with extensions block." },
{ "id": "state_json_shape", "weight": 5, "description": "stardust/state.json exists with _provenance first, site (originUrl/extractedAt/pageCap/totalDiscovered/crawled), direction (resolvedAt null at this stage), and pages[] (one entry per crawled page, status=extracted)." },
{ "id": "provenance_stamped", "weight": 5, "description": "Every artifact carries a stardust:provenance block per artifact-map.md (first <head> child for HTML, first line above frontmatter for markdown, _provenance first key for JSON)." },
{ "id": "no_eds_references", "weight": 5, "description": "No mention of EDS, AEM, localhost:3000, dev servers, or framework targets in outputs or narration." }
"context": "Phase 1 (extract): tests a multi-page crawl with a cap using Playwright (not WebFetch), correct output file shapes, and direct authoring of the current PRODUCT.md / DESIGN.md.",
"type": "weighted_checklist",
"checklist": [
{
"name": "activated",
"max_score": 5,
"description": "The stardust:extract skill was invoked."
},
{
"name": "impeccable_dep_check",
"max_score": 5,
"description": "Master skill setup ran first; impeccable was confirmed installed before extract proceeded."
},
{
"name": "discovery_before_crawl",
"max_score": 10,
"description": "Discovery (sitemap.xml -> sitemap_index -> robots.txt -> BFS crawl) happened before any per-page rendering. The discovered URL list was visible in the agent's narration."
},
{
"name": "page_cap_confirmation",
"max_score": 10,
"description": "The default 25-page cap was applied and the user was shown the kept list AND the cut list before crawling proceeded. Confirmation was solicited (no silent crawling beyond the cap)."
},
{
"name": "playwright_over_webfetch",
"max_score": 10,
"description": "Playwright was used for per-page rendering (1440x900 @ 2x DPR, networkidle + 1.5s, scroll-to-bottom pass). WebFetch / curl was NOT used as a substitute."
},
{
"name": "per_page_json_shape",
"max_score": 10,
"description": "Each stardust/current/pages/<slug>.json contains all required top-level keys: _provenance (first key), slug, url, finalUrl, title, og, themeColor, headings, landmarks, ctas, links, media, forms, widgets, perSectionStyle."
},
{
"name": "brand_extraction_shape",
"max_score": 10,
"description": "stardust/current/_brand-extraction.json exists with all sections: logo, palette (with role names + occurrences + sourceSelectors), type (heading + body families with weights / sizes), spacing, motifs (borderRadius / shadows / patterns), componentStyle (v1 fields preserved), voice samples, register guess."
},
{
"name": "logo_locator_chain",
"max_score": 5,
"description": "Logo extracted via the priority chain. _brand-extraction.json.logo.source names the method used (inline-svg | img | apple-touch-icon | og-image | favicon | synthesized). File saved under stardust/current/assets/ (NOT icons/)."
},
{
"name": "current_product_md_direct",
"max_score": 10,
"description": "stardust/current/PRODUCT.md was authored directly (sections per impeccable's teach.md format spec). The agent did NOT invoke $impeccable teach for the current-state file."
},
{
"name": "current_design_md_direct",
"max_score": 10,
"description": "stardust/current/DESIGN.md and DESIGN.json were authored directly. DESIGN.md uses Stitch frontmatter (colors, typography, rounded, spacing, components) and the 6 canonical sections. DESIGN.json schemaVersion 2 with extensions block."
},
{
"name": "state_json_shape",
"max_score": 5,
"description": "stardust/state.json exists with _provenance first, site (originUrl/extractedAt/pageCap/totalDiscovered/crawled), direction (resolvedAt null at this stage), and pages[] (one entry per crawled page, status=extracted)."
},
{
"name": "provenance_stamped",
"max_score": 5,
"description": "Every artifact carries a stardust:provenance block per artifact-map.md (first <head> child for HTML, first line above frontmatter for markdown, _provenance first key for JSON)."
},
{
"name": "no_eds_references",
"max_score": 5,
"description": "No mention of EDS, AEM, localhost:3000, dev servers, or framework targets in outputs or narration."
}
]
}
Loading
Loading