Improve Najm example against eval audit#14
Conversation
Build a deterministic slop detector grounded in Impeccable's 37 patterns plus Hallmark's own gates, score self-contained fixtures across genres, and run a 10-cycle eval-driven hillclimb. Phase 1 (v1, cycles 1-5): close gaps the detector found, adding gates 70-77 to references/slop-test.md; fixtures climb 74.2 -> 98.3. Cycle 6: per "Your Evals Will Break", upgrade the eval to v2 -- six new detector rules (incl. hero-float/gate 54 the v1-perfect fixtures had been violating), a cross-fixture order parameter (macrostructure reuse), and two adversarial fixtures. Score honestly drops to 76.4. Phase 2 (v2, cycles 7-10): add gates 78-84 and climb back to 98.7, resisting a dark/neon/metric-hero brief. The skill gained 15 gates motivated by what the eval could measure. Full curve in evals/results/history.md.
Audit the in-repo Hallmark corpus (homepage + examples) surfaced a real false-positive rate. Fix the worst offenders so the signal is trustworthy: - placeholder-names: only flag actual placeholder names (Jane Doe, Acme, lorem ipsum), not ordinary words like "seamless"/"unleash" in prose. - ai-palette: require the violet->cyan *ramp*, not a single deliberate brand hue, so a midnight-violet brand is no longer flagged. - font counting: count a monospace family toward the budget only when used outside code (per gate 39); stop counting unused --font-mono tokens. - multi-theme scoping: resolve tokens from the active [data-theme] only, and label 22-theme / component-library stylesheets low-confidence instead of scoring them as one page. evals/audit-site.mjs inlines a page's linked stylesheets so the detector can score real shipped pages. Fixtures unchanged (all still 5.00/5 on v1 and v2); true positives (Inter in hyperlane, gradient text in bananastudio, "Acme" in tally) are retained while the false positives are removed.
|
@adewale is attempting to deploy a commit to the Together AI Team on Vercel. A member of the Team first needs to authorize it. |
Hallmark uses a strict suite of automated tests (references/slop-test.md) to block typical LLM UI layout "tells." Your PR will not pass if it introduces any code, updates templates, or changes themes and fails the following hard-coded tests:
Multi-Assistant Paths and Frontmatter Errors It’s Cross Compatible! Hallmark is compatible across Claude Code, Cursor and Codex so you have to change in structurallayout in one and it should make a corresponding change in the other parsing environment!
/ Hallmark pre-emit critique: P5 H4 E5 S4 R5 V5 / Together AI Custom Mode Fallback |
Draft stacked PR after #12 and #13.
Because I do not have write access to create stack base branches in
Nutlope/hallmark, this PR targetsmain. Until the parent PRs merge, GitHub's default Files changed view will include parent diffs. The intended child diff is:adewale/hallmark@pr/tally-eval-audit...pr/najm-eval-audit
What
Improves the Najm example using the eval audit from #12 while preserving the Moroccan fashion/drop direction.
Targeted changes:
overflow-x: clipWhy
Najm was another low-scoring real example in the audit. This PR is the second proof-of-value pass: the eval harness can identify concrete, fixable issues in existing examples without requiring a wholesale redesign.
Audit evidence
Before this PR:
After this PR:
Testing
Observed:
Risk
This touches a visual example, so browser review is still needed. The largest semantic change is the cart drawer role: the drawer was labelled as modal, but the implementation does not trap focus, so
role="region"better matches the current behavior.