[goosewobbler]

Implements the LLM-notes eval harness from #542 — the deterministic quality net for releasekit's clearest differentiator.

The gap this closes

Nothing measured whether LLM notes were good, only structurally valid. Prompt drift was invisible in review diffs (no prompt was snapshot-tested), and a wording tweak silently busted the entire response cache with no signal.

What's here

test/eval/ harness

• Golden fixtures — a real-shaped commit set (release-notes-basic.json) with a fixed date, so cache keys are stable.
• Record/replay on the existing cache format — responses are replayed through withContentHashCache's on-disk format, so CI is deterministic with no API keys. A cache miss (i.e. a changed prompt/input → changed key) fails loudly with a re-record hint — the missing fixture is the drift signal.
• Deterministic assertions (pure, reusable) — no <!-- releasekit- --> marker leaks, no duplicated "Updated dependencies" churn, length bounds, past-tense leaning. These are the regression net and hold for both replayed and live output.
• Opt-in live mode — RELEASEKIT_EVAL=1 runs a real Ollama model (recording as it goes); RELEASEKIT_EVAL_RECORD=1 re-seeds fixtures from the human-readable *.recorded.md.

Prompt snapshot tests

Pins the five default task system prompts (enhance, categorize, enhanceAndCategorize, releaseNotes, summarize). A prompt edit now shows up as a reviewed snapshot diff in the same PR instead of a silent change that reshapes every generation. The task prompt builders are exported for this (module-level, not public package API).

Scope note

The first eval case covers the free-text release-notes path. The structured enhance-and-categorize path (which unlocks the category-distribution assertion) is the next case the harness is deliberately built to take — the record/replay + assertion infrastructure is path-agnostic.

Testing

• Full notes suite green (441 passed, 2 opt-in e2e skipped), incl. the new eval (replay) + 5 prompt snapshots.
• Verified the record → strict-replay round trip: RELEASEKIT_EVAL_RECORD=1 seeds the committed cache fixture, and the default run replays it deterministically.
• typecheck + biome check clean; coverage only instruments src/**, so the eval helpers don't affect thresholds.

Part of #542. Independent of the #544 CLI-envelope work (#608) — touches only packages/notes.

Greptile Summary

This PR implements the LLM-notes eval harness from #542, adding a golden-fixture record/replay infrastructure and prompt snapshot tests to catch quality regressions and prompt drift in the release-notes pipeline.

• Eval harness (test/eval/): golden commit sets replay through the real pipeline against committed cache fixtures using withContentHashCache's on-disk format, so CI is deterministic with no API keys. A changed prompt busts the cache key and fails loudly. Opt-in live (Ollama) and record-from-markdown modes are available. The refresh flag added to withContentHashCache is the enabler for write-through recording.
• Deterministic quality assertions (assertions.ts): four pure checks — marker leak detection, duplicate dependency churn, length bounds, and past-tense leaning — serve as the regression net for both replayed and live output.
• Prompt snapshot tests (prompts-snapshot.spec.ts): all five default task system prompts are pinned as Vitest snapshots, making any prompt edit an explicit, reviewable CI event rather than a silent cache-busting change.

Confidence Score: 5/5

Safe to merge — changes are purely additive test infrastructure and narrow source exports; no production behaviour is altered.

The refresh flag in withContentHashCache is a simple, well-tested write-through option. All source changes are visibility-only exports for testing. The eval harness, assertions, fixtures, and snapshot tests introduce no runtime risk. The two observations are about edge-case gaps in the tense-detection heuristic that don't affect correctness of the current fixture or the replay pipeline.

Files Needing Attention: No files require special attention.

Important Files Changed

Filename	Overview
packages/notes/src/llm/cache.ts	Adds a refresh option to withContentHashCache — skips the read and always calls the provider, allowing record/replay modes to overwrite existing entries. Well-tested with a new unit case.
packages/notes/test/eval/harness.ts	Eval harness wiring three modes (replay / record-from-markdown / live-Ollama) through withContentHashCache. Fixed provider name and capabilities ensure record→replay keys match.
packages/notes/test/eval/assertions.ts	Pure, content-based quality checks for generated notes: marker leak detection, duplicate dependency churn, length bounds, and past-tense leaning. The IRREGULAR_PAST set is small but the 0.6 ratio threshold provides tolerance for missed irregular verbs.
packages/notes/test/eval/notes-eval.spec.ts	Two tests: golden-fixture replay through the real pipeline with four quality assertions, and a capabilities-fixation unit test that guards the record→replay key-stability invariant.
packages/notes/test/unit/prompts-snapshot.spec.ts	Snapshots all five default task system prompts to make prompt drift an explicit, reviewable event in CI.
packages/notes/test/unit/llm/cache.spec.ts	New refresh test verifies that the write-through path replaces a stale entry and that a subsequent non-refresh read finds the fresh value.
biome.jsonc	Adds the eval cache fixtures directory to Biome's ignore list so compact provider-response blobs are not reformatted on re-record.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Test run] --> B{Env flags?}
    B -->|default| C[strictOfflineProvider]
    B -->|RELEASEKIT_EVAL_RECORD=1| D[loadRecorded\n*.recorded.md]
    B -->|RELEASEKIT_EVAL=1| E[OllamaProvider\nlive model]

    C --> F[withContentHashCache\nrefresh=false]
    D --> G[cannedProvider\nfixed content]
    G --> H[withContentHashCache\nrefresh=true]
    E --> I[asEvalProvider\nre-brand + tee to *.recorded.md]
    I --> J[withContentHashCache\nrefresh=true]

    F -->|cache hit| K[return fixture]
    F -->|cache miss| L[throw with re-record hint]
    H -->|always writes| M[cache fixture]
    J -->|always writes| M

    K --> N[generateReleaseNotes]
    M --> N

    N --> O{Quality assertions}
    O --> P[findMarkerLeaks]
    O --> Q[findDuplicateDependencyChurn]
    O --> R[checkLengthBounds]
    O --> S[checkPastTenseLeaning]

Loading

_{Reviews (3): Last reviewed commit: "test(notes): pin eval capabilities and w..." | Re-trigger Greptile}

[github-actions]

Standing release PR: #614 · v0.41.3 queued (4 packages) · open 15m · ✅ ready to merge

Release Preview — v0.41.2 → v0.41.3

Note: Labels on this PR are advisory in standing-pr mode. Bumps come from conventional commits in the standing PR; override by editing labels on the standing PR itself. Add release:immediate to bypass the standing PR and release this PR directly.

These changes will be added to the release PR (#614) when merged:

Changelog

monorepo 0.41.2 → 0.41.3

Changed

• pin eval capabilities and write through the cache instead of clearing it (notes)
• bump the production-dependencies group across 1 directory with 6 updates (deps) (PR #613)
• bump anthropics/claude-code-action from 1.0.179 to 1.0.183 (deps) (PR #610)
• bump the development-dependencies group with 3 updates (deps-dev) (PR #612)
• address eval-harness review — refresh fixtures, tense-check numbered lists (notes)
• add LLM-notes eval harness and prompt snapshot tests (notes)

All packages → 0.41.3 (sync versioning)

• @releasekit/version
• @releasekit/notes
• @releasekit/publish
• @releasekit/release

After merge — predicted release

No version escalation — this PR's changes will be included in the queued release without affecting the projected versions.

Package	Standing PR	This PR	After merge
monorepo	0.41.3	0.41.3	0.41.3

---

Updated automatically by ReleaseKit