Repository navigation
test(notes): add the structured enhance-and-categorize eval case - #617
Merged
Merged
Conversation
Completes the assertion scope of #542. The harness shipped with the free-text release-notes path; the structured path is the case it was built to take next, and the one that unlocks the category-distribution assertion the issue lists. Two assertions the free-text case can't reach: `checkCategoryDistribution` — grouping has to discriminate, not merely be legal. The task validator already rejects category names outside the configured set, so a model that drops every entry into one bucket passes validation and still produces a useless grouping. Catches too few populated categories, an emitted-but-empty category, and one category swallowing most of the entries. The share bound is skipped below four entries, where a lopsided split isn't evidence of anything. `checkNoEntryLoss` — every entry survives categorization exactly once, catching both drops and duplication across buckets. `loadGoldenCase` is now generic in its context type, since the tasks take different ones — release notes a ReleaseNotesContext, this an EnhanceContext & CategorizeContext. Also adds unit tests for the assertion module. These are the regression net, so each needs a case that actually trips it; without one the eval would pass every run while catching nothing. Verified the drift signal fires on the new case: editing the enhance-and-categorize system prompt changes the cache key and the run fails with the re-record hint rather than silently replaying stale output. Recording the new case left the existing release-notes fixture untouched, confirming the per-case refresh. Part of #542. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FADRmu9uhtCk3StMoqwV27
Contributor
|
Standing release PR: #614 · v0.42.0 queued (4 packages) · open 82d 20h 47m · ✅ ready to merge Release Preview — v0.41.2 → v0.42.0
These changes will be added to the release PR (#614) when merged: Changelogmonorepo 0.41.2 → 0.42.0
After merge — predicted release
Updated automatically by ReleaseKit |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
From Greptile review of #617, and a real defect in the fixture. `LLMCategory.description` is required, and `buildCategorySection` renders each category as `- "<name>": <description>`. The fixture omitted them, so the prompt the eval measured against contained `- "New": undefined` — a shape production never produces. The case was pinning malformed categorization guidance. Re-recorded from a clean cache, since the corrected prompt changes the key. That also drops the entry orphaned under the old key — the clean-slate procedure the harness header documents, now exercised for the first time. The release-notes fixture is untouched, confirming per-case recording. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FADRmu9uhtCk3StMoqwV27
…aped golden input Review follow-ups for the structured eval case: - `checkNoEntryLoss` compared totals only, so a dropped entry and a duplicated one cancelled out (a groupByCategory mutation that did exactly that passed the eval). It now checks identity: every enhanced entry sits in exactly one bucket once, no bucket holds anything else, and enhancement kept the golden entry count. The same mutation now fails with "entry 0 was categorized 2 times / entry 1 was dropped". - The golden input kept raw `feat(api): ...` subjects with no `scope`, a shape neither commit parser produces, so the pinned prompt wasn't the one production builds. Use the parsed shape (bare subject, `scope`, `originalType`, no empty `issueIds`) and re-key the cache from the hand-authored recorded response. - A cache miss on a corrective retry now says the recorded response was rejected, instead of advising a re-record that would fail identically. - Note that the case exercises the text -> JSON path, not the structured-output branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HQN3v9Z4f1mu3xK4Z7K44e
- Record mode answered corrective retries with the canned markdown, caching orphan entries that replay then served, so a rejected recording failed later on a misleading distribution assertion. Both offline modes now refuse a corrective retry at the source. - checkNoEntryLoss reports grouping that copies entries as a broken identity assumption instead of N phantom "dropped" entries. - Restrict the golden case's scopes to api/config, so grouping has to use the scope-validated entries: grouping the pre-validation entries now fails the eval (it passed before). Prompt and cache key unchanged. - Retarget the prefix-leak guard to a leading `word(scope):` / `word:` prefix; the old pattern only matched tokens the prompt never contains. - checkCategoryDistribution: drop the possibly-undefined reduce seed (TS18048 under tsconfig.test.json). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HQN3v9Z4f1mu3xK4Z7K44e
…ches
Third-review follow-ups:
- checkNoEntryLoss's "copies" diagnosis matched on counts alone, so
bucketing the raw, un-enhanced input (a real regression) was reported
as a harmless copy and the developer told to relax the check. It now
requires each stray to be bucketed once and to match a distinct
missing entry by content; anything else reads as loss. Partial copies
are named as copies; unrelated objects and duplicated copies are not.
- Assert directly that no disallowed scope reaches a bucket, so grouping
the unvalidated response fails on its own, not only via identity.
- Build the prefix-leak pattern from the golden case's own type,
originalType and scope labels, so a legitimate lead-in ("CLI: ...")
in a live run isn't flagged.
- checkCategoryDistribution's doc listed two failure modes; it checks
three.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQN3v9Z4f1mu3xK4Z7K44e
Fourth-review nits: - Cover the length guard in isOneToOneCopy: one copied entry plus one genuinely dropped entry must read as loss, not as a clean copy that hides the drop. - Compare copies by key-sorted JSON, so a copy rebuilt with its keys in another order is still recognised. - Match scopes the way production validates them (case-insensitive unless the config says otherwise), so a live model's "API" for an allowed "api" is not a false failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HQN3v9Z4f1mu3xK4Z7K44e
Remove the copy diagnosis in checkNoEntryLoss (isOneToOneCopy, canonicalJson and their spec cases) — the check fails either way, it only changed the message for a refactor that doesn't exist — the regex escaping of the fixture's fixed labels, and the case-insensitive scope match, which only mattered for live-model runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HQN3v9Z4f1mu3xK4Z7K44e
This was referenced Oct 22, 2026
main now enforces style/useBlockStatements (#679). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HQN3v9Z4f1mu3xK4Z7K44e
7 of 20 tasks
Keep the non-obvious why; drop restated mechanics and examples the code already shows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 23, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the assertion scope of #542. The harness shipped covering the free-text release-notes path; the structured
enhance-and-categorizepath is the case it was deliberately built to take next, and the one that unlocks the category-distribution assertion the issue lists.Two assertions the free-text case can't reach
checkCategoryDistribution— grouping has to discriminate, not merely be legal. The task validator already rejects category names outside the configured set, so a model that drops every entry into one bucket passes validation and still produces a useless grouping. This catches the three ways that degrades: too few populated categories, an emitted-but-empty category, and one category swallowing most of the entries. The share bound is skipped below four entries, where a lopsided split isn't evidence of laziness.checkNoEntryLoss— every enhanced entry survives categorization exactly once. It compares by identity, not by count (a dropped entry and a duplicated one cancel out in a total): the task'scategories[].entriesare the same objects as itsenhancedEntries, so each must sit in exactly one bucket, once, and no bucket may hold anything else.The golden case
The input is production-shaped — the bare subject plus
scopeandoriginalType, as both commit parsers emit — so the pinned prompt is the one production builds. Its scopes are restricted toapi/config, and the eval asserts that no disallowed scope reaches a bucket, so grouping unvalidated entries fails it. The recorded response is hand-authored and re-keyed withRELEASEKIT_EVAL_RECORD=1.The eval capabilities advertise no structured outputs, so the case exercises the text → JSON parse path, not the schema/tool-call branch. The prefix-leak guard is built from the case's own type/scope labels, so a legitimate lead-in in a live run isn't flagged.
loadGoldenCaseis now generic in its context type, since the tasks take different ones — release notes aReleaseNotesContext, this anEnhanceContext & CategorizeContext.Harness
Both offline modes refuse a corrective retry (the task rejected the recorded response): replay can't answer it, and a re-seed that answered with the same canned markdown would cache orphan entries that replay then serves. The error now says the recording was rejected instead of advising a re-record that would fail the same way.
Assertion unit tests
Added
assertions.spec.ts. The assertions are the regression net, so each needs a case that actually trips it — a check that only ever returns[]passes every eval run while catching nothing. Covers the pre-existing assertions too.Verification
entries[i === 1 ? 0 : i]ingroupByCategory); grouping the pre-scope-validation entries; grouping the raw, un-enhanced input.Closes #542.
🤖 Generated with Claude Code