Skip to content

fix(evals): swap cart classifier to claude-haiku-4.5 (gpt-5-mini drifted to 75%) - #243

Merged
BSalaeddin merged 3 commits into
mainfrom
eval/cc-anthropic-claude-haiku-4-5
Sep 14, 2026
Merged

BSalaeddin merged 3 commits into
mainfrom
eval/cc-anthropic-claude-haiku-4-5

Conversation

@BSalaeddin

Copy link
Copy Markdown
Collaborator

Why

openai/gpt-5-mini has drifted below the eval gate. No commit touched cartClassifier.ts, openrouter.ts, the eval fixtures, or the scorers since the last green run on 2026-07-14, so the drop from 97.5% -> 75.0% on the 2026-09-14 nightly (#242, and the recurring nightly RED since 2026-08-28) is provider-side model drift — exactly what the nightly gate exists to catch.

The swap

Single source of truth is env.OPENROUTER_MODEL (CI leaves it unset so the code default is what gets tested). Default changed openai/gpt-5-mini -> anthropic/claude-haiku-4.5.

Gated sweep (live CI workflow_dispatch, production classifyCart, temp 0, 40 cases)

Model Primary-match Schema-valid p50 p95
anthropic/claude-haiku-4.5 100.0% 100% 848-878ms 1693-1957ms
openai/gpt-5.1 92.5% 100% 1493ms 3228ms
google/gemini-3.8-flash 90.0% 95% 2388ms 7000ms

Haiku 4.5 wins on every axis and is cheaper than gpt-5.1 ($1.00·$5.00 vs $1.25·$10.00 per M). It is already 100% on the sibling coupons classifier.

Eval-gate compliance

  • Green x2 on the shipped model: runs 34889586347 and 34889877905, both exit 0, both 100%.
  • Scoreboard row added (evals/SCOREBOARD.md) with the dated numbers and the sweep.
  • Regression proof on record: the 75.0% nightly Nightly AI eval failed: cart classifier (2026-09-14) #242 is the gate biting the drifted model.

Prod picks this up on the next deploy. Closes the #242 series once merged and the nightly runs green.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant