Add Anthropic Fable 5 model to benchmark list - #4
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughWalkthroughAdds ChangesBench configuration and runtime guards
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
scripts/run-prod.sh (1)
5-29: 🏗️ Heavy liftConsider consolidating the model list into a single source.
The
modelsarray in this file duplicates the list indata/models.json. Both files now contain the same 24 models in the same order, but maintaining them separately risks drift over time (e.g., a model added to one but not the other).Since
scripts/run-all.tsalready imports fromdata/models.json, consider refactoring this script to read from that same source. For example:# Read models from JSON and iterate node -pe "require('./data/models.json').join('\n')" | while read -r model; do echo "\n▶ Running $model..." npm run bench "$model" || echo "⚠ $model failed, continuing..." doneAlternatively, migrate
run-prod.shto a TypeScript/Node script that importsmodels.jsondirectly (similar torun-all.ts).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/run-prod.sh` around lines 5 - 29, The models array in scripts/run-prod.sh is duplicated from data/models.json; replace the hardcoded models list by reading/importing the canonical list from data/models.json (the same source used by run-all.ts) so the two can't drift: update run-prod.sh to load models from data/models.json (or convert it to a small Node/TypeScript script like run-all.ts) and iterate that list to call the existing npm run bench "$model" loop (preserve the existing echo/log and error-handling behavior).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@scripts/run-prod.sh`:
- Around line 5-29: The models array in scripts/run-prod.sh is duplicated from
data/models.json; replace the hardcoded models list by reading/importing the
canonical list from data/models.json (the same source used by run-all.ts) so the
two can't drift: update run-prod.sh to load models from data/models.json (or
convert it to a small Node/TypeScript script like run-all.ts) and iterate that
list to call the existing npm run bench "$model" loop (preserve the existing
echo/log and error-handling behavior).
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: c466cd2f-752e-4924-9248-0748e715bda3
📒 Files selected for processing (2)
data/models.jsonscripts/run-prod.sh
- run.ts: abort before finalizeRun if no prompts succeeded, so an unavailable model slug marks the run failed instead of writing a NaN/empty run to the production backend - add .env.local.example documenting OPENROUTER_API_KEY, VITE_CONVEX_URL, BENCHMARK_SECRET
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.env.local.example:
- Line 5: Replace the hardcoded production Convex URL in the example env by
changing the VITE_CONVEX_URL value to a placeholder (e.g., a descriptive token
like YOUR_CONVEX_URL or a sample host) so developers must supply their own URL
when copying .env.local; update the VITE_CONVEX_URL entry to use that
placeholder.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: b2a69cc6-2db7-407a-9641-da81795a092a
📒 Files selected for processing (2)
.env.local.examplescripts/run.ts
| # OpenRouter API key — used to call models. Spends real money per run. | ||
| OPENROUTER_API_KEY= | ||
| # Production Convex deployment URL the bench writes to. | ||
| VITE_CONVEX_URL=https://uncommon-sandpiper-321.convex.cloud |
There was a problem hiding this comment.
Replace production URL with a placeholder.
Example files should use placeholders rather than hardcoded production values. Developers copying this to .env.local might not realize they need to change it, potentially writing test data to production (though BENCHMARK_SECRET provides a gate).
📝 Suggested fix
-VITE_CONVEX_URL=https://uncommon-sandpiper-321.convex.cloud
+VITE_CONVEX_URL=https://your-deployment.convex.cloud📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| VITE_CONVEX_URL=https://uncommon-sandpiper-321.convex.cloud | |
| VITE_CONVEX_URL=https://your-deployment.convex.cloud |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.env.local.example at line 5, Replace the hardcoded production Convex URL in
the example env by changing the VITE_CONVEX_URL value to a placeholder (e.g., a
descriptive token like YOUR_CONVEX_URL or a sample host) so developers must
supply their own URL when copying .env.local; update the VITE_CONVEX_URL entry
to use that placeholder.
2 participants
Summary
Adds Anthropic's new Fable 5 model (
anthropic/claude-fable-5) to the SlopBench model list so it gets benchmarked alongside the other models.Changes
data/models.json— addedanthropic/claude-fable-5(consumed byscripts/run-all.ts)scripts/run-prod.sh— added the same slug to themodelsarrayPlaced at the top of the Anthropic group since Fable 5 is Anthropic's most capable tier (above Opus). No change needed in
src/utils/providers.js— theanthropicprovider already maps to a logo/domain, so favicon and label rendering work automatically.Not done yet — running the benchmark
The actual bench run for Fable 5 has not been executed. Running
npm run bench anthropic/claude-fable-5requires credentials and network egress that aren't available in the CI/sandbox environment:OPENROUTER_API_KEY,VITE_CONVEX_URL,BENCHMARK_SECRET(none present)openrouter.ai(blocked — returns 403 here)The run needs to be kicked off from an environment that has those. Once it writes to the production Convex backend, the live site picks it up reactively (no separate frontend deploy needed).
https://claude.ai/code/session_017Gbnqqz3QbMnhQ6xiZRDuZ
Generated by Claude Code
Note
Low Risk
Model list and local env documentation changes plus a bench-script safety check; no auth or production app logic changes.
Overview
Adds
anthropic/claude-fable-5to the benchmark model lists indata/models.jsonandscripts/run-prod.sh, positioned at the top of the Anthropic group so Fable 5 is included in batch and prod bench runs.Introduces
.env.local.exampledocumentingOPENROUTER_API_KEY,VITE_CONVEX_URL, andBENCHMARK_SECRETfornpm run bench.scripts/run.tsnow fails fast when every prompt fails for a model (e.g. bad OpenRouter slug): it throws beforefinalizeRuninstead of finalizing an empty run.Reviewed by Cursor Bugbot for commit f8f4626. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation