Skip to content

Retarget delegation models from measured coding-agent cost/task - #19

Draft
kelchm wants to merge 1 commit into
masterfrom
t3code/retarget-delegation-models
Draft

Retarget delegation models from measured coding-agent cost/task#19
kelchm wants to merge 1 commit into
masterfrom
t3code/retarget-delegation-models

Conversation

@kelchm

@kelchm kelchm commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Summary

  • retarget the model ranking and delegation pins to Grok 4.6, Fable 5.1, GPT-6 Astra, and Opus 5
  • rank cost and intelligence from measured coding-agent cost/task, not list price or the old Grok deal: DeepSWE and the Artificial Analysis Coding Agent Index
  • keep taste as the remaining judgment axis
  • add a start effort column instead of a model×effort matrix: scores assume that operating point
  • escalate by raising one effort step on the same model, then switching models
  • pin those start efforts on the CLI invocations the skills actually run (grok --effort medium, Codex model_reasoning_effort=xhigh, claude --effort high)
  • preserve the live @RTK.md include in CLAUDE.md so chezmoi apply does not drop it

Why these start efforts

  • grok-4.6 / medium — DeepSWE peak (67.5% at $3.45). xhigh costs more and scores worse.
  • gpt-6-astra / xhigh — DeepSWE peak (74.1% at $6.52). max is ~2x the cost with no gain.
  • fable-5.1 / opus-5 / high — Claude's product default. Fable 5.1 max is the expensive last step (~1.7x Fable 5 tokens on the Intelligence Index).
  • gpt-5.6-luna / max — the exception: lower efforts collapse on DeepSWE (high 44%, medium 11%).

Sonnet-5 stays in the table with honest scores (cost 2, intelligence 4). Measured DeepSWE cost/task at max is $26.40 for 54%; it is not a cheap coding agent.

Out of scope

  • interactive Codex default in ~/.codex/config.toml is still gpt-5.6-sol / high (not chezmoi-managed)
  • historical grok-4.5 canary in docs/cross-harness-delegation.md

Validation

  • git diff --check passed
  • chezmoi applied the rendered CLAUDE.md, Codex AGENTS.md, and affected skills locally
  • live six-cell matrix not re-run in this PR (model/effort pins only; guards unchanged)

Review notes

The start column is the bet that effort is a cheaper lever than switching models, without turning the ranking into a 2D grid. If a score should move (especially Astra taste 6, Sonnet remaining in the table, or Luna starting at max), that is the discussion this draft is for.

Rank Grok 4.6, Fable 5.1, GPT-6 Astra, and Opus 5 from DeepSWE and the
Artificial Analysis Coding Agent Index. Pin each model's start effort
and raise effort before switching models.
@coderabbitai

coderabbitai Bot commented Sep 6, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant