Repository navigation
Cost you can check, a step-by-step guide, and scripts safe to edit mid-run - #20
Merged
Merged
Conversation
…d-run - score.py lists what Reason charged per attempt (the cost_usd Harbor uploads: list price + 25% on Reason-paid models) beside the list-price cost, labels its fallback as Reason's charge, and prices Anthropic one-hour cache writes at 2x input. The price table's comment no longer claims Reason bills at list price. - METHODOLOGY 1.10: both costs, and how tokens and cost were checked against the inference gateway and OpenAI's usage and cost APIs; tool names in trajectories. - docs/GUIDE.md: how a run works, step by step, what each file does, the decisions and why, and the cost checks. - run.sh and run-pass.sh run as one function, so pulling or editing the repo during a run can't change what it executes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hows its count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
From an end-to-end audit of a six-task smoke run (gpt-6-luna xhigh, adapter 0.3.44, Harbor 0.25):
Verified
agent_resultand the Reason API's Session usage on every attempt.score.pycost equals the gateway's provider cost.PRICESmatches OpenAI's published price page.Fixed here
cost_usd(what Hub shows) is Reason's charge, list price plus 25% on Reason-paid models, but the docs said Reason bills at list.score.pynow lists both, and the methodology says which is which.run.sh/run-pass.share wrapped inmain(). Editing the checkout during a run made bash resume at a stale offset and run the wrong lines; reproduced, and fixed with a fakeharborwhile rewriting the file mid-run.docs/GUIDE.md, linked from the README.Related monorepo fixes: #31055 (trajectories named the
reasontoolcodemode), #31058 (one-hour cache writes in exports), #31063 (adapter shows every error code, Harbor 0.3.45).Tests: 22 pass; shellcheck clean.
🤖 Generated with Claude Code