Skip to content

fix(ci): unpin upstream LiteLLM prices that are failing every open PR - #150

Merged
ohong merged 2 commits into
mainfrom
oh-review-followups
Aug 25, 2026
Merged

fix(ci): unpin upstream LiteLLM prices that are failing every open PR#150
ohong merged 2 commits into
mainfrom
oh-review-followups

Conversation

@ohong

@ohong ohong commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Why this is urgent

CI is red on every open PR — #143, #145, #146 and #149 — for one shared reason that none of them caused:

AssertionError: expected 0.28400000000000003 to be close to 0.355
  ❯ __tests__/ccusage-pricing.integration.test.ts:94:34

packages/cli/__tests__/ccusage-pricing.integration.test.ts pins hardcoded dollar amounts for the GPT-5.6 family. Those rates are owned by LiteLLM upstream and are fetched live by the bundled ccusage binary. LiteLLM moved gpt-5.6-terra from $0.355 to $0.284, and the whole repo's Test (cli) job went red on the next run. 202 of 203 CLI tests pass; this one assertion is the blocker.

#149 is the clearest proof it isn't the PRs: it changes .gitignore and deletes committed artifacts, touches no CLI code at all, and still fails.

What changed

The test now asserts the invariant we actually own instead of the number upstream owns:

  • every member of the GPT-5.6 family resolves to a non-zero LiteLLM price (catches an unpriced or renamed model, which is the real regression this test exists to catch);
  • the day total equals the sum of the per-model breakdown (catches aggregation bugs);
  • the 440,000-token fixture total stays pinned, since token counts are ours, not upstream's.

What it no longer does is fail when a vendor reprices a model overnight.

docs/CHANGELOG.md had the old $1.917 figure written into the ccusage entry as a durable claim; corrected to describe the invariant.

Also in here

docs/ROADMAP.md picks up two findings from reviewing the open PRs:

  • Straude already ingests Gemini, Qwen, Kimi, Copilot and the rest of ccusage's sources. The collector runs unscoped ccusage daily --json, CcusageAgent is string, and /api/usage/submit has no agent allowlist. feat: add Gemini CLI usage tracking via gemistat #77 and feat: add Gemini, Qwen Code, and Mistral Vibe providers #22 both hand-wrote parsers for capability that already ships. What's missing is prettifyModel cases, model colours, and copy telling users.
  • calculate_user_streak is SECURITY DEFINER with EXECUTE granted to anon, so anyone can read any user's streak by id, private profiles included. Long-standing, from 20260430172022, not from an open PR.

Validation

bun test __tests__/ccusage-pricing.integration.test.ts in packages/cli — 1 pass, 13 assertions, against the real bundled binary and live pricing.

Worth merging before the others so their CI can go green.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes

    • Improved GPT-5.6 pricing validation to ensure all supported models return valid pricing and totals remain consistent with model-level costs.
  • Documentation

    • Updated changelog guidance to reflect resilient pricing validation without relying on fixed dollar amounts.
    • Added roadmap items for multi-agent CLI support and strengthening access controls around user streak calculations.

ohong and others added 2 commits August 8, 2026 20:34
The GPT-5.6 family costs are owned upstream by LiteLLM and change without
notice, so hardcoded rates make this integration test fail on price moves
rather than on regressions we control. Assert the invariant instead: every
model in the family resolves to a non-zero price, and the day total equals
the sum of the per-model breakdown.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two items surfaced while reviewing the open PRs:

- Straude already ingests Gemini, Qwen, Kimi, Copilot and the rest of
  ccusage's sources, because the collector runs unscoped `ccusage daily`
  and nothing downstream filters on agent name. PRs #77 and #22 hand-wrote
  parsers for capability we already have. What's missing is prettification,
  model colours, and copy that tells users.
- `calculate_user_streak` is SECURITY DEFINER with EXECUTE granted to anon,
  so any caller can read any user's streak by id, private profiles included.
  Long-standing, not from an open PR.

Also corrects the ccusage CHANGELOG entry, which claimed the GPT-5.6 fixture
pins a $1.917 total; it now asserts the pricing invariant instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Aug 9, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
straude Ready Ready Preview Aug 9, 2026 8:09am

Request Review

@coderabbitai

coderabbitai Bot commented Aug 9, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The GPT-5.6 pricing test now checks model coverage, positive resolved pricing, and cost consistency without fixed dollar values. The changelog records this behavior. The roadmap adds multi-agent CLI support and a security item about streak RPC grants.

Changes

GPT-5.6 pricing validation

Layer / File(s) Summary
Pricing assertions and changelog
packages/cli/__tests__/ccusage-pricing.integration.test.ts, docs/CHANGELOG.md
The test validates all GPT-5.6 variants, positive LiteLLM pricing, and consistency between model, daily, and summary totals. The changelog documents the non-pinned dollar values.

Multi-agent CLI roadmap

Layer / File(s) Summary
Multi-agent CLI support
docs/ROADMAP.md
The roadmap adds planned model naming, color support, supported-agent documentation, and Mistral Vibe support.

Streak RPC security roadmap

Layer / File(s) Summary
Streak RPC grant review
docs/ROADMAP.md
The roadmap documents arbitrary-user streak access through anon and authenticated grants and proposes call-site verification followed by grant revocation.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: alexesprit

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: removing hardcoded upstream LiteLLM price assertions from the CI pricing test.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch oh-review-followups

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

ESLint install failed. For unrecoverable errors, disable the tool in CodeRabbit configuration.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
docs/ROADMAP.md (1)

40-40: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Define the remaining documentation deliverable.

packages/cli/README.md, Lines 15-20, already lists Kimi, Qwen, Copilot CLI, and Gemini CLI. State whether this item adds landing-page copy, a dedicated supported-agents section, or an explicit support guarantee. Link to the existing README section. This gives future work a clear, non-duplicative acceptance target.

As per coding guidelines, docs/ROADMAP.md must include feature requests, ideas, and improvements with sufficient context for future pickup.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/ROADMAP.md` at line 40, Clarify the roadmap item about supported-agent
messaging by specifying the exact documentation deliverable, such as
landing-page copy or a linked supported-agents section. Reference the existing
supported-agents section in packages/cli/README.md and define a non-duplicative
acceptance target rather than leaving the scope as general README and
landing-page copy.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@docs/ROADMAP.md`:
- Line 40: Clarify the roadmap item about supported-agent messaging by
specifying the exact documentation deliverable, such as landing-page copy or a
linked supported-agents section. Reference the existing supported-agents section
in packages/cli/README.md and define a non-duplicative acceptance target rather
than leaving the scope as general README and landing-page copy.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: b9a7675a-4924-4d96-a7b7-718a1d92cc44

📥 Commits

Reviewing files that changed from the base of the PR and between f007465 and 17593a3.

📒 Files selected for processing (3)
  • docs/CHANGELOG.md
  • docs/ROADMAP.md
  • packages/cli/__tests__/ccusage-pricing.integration.test.ts

@ohong
ohong merged commit 994c8e7 into main Aug 25, 2026
5 checks passed
@ohong
ohong deleted the oh-review-followups branch August 30, 2026 09:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant