Reprice the analyzed org on the other vendor's tiers - #7
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Adds one org-level INFO finding per report that reprices the whole analyzed org on the other vendor's nearest-equivalent tiers. Both pricing tables already lived in the repo; neither engine used the other's until now. New module src/lib/crossVendor.ts owns the tier mapping (both directions, round-tripping on current-generation models) and the comparison math. Both engines import it; neither reaches into the other vendor's pricing table directly. Both sides are priced from the pricing tables on the row's input and output tokens. The OpenAI engine normally prefers the billed figure from the Costs API, but the Anthropic side has no equivalent, so mixing them would compare two different things. Billed cost remains the source of truth for cur on every other finding. The finding never contributes savings: sav is always 0, opt always equals cur, severity is always INFO. It gates itself off entirely below $50/mo of org spend or a delta under 10% of that spend, since "roughly the same" is noise. KEEP_ZERO_SAVINGS in savingsCap.ts gains cross-vendor, without which capRowSavings would drop the finding for having no savings. The NIM category set is deliberately unchanged: the LLM sees one vendor's data and cannot propose this, so it stays rules-only.
nitrimandylis
force-pushed
the
cross-vendor-comparison
branch
from
August 8, 2026 17:50
b4f1993 to
6942231
Compare
nitrimandylis
marked this pull request as ready for review
August 8, 2026 17:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One org-level INFO finding per report that reprices the whole analyzed org on
the other vendor's nearest-equivalent tiers. Both pricing tables already lived
in the repo; neither engine used the other's until now.
What's here
src/lib/crossVendor.tsis the only new module. It owns the tier mapping andthe comparison math, and is imported by both
findIssuesandfindIssuesOpenAI— neither engine reaches into the other vendor's pricingtable directly.
Tier mapping, current-generation model per capability class, both
directions:
opus-*→o3o1,o1-preview,o3→opus-4-6sonnet-*→gpt-4oo1-mini,o3-mini→sonnet-4-6haiku-*→gpt-4o-minigpt-4o,gpt-4-turbo,gpt-4,gpt-4-32k→sonnet-4-6gpt-4o-mini,gpt-3.5-turbo→haiku-4-5It round-trips on current-gen models (
opus→o3→opus,gpt-4o→sonnet→gpt-4o,haiku→mini→haiku), pinned as a test. The Anthropic direction reads thepricing table's tier, so dated ids (
claude-opus-4-6-20260101) map the same asbare ones. The OpenAI direction matches the mini variants by name first,
because
OpenAIModelTierlumpso1witho1-miniando3witho3-mini.Comparison math. Both sides are priced from the pricing tables via
tc()/tcOpenAI()on the row'sinpandout, summed per tier class and then to anorg total. Cached and cache-created tokens are ignored throughout, matching how
tc()computescur.The OpenAI engine normally prefers the real billed figure from the Costs API
(
r.cost), but this comparison deliberately does not: the Anthropic side hasno billed equivalent, so mixing them would compare two different things.
r.costremains the source of truth forcuron every other finding.The finding is
sev: INFO,sav: 0,opt: cur— always. It nevercontributes to
report.savingsorreport.highConfSavings, which is pinned asa test. Confidence comes from
confidenceScore()over four weighted signals(delta ≥ 40% of spend, spend ≥ $500/mo, ≥ 80% of spend on directly matched
tiers, cache rate < 10%), so it carries a "Based on:" trail like every other
rule. The action names three caveats: that this is a repricing of identical
token volumes rather than a quality claim, that migration and re-evaluation
cost are excluded, and — only when org cache rate ≥ 10% — that Anthropic cache
reads are 90% off against OpenAI's 50%, so a cache-heavy workload keeps less of
the delta than the numbers show.
The gate: no finding below $50/mo of org spend, or when |delta| is under
10% of it. A comparison that says "roughly the same" is noise.
Sample output from the seeded enterprise demo org:
Also
KEEP_ZERO_SAVINGSinsavingsCap.tsgainscross-vendor. Without itcapRowSavingsdrops the finding for having no savings; there's a regressiontest on the regex change.
vendor's data and cannot propose this, so it stays rules-only,
source: "rules".PRODUCT.md(plus the "Where it's headed" bulletmoved to shipped),
README.md, andCLAUDE.md: 22 → 24, 9 → 10 Anthropic,13 → 14 OpenAI.
PRODUCT.md's test count was stale at 96; it's now 124.The bug this surfaced, now fixed
Building this exposed that
prOpenAImatched pricing keys in insertion order,so
gpt-4o-miniresolved to GPT-4o at $2.50/$10 instead of $0.15/$0.60. Thatis fixed in #8, merged ahead of this branch, and this PR is rebased on it. The
sample above reflects the corrected pricing: the Haiku line reads
→ GPT-4o Mini $7.36 (-83%), where before the fix it read→ GPT-4o $120.80 (+178%).done-when
Run locally, all green:
npm run calibrate— not in CI, and the sweep has choked on new rules before.A zero-savings INFO finding shouldn't move either engine's numbers, and it
didn't: every pre-existing category holds its previous confidence and
persistence, no inversions, no overconfidence flags.
Tests
src/__tests__/cross-vendor.test.ts, 21 cases: mapping round-trips bothdirections; report totals identical with and without the finding; the
capRowSavingsregression; both gate arms (under $50/mo, and a <10% delta);and an OpenAI row whose
r.costis 10× its table price still comparing againstthe table figure. Plus per-tier rollup, cached-token neutrality, and the
conditional cache caveat.