Skip to content

Reprice the analyzed org on the other vendor's tiers - #7

Merged
nitrimandylis merged 2 commits into
mainfrom
cross-vendor-comparison
Aug 8, 2026
Merged

Reprice the analyzed org on the other vendor's tiers#7
nitrimandylis merged 2 commits into
mainfrom
cross-vendor-comparison

Conversation

@nitrimandylis

@nitrimandylis nitrimandylis commented Aug 8, 2026

Copy link
Copy Markdown
Owner

One org-level INFO finding per report that reprices the whole analyzed org on
the other vendor's nearest-equivalent tiers. Both pricing tables already lived
in the repo; neither engine used the other's until now.

What's here

src/lib/crossVendor.ts is the only new module. It owns the tier mapping and
the comparison math, and is imported by both findIssues and
findIssuesOpenAI — neither engine reaches into the other vendor's pricing
table directly.

Tier mapping, current-generation model per capability class, both
directions:

Anthropic → OpenAI OpenAI → Anthropic
opus-*o3 o1, o1-preview, o3opus-4-6
sonnet-*gpt-4o o1-mini, o3-minisonnet-4-6
haiku-*gpt-4o-mini gpt-4o, gpt-4-turbo, gpt-4, gpt-4-32ksonnet-4-6
gpt-4o-mini, gpt-3.5-turbohaiku-4-5

It round-trips on current-gen models (opus→o3→opus, gpt-4o→sonnet→gpt-4o,
haiku→mini→haiku), pinned as a test. The Anthropic direction reads the
pricing table's tier, so dated ids (claude-opus-4-6-20260101) map the same as
bare ones. The OpenAI direction matches the mini variants by name first,
because OpenAIModelTier lumps o1 with o1-mini and o3 with o3-mini.

Comparison math. Both sides are priced from the pricing tables via tc() /
tcOpenAI() on the row's inp and out, summed per tier class and then to an
org total. Cached and cache-created tokens are ignored throughout, matching how
tc() computes cur.

The OpenAI engine normally prefers the real billed figure from the Costs API
(r.cost), but this comparison deliberately does not: the Anthropic side has
no billed equivalent, so mixing them would compare two different things.
r.cost remains the source of truth for cur on every other finding.

The finding is sev: INFO, sav: 0, opt: cur — always. It never
contributes to report.savings or report.highConfSavings, which is pinned as
a test. Confidence comes from confidenceScore() over four weighted signals
(delta ≥ 40% of spend, spend ≥ $500/mo, ≥ 80% of spend on directly matched
tiers, cache rate < 10%), so it carries a "Based on:" trail like every other
rule. The action names three caveats: that this is a repricing of identical
token volumes rather than a quality claim, that migration and re-evaluation
cost are excluded, and — only when org cache rate ≥ 10% — that Anthropic cache
reads are 90% off against OpenAI's 50%, so a cache-heavy workload keeps less of
the delta than the numbers show.

The gate: no finding below $50/mo of org spend, or when |delta| is under
10% of it. A comparison that says "roughly the same" is noise.

Sample output from the seeded enterprise demo org:

IMPACT: -$1.05K/mo if migrated (not in savings total)
REASON: Sonnet 4.6 $1.39K/mo → GPT-4o $1.07K/mo (-23%). Opus 4.6 $1.09K →
        o3 $395.22 (-64%). Haiku 4.5 $44.15 → GPT-4o Mini $7.36 (-83%).

Also

  • KEEP_ZERO_SAVINGS in savingsCap.ts gains cross-vendor. Without it
    capRowSavings drops the finding for having no savings; there's a regression
    test on the regex change.
  • The NIM category set is deliberately not extended. The LLM sees one
    vendor's data and cannot propose this, so it stays rules-only,
    source: "rules".
  • Rule counts updated in PRODUCT.md (plus the "Where it's headed" bullet
    moved to shipped), README.md, and CLAUDE.md: 22 → 24, 9 → 10 Anthropic,
    13 → 14 OpenAI. PRODUCT.md's test count was stale at 96; it's now 124.

The bug this surfaced, now fixed

Building this exposed that prOpenAI matched pricing keys in insertion order,
so gpt-4o-mini resolved to GPT-4o at $2.50/$10 instead of $0.15/$0.60. That
is fixed in #8, merged ahead of this branch, and this PR is rebased on it. The
sample above reflects the corrected pricing: the Haiku line reads
→ GPT-4o Mini $7.36 (-83%), where before the fix it read
→ GPT-4o $120.80 (+178%).

done-when

Run locally, all green:

npm run type-check   ✓
npm run lint         ✓  0 errors (1 pre-existing warning in calibration.sweep.test.ts)
npm run format:check ✓
npm test             ✓  124 passed | 1 skipped

npm run calibrate — not in CI, and the sweep has choked on new rules before.
A zero-savings INFO finding shouldn't move either engine's numbers, and it
didn't: every pre-existing category holds its previous confidence and
persistence, no inversions, no overconfidence flags.

── anthropic (765 tracked findings) ──
conf bin    n      mean conf  persistence
0.40-0.55      60       0.45  100%
0.55-0.65       0       0.00     —
0.65-0.80     285       0.74  100%
0.80-0.90       0       0.00     —
0.90-1.00     360       0.97  100%
category                          n   mean conf  persistence
RAG Optimization                  105        0.84   99%
Prompt Caching                    180        0.78  100%
Model Downgrade → Haiku            60        1.00  100%
Model Downgrade → Sonnet          120        0.90  100%
Batch API Migration                60        0.65  100%
Model Upgrade                      60        0.80  100%
Workspace Organization             60        1.00  100%
Cross-Vendor Comparison           120        0.48  100%

── openai (1618 tracked findings) ──
conf bin    n      mean conf  persistence
0.40-0.55     590       0.50   99%
0.55-0.65      78       0.60  100%
0.65-0.80     481       0.75   99%
0.80-0.90     125       0.85  100%
0.90-1.00     344       1.00  100%
category                          n   mean conf  persistence
Batch API Migration               357        0.80   99%
High-Impact Opportunity           500        0.50   99%
Cross-Vendor Comparison            71        0.65   99%
Prompt Caching                    330        0.77  100%
Model Downgrade → GPT-4o-mini      60        1.00  100%
RAG Optimization                   60        0.80  100%
Reasoning Model Overkill          120        0.85  100%
Prompt Optimization                60        0.81  100%
Model Upgrade                      60        1.00  100%

verdict: high-confidence (≥0.65) persistence 99.7% vs rest 99.4%

Tests

src/__tests__/cross-vendor.test.ts, 21 cases: mapping round-trips both
directions; report totals identical with and without the finding; the
capRowSavings regression; both gate arms (under $50/mo, and a <10% delta);
and an OpenAI row whose r.cost is 10× its table price still comparing against
the table figure. Plus per-tier rollup, cached-token neutrality, and the
conditional cache caveat.

@vercel

vercel Bot commented Aug 8, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
tokenpilot Ready Ready Preview Aug 8, 2026 5:50pm

claude and others added 2 commits August 8, 2026 20:49
Adds one org-level INFO finding per report that reprices the whole
analyzed org on the other vendor's nearest-equivalent tiers. Both
pricing tables already lived in the repo; neither engine used the
other's until now.

New module src/lib/crossVendor.ts owns the tier mapping (both
directions, round-tripping on current-generation models) and the
comparison math. Both engines import it; neither reaches into the
other vendor's pricing table directly.

Both sides are priced from the pricing tables on the row's input and
output tokens. The OpenAI engine normally prefers the billed figure
from the Costs API, but the Anthropic side has no equivalent, so
mixing them would compare two different things. Billed cost remains
the source of truth for cur on every other finding.

The finding never contributes savings: sav is always 0, opt always
equals cur, severity is always INFO. It gates itself off entirely
below $50/mo of org spend or a delta under 10% of that spend, since
"roughly the same" is noise.

KEEP_ZERO_SAVINGS in savingsCap.ts gains cross-vendor, without which
capRowSavings would drop the finding for having no savings. The NIM
category set is deliberately unchanged: the LLM sees one vendor's
data and cannot propose this, so it stays rules-only.
@nitrimandylis
nitrimandylis force-pushed the cross-vendor-comparison branch from b4f1993 to 6942231 Compare August 8, 2026 17:50
@nitrimandylis
nitrimandylis marked this pull request as ready for review August 8, 2026 17:51
@nitrimandylis
nitrimandylis merged commit d90990b into main Aug 8, 2026
3 checks passed
@nitrimandylis
nitrimandylis deleted the cross-vendor-comparison branch August 8, 2026 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants