Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Aggregation (agg/aggOpenAI)
└─ Consolidates usage buckets by model/key/workspace
Analysis Engine (findIssues/findIssuesOpenAI)
└─ Runs the vendor's rule set (9 Anthropic / 13 OpenAI) with confidence
└─ Runs the vendor's rule set (10 Anthropic / 14 OpenAI) with confidence
scoring; optional NIM consensus merges LLM proposals on top, priced
by lib/{vendor}/costing.ts — the LLM never emits dollar figures
Expand Down Expand Up @@ -78,7 +78,7 @@ History page → Detail routes → /recommendations, /analytics, /raw-data
- Service-level analytics: tracks spending by service type (embeddings, completions, audio, images, etc.)
- Per-project service breakdown charts showing monthly usage across all services

### Analysis Engine (22 rules: 9 Anthropic, 13 OpenAI)
### Analysis Engine (24 rules: 10 Anthropic, 14 OpenAI)

Located in `lib/anthropic/analysis.ts` and `lib/openai/analysis.ts` — the
`RULE n:` comment headers in those files are the authoritative list. The core
Expand Down Expand Up @@ -110,6 +110,12 @@ categories:
6. **Legacy Model** (Anthropic only)
- Outdated generation (g=1) still in use

7. **Cross-Vendor Comparison** (both vendors, one org-level finding per report)
- Reprices the whole org on the other vendor's nearest-equivalent tiers
- Both sides priced from the pricing tables (never OpenAI's billed `r.cost`)
- Always INFO, always `sav: 0` — never contributes to report savings
- Gated off below $50/mo org spend or a delta under 10% of that spend

**Confidence Scoring:** Multi-signal weighted system (0-1 scale)

- Signals: usage volume, consistency (CoV), active days, input variance, temporal patterns
Expand All @@ -126,6 +132,7 @@ categories:
- `lib/anthropic/costing.ts` - deterministic optimized-cost formulas per category (OpenAI parallel in `lib/openai/costing.ts`)
- `lib/nim/analysis.ts` - NIM LLM proposals, deterministic pricing, consensus merge (`source: rules|llm|both`)
- `lib/savingsCap.ts` - caps cumulative per-row savings at the row's spend, applied by every engine output
- `lib/crossVendor.ts` - cross-vendor tier mapping + comparison math, imported by both engines (rules-only, never NIM)
- `lib/anthropic/pricing.ts` - Model pricing table, cost calculation (`tc()` function)
- `lib/openai/*` - Parallel implementations for OpenAI
- `lib/formatters.ts` - Currency (`$`) and percentage (`P`) formatters
Expand Down
13 changes: 9 additions & 4 deletions PRODUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Built as part of Nick's Omilia internship; doubles as a portfolio project.

## Where it stands (2026-08-06)

- **22 detection rules** (9 Anthropic, 13 OpenAI) with weighted-signal
- **24 detection rules** (10 Anthropic, 14 OpenAI) with weighted-signal
confidence scoring and per-finding signal trails.
- **Consensus engine**: optional NVIDIA NIM augmentation runs alongside the
rules, proposes findings from a fixed category set, and never prices —
Expand All @@ -25,7 +25,7 @@ Built as part of Nick's Omilia internship; doubles as a portfolio project.
incident month) or a lean startup (1-2 minor findings, on purpose). Each
run is internally coherent across its 6 months; the generators are pure
and clock-free, reproducible via explicit seeds (`DEMO_SEED` in tests).
- 117 vitest tests pin the money math; CI runs type-check, lint, format, test.
- 138 vitest tests pin the money math; CI runs type-check, lint, format, test.
- AGPL-3.0.

Corrected 2026-08-08: `prOpenAI` matched pricing-table keys in insertion
Expand All @@ -47,8 +47,13 @@ Next candidates, in rough order (none committed):
responses; needs a real key and a real llama round-trip. The demo now
makes exactly one NIM call per run (current month only), so it doubles
as the cheapest live test.
2. **Cross-vendor comparison** — both pricing tables are in the repo; "this
workload on the other vendor" is a finding single-vendor tools can't make.

Shipped from this list 2026-08-08: **cross-vendor comparison** — one org-level
INFO finding per report reprices the whole analyzed org on the other vendor's
nearest-equivalent tiers (`src/lib/crossVendor.ts`, imported by both engines).
Both sides are priced from the pricing tables, never from OpenAI's billed
`r.cost`, which has no Anthropic counterpart. It contributes zero savings by
construction and gates itself off below $50/mo or a sub-10% delta.

Shipped from this list 2026-08-06: temporal/scheduling rules (weekly-cadence
5d/4c, steady-batch 5c with a weekend-parity gate that keeps batch advice off
Expand Down
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ _an LLM spend auditor that reads your Admin API and tells you where the money le
![telemetry](https://img.shields.io/badge/telemetry-0-3fb950?style=flat-square&labelColor=111111)
![keys](https://img.shields.io/badge/your_API_keys-never_leave_the_browser-3fb950?style=flat-square&labelColor=111111)
![access](https://img.shields.io/badge/access-read--only._we_touch_nothing-d4a017?style=flat-square&labelColor=111111)
![rules](https://img.shields.io/badge/detection_rules-22-d4a017?style=flat-square&labelColor=111111)
![rules](https://img.shields.io/badge/detection_rules-24-d4a017?style=flat-square&labelColor=111111)
![engine](https://img.shields.io/badge/AI_augmentation-detects%2C_never_prices-3fb950?style=flat-square&labelColor=111111)
![savings](<https://img.shields.io/badge/savings-real._(your_CFO_may_weep)-3fb950?style=flat-square&labelColor=111111>)

Expand All @@ -32,7 +32,7 @@ _an LLM spend auditor that reads your Admin API and tells you where the money le

Most teams discover their LLM bill the way you discover a parking ticket.
TokenPilot reads your **Anthropic and OpenAI Admin APIs**, runs your actual
token volumes through a **22-rule detection engine** (9 Anthropic, 13 OpenAI),
token volumes through a **24-rule detection engine** (10 Anthropic, 14 OpenAI),
and hands back confidence-scored savings recommendations in about 60 seconds.
No theoretical benchmarks — every dollar figure is computed from _your_ usage.

Expand All @@ -41,7 +41,7 @@ the tab does, and never gets sent anywhere except the vendor it belongs to.

```console
nick@tokenpilot:~$ audit --vendor anthropic
[✓] 9 rules executed. 4 findings. estimated waste: not zero.
[✓] 10 rules executed. 4 findings. estimated waste: not zero.
[i] your keys stayed in this tab. as is tradition.
```

Expand All @@ -58,6 +58,7 @@ nick@tokenpilot:~$ audit --vendor anthropic
| 07 | **reasoning-model overkill** | o-series reasoning premiums spent on non-reasoning work |
| 08 | **quality upgrade** | the reverse case — places a smarter model would pay for itself |
| 09 | **legacy model usage** | deprecated models quietly costing more for less |
| 10 | **cross-vendor comparison** | your exact token volumes repriced on the other vendor's tiers |

Each finding gets **multi-signal confidence scoring** — volume, consistency,
active days, temporal patterns — and carries its **signal trail** ("based on:
Expand Down
Loading
Loading