diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f9fa560..493f478 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -64,7 +64,9 @@ jobs: # The llm group (anthropic SDK) is installed so the Haiku runner's # retry and key-redaction tests use the SDK's real exception classes. # They run against a fake client; no key is set and no API is called. - run: uv sync --locked --group llm + # The figures group (matplotlib) lets the figure tests draw from the + # committed JSON. + run: uv sync --locked --group llm --group figures - name: make lint (ruff check, ruff format --check, em dash check) run: make lint diff --git a/.gitignore b/.gitignore index 9e19bed..0bc8d1b 100644 --- a/.gitignore +++ b/.gitignore @@ -26,11 +26,12 @@ checkpoints/ *.pt *.onnx -# Run outputs. Committed: results/runs/*.json, results/ac2.json and -# results/logits-manifest.json (SHA-256 of every logits archive). Not +# Run outputs. Committed: results/runs/*.json, results/ac2.json, +# results/logits-manifest.json (SHA-256 of every logits archive) and +# everything `make report` reads or writes (analysis, efficiency, cost, +# figures, report.md). Not # committed: the archives themselves (about 5 MB each, about 40 runs); # they are attached to a GitHub Release and checked with `make verify-logits`. -results/summary.md results/logits/ # Haiku replies: the journal and the predictions file (about 3 MB) go to the # GitHub Release like the logits; results/llm-manifest.json and diff --git a/DEVLOG.md b/DEVLOG.md index a9ff26b..d213f6a 100644 --- a/DEVLOG.md +++ b/DEVLOG.md @@ -4,6 +4,67 @@ --- +## 2026-09-29（深夜，五）：PR #18 審查修正（R1 到 R6） + +### 本次工作 / 執行摘要 +- 狀態：RQ1 到 RQ5 完成；**Tier 1 驗收未完成**。Drew 2026-09-29 決定：AC1 維持原定義（改稱 AC1b：乾淨 clone 後 `make setup && make reproduce` 完整實跑一次，含訓練與 Haiku），必須實跑成功才算達成；另新增 AC1a（從 Release artifacts 驗證 SHA-256 並離線重建分析與 README），作為快速日常驗證層，**不取代 AC1b**。AC1b 成功前 Tier 1 驗收狀態維持未完成。本 PR 不實作 AC1。 +- **R1**：原本「README 與重算一致」只守一致性，欄位接錯後重生 README，CI 仍綠（審查突變 X3、X4 存活）。新增表格驅動測試：首屏 12 個數字各自對應 `summary.json` 的明確路徑，測試自己格式化，不經 report.py，斷言渲染字串等於該欄位的值。 +- **R2**：k=10 hybrid 補上三個 seed 的絕對呼叫次數：1,235、1,157、1,548（test 共 5,500 筆），並納入 R1 的測試。 +- **R3**：routers 圖 (a) 原本是長條圖且 y 軸截斷在 70，會放大 91.9 對 92.1 的差距。改成點圖加誤差棒，y 軸從 75% 起，圖說寫明「看點的距離，不看長條長度」；(b) 仍是從 0 開始的長條圖。 +- R4：「約 87%」改寫為「把 test 加權到 validation 的 OOS 比例後，validation 與 test 的 risk 差距縮小約 87%」，屬描述性寫法。R5：效率表註腳改為「準確率與 OOS recall 用 validation 選的 8 類聚合，ECE 是 151 類」。R6：Haiku 延遲註明不是同條件比較（不同機器，中間有網路）。 +- 延遲的同架構比對：計時模型改用訓練時同一個 loader（`train.load_model_and_tokenizer`）建立；model revision、max_length、torch 與 transformers 版本必須等於訓練 run 的記錄，否則停止。另記錄 `attention_implementation`（兩個模型都是 sdpa；訓練 run 沒有記錄這個欄位，因為 loader 與 transformers 版本相同，選法一致）。 +- 因為延遲改了程式，重跑 `make bench-cpu`，接著重生 cost、report、figures。 + +### 核心發現 / 數據 +- 重量後 CPU p50 / p95：BERT 15.5 / 18.4 ms，ModernBERT 20.9 / 28.5 ms（上一則是 15.6 / 17.0 與 20.2 / 23.2）。p50 差不到 1 ms，p95 對同機其他負載較敏感，重量時差了約 5 ms；這是同一台機器兩次量測的差異，不是程式造成的。 +- 本機推論成本隨延遲重量小幅改變（每筆 1.121e-06 → 1.159e-06 US$），連帶影響損益兩平：只算訓練的表只有 ModernBERT k=100 hybrid US$2/h 一格變動（1,830 → 1,831）；標註敏感度表 6 格都小幅變動（均小於 0.02%，例如 k=100、每筆 US$0.2：8,387,456 → 8,388,334）。`summary.json`、`curves.json`、`haiku.json` 與上一版逐欄相同。 + +### Blockers / 遇到的問題 +- (無) + +### Next +- [ ] PLAN §5 記錄決策：AC1 改稱 AC1b（定義不變、不弱化），新增 AC1a，並寫入凍結的 AC1b 通過標準 +- [ ] AC1a：`make reproduce-artifacts`（或同等名稱）從 Release 驗證並重建，接 CI +- [ ] AC1b：`make reproduce` 在乾淨 clone（固定已合併 commit、lockfile、資料與模型 revision）完整實跑一次；Haiku 另設 US$5 的 reproduction-validation 上限，與原始實驗的 US$3.19 分開記錄 +- [ ] Tier 2（步驟 6） + +### Files / Budget +- `src/tinyrouter/report.py`、`figures.py`、`latency.py`；`tests/test_report.py`、`test_latency.py`；`results/efficiency/cpu_latency.json`、`results/cost/cost.json`、`results/figures/*.png`、`results/report.md`；`README.md`；`DEVLOG.md` +- API 花費：US$0 + +--- + +## 2026-09-29（深夜，四）：步驟 5，效率表、成本、圖與 README 首屏自動產生 + +### 本次工作 / 執行摘要 +- **CPU 延遲（AC5，`make bench-cpu`）**：曲線權重已刪，改用同架構量：鎖定 revision 的預訓練骨架 + 151 類分類頭（seed 42 隨機初始化）+ 同 tokenizer 與 max_length。延遲只取決於架構與輸入形狀，不取決於權重數值；程式比對參數量必須等於 k=100 訓練 run 記錄的值（BERT 109,598,359、ModernBERT 149,720,983），不等就停。量法：CPU、batch 1、`torch.inference_mode()`、4 個 intra-op 執行緒、interop 1、暖機 50 筆，再依序量 validation 前 500 筆；報 tokenization + forward + argmax 與 forward-only 兩種。 +- **Haiku 延遲（`make llm-latency`）**：journal 每筆的 `latency_ms` 是用戶端量的最後一次嘗試時間，含網路往返與 API 排隊，執行時最多 6 個呼叫同時進行；8,600 筆沒有任何重試。 +- **成本（RQ5，`make cost`）**：`results/cost/cost.json` 把實測（M4 訓練 wall-clock、CPU 延遲、Haiku token 與花費、各 router 在 test 的實際 Haiku 花費）與假設（accelerator 每小時 US$0.5、1、2；標註每筆 US$0.05、0.2、1；本機推論每 vCPU-hour US$0.05、滿載）分開存。不把 Mac 購買價算進 run 的成本。損益兩平 = 一次性成本 ÷（LLM-only 每筆 − router 每筆），對 ModernBERT k=10、k=100 的 small-only 與 hybrid、每個價格情境都算，標為情境敏感度。 +- **`make report`**：從 commit 的 JSON 產生 `results/report.md` 與 README 標記之間的區塊（首屏雙欄、router 表、效率表、成本表、聚合方式、圖、Limitations）。`tests/test_report.py` 在 CI 重算並比對，不一致就紅。 +- **`make figures`**：四張 PNG（學習曲線、risk-coverage、router 比較、threshold transfer），Okabe-Ito 配色加線型與標記，300 dpi，拿掉 PNG 的 Software 欄位，兩次輸出逐位元組相同（有測試）。matplotlib 3.11.2 放在新的 `figures` group，CI 的 test job 一併安裝。 +- README 另加隱私說明：Release 存的是公開 CLINC 查詢的 hash 與 LLM 回覆，這個 journal 設計不應原封不動套到含私人查詢的產品。 + +### 核心發現 / 數據 +（取自 `results/efficiency/*.json`、`results/cost/cost.json`、`results/analysis/summary.json`） +- CPU p50 / p95（Apple M4，4 執行緒，端到端）：BERT 15.6 / 17.0 ms，ModernBERT 20.2 / 23.2 ms；Haiku test 669 / 916 ms（含網路）。 +- 訓練（k=100，M4 MPS）：BERT 852 ± 33 s、峰值 4.31 GiB；ModernBERT 1,198 ± 1 s、峰值 6.24 GiB（MPS driver 記憶體取樣）。ECE（151 類，test）溫度校準前後：BERT 3.90 → 3.80，ModernBERT 3.95 → 2.45。 +- Haiku 每 1K test 查詢 US$0.369（約 340K 輸入、5.9K 輸出 token）。 +- 損益兩平（只算訓練，US$1/h）：ModernBERT k=10 hybrid 198 筆、k=100 hybrid 915 筆。訓練算力只值幾分錢；一旦要付標註費，標註主導（k=100、每筆 US$0.2 時約 839 萬筆）。 +- k=100 的 hybrid 在三個 seed 各呼叫 Haiku 170、9、31 次（共 5,500 筆 test），8 類準確率 91.9 → 92.1，是安全與診斷槓桿，不是準確率主要來源；k=10 時 81.5 → 88.0（Haiku 呼叫 23.9%）才是 fallback 價值的主要證據。 + +### Blockers / 遇到的問題 +- (無) + +### Next +- [ ] Tier 2（步驟 6）：ONNX int8 + FastAPI + Docker + 壓測，完成後把 RQ5 的本機延遲換成 ONNX 實測 + +### Files / Budget +- 新增：`src/tinyrouter/latency.py`、`cost.py`、`figures.py`；`tests/test_latency.py`、`test_cost.py`、`test_report.py`、`test_figures.py`；`results/efficiency/*.json`、`results/cost/cost.json`、`results/figures/*.png`、`results/report.md` +- 修改：`src/tinyrouter/report.py`（改寫）、`Makefile`、`README.md`、`pyproject.toml`、`uv.lock`、`.github/workflows/ci.yml`、`.gitignore`、`docs/OPERATIONS.md`、`DEVLOG.md` +- API 花費：US$0 + +--- + ## 2026-09-29（深夜，三）：PR #17 複查小修（R1 到 R4） ### 本次工作 / 執行摘要 diff --git a/Makefile b/Makefile index cf174c4..f2c3498 100644 --- a/Makefile +++ b/Makefile @@ -1,5 +1,6 @@ .PHONY: setup lint format test test-network smoke train evaluate ac2 pilot-lr pilot-steps baselines \ - curve oos-ablation verify-logits llm-smoke llm verify-llm analysis report clean-checkpoints + curve oos-ablation verify-logits llm-smoke llm verify-llm analysis bench-cpu llm-latency cost figures report \ + clean-checkpoints CONFIG ?= configs/bert-base.yaml SEED ?= 42 @@ -141,6 +142,30 @@ verify-llm: analysis: uv run python -m tinyrouter.analysis_run --quiet +# AC5 latency. bench-cpu: both encoders on CPU, batch 1, validation rows +# 0-499, 4 threads, the pretrained backbone with a 151-way head (latency +# depends on shapes, not weight values; see src/tinyrouter/latency.py); +# downloads the two base models; writes results/efficiency/cpu_latency.json. +# llm-latency: Haiku's per-call latency from results/llm/haiku-8way.jsonl +# (Release; check it with verify-llm); writes results/efficiency/haiku_latency.json. +bench-cpu: + uv run $(UV_ENV) python -m tinyrouter.latency cpu + +llm-latency: + uv run python -m tinyrouter.latency haiku + +# RQ5 from committed JSON only: measured numbers and assumed prices kept +# apart, break-even per scenario; writes results/cost/cost.json. +cost: + uv run python -m tinyrouter.cost + +# README figures from results/analysis/*.json; writes results/figures/*.png. +figures: + uv run --group figures python -m tinyrouter.figures + +# results/report.md and the README block between the BEGIN/END GENERATED +# markers, from committed JSON. tests/test_report.py fails when either is stale. +# Order after new results: analysis, bench-cpu, llm-latency, cost, figures, report. report: uv run python -m tinyrouter.report diff --git a/README.md b/README.md index 6a5092e..eb90c9a 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,101 @@ A small fine-tuned encoder that routes CLINC150 queries to 7 agents (or out-of-s ## Results -Not produced yet. Every number in this section will be generated by `make report` from `results/*.json`; nothing here is written by hand. + +Test split, 8-way routing (7 agents + out-of-scope), mean ± std over seeds 42, 43, 44. + +| What transferred | What did not transfer | +|---|---| +| ModernBERT-base fine-tuned on all 100 examples per intent routes 91.9 ± 0.1% of test queries correctly in the 8-way space (Claude Haiku 4.5 zero-shot: 82.1%). | The deferral threshold chosen on validation for 2% selective risk gives 7.34 ± 0.86% selective risk on test (ModernBERT k=100; validation: 1.50 ± 0.13%). | +| With 10 examples per intent, deferring low-confidence queries to Haiku lifts 8-way accuracy from 81.5 ± 0.7% (small model alone) to 88.0 ± 0.4%, with Haiku called on 23.9 ± 3.8% of queries (1,235, 1,157, 1,548 of 5,500 for seeds 42, 43, 44). | Validation is 3.2% OOS and test 18.2%. In a reweighting diagnostic, weighting test to the validation OOS share closes about 87% of the gap between validation and test risk; the remaining difference is consistent with a higher conditional error rate on test OOS queries that the model keeps (36.5 ± 3.0% vs 19.9 ± 2.9% on validation). | + +**Engineering conclusion.** A deferral threshold has to be calibrated on labelled data that represents the traffic it will see. A benchmark's validation split is not a risk guarantee for production. + +**How to read the hybrid.** With few labels the fallback is worth the most: at k=10 it is the best evidence for the cascade (numbers above). With enough labels the local model covers almost all traffic: at k=100 the hybrid sends 1.3 ± 1.6% of test queries to Haiku (170, 9, 31 of 5,500 for seeds 42, 43, 44) and moves 8-way accuracy from 91.9 ± 0.1% to 92.1 ± 0.1%. There the fallback is a small safety and diagnostic lever, not the main source of accuracy. + +### Routers (test, 8-way) + +| router | 8-way acc (%) | OOS recall (%) | high-confidence OOS misroute (%) | Haiku calls (%) | Haiku US$ per 1K queries | +|---|---|---|---|---|---| +| LLM-only (Haiku 4.5) | 82.1 | 56.8 | n/a | 100.0 | 0.369 | +| ModernBERT k=10, small-only | 81.5 ± 0.7 | 23.1 ± 3.2 | 76.9 ± 3.2 | 0.0 ± 0.0 | 0.000 ± 0.000 | +| ModernBERT k=10, hybrid (target 2%) | 88.0 ± 0.4 | 53.5 ± 1.5 | 15.4 ± 4.7 | 23.9 ± 3.8 | 0.088 ± 0.014 | +| ModernBERT k=10, oracle (upper bound) | 91.5 ± 0.2 | 61.2 ± 0.7 | 0.0 ± 0.0 | 18.5 ± 0.7 | 0.068 ± 0.003 | +| ModernBERT k=100, small-only | 91.9 ± 0.1 | 61.1 ± 0.4 | 38.9 ± 0.4 | 0.0 ± 0.0 | 0.000 ± 0.000 | +| ModernBERT k=100, hybrid (target 2%) | 92.1 ± 0.1 | 62.4 ± 1.3 | 34.5 ± 5.5 | 1.3 ± 1.6 | 0.005 ± 0.006 | +| ModernBERT k=100, oracle (upper bound) | 95.1 ± 0.1 | 75.6 ± 0.6 | 0.0 ± 0.0 | 8.1 ± 0.1 | 0.030 ± 0.000 | + +### Efficiency (AC5) + +| | BERT-base (historical baseline) | ModernBERT-base (main model) | Claude Haiku 4.5 | +|---|---|---|---| +| parameters | 109.6M | 149.7M | n/a | +| training time, k=100 (Apple M4, MPS) | 852 ± 33 s | 1,198 ± 1 s | none (zero-shot) | +| training peak memory, k=100 (MPS driver, sampled) | 4.31 ± 0.03 GiB | 6.24 ± 0.60 GiB | n/a | +| 8-way accuracy, test (%) | 91.1 ± 0.5 | 91.9 ± 0.1 | 82.1 | +| OOS recall, test (%) | 56.1 ± 2.6 | 61.1 ± 0.4 | 56.8 | +| ECE 151-way, test, before → after temperature (%) | 3.90 ± 0.34 → 3.80 ± 0.44 | 3.95 ± 0.20 → 2.45 ± 0.29 | n/a | +| latency p50 / p95, batch 1 | 15.5 / 18.4 ms | 20.9 / 28.5 ms | 669 / 916 ms (API, incl. network) | + +Encoder latency: Apple M4 CPU, batch 1, 4 threads, torch 2.14.0, `torch.inference_mode()`, 50 warm-up queries, then validation rows 0 to 499, in order; tokenization + forward + argmax. It is timed with the same architecture, not the fine-tuned weights (deleted to save disk): the pinned pretrained backbone, a 151-way head, the same tokenizer and max_length. Latency depends on shapes, not weight values. The timed model is built by the training loader with the model revision, max_length and torch and transformers versions of the trained run, and its parameter count equals the trained run's. Haiku: client-side time per call on the 5,500 test queries, including the network round trip, with up to 6 calls in flight; it is not a like-for-like comparison with the encoders (different machine, a network in between). Peak memory samples `torch.mps.driver_allocated_memory()` after each backward pass and optimizer step (includes the allocator cache). Accuracy and OOS recall use the 8-way aggregation chosen on validation; ECE is over the 151 intents. Values are mean ± std over seeds 42, 43, 44. + +### Cost (RQ5) + +Measured: Haiku costs US$0.369 per 1K test queries (340.2K input and 5.89K output tokens per 1K; the whole run of 8,600 calls cost US$3.18). One ModernBERT training run takes 3.3 min at k=10 and 20.0 min at k=100 on an Apple M4. + +Break-even queries against LLM-only, for **assumed** accelerator prices (training only, no labelling cost, local inference priced as below). A scenario sensitivity, not a forecast: + +| point | router | at US$0.5/h | at US$1/h | at US$2/h | +|---|---|---|---|---| +| ModernBERT k=10 | small-only | 75 | 151 | 302 | +| ModernBERT k=10 | hybrid | 99 | 198 | 396 | +| ModernBERT k=100 | small-only | 452 | 904 | 1,807 | +| ModernBERT k=100 | hybrid | 458 | 915 | 1,831 | + +Break-even compares cost only: at k=10 the small model alone is less accurate than Haiku and the hybrid is more accurate than Haiku (router table). + +Training compute costs cents; labelling dominates once it has to be paid for. CLINC150 is an existing dataset, so labelling cost is an assumed sensitivity only (break-even queries, training plus labelling): + +| hybrid, US$1/h | labelled rows | at US$0.05 per label | at US$0.2 per label | at US$1 per label | +|---|---|---|---|---| +| ModernBERT-base k=10 | 1,525 | 272,134 | 1,087,942 | 5,438,918 | +| ModernBERT-base k=100 | 15,250 | 2,097,770 | 8,388,334 | 41,938,009 | + +Local inference is assumed to cost US$0.05 per vCPU-hour at full utilisation, times the measured p50 latency and thread count. Full grid and every input: `results/cost/cost.json`. + +**8-way aggregation.** The model predicts 151 intents. Two ways map that to 8 agents: take the argmax intent and map it (`argmax`), or sum the probabilities per agent and take the largest (`summed`). Each run picks one by validation 8-way accuracy only (ties go to argmax). BERT-base: k=1 summed, k=5 summed, k=10 per seed summed/argmax/argmax, k=25 argmax, k=50 per seed summed/summed/argmax, k=100 per seed argmax/summed/argmax. ModernBERT-base: k=1 summed, k=5 summed, k=10 summed, k=25 per seed summed/argmax/summed, k=50 argmax, k=100 argmax. + +### Figures + +![Learning curves](results/figures/learning_curves.png) + +*Test 8-way accuracy and OOS recall against labelled examples per intent (log scale), small model alone, 8-way aggregation chosen on validation; bands are mean ± std over three seeds. The dash-dot line is Claude Haiku 4.5 zero-shot on the same queries. Shaded: k ≤ 10 trains for the fixed 400-step budget.* + +![Risk-coverage](results/figures/risk_coverage.png) + +*Test selective risk (8-way error among the queries the small model keeps) against coverage, for ModernBERT at k=10 and k=100, for each confidence signal; the band is the std of the temperature-scaled MSP, the main signal. The four signals nearly coincide.* + +![Router comparison](results/figures/routers.png) + +*Test 8-way accuracy (a) and share of queries sent to Haiku (b) for LLM-only, small-only, the hybrid (threshold chosen on validation for 2% selective risk) and the oracle, which defers exactly the small model's errors. Panel (a) uses points, not bars, and its axis starts at 75%, so vertical distance is the difference in accuracy points and nothing is read from bar length; panel (b) starts at 0. Error bars are std over three seeds.* + +![Threshold transfer](results/figures/threshold_transfer.png) + +*Selective risk of the hybrid whose threshold was chosen on validation for 2% risk: on validation, on test, and on test reweighted to the validation OOS share (a diagnostic, not a method). Points appear where all three seeds had a feasible threshold.* + +## Limitations + +- **Hyperparameters at the edge of their grids.** The validation pilots chose values at the upper end: learning rate 5e-05 is the top of {1e-05, 2e-05, 5e-05}; S_min 400 is the top of {100, 200, 400}. A better value may lie beyond. +- **Small k is a fixed step budget.** k in {1, 5, 10} trains for S_min = 400 steps (more than 5 epochs), so those points show performance at that budget. +- **Latency uses the same architecture, not the trained weights** (see the efficiency table note). +- **Validation thresholds do not transfer** to test (headline above). +- **One benchmark.** CLINC150, English, one domain mix; BANKING77-OOS is planned for v0.2. +- Training on Apple MPS is not bit-for-bit deterministic, hence three seeds. + + +### Privacy of the stored LLM replies + +The `haiku-predictions` Release stores, for every query, a SHA-256 of the query text, Haiku's reply, token counts and latency. That is fine here because every query is public CLINC150 text. The same journal design should not be copied unchanged into a product that sees private queries: a hash of a short, guessable query can be reversed by hashing candidates, and replies can echo what the user wrote. There it needs a retention limit, a keyed hash or no query identifier at all, and access control. ## Quick start @@ -39,7 +133,11 @@ Other targets: | `make verify-llm` | check `results/llm/haiku-8way.jsonl`: every row once, SHA-256 equal in the file, the summary and the manifest | | `make verify-logits` | check every archive in `results/logits/` against `results/logits-manifest.json` | | `make analysis` | RQ2 to RQ4 from the stored logits and Haiku predictions, no training and no API calls; every temperature, aggregation, threshold and signal is chosen on validation; writes `results/analysis/{summary,curves,haiku}.json` and ends with `completed analysis (75/75 archives, 8600/8600 llm rows, 25 groups)` | -| `make report` | build `results/summary.md` from `results/runs/*.json` | +| `make bench-cpu` | CPU batch-1 latency of both encoders on validation rows 0-499 (4 threads, 50 warm-up queries), with the pinned pretrained backbone and a 151-way head; stops unless the parameter count equals the trained k=100 run's; writes `results/efficiency/cpu_latency.json` | +| `make llm-latency` | Haiku's per-call latency from `results/llm/haiku-8way.jsonl`; writes `results/efficiency/haiku_latency.json` | +| `make cost` | RQ5 from committed JSON: measured training time, latency and Haiku spend, assumed prices, break-even per scenario; writes `results/cost/cost.json` | +| `make figures` | the four README figures from `results/analysis/*.json`; writes `results/figures/*.png` (byte-identical on rerun with the same matplotlib) | +| `make report` | `results/report.md` and the generated block of this README, from committed JSON; `tests/test_report.py` fails when either is stale | | `make clean-checkpoints` | delete all trained weights | Run every target from the repository root: `checkpoint_root` and `results_root` in the configs are relative paths. @@ -94,7 +192,10 @@ src/tinyrouter/ analysis.py one archive: 8-way aggregation, OOS, signals, ECE, fallback, oracle diagnostics.py why validation thresholds miss the target on test (OOS share, reweighting) analysis_run.py `make analysis`: completion checks, label cross-checks, mean and std over seeds - report.py results/*.json -> results/summary.md + latency.py CPU batch-1 latency of the encoders; Haiku latency from the journal + cost.py RQ5: measured costs, assumed prices, break-even + figures.py README figures from results/analysis/*.json + report.py results/*.json -> results/report.md and the README's generated block smoke.py end-to-end wiring check tests/ pytest; `network` marker for Hub downloads ``` diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 0512ec0..68c8d5f 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -8,7 +8,7 @@ TinyRouter is a research repository with no deployment target: nothing runs as a | job | runs | required to merge | |---|---|---| -| `test` | tracked-files guard, `uv sync --locked --group llm` (the SDK's exception classes for the Haiku runner tests; no key, no API call), `make lint`, `make test` (offline) | yes | +| `test` | tracked-files guard, `uv sync --locked --group llm --group figures` (the SDK's exception classes for the Haiku runner tests, no key and no API call; matplotlib for the figure tests), `make lint`, `make test` (offline) | yes | | `commit-hygiene` | rejects tool-attribution trailers in commit messages (patterns in `.github/disallowed-trailers.txt`) | yes | | `network` | `make test-network` and `make smoke` against the Hugging Face Hub | no | | `pr-text-hygiene` (`pr-text.yml`) | rejects the same patterns in the PR title and body, on open, edit, push and reopen | yes (added to the required checks once it is on `main`) | diff --git a/pyproject.toml b/pyproject.toml index 5210691..0c87a41 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -49,6 +49,11 @@ onnx = [ "onnxruntime==1.30.0", "onnxscript==0.7.2", ] +# `make figures` (results/figures/*.png). Not installed by `uv sync`; CI's +# test job installs it so the figure tests run there. +figures = [ + "matplotlib==3.11.2", +] [tool.uv] default-groups = ["dev"] diff --git a/results/cost/cost.json b/results/cost/cost.json new file mode 100644 index 0000000..bd3740f --- /dev/null +++ b/results/cost/cost.json @@ -0,0 +1,843 @@ +{ + "format_version": 1, + "inputs_sha256": { + "analysis/summary.json": "6d108f1876514e00a6312858fef47473a34e610bd9b04617ab71fae72a149f06", + "analysis/haiku.json": "d91a1265d0f68fea2f3756a44c9d4d68ac337074982cfb738cd6330e45831acc", + "llm/haiku-8way.json": "d2423251d948f730a50cadca163327edca1ea294d6f5bfe91c334a4e8facfa99", + "efficiency/cpu_latency.json": "894d7f1d3ec6bf3c5e63798342bf16a788f4c104bc7c3eb30b635ba7b8c304f9", + "curves/bert.json": "35b46d514e6b47728501678dd6e029fb6a670e9b57d4514d6a2e73827aa987a5", + "curves/modernbert.json": "515c153596bfb0115b39b2a6166999135014ef2ca68361a35764aa1866e9ffc4" + }, + "measured": { + "training": { + "bert": { + "1": { + "train_wall_seconds": { + "mean": 136.701333, + "std": 3.288993, + "values": [ + 133.46, + 136.608, + 140.036 + ] + }, + "train_rows": 153, + "steps": 400, + "device": "mps" + }, + "5": { + "train_wall_seconds": { + "mean": 136.528, + "std": 0.787443, + "values": [ + 136.13, + 137.435, + 136.019 + ] + }, + "train_rows": 763, + "steps": 400, + "device": "mps" + }, + "10": { + "train_wall_seconds": { + "mean": 137.411667, + "std": 0.815936, + "values": [ + 137.968, + 137.792, + 136.475 + ] + }, + "train_rows": 1525, + "steps": 400, + "device": "mps" + }, + "25": { + "train_wall_seconds": { + "mean": 206.634, + "std": 0.366004, + "values": [ + 207.001, + 206.269, + 206.632 + ] + }, + "train_rows": 3813, + "steps": 600, + "device": "mps" + }, + "50": { + "train_wall_seconds": { + "mean": 411.097333, + "std": 1.184395, + "values": [ + 412.406, + 410.787, + 410.099 + ] + }, + "train_rows": 7625, + "steps": 1195, + "device": "mps" + }, + "100": { + "train_wall_seconds": { + "mean": 851.581, + "std": 32.537189, + "values": [ + 886.184, + 846.954, + 821.605 + ] + }, + "train_rows": 15250, + "steps": 2385, + "device": "mps" + } + }, + "modernbert": { + "1": { + "train_wall_seconds": { + "mean": 196.575, + "std": 3.053313, + "values": [ + 193.341, + 196.976, + 199.408 + ] + }, + "train_rows": 153, + "steps": 400, + "device": "mps" + }, + "5": { + "train_wall_seconds": { + "mean": 199.517, + "std": 0.764402, + "values": [ + 199.008, + 199.147, + 200.396 + ] + }, + "train_rows": 763, + "steps": 400, + "device": "mps" + }, + "10": { + "train_wall_seconds": { + "mean": 199.971667, + "std": 1.186576, + "values": [ + 200.03, + 201.128, + 198.757 + ] + }, + "train_rows": 1525, + "steps": 400, + "device": "mps" + }, + "25": { + "train_wall_seconds": { + "mean": 302.031667, + "std": 1.210153, + "values": [ + 301.341, + 301.325, + 303.429 + ] + }, + "train_rows": 3813, + "steps": 600, + "device": "mps" + }, + "50": { + "train_wall_seconds": { + "mean": 600.910667, + "std": 1.226663, + "values": [ + 602.167, + 600.849, + 599.716 + ] + }, + "train_rows": 7625, + "steps": 1195, + "device": "mps" + }, + "100": { + "train_wall_seconds": { + "mean": 1198.300667, + "std": 1.482022, + "values": [ + 1199.467, + 1198.802, + 1196.633 + ] + }, + "train_rows": 15250, + "steps": 2385, + "device": "mps" + } + } + }, + "cpu_latency": { + "intra_op_threads": 4, + "hardware": "Apple M4", + "end_to_end_ms": { + "bert": { + "p50_ms": 15.501, + "p95_ms": 18.379, + "mean_ms": 15.652 + }, + "modernbert": { + "p50_ms": 20.864, + "p95_ms": 28.504, + "mean_ms": 21.543 + } + } + }, + "haiku": { + "model": "claude-haiku-4-5-20251001", + "calls": 8600, + "input_tokens": 2925596, + "output_tokens": 50630, + "cost_usd": 3.178746, + "input_tokens_per_1k_queries": 340185.581, + "output_tokens_per_1k_queries": 5887.209, + "cost_usd_per_1k_queries": { + "validation": 0.369852, + "test": 0.369492 + } + }, + "routers": { + "modernbert/k10": { + "small_only": { + "accuracy_8": 0.81503, + "llm_call_rate": 0.0, + "llm_cost_usd_per_1k": 0.0 + }, + "hybrid": { + "accuracy_8": 0.879515, + "llm_call_rate": 0.238788, + "llm_cost_usd_per_1k": 0.087936 + } + }, + "modernbert/k100": { + "small_only": { + "accuracy_8": 0.918727, + "llm_call_rate": 0.0, + "llm_cost_usd_per_1k": 0.0 + }, + "hybrid": { + "accuracy_8": 0.921273, + "llm_call_rate": 0.012727, + "llm_cost_usd_per_1k": 0.004693 + } + } + }, + "router_target_risk": 0.02 + }, + "assumptions": { + "accelerator_usd_per_hour": [ + 0.5, + 1.0, + 2.0 + ], + "label_usd_per_example": [ + 0.05, + 0.2, + 1.0 + ], + "vcpu_usd_per_hour": 0.05, + "notes": { + "training": "measured M4 wall-clock hours of one training run, priced as if billed at each accelerator-hour rate; these are assumed rates, not money spent (the runs used a Mac already owned, and its purchase price is not charged to a run)", + "labelling": "CLINC150 is an existing dataset, so no labelling was paid; cost = labelled training rows x an assumed price per label, a sensitivity only", + "local_inference": "measured CPU p50 end-to-end latency x intra-op threads (vCPUs) x an assumed price per vCPU-hour, at full utilisation; idle capacity would cost more" + } + }, + "break_even": { + "formula": "one_time_usd / (llm_only_usd_per_query - router_usd_per_query); null when the router saves nothing per query", + "label": "scenario sensitivity, not a forecast", + "rows": [ + { + "point": "modernbert/k10", + "router": "small_only", + "accelerator_usd_per_hour": 0.5, + "label_usd_per_example": 0.0, + "one_time_usd": 0.027774, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 75.4, + "break_even_queries_api_only": 75.2 + }, + { + "point": "modernbert/k10", + "router": "small_only", + "accelerator_usd_per_hour": 0.5, + "label_usd_per_example": 0.05, + "one_time_usd": 76.277774, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 207089.2, + "break_even_queries_api_only": 206439.6 + }, + { + "point": "modernbert/k10", + "router": "small_only", + "accelerator_usd_per_hour": 0.5, + "label_usd_per_example": 0.2, + "one_time_usd": 305.027774, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 828130.7, + "break_even_queries_api_only": 825532.8 + }, + { + "point": "modernbert/k10", + "router": "small_only", + "accelerator_usd_per_hour": 0.5, + "label_usd_per_example": 1.0, + "one_time_usd": 1525.027774, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 4140351.9, + "break_even_queries_api_only": 4127363.4 + }, + { + "point": "modernbert/k10", + "router": "small_only", + "accelerator_usd_per_hour": 1.0, + "label_usd_per_example": 0.0, + "one_time_usd": 0.055548, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 150.8, + "break_even_queries_api_only": 150.3 + }, + { + "point": "modernbert/k10", + "router": "small_only", + "accelerator_usd_per_hour": 1.0, + "label_usd_per_example": 0.05, + "one_time_usd": 76.305548, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 207164.6, + "break_even_queries_api_only": 206514.7 + }, + { + "point": "modernbert/k10", + "router": "small_only", + "accelerator_usd_per_hour": 1.0, + "label_usd_per_example": 0.2, + "one_time_usd": 305.055548, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 828206.1, + "break_even_queries_api_only": 825608.0 + }, + { + "point": "modernbert/k10", + "router": "small_only", + "accelerator_usd_per_hour": 1.0, + "label_usd_per_example": 1.0, + "one_time_usd": 1525.055548, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 4140427.3, + "break_even_queries_api_only": 4127438.6 + }, + { + "point": "modernbert/k10", + "router": "small_only", + "accelerator_usd_per_hour": 2.0, + "label_usd_per_example": 0.0, + "one_time_usd": 0.111095, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 301.6, + "break_even_queries_api_only": 300.7 + }, + { + "point": "modernbert/k10", + "router": "small_only", + "accelerator_usd_per_hour": 2.0, + "label_usd_per_example": 0.05, + "one_time_usd": 76.361095, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 207315.4, + "break_even_queries_api_only": 206665.1 + }, + { + "point": "modernbert/k10", + "router": "small_only", + "accelerator_usd_per_hour": 2.0, + "label_usd_per_example": 0.2, + "one_time_usd": 305.111095, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 828356.9, + "break_even_queries_api_only": 825758.3 + }, + { + "point": "modernbert/k10", + "router": "small_only", + "accelerator_usd_per_hour": 2.0, + "label_usd_per_example": 1.0, + "one_time_usd": 1525.111095, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 4140578.1, + "break_even_queries_api_only": 4127588.9 + }, + { + "point": "modernbert/k10", + "router": "hybrid", + "accelerator_usd_per_hour": 0.5, + "label_usd_per_example": 0.0, + "one_time_usd": 0.027774, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 8.7936e-05, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 99.1, + "break_even_queries_api_only": 98.6 + }, + { + "point": "modernbert/k10", + "router": "hybrid", + "accelerator_usd_per_hour": 0.5, + "label_usd_per_example": 0.05, + "one_time_usd": 76.277774, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 8.7936e-05, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 272035.0, + "break_even_queries_api_only": 270915.1 + }, + { + "point": "modernbert/k10", + "router": "hybrid", + "accelerator_usd_per_hour": 0.5, + "label_usd_per_example": 0.2, + "one_time_usd": 305.027774, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 8.7936e-05, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 1087842.9, + "break_even_queries_api_only": 1083364.5 + }, + { + "point": "modernbert/k10", + "router": "hybrid", + "accelerator_usd_per_hour": 0.5, + "label_usd_per_example": 1.0, + "one_time_usd": 1525.027774, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 8.7936e-05, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 5438818.5, + "break_even_queries_api_only": 5416427.9 + }, + { + "point": "modernbert/k10", + "router": "hybrid", + "accelerator_usd_per_hour": 1.0, + "label_usd_per_example": 0.0, + "one_time_usd": 0.055548, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 8.7936e-05, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 198.1, + "break_even_queries_api_only": 197.3 + }, + { + "point": "modernbert/k10", + "router": "hybrid", + "accelerator_usd_per_hour": 1.0, + "label_usd_per_example": 0.05, + "one_time_usd": 76.305548, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 8.7936e-05, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 272134.1, + "break_even_queries_api_only": 271013.8 + }, + { + "point": "modernbert/k10", + "router": "hybrid", + "accelerator_usd_per_hour": 1.0, + "label_usd_per_example": 0.2, + "one_time_usd": 305.055548, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 8.7936e-05, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 1087942.0, + "break_even_queries_api_only": 1083463.1 + }, + { + "point": "modernbert/k10", + "router": "hybrid", + "accelerator_usd_per_hour": 1.0, + "label_usd_per_example": 1.0, + "one_time_usd": 1525.055548, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 8.7936e-05, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 5438917.5, + "break_even_queries_api_only": 5416526.5 + }, + { + "point": "modernbert/k10", + "router": "hybrid", + "accelerator_usd_per_hour": 2.0, + "label_usd_per_example": 0.0, + "one_time_usd": 0.111095, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 8.7936e-05, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 396.2, + "break_even_queries_api_only": 394.6 + }, + { + "point": "modernbert/k10", + "router": "hybrid", + "accelerator_usd_per_hour": 2.0, + "label_usd_per_example": 0.05, + "one_time_usd": 76.361095, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 8.7936e-05, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 272332.2, + "break_even_queries_api_only": 271211.0 + }, + { + "point": "modernbert/k10", + "router": "hybrid", + "accelerator_usd_per_hour": 2.0, + "label_usd_per_example": 0.2, + "one_time_usd": 305.111095, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 8.7936e-05, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 1088140.1, + "break_even_queries_api_only": 1083660.4 + }, + { + "point": "modernbert/k10", + "router": "hybrid", + "accelerator_usd_per_hour": 2.0, + "label_usd_per_example": 1.0, + "one_time_usd": 1525.111095, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 8.7936e-05, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 5439115.6, + "break_even_queries_api_only": 5416723.8 + }, + { + "point": "modernbert/k100", + "router": "small_only", + "accelerator_usd_per_hour": 0.5, + "label_usd_per_example": 0.0, + "one_time_usd": 0.166431, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 451.8, + "break_even_queries_api_only": 450.4 + }, + { + "point": "modernbert/k100", + "router": "small_only", + "accelerator_usd_per_hour": 0.5, + "label_usd_per_example": 0.05, + "one_time_usd": 762.666431, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 2070590.1, + "break_even_queries_api_only": 2064094.6 + }, + { + "point": "modernbert/k100", + "router": "small_only", + "accelerator_usd_per_hour": 0.5, + "label_usd_per_example": 0.2, + "one_time_usd": 3050.166431, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 8281004.8, + "break_even_queries_api_only": 8255027.0 + }, + { + "point": "modernbert/k100", + "router": "small_only", + "accelerator_usd_per_hour": 0.5, + "label_usd_per_example": 1.0, + "one_time_usd": 15250.166431, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 41403216.7, + "break_even_queries_api_only": 41273333.2 + }, + { + "point": "modernbert/k100", + "router": "small_only", + "accelerator_usd_per_hour": 1.0, + "label_usd_per_example": 0.0, + "one_time_usd": 0.332861, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 903.7, + "break_even_queries_api_only": 900.9 + }, + { + "point": "modernbert/k100", + "router": "small_only", + "accelerator_usd_per_hour": 1.0, + "label_usd_per_example": 0.05, + "one_time_usd": 762.832861, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 2071041.9, + "break_even_queries_api_only": 2064545.0 + }, + { + "point": "modernbert/k100", + "router": "small_only", + "accelerator_usd_per_hour": 1.0, + "label_usd_per_example": 0.2, + "one_time_usd": 3050.332861, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 8281456.7, + "break_even_queries_api_only": 8255477.4 + }, + { + "point": "modernbert/k100", + "router": "small_only", + "accelerator_usd_per_hour": 1.0, + "label_usd_per_example": 1.0, + "one_time_usd": 15250.332861, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 41403668.6, + "break_even_queries_api_only": 41273783.6 + }, + { + "point": "modernbert/k100", + "router": "small_only", + "accelerator_usd_per_hour": 2.0, + "label_usd_per_example": 0.0, + "one_time_usd": 0.665723, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 1807.4, + "break_even_queries_api_only": 1801.7 + }, + { + "point": "modernbert/k100", + "router": "small_only", + "accelerator_usd_per_hour": 2.0, + "label_usd_per_example": 0.05, + "one_time_usd": 763.165723, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 2071945.6, + "break_even_queries_api_only": 2065445.9 + }, + { + "point": "modernbert/k100", + "router": "small_only", + "accelerator_usd_per_hour": 2.0, + "label_usd_per_example": 0.2, + "one_time_usd": 3050.665723, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 8282360.4, + "break_even_queries_api_only": 8256378.3 + }, + { + "point": "modernbert/k100", + "router": "small_only", + "accelerator_usd_per_hour": 2.0, + "label_usd_per_example": 1.0, + "one_time_usd": 15250.665723, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 0.0, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 41404572.3, + "break_even_queries_api_only": 41274684.5 + }, + { + "point": "modernbert/k100", + "router": "hybrid", + "accelerator_usd_per_hour": 0.5, + "label_usd_per_example": 0.0, + "one_time_usd": 0.166431, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 4.693e-06, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 457.7, + "break_even_queries_api_only": 456.2 + }, + { + "point": "modernbert/k100", + "router": "hybrid", + "accelerator_usd_per_hour": 0.5, + "label_usd_per_example": 0.05, + "one_time_usd": 762.666431, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 4.693e-06, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 2097312.4, + "break_even_queries_api_only": 2090648.4 + }, + { + "point": "modernbert/k100", + "router": "hybrid", + "accelerator_usd_per_hour": 0.5, + "label_usd_per_example": 0.2, + "one_time_usd": 3050.166431, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 4.693e-06, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 8387876.4, + "break_even_queries_api_only": 8361224.8 + }, + { + "point": "modernbert/k100", + "router": "hybrid", + "accelerator_usd_per_hour": 0.5, + "label_usd_per_example": 1.0, + "one_time_usd": 15250.166431, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 4.693e-06, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 41937551.1, + "break_even_queries_api_only": 41804298.9 + }, + { + "point": "modernbert/k100", + "router": "hybrid", + "accelerator_usd_per_hour": 1.0, + "label_usd_per_example": 0.0, + "one_time_usd": 0.332861, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 4.693e-06, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 915.4, + "break_even_queries_api_only": 912.5 + }, + { + "point": "modernbert/k100", + "router": "hybrid", + "accelerator_usd_per_hour": 1.0, + "label_usd_per_example": 0.05, + "one_time_usd": 762.832861, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 4.693e-06, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 2097770.0, + "break_even_queries_api_only": 2091104.6 + }, + { + "point": "modernbert/k100", + "router": "hybrid", + "accelerator_usd_per_hour": 1.0, + "label_usd_per_example": 0.2, + "one_time_usd": 3050.332861, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 4.693e-06, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 8388334.0, + "break_even_queries_api_only": 8361681.0 + }, + { + "point": "modernbert/k100", + "router": "hybrid", + "accelerator_usd_per_hour": 1.0, + "label_usd_per_example": 1.0, + "one_time_usd": 15250.332861, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 4.693e-06, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 41938008.8, + "break_even_queries_api_only": 41804755.1 + }, + { + "point": "modernbert/k100", + "router": "hybrid", + "accelerator_usd_per_hour": 2.0, + "label_usd_per_example": 0.0, + "one_time_usd": 0.665723, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 4.693e-06, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 1830.7, + "break_even_queries_api_only": 1824.9 + }, + { + "point": "modernbert/k100", + "router": "hybrid", + "accelerator_usd_per_hour": 2.0, + "label_usd_per_example": 0.05, + "one_time_usd": 763.165723, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 4.693e-06, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 2098685.4, + "break_even_queries_api_only": 2092017.0 + }, + { + "point": "modernbert/k100", + "router": "hybrid", + "accelerator_usd_per_hour": 2.0, + "label_usd_per_example": 0.2, + "one_time_usd": 3050.665723, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 4.693e-06, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 8389249.4, + "break_even_queries_api_only": 8362593.4 + }, + { + "point": "modernbert/k100", + "router": "hybrid", + "accelerator_usd_per_hour": 2.0, + "label_usd_per_example": 1.0, + "one_time_usd": 15250.665723, + "llm_only_usd_per_query": 0.000369492, + "router_api_usd_per_query": 4.693e-06, + "router_local_usd_per_query": 1.159e-06, + "break_even_queries": 41938924.2, + "break_even_queries_api_only": 41805667.6 + } + ] + } +} diff --git a/results/efficiency/cpu_latency.json b/results/efficiency/cpu_latency.json new file mode 100644 index 0000000..991c985 --- /dev/null +++ b/results/efficiency/cpu_latency.json @@ -0,0 +1,95 @@ +{ + "format_version": 1, + "method": { + "device": "cpu", + "batch_size": 1, + "mode": "torch.inference_mode", + "queries": "validation rows 0 to 499, in order", + "warmup_queries": 50, + "intra_op_threads": 4, + "inter_op_threads": 1, + "percentiles": "numpy.percentile, method=linear", + "end_to_end": "tokenization + forward + argmax", + "forward": "forward pass only", + "weights": "timed with the pinned pretrained backbone and a randomly initialised 151-way head, not the fine-tuned weights (deleted after their logits were archived); latency depends on the architecture and input shapes, not on the weight values, and the parameter count equals the trained k=100 run's" + }, + "hardware": { + "cpu": "Apple M4", + "machine": "arm64", + "platform": "macOS-26.6.2-arm64-arm-64bit" + }, + "versions": { + "python": "3.12.12", + "torch": "2.14.0", + "transformers": "5.17.0" + }, + "measured_at": "2026-09-29T06:13:26+00:00", + "models": { + "bert": { + "model_name": "google-bert/bert-base-uncased", + "model_revision": "86b5e0934494bd15c9632b12f734a8a67f723594", + "num_labels": 151, + "max_length": 64, + "parameters_total": 109598359, + "parameters_match_trained_run": true, + "matches_trained_run": { + "model_name": "google-bert/bert-base-uncased", + "model_revision": "86b5e0934494bd15c9632b12f734a8a67f723594", + "max_length": "64", + "torch": "2.14.0", + "transformers": "5.17.0" + }, + "loader": "tinyrouter.train.load_model_and_tokenizer (the training loader)", + "attention_implementation": "sdpa", + "end_to_end": { + "n": 500, + "mean_ms": 15.652, + "p50_ms": 15.501, + "p95_ms": 18.379, + "min_ms": 12.606, + "max_ms": 45.588 + }, + "forward": { + "n": 500, + "mean_ms": 15.523, + "p50_ms": 15.377, + "p95_ms": 18.246, + "min_ms": 12.481, + "max_ms": 45.461 + } + }, + "modernbert": { + "model_name": "answerdotai/ModernBERT-base", + "model_revision": "8949b909ec900327062f0ebf497f51aef5e6f0c8", + "num_labels": 151, + "max_length": 64, + "parameters_total": 149720983, + "parameters_match_trained_run": true, + "matches_trained_run": { + "model_name": "answerdotai/ModernBERT-base", + "model_revision": "8949b909ec900327062f0ebf497f51aef5e6f0c8", + "max_length": "64", + "torch": "2.14.0", + "transformers": "5.17.0" + }, + "loader": "tinyrouter.train.load_model_and_tokenizer (the training loader)", + "attention_implementation": "sdpa", + "end_to_end": { + "n": 500, + "mean_ms": 21.543, + "p50_ms": 20.864, + "p95_ms": 28.504, + "min_ms": 15.711, + "max_ms": 66.021 + }, + "forward": { + "n": 500, + "mean_ms": 21.409, + "p50_ms": 20.738, + "p95_ms": 28.335, + "min_ms": 15.608, + "max_ms": 65.883 + } + } + } +} diff --git a/results/efficiency/haiku_latency.json b/results/efficiency/haiku_latency.json new file mode 100644 index 0000000..0642fbe --- /dev/null +++ b/results/efficiency/haiku_latency.json @@ -0,0 +1,32 @@ +{ + "format_version": 1, + "source": "haiku-8way.jsonl", + "measures": "client-side wall time of the last attempt of each messages.create call; includes the network round trip and API queueing; up to 6 calls were in flight", + "calls_retried": 0, + "all": { + "n": 8600, + "mean_ms": 715.513, + "p50_ms": 671.0, + "p95_ms": 921.0, + "min_ms": 534.0, + "max_ms": 7453.0 + }, + "splits": { + "test": { + "n": 5500, + "mean_ms": 710.406, + "p50_ms": 669.0, + "p95_ms": 916.0, + "min_ms": 534.0, + "max_ms": 5601.0 + }, + "validation": { + "n": 3100, + "mean_ms": 724.574, + "p50_ms": 676.0, + "p95_ms": 939.05, + "min_ms": 546.0, + "max_ms": 7453.0 + } + } +} diff --git a/results/figures/learning_curves.png b/results/figures/learning_curves.png new file mode 100644 index 0000000..16eec6f Binary files /dev/null and b/results/figures/learning_curves.png differ diff --git a/results/figures/risk_coverage.png b/results/figures/risk_coverage.png new file mode 100644 index 0000000..db49340 Binary files /dev/null and b/results/figures/risk_coverage.png differ diff --git a/results/figures/routers.png b/results/figures/routers.png new file mode 100644 index 0000000..bd048d4 Binary files /dev/null and b/results/figures/routers.png differ diff --git a/results/figures/threshold_transfer.png b/results/figures/threshold_transfer.png new file mode 100644 index 0000000..72b73c9 Binary files /dev/null and b/results/figures/threshold_transfer.png differ diff --git a/results/report.md b/results/report.md new file mode 100644 index 0000000..6fbaa3d --- /dev/null +++ b/results/report.md @@ -0,0 +1,124 @@ +# TinyRouter results + +Generated by `make report` from `results/*.json`. Do not edit by hand. + + +Test split, 8-way routing (7 agents + out-of-scope), mean ± std over seeds 42, 43, 44. + +| What transferred | What did not transfer | +|---|---| +| ModernBERT-base fine-tuned on all 100 examples per intent routes 91.9 ± 0.1% of test queries correctly in the 8-way space (Claude Haiku 4.5 zero-shot: 82.1%). | The deferral threshold chosen on validation for 2% selective risk gives 7.34 ± 0.86% selective risk on test (ModernBERT k=100; validation: 1.50 ± 0.13%). | +| With 10 examples per intent, deferring low-confidence queries to Haiku lifts 8-way accuracy from 81.5 ± 0.7% (small model alone) to 88.0 ± 0.4%, with Haiku called on 23.9 ± 3.8% of queries (1,235, 1,157, 1,548 of 5,500 for seeds 42, 43, 44). | Validation is 3.2% OOS and test 18.2%. In a reweighting diagnostic, weighting test to the validation OOS share closes about 87% of the gap between validation and test risk; the remaining difference is consistent with a higher conditional error rate on test OOS queries that the model keeps (36.5 ± 3.0% vs 19.9 ± 2.9% on validation). | + +**Engineering conclusion.** A deferral threshold has to be calibrated on labelled data that represents the traffic it will see. A benchmark's validation split is not a risk guarantee for production. + +**How to read the hybrid.** With few labels the fallback is worth the most: at k=10 it is the best evidence for the cascade (numbers above). With enough labels the local model covers almost all traffic: at k=100 the hybrid sends 1.3 ± 1.6% of test queries to Haiku (170, 9, 31 of 5,500 for seeds 42, 43, 44) and moves 8-way accuracy from 91.9 ± 0.1% to 92.1 ± 0.1%. There the fallback is a small safety and diagnostic lever, not the main source of accuracy. + +### Routers (test, 8-way) + +| router | 8-way acc (%) | OOS recall (%) | high-confidence OOS misroute (%) | Haiku calls (%) | Haiku US$ per 1K queries | +|---|---|---|---|---|---| +| LLM-only (Haiku 4.5) | 82.1 | 56.8 | n/a | 100.0 | 0.369 | +| ModernBERT k=10, small-only | 81.5 ± 0.7 | 23.1 ± 3.2 | 76.9 ± 3.2 | 0.0 ± 0.0 | 0.000 ± 0.000 | +| ModernBERT k=10, hybrid (target 2%) | 88.0 ± 0.4 | 53.5 ± 1.5 | 15.4 ± 4.7 | 23.9 ± 3.8 | 0.088 ± 0.014 | +| ModernBERT k=10, oracle (upper bound) | 91.5 ± 0.2 | 61.2 ± 0.7 | 0.0 ± 0.0 | 18.5 ± 0.7 | 0.068 ± 0.003 | +| ModernBERT k=100, small-only | 91.9 ± 0.1 | 61.1 ± 0.4 | 38.9 ± 0.4 | 0.0 ± 0.0 | 0.000 ± 0.000 | +| ModernBERT k=100, hybrid (target 2%) | 92.1 ± 0.1 | 62.4 ± 1.3 | 34.5 ± 5.5 | 1.3 ± 1.6 | 0.005 ± 0.006 | +| ModernBERT k=100, oracle (upper bound) | 95.1 ± 0.1 | 75.6 ± 0.6 | 0.0 ± 0.0 | 8.1 ± 0.1 | 0.030 ± 0.000 | + +### Efficiency (AC5) + +| | BERT-base (historical baseline) | ModernBERT-base (main model) | Claude Haiku 4.5 | +|---|---|---|---| +| parameters | 109.6M | 149.7M | n/a | +| training time, k=100 (Apple M4, MPS) | 852 ± 33 s | 1,198 ± 1 s | none (zero-shot) | +| training peak memory, k=100 (MPS driver, sampled) | 4.31 ± 0.03 GiB | 6.24 ± 0.60 GiB | n/a | +| 8-way accuracy, test (%) | 91.1 ± 0.5 | 91.9 ± 0.1 | 82.1 | +| OOS recall, test (%) | 56.1 ± 2.6 | 61.1 ± 0.4 | 56.8 | +| ECE 151-way, test, before → after temperature (%) | 3.90 ± 0.34 → 3.80 ± 0.44 | 3.95 ± 0.20 → 2.45 ± 0.29 | n/a | +| latency p50 / p95, batch 1 | 15.5 / 18.4 ms | 20.9 / 28.5 ms | 669 / 916 ms (API, incl. network) | + +Encoder latency: Apple M4 CPU, batch 1, 4 threads, torch 2.14.0, `torch.inference_mode()`, 50 warm-up queries, then validation rows 0 to 499, in order; tokenization + forward + argmax. It is timed with the same architecture, not the fine-tuned weights (deleted to save disk): the pinned pretrained backbone, a 151-way head, the same tokenizer and max_length. Latency depends on shapes, not weight values. The timed model is built by the training loader with the model revision, max_length and torch and transformers versions of the trained run, and its parameter count equals the trained run's. Haiku: client-side time per call on the 5,500 test queries, including the network round trip, with up to 6 calls in flight; it is not a like-for-like comparison with the encoders (different machine, a network in between). Peak memory samples `torch.mps.driver_allocated_memory()` after each backward pass and optimizer step (includes the allocator cache). Accuracy and OOS recall use the 8-way aggregation chosen on validation; ECE is over the 151 intents. Values are mean ± std over seeds 42, 43, 44. + +### Cost (RQ5) + +Measured: Haiku costs US$0.369 per 1K test queries (340.2K input and 5.89K output tokens per 1K; the whole run of 8,600 calls cost US$3.18). One ModernBERT training run takes 3.3 min at k=10 and 20.0 min at k=100 on an Apple M4. + +Break-even queries against LLM-only, for **assumed** accelerator prices (training only, no labelling cost, local inference priced as below). A scenario sensitivity, not a forecast: + +| point | router | at US$0.5/h | at US$1/h | at US$2/h | +|---|---|---|---|---| +| ModernBERT k=10 | small-only | 75 | 151 | 302 | +| ModernBERT k=10 | hybrid | 99 | 198 | 396 | +| ModernBERT k=100 | small-only | 452 | 904 | 1,807 | +| ModernBERT k=100 | hybrid | 458 | 915 | 1,831 | + +Break-even compares cost only: at k=10 the small model alone is less accurate than Haiku and the hybrid is more accurate than Haiku (router table). + +Training compute costs cents; labelling dominates once it has to be paid for. CLINC150 is an existing dataset, so labelling cost is an assumed sensitivity only (break-even queries, training plus labelling): + +| hybrid, US$1/h | labelled rows | at US$0.05 per label | at US$0.2 per label | at US$1 per label | +|---|---|---|---|---| +| ModernBERT-base k=10 | 1,525 | 272,134 | 1,087,942 | 5,438,918 | +| ModernBERT-base k=100 | 15,250 | 2,097,770 | 8,388,334 | 41,938,009 | + +Local inference is assumed to cost US$0.05 per vCPU-hour at full utilisation, times the measured p50 latency and thread count. Full grid and every input: `results/cost/cost.json`. + +**8-way aggregation.** The model predicts 151 intents. Two ways map that to 8 agents: take the argmax intent and map it (`argmax`), or sum the probabilities per agent and take the largest (`summed`). Each run picks one by validation 8-way accuracy only (ties go to argmax). BERT-base: k=1 summed, k=5 summed, k=10 per seed summed/argmax/argmax, k=25 argmax, k=50 per seed summed/summed/argmax, k=100 per seed argmax/summed/argmax. ModernBERT-base: k=1 summed, k=5 summed, k=10 summed, k=25 per seed summed/argmax/summed, k=50 argmax, k=100 argmax. + +### Figures + +![Learning curves](figures/learning_curves.png) + +*Test 8-way accuracy and OOS recall against labelled examples per intent (log scale), small model alone, 8-way aggregation chosen on validation; bands are mean ± std over three seeds. The dash-dot line is Claude Haiku 4.5 zero-shot on the same queries. Shaded: k ≤ 10 trains for the fixed 400-step budget.* + +![Risk-coverage](figures/risk_coverage.png) + +*Test selective risk (8-way error among the queries the small model keeps) against coverage, for ModernBERT at k=10 and k=100, for each confidence signal; the band is the std of the temperature-scaled MSP, the main signal. The four signals nearly coincide.* + +![Router comparison](figures/routers.png) + +*Test 8-way accuracy (a) and share of queries sent to Haiku (b) for LLM-only, small-only, the hybrid (threshold chosen on validation for 2% selective risk) and the oracle, which defers exactly the small model's errors. Panel (a) uses points, not bars, and its axis starts at 75%, so vertical distance is the difference in accuracy points and nothing is read from bar length; panel (b) starts at 0. Error bars are std over three seeds.* + +![Threshold transfer](figures/threshold_transfer.png) + +*Selective risk of the hybrid whose threshold was chosen on validation for 2% risk: on validation, on test, and on test reweighted to the validation OOS share (a diagnostic, not a method). Points appear where all three seeds had a feasible threshold.* + +## Limitations + +- **Hyperparameters at the edge of their grids.** The validation pilots chose values at the upper end: learning rate 5e-05 is the top of {1e-05, 2e-05, 5e-05}; S_min 400 is the top of {100, 200, 400}. A better value may lie beyond. +- **Small k is a fixed step budget.** k in {1, 5, 10} trains for S_min = 400 steps (more than 5 epochs), so those points show performance at that budget. +- **Latency uses the same architecture, not the trained weights** (see the efficiency table note). +- **Validation thresholds do not transfer** to test (headline above). +- **One benchmark.** CLINC150, English, one domain mix; BANKING77-OOS is planned for v0.2. +- Training on Apple MPS is not bit-for-bit deterministic, hence three seeds. + + +## Learning curves (test, target 2% for the hybrid) + +| model | k | 8-way acc (%) | OOS recall (%) | hybrid acc (%) | hybrid Haiku calls (%) | +|---|---|---|---|---|---| +| modernbert | 1 | 41.7 ± 2.5 | 1.7 ± 0.7 | 82.1 ± 0.0 | 100.0 ± 0.0 | +| modernbert | 5 | 71.0 ± 3.4 | 11.0 ± 4.3 | 84.4 ± 0.9 | 60.0 ± 6.7 | +| modernbert | 10 | 81.5 ± 0.7 | 23.1 ± 3.2 | 88.0 ± 0.4 | 23.9 ± 3.8 | +| modernbert | 25 | 86.3 ± 0.5 | 36.4 ± 3.0 | 89.7 ± 0.4 | 14.3 ± 1.3 | +| modernbert | 50 | 90.4 ± 0.4 | 53.9 ± 2.3 | 91.2 ± 0.5 | 3.9 ± 1.7 | +| modernbert | 100 | 91.9 ± 0.1 | 61.1 ± 0.4 | 92.1 ± 0.1 | 1.3 ± 1.6 | +| bert | 1 | 54.9 ± 1.3 | 9.9 ± 6.2 | 82.1 ± 0.0 | 98.4 ± 2.7 | +| bert | 5 | 79.0 ± 1.6 | 19.1 ± 7.0 | 86.4 ± 0.7 | 41.8 ± 0.9 | +| bert | 10 | 81.9 ± 1.2 | 24.1 ± 4.2 | 86.9 ± 0.3 | 41.5 ± 9.5 | +| bert | 25 | 86.5 ± 0.6 | 38.0 ± 3.2 | 89.7 ± 0.3 | 16.7 ± 3.8 | +| bert | 50 | 88.8 ± 0.3 | 44.8 ± 1.6 | 90.8 ± 0.2 | 8.5 ± 1.6 | +| bert | 100 | 91.1 ± 0.5 | 56.1 ± 2.6 | 91.9 ± 0.4 | 4.8 ± 1.0 | +| tfidf-centroid | 1 | 49.8 ± 1.2 | 0.1 ± 0.1 | 82.1 ± 0.0 | 100.0 ± 0.0 | +| tfidf-centroid | 5 | 68.3 ± 0.5 | 0.7 ± 1.1 | 83.6 ± 0.3 | 75.9 ± 1.9 | +| tfidf-centroid | 10 | 73.0 ± 0.6 | 1.8 ± 1.9 | 84.1 ± 0.3 | 70.5 ± 3.9 | +| tfidf-centroid | 25 | 75.3 ± 0.9 | 3.4 ± 4.4 | 85.6 ± 0.4 | 53.3 ± 6.5 | +| tfidf-centroid | 50 | 76.5 ± 0.4 | 6.2 ± 1.9 | 85.4 ± 0.5 | 57.7 ± 5.7 | +| tfidf-centroid | 100 | 77.6 ± 0.0 | 9.6 ± 0.0 | 86.0 ± 0.0 | 54.8 ± 0.0 | +| majority | 1 | 20.7 ± 0.0 | 0.0 ± 0.0 | n/a | n/a | +| majority | 5 | 20.7 ± 0.0 | 0.0 ± 0.0 | n/a | n/a | +| majority | 10 | 20.7 ± 0.0 | 0.0 ± 0.0 | n/a | n/a | +| majority | 25 | 20.7 ± 0.0 | 0.0 ± 0.0 | n/a | n/a | +| majority | 50 | 20.7 ± 0.0 | 0.0 ± 0.0 | n/a | n/a | +| majority | 100 | 20.7 ± 0.0 | 0.0 ± 0.0 | n/a | n/a | diff --git a/src/tinyrouter/cost.py b/src/tinyrouter/cost.py new file mode 100644 index 0000000..4a960bb --- /dev/null +++ b/src/tinyrouter/cost.py @@ -0,0 +1,247 @@ +"""RQ5 cost: measured numbers and assumed prices, kept apart (docs/PLAN.md section 3, RQ5). + +``make cost`` reads committed JSON only and writes ``results/cost/cost.json``: + +- ``measured``: M4 (MPS) training wall-clock per encoder and k, CPU latency + (``results/efficiency/cpu_latency.json``), Haiku's real tokens and spend + (``results/llm/haiku-8way.json``, ``results/analysis/haiku.json``) and each + router's real Haiku spend on test (``results/analysis/summary.json``). +- ``assumptions``: prices nobody paid. Training is priced as if the measured + M4 wall-clock hours were billed at a few accelerator-hour rates; the Mac's + purchase price is not used. Labelling cost is a sensitivity only (CLINC150 + is an existing dataset). Local inference is priced per vCPU-hour. +- ``break_even``: queries after which a one-time cost (training, optionally + labelling) is paid back by the per-query saving against LLM-only: + ``one_time_usd / (llm_only_per_query - router_per_query)``. A scenario + sensitivity, not a forecast. + +Every number is a mean over seeds 42, 43, 44 where seeds exist. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +from pathlib import Path + +import numpy as np + +ACCELERATOR_USD_PER_HOUR = (0.5, 1.0, 2.0) +LABEL_USD_PER_EXAMPLE = (0.05, 0.2, 1.0) +VCPU_USD_PER_HOUR = 0.05 +TARGET = "0.02" +MODELS = ("bert", "modernbert") +KS = (1, 5, 10, 25, 50, 100) +POINTS = (("modernbert", 10), ("modernbert", 100)) +ROUTERS = ("small_only", "hybrid") +INPUTS = ( + "analysis/summary.json", + "analysis/haiku.json", + "llm/haiku-8way.json", + "efficiency/cpu_latency.json", + "curves/bert.json", + "curves/modernbert.json", +) +ASSUMPTION_NOTES = { + "training": ( + "measured M4 wall-clock hours of one training run, priced as if billed at each " + "accelerator-hour rate; these are assumed rates, not money spent (the runs used a Mac " + "already owned, and its purchase price is not charged to a run)" + ), + "labelling": ( + "CLINC150 is an existing dataset, so no labelling was paid; cost = labelled training " + "rows x an assumed price per label, a sensitivity only" + ), + "local_inference": ( + "measured CPU p50 end-to-end latency x intra-op threads (vCPUs) x an assumed price per " + "vCPU-hour, at full utilisation; idle capacity would cost more" + ), +} + + +def training_cost_usd(wall_seconds: float, usd_per_hour: float) -> float: + return wall_seconds / 3600.0 * usd_per_hour + + +def local_inference_usd_per_query(latency_ms: float, vcpus: int, usd_per_vcpu_hour: float) -> float: + return latency_ms / 1000.0 / 3600.0 * vcpus * usd_per_vcpu_hour + + +def break_even_queries( + one_time_usd: float, llm_only_per_query: float, router_per_query: float +) -> float | None: + """Queries until the one-time cost is repaid; None when the router saves nothing per query.""" + saving = llm_only_per_query - router_per_query + if saving <= 0: + return None + return one_time_usd / saving + + +def mean_std(values: list[float]) -> dict[str, object]: + arr = np.asarray(values, dtype=np.float64) + return { + "mean": round(float(arr.mean()), 6), + "std": round(float(arr.std(ddof=1)), 6) if arr.size > 1 else None, + "values": [round(float(v), 6) for v in values], + } + + +def load(root: Path, name: str) -> dict: + return json.loads((root / name).read_text(encoding="utf-8")) + + +def training_runs(root: Path, model: str) -> dict[int, list[dict]]: + """Each k's run JSON ``training`` blocks, from the curve index (BERT k=100 is AC2).""" + index = load(root, f"curves/{model}.json") + runs: dict[int, list[dict]] = {} + for point in sorted(index["points"], key=lambda p: (p["k"], p["seed"])): + run = load(root, f"runs/{point['run_name']}.json") + runs.setdefault(point["k"], []).append(run["training"]) + return runs + + +def measured_training(root: Path) -> dict[str, dict[str, object]]: + out: dict[str, dict[str, object]] = {} + for model in MODELS: + out[model] = {} + for k, blocks in training_runs(root, model).items(): + devices = {b["device"] for b in blocks} + out[model][str(k)] = { + "train_wall_seconds": mean_std([b["train_wall_seconds"] for b in blocks]), + "train_rows": blocks[0]["train_rows"], + "steps": blocks[0]["global_step"], + "device": devices.pop() if len(devices) == 1 else sorted(devices), + } + return out + + +def measured_haiku(root: Path) -> dict[str, object]: + totals = load(root, "llm/haiku-8way.json")["totals"] + splits = load(root, "analysis/haiku.json")["splits"] + calls = totals["calls"] + return { + "model": load(root, "llm/haiku-8way.json")["identity"]["model"], + "calls": calls, + "input_tokens": totals["input_tokens"], + "output_tokens": totals["output_tokens"], + "cost_usd": totals["cost_usd"], + "input_tokens_per_1k_queries": round(1000 * totals["input_tokens"] / calls, 3), + "output_tokens_per_1k_queries": round(1000 * totals["output_tokens"] / calls, 3), + "cost_usd_per_1k_queries": {s: splits[s]["cost_usd_per_1k_queries"] for s in splits}, + } + + +def router_block(summary: dict, model: str, k: int) -> dict[str, dict[str, object]]: + final = summary["groups"][f"{model}/k{k}"]["result"]["final"] + blocks = {"small_only": final["small_only"], "hybrid": final["fallback"][TARGET]["hybrid"]} + out = {} + for name, block in blocks.items(): + metrics = block if name == "small_only" else block["test"] + out[name] = { + key: metrics[key]["mean"] + for key in ("accuracy_8", "llm_call_rate", "llm_cost_usd_per_1k") + } + return out + + +def cpu_latency(root: Path) -> dict[str, object]: + bench = load(root, "efficiency/cpu_latency.json") + return { + "intra_op_threads": bench["method"]["intra_op_threads"], + "hardware": bench["hardware"]["cpu"], + "end_to_end_ms": { + m: {q: bench["models"][m]["end_to_end"][q] for q in ("p50_ms", "p95_ms", "mean_ms")} + for m in MODELS + }, + } + + +def break_even_rows(measured: dict[str, object]) -> list[dict[str, object]]: + llm_only = measured["haiku"]["cost_usd_per_1k_queries"]["test"] / 1000.0 # type: ignore[index] + latency = measured["cpu_latency"] # type: ignore[assignment] + rows = [] + for model, k in POINTS: + training = measured["training"][model][str(k)] # type: ignore[index] + seconds = training["train_wall_seconds"]["mean"] + local = local_inference_usd_per_query( + latency["end_to_end_ms"][model]["p50_ms"], + latency["intra_op_threads"], + VCPU_USD_PER_HOUR, + ) + for router in ROUTERS: + api = measured["routers"][f"{model}/k{k}"][router]["llm_cost_usd_per_1k"] / 1000.0 # type: ignore[index] + for price in ACCELERATOR_USD_PER_HOUR: + for label in (0.0, *LABEL_USD_PER_EXAMPLE): + one_time = training_cost_usd(seconds, price) + label * training["train_rows"] + rows.append( + { + "point": f"{model}/k{k}", + "router": router, + "accelerator_usd_per_hour": price, + "label_usd_per_example": label, + "one_time_usd": round(one_time, 6), + "llm_only_usd_per_query": round(llm_only, 9), + "router_api_usd_per_query": round(api, 9), + "router_local_usd_per_query": round(local, 9), + "break_even_queries": rounded( + break_even_queries(one_time, llm_only, api + local) + ), + "break_even_queries_api_only": rounded( + break_even_queries(one_time, llm_only, api) + ), + } + ) + return rows + + +def rounded(value: float | None) -> float | None: + return None if value is None else round(value, 1) + + +def input_hashes(root: Path) -> dict[str, str]: + return {name: hashlib.sha256((root / name).read_bytes()).hexdigest() for name in INPUTS} + + +def build(root: Path) -> dict[str, object]: + summary = load(root, "analysis/summary.json") + measured = { + "training": measured_training(root), + "cpu_latency": cpu_latency(root), + "haiku": measured_haiku(root), + "routers": {f"{m}/k{k}": router_block(summary, m, k) for m, k in POINTS}, + "router_target_risk": float(TARGET), + } + assumptions = { + "accelerator_usd_per_hour": list(ACCELERATOR_USD_PER_HOUR), + "label_usd_per_example": list(LABEL_USD_PER_EXAMPLE), + "vcpu_usd_per_hour": VCPU_USD_PER_HOUR, + "notes": ASSUMPTION_NOTES, + } + return { + "format_version": 1, + "inputs_sha256": input_hashes(root), + "measured": measured, + "assumptions": assumptions, + "break_even": { + "formula": "one_time_usd / (llm_only_usd_per_query - router_usd_per_query); " + "null when the router saves nothing per query", + "label": "scenario sensitivity, not a forecast", + "rows": break_even_rows(measured), + }, + } + + +def main(argv: list[str] | None = None) -> None: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--results-root", default="results") + args = parser.parse_args(argv) + root = Path(args.results_root) + out = root / "cost" / "cost.json" + out.parent.mkdir(parents=True, exist_ok=True) + out.write_text(json.dumps(build(root), indent=2) + "\n", encoding="utf-8") + print(f"wrote {out}") + + +if __name__ == "__main__": + main() diff --git a/src/tinyrouter/figures.py b/src/tinyrouter/figures.py new file mode 100644 index 0000000..9ccd76c --- /dev/null +++ b/src/tinyrouter/figures.py @@ -0,0 +1,371 @@ +"""``make figures``: the four README figures, drawn from committed JSON only. + +Reads ``results/analysis/summary.json`` and ``curves.json`` and writes PNGs +(300 dpi) to ``results/figures/``. Nothing is random, and the PNG +``Software`` tag (the matplotlib version) is dropped, so two runs with the +same matplotlib give byte-identical files. Colours are the Okabe-Ito +palette, which stays distinguishable under the common colour-vision +deficiencies; line styles and markers differ too, so the figures also work +in greyscale. Bands and error bars are mean ± sample std over seeds 42, 43, 44. +""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +KS = (1, 5, 10, 25, 50, 100) +TARGET = "0.02" +OKABE_ITO = { + "black": "#000000", + "orange": "#E69F00", + "sky": "#56B4E9", + "green": "#009E73", + "blue": "#0072B2", + "vermillion": "#D55E00", + "purple": "#CC79A7", + "grey": "#999999", +} +STYLE = { + "font.size": 9, + "font.family": "serif", + "axes.labelsize": 9, + "axes.titlesize": 10, + "xtick.labelsize": 8, + "ytick.labelsize": 8, + "legend.fontsize": 8, + "figure.dpi": 300, + "savefig.dpi": 300, + "savefig.bbox": "tight", + "axes.spines.top": False, + "axes.spines.right": False, +} +MODELS = { + "modernbert": ("ModernBERT-base", OKABE_ITO["blue"], "-", "o"), + "bert": ("BERT-base", OKABE_ITO["vermillion"], "--", "s"), + "tfidf-centroid": ("TF-IDF centroid", OKABE_ITO["green"], ":", "^"), +} +SIGNALS = { + "msp_t": ("MSP, temperature-scaled", OKABE_ITO["blue"], "-", 1.8), + "msp": ("MSP", OKABE_ITO["sky"], "--", 1.0), + "entropy": ("negative entropy", OKABE_ITO["orange"], "-.", 1.0), + "margin": ("top-2 margin", OKABE_ITO["purple"], ":", 1.2), +} + + +def load(root: Path, name: str) -> dict: + return json.loads((root / name).read_text(encoding="utf-8")) + + +def pyplot(): + import matplotlib + + matplotlib.use("Agg") + import matplotlib.pyplot as plt + + plt.rcParams.update(STYLE) + return plt + + +def final(summary: dict, group: str) -> dict: + return summary["groups"][group]["result"]["final"] + + +def series(summary: dict, model: str, metric: str) -> tuple[list[float], list[float]]: + """Per-k (mean, std) in percent of the small-only ``metric``.""" + stats = [final(summary, f"{model}/k{k}")["small_only"][metric] for k in KS] + return [100 * s["mean"] for s in stats], [100 * (s["std"] or 0.0) for s in stats] + + +def log_k_axis(ax) -> None: # noqa: ANN001 + ax.set_xscale("log") + ax.set_xticks(KS, [str(k) for k in KS]) + ax.minorticks_off() + ax.set_xlabel("labelled examples per intent, k (log scale)") + + +def learning_curves(summary: dict, plt) -> object: # noqa: ANN001 + fig, axes = plt.subplots(1, 2, figsize=(6.8, 3.0), layout="constrained") + haiku = summary["llm_only_test"] + panels = ( + ("accuracy_8", "8-way routing accuracy (%)", haiku["accuracy_8"]), + ("oos_recall", "OOS recall (%)", haiku["oos"]["recall"]), + ) + for ax, (metric, ylabel, haiku_value) in zip(axes, panels, strict=True): + ax.axvspan(0.85, 10 * 1.2, color=OKABE_ITO["grey"], alpha=0.15, lw=0) + for model, (label, colour, style, marker) in MODELS.items(): + mean, std = series(summary, model, metric) + ax.plot(KS, mean, style, color=colour, marker=marker, ms=3.5, lw=1.4, label=label) + ax.fill_between( + KS, + [m - s for m, s in zip(mean, std, strict=True)], + [m + s for m, s in zip(mean, std, strict=True)], + color=colour, + alpha=0.2, + lw=0, + ) + ax.axhline( + 100 * haiku_value, + color=OKABE_ITO["black"], + ls="-.", + lw=1.0, + label="Claude Haiku 4.5, zero-shot", + ) + ax.text( + 1.0, + 0.98 if metric == "accuracy_8" else 0.9, + "k ≤ 10: fixed\n400-step budget", + transform=ax.get_xaxis_transform(), + fontsize=7, + va="top", + ) + log_k_axis(ax) + ax.set_ylabel(ylabel) + ax.set_ylim(0, 100) + axes[0].set_title("(a) 8-way accuracy, test") + axes[1].set_title("(b) OOS recall, test") + fig.legend( + *axes[0].get_legend_handles_labels(), loc="outside lower center", ncol=4, frameon=False + ) + return fig + + +def selected_aggregation(summary: dict, group: str) -> str: + chosen = summary["groups"][group]["result"]["selected_aggregation"] + if not isinstance(chosen, str): + raise ValueError(f"{group}: seeds chose different aggregations {chosen}; pick one first") + return chosen + + +def risk_coverage(summary: dict, curves: dict, plt) -> object: # noqa: ANN001 + fig, axes = plt.subplots(1, 2, figsize=(6.8, 3.0), sharey=True, layout="constrained") + grid = [100 * c for c in curves["coverage_grid"]] + for ax, k in zip(axes, (10, 100), strict=True): + group = f"modernbert/k{k}" + aggregation = selected_aggregation(summary, group) + block = curves["groups"][group]["result"][aggregation] + for signal, (label, colour, style, width) in SIGNALS.items(): + points = block[signal]["risk_coverage"] + mean = [100 * p["mean"] for p in points] + ax.plot(grid, mean, style, color=colour, lw=width, label=label) + if signal == "msp_t": + std = [100 * (p["std"] or 0.0) for p in points] + ax.fill_between( + grid, + [m - s for m, s in zip(mean, std, strict=True)], + [m + s for m, s in zip(mean, std, strict=True)], + color=colour, + alpha=0.2, + lw=0, + ) + ax.axhline( + 100 * float(TARGET), + color=OKABE_ITO["grey"], + ls="--", + lw=0.9, + label=f"{100 * float(TARGET):.0f}% target risk", + ) + ax.set_title(f"({'ab'[k == 100]}) ModernBERT k={k} ({aggregation})") + ax.set_xlabel("coverage: queries kept by the small model (%)") + ax.set_xlim(0, 100) + axes[0].set_ylabel("selective risk on test (%)") + fig.legend( + *axes[0].get_legend_handles_labels(), loc="outside lower center", ncol=5, frameon=False + ) + return fig + + +def router_stats(summary: dict, k: int) -> dict[str, tuple[float, float, float, float]]: + """Router -> (accuracy mean, std, call-rate mean, std) in percent.""" + haiku = summary["llm_only_test"] + block = final(summary, f"modernbert/k{k}") + metrics = { + "small-only": block["small_only"], + f"hybrid ({100 * float(TARGET):.0f}% target)": block["fallback"][TARGET]["hybrid"]["test"], + "oracle": block["oracle"], + } + out = {"LLM-only (Haiku)": (100 * haiku["accuracy_8"], 0.0, 100.0, 0.0)} + for name, m in metrics.items(): + acc, calls = m["accuracy_8"], m["llm_call_rate"] + out[name] = ( + 100 * acc["mean"], + 100 * (acc["std"] or 0.0), + 100 * calls["mean"], + 100 * (calls["std"] or 0.0), + ) + return out + + +ROUTER_STYLE = ( + (OKABE_ITO["grey"], "D", ""), + (OKABE_ITO["sky"], "s", "//"), + (OKABE_ITO["blue"], "o", ""), + (OKABE_ITO["orange"], "^", ".."), +) +ROUTER_OFFSET = 0.2 + + +def accuracy_dots(ax, stats: dict, names: list[str]) -> None: # noqa: ANN001 + """Panel (a): points with error bars, so the y range does not turn into bar lengths.""" + for i, name in enumerate(names): + colour, marker, _ = ROUTER_STYLE[i] + xs = [j + (i - 1.5) * ROUTER_OFFSET for j in range(2)] + means = [stats[k][name][0] for k in (10, 100)] + errors = [stats[k][name][1] for k in (10, 100)] + ax.errorbar( + xs, + means, + yerr=errors, + fmt=marker, + color=colour, + mec="black", + mew=0.5, + ms=6, + capsize=2, + lw=1.0, + label=name, + ) + for x, m in zip(xs, means, strict=True): + ax.annotate( + f"{m:.1f}", + (x, m), + xytext=(0, 6), + textcoords="offset points", + ha="center", + fontsize=6, + ) + ax.set_ylabel("8-way accuracy on test (%)") + ax.set_ylim(75, 100) + ax.grid(axis="y", color=OKABE_ITO["grey"], alpha=0.3, lw=0.5) + + +def call_bars(ax, stats: dict, names: list[str]) -> None: # noqa: ANN001 + """Panel (b): bars from 0, since the call rate is a share of all queries.""" + for i, name in enumerate(names): + colour, _, hatch = ROUTER_STYLE[i] + xs = [j + (i - 1.5) * ROUTER_OFFSET for j in range(2)] + bars = ax.bar( + xs, + [stats[k][name][2] for k in (10, 100)], + ROUTER_OFFSET, + yerr=[stats[k][name][3] for k in (10, 100)], + capsize=2, + color=colour, + hatch=hatch, + edgecolor="black", + lw=0.5, + label=name, + ) + ax.bar_label(bars, fmt="%.1f", fontsize=6, padding=1) + ax.set_ylabel("queries sent to Haiku (%)") + ax.set_ylim(0, 110) + + +def routers(summary: dict, plt) -> object: # noqa: ANN001 + fig, axes = plt.subplots(1, 2, figsize=(6.8, 3.0), layout="constrained") + stats = {k: router_stats(summary, k) for k in (10, 100)} + names = list(stats[10]) + accuracy_dots(axes[0], stats, names) + call_bars(axes[1], stats, names) + for ax in axes: + ax.set_xticks([0, 1], ["ModernBERT k=10", "ModernBERT k=100"]) + ax.set_xlim(-0.5, 1.5) + axes[0].set_title("(a) accuracy (points; axis from 75%)") + axes[1].set_title("(b) LLM call rate") + handles = axes[0].get_legend_handles_labels() + fig.legend(*handles, loc="outside lower center", ncol=4, frameon=False) + return fig + + +def transfer_series(summary: dict, model: str) -> dict[str, list[tuple[int, float, float]]]: + """Per feasible k: (k, mean, std) in percent of val, test and reweighted-test risk.""" + keys = { + "validation": ("validation", "selective_risk"), + "test": ("test", "selective_risk"), + "reweighted": ("test_reweighted_to_validation_oos_share", "selective_risk"), + } + out: dict[str, list[tuple[int, float, float]]] = {name: [] for name in keys} + for k in KS: + diag = summary["groups"][f"{model}/k{k}"]["result"]["diagnostics"][TARGET] + if diag is None: + continue + for name, (block, metric) in keys.items(): + stat = diag[block][metric] + if stat is None or stat.get("n") is not None: + continue + out[name].append((k, 100 * stat["mean"], 100 * (stat["std"] or 0.0))) + return out + + +def threshold_transfer(summary: dict, plt) -> object: # noqa: ANN001 + fig, axes = plt.subplots(1, 2, figsize=(6.8, 3.0), sharey=True, layout="constrained") + lines = { + "validation": ("validation (where the threshold was chosen)", OKABE_ITO["sky"], "--", "o"), + "test": ("test", OKABE_ITO["vermillion"], "-", "s"), + "reweighted": ("test reweighted to the validation OOS share", OKABE_ITO["green"], ":", "^"), + } + for ax, model in zip(axes, ("modernbert", "bert"), strict=True): + for name, points in transfer_series(summary, model).items(): + label, colour, style, marker = lines[name] + ks, mean, std = zip(*points, strict=True) + ax.errorbar( + ks, + mean, + yerr=std, + fmt=style, + color=colour, + marker=marker, + ms=3.5, + lw=1.3, + capsize=2, + label=label, + ) + ax.axhline( + 100 * float(TARGET), + color=OKABE_ITO["black"], + ls="-.", + lw=0.9, + label=f"{100 * float(TARGET):.0f}% target", + ) + log_k_axis(ax) + ax.set_title(f"({'ab'[model == 'bert']}) {MODELS[model][0]}") + axes[0].set_ylabel("selective risk of the hybrid (%)") + fig.legend( + *axes[0].get_legend_handles_labels(), loc="outside lower center", ncol=2, frameon=False + ) + return fig + + +def build(root: Path, out: Path) -> list[Path]: + plt = pyplot() + summary = load(root, "analysis/summary.json") + curves = load(root, "analysis/curves.json") + figures = { + "learning_curves.png": learning_curves(summary, plt), + "risk_coverage.png": risk_coverage(summary, curves, plt), + "routers.png": routers(summary, plt), + "threshold_transfer.png": threshold_transfer(summary, plt), + } + out.mkdir(parents=True, exist_ok=True) + written = [] + for name, fig in figures.items(): + path = out / name + fig.savefig(path, format="png", metadata={"Software": None}) + plt.close(fig) + written.append(path) + return written + + +def main(argv: list[str] | None = None) -> None: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--results-root", default="results") + parser.add_argument("--out", default=None, help="default: /figures") + args = parser.parse_args(argv) + root = Path(args.results_root) + for path in build(root, Path(args.out) if args.out else root / "figures"): + print(f"wrote {path}") + + +if __name__ == "__main__": + main() diff --git a/src/tinyrouter/latency.py b/src/tinyrouter/latency.py new file mode 100644 index 0000000..87ef841 --- /dev/null +++ b/src/tinyrouter/latency.py @@ -0,0 +1,296 @@ +"""CPU batch-1 inference latency of the two encoders, and Haiku's measured latency (AC5). + +The curve weights were deleted after their logits were archived (disk, +docs/PLAN.md section 6). Latency does not depend on the weight values, +only on the architecture and the input shapes, so each encoder is timed +with the same architecture it was fine-tuned from: the pinned pretrained +checkpoint (``model_name`` at ``model_revision`` from its config), a +151-way classification head (randomly initialised, seed 42), and the same +tokenizer and ``max_length``. Fine-tuning changes values, not shapes. To +show the timed model is the trained one in structure, its parameter count +must equal the one the k=100 run recorded, or the benchmark stops. + +Method: CPU, batch 1, ``torch.inference_mode()``, a fixed number of +intra-op threads (recorded), ``WARMUP`` untimed queries, then the first +``N`` validation queries in order. Each query is timed twice from the same +pass: ``end_to_end_ms`` is tokenization plus forward plus argmax (what a +router pays per request) and ``forward_ms`` is the forward pass alone. +Percentiles use numpy's default linear interpolation. + +Haiku: ``latency_ms`` in the journal is the client-side wall time of the +last attempt of each ``messages.create`` call. It includes the network +round trip and any API queueing, and the run kept up to 6 calls in flight. +""" + +from __future__ import annotations + +import argparse +import json +import platform +import subprocess +import time +from collections.abc import Sequence +from pathlib import Path + +import numpy as np + +from tinyrouter.archive import utc_now +from tinyrouter.config import RunConfig, load_config +from tinyrouter.data import load_split +from tinyrouter.labels import load_label_space + +ENCODERS = {"bert": "configs/bert-base.yaml", "modernbert": "configs/modernbert-base.yaml"} +DEFAULT_QUERIES = 500 +WARMUP = 50 +DEFAULT_THREADS = 4 +HEAD_SEED = 42 +PERCENTILE_METHOD = "linear" +WEIGHTS_NOTE = ( + "timed with the pinned pretrained backbone and a randomly initialised 151-way head, not the " + "fine-tuned weights (deleted after their logits were archived); latency depends on the " + "architecture and input shapes, not on the weight values, and the parameter count equals " + "the trained k=100 run's" +) +HAIKU_NOTE = ( + "client-side wall time of the last attempt of each messages.create call; includes the " + "network round trip and API queueing; up to 6 calls were in flight" +) + + +class ArchitectureMismatchError(RuntimeError): + """The timed model does not have the parameter count of the trained run.""" + + +def latency_summary(values_ms: Sequence[float]) -> dict[str, float | int]: + """n, mean, p50, p95, min and max of per-query latencies in milliseconds.""" + values = np.asarray(values_ms, dtype=np.float64) + if values.ndim != 1 or values.size == 0: + raise ValueError(f"need a non-empty 1-d list of latencies, got shape {values.shape}") + p50, p95 = np.percentile(values, [50, 95], method=PERCENTILE_METHOD) + return { + "n": int(values.size), + "mean_ms": round(float(values.mean()), 3), + "p50_ms": round(float(p50), 3), + "p95_ms": round(float(p95), 3), + "min_ms": round(float(values.min()), 3), + "max_ms": round(float(values.max()), 3), + } + + +def haiku_latency(journal: Path) -> dict[str, object]: + """Latency summary of the stored Haiku calls, per split and over both.""" + by_split: dict[str, list[float]] = {} + retried = 0 + for line in journal.read_text(encoding="utf-8").splitlines(): + row = json.loads(line) + by_split.setdefault(row["split"], []).append(float(row["latency_ms"])) + retried += int(row["attempts"]) > 1 + every = [v for split in sorted(by_split) for v in by_split[split]] + return { + "source": journal.name, + "measures": HAIKU_NOTE, + "calls_retried": retried, + "all": latency_summary(every), + "splits": {split: latency_summary(v) for split, v in sorted(by_split.items())}, + } + + +def cpu_name() -> str: + if platform.system() == "Darwin": + out = subprocess.run( + ["sysctl", "-n", "machdep.cpu.brand_string"], capture_output=True, text=True + ) + if out.returncode == 0 and out.stdout.strip(): + return out.stdout.strip() + cpuinfo = Path("/proc/cpuinfo") + if cpuinfo.is_file(): + for line in cpuinfo.read_text().splitlines(): + if line.startswith("model name"): + return line.split(":", 1)[1].strip() + return platform.processor() or "unknown" + + +def trained_run(results_root: Path, model: str) -> dict: + """The run JSON of the k=100 seed-42 point of ``model``'s curve.""" + index = json.loads((results_root / "curves" / f"{model}.json").read_text(encoding="utf-8")) + point = next(p for p in index["points"] if p["k"] == 100 and p["seed"] == 42) + return json.loads((results_root / "runs" / f"{point['run_name']}.json").read_text("utf-8")) + + +def trained_parameter_count(results_root: Path, model: str) -> int: + """Parameter total recorded by the k=100 seed-42 run of ``model``'s curve.""" + return int(trained_run(results_root, model)["training"]["parameters"]["total"]) + + +def public(version: str) -> str: + """Drop a local version label: the Linux CPU wheel of torch 2.14.0 reports ``2.14.0+cpu``.""" + return version.split("+", 1)[0] + + +def check_same_setup(model: str, run: dict, installed: dict[str, str]) -> dict[str, str]: + """The trained run's model, revision and library versions must equal the timed setup's. + + The run did not record the attention implementation. The timed model is + built by the same loader (``train.load_model_and_tokenizer``) with the + same transformers version, so transformers picks it the same way; the + benchmark records the one it got. + """ + config = load_config(ENCODERS[model]) + recorded = { + "model_name": run["config"]["model_name"], + "model_revision": run["config"]["model_revision"], + "max_length": str(run["config"]["max_length"]), + "torch": run["environment"]["torch"], + "transformers": run["environment"]["transformers"], + } + timed = { + "model_name": config.model_name, + "model_revision": config.model_revision, + "max_length": str(config.max_length), + **installed, + } + differ = sorted(k for k in recorded if public(recorded[k]) != public(timed[k])) + if differ: + raise ArchitectureMismatchError( + f"{model}: timed setup differs from the trained run in {differ}: " + f"{ {k: (recorded[k], timed[k]) for k in differ} }" + ) + return recorded + + +def load_timed_model(config: RunConfig) -> tuple[object, object]: + """The training loader's model on CPU; the pretrained backbone and a seeded 151-way head.""" + import torch + + from tinyrouter.train import load_model_and_tokenizer + + torch.manual_seed(HEAD_SEED) + model, tokenizer = load_model_and_tokenizer(config, load_label_space()) + return model.to("cpu").eval(), tokenizer # type: ignore[attr-defined] + + +def time_queries( + model: object, tokenizer: object, texts: Sequence[str], max_length: int +) -> tuple[list[float], list[float]]: + """(end-to-end ms, forward-only ms) per query, batch 1, in order.""" + import torch + + end_to_end, forward = [], [] + with torch.inference_mode(): + for text in texts: + started = time.perf_counter_ns() + batch = tokenizer( # type: ignore[operator] + [text], truncation=True, max_length=max_length, return_tensors="pt" + ) + forward_started = time.perf_counter_ns() + logits = model(**batch).logits # type: ignore[operator] + forward_done = time.perf_counter_ns() + int(logits.argmax(dim=-1)[0]) + finished = time.perf_counter_ns() + end_to_end.append((finished - started) / 1e6) + forward.append((forward_done - forward_started) / 1e6) + return end_to_end, forward + + +def benchmark_encoder( + name: str, results_root: Path, texts: Sequence[str], threads: int +) -> dict[str, object]: + import torch + import transformers + + config = load_config(ENCODERS[name]) + run = trained_run(results_root, name) + same = check_same_setup( + name, run, {"torch": torch.__version__, "transformers": transformers.__version__} + ) + model, tokenizer = load_timed_model(config) + total = sum(p.numel() for p in model.parameters()) # type: ignore[attr-defined] + trained = int(run["training"]["parameters"]["total"]) + if total != trained: + raise ArchitectureMismatchError( + f"{name}: timed model has {total} parameters, the trained k=100 run {trained}" + ) + torch.set_num_threads(threads) + time_queries(model, tokenizer, texts[:WARMUP], config.max_length) + end_to_end, forward = time_queries(model, tokenizer, texts, config.max_length) + return { + "model_name": config.model_name, + "model_revision": config.model_revision, + "num_labels": load_label_space().num_intents, + "max_length": config.max_length, + "parameters_total": total, + "parameters_match_trained_run": True, + "matches_trained_run": same, + "loader": "tinyrouter.train.load_model_and_tokenizer (the training loader)", + "attention_implementation": getattr(model.config, "_attn_implementation", None), # type: ignore[attr-defined] + "end_to_end": latency_summary(end_to_end), + "forward": latency_summary(forward), + } + + +def run_cpu_benchmark(results_root: Path, queries: int, threads: int) -> dict[str, object]: + import torch + import transformers + + texts = load_split("validation").texts[:queries] + torch.set_num_interop_threads(1) + models = {name: benchmark_encoder(name, results_root, texts, threads) for name in ENCODERS} + return { + "format_version": 1, + "method": { + "device": "cpu", + "batch_size": 1, + "mode": "torch.inference_mode", + "queries": f"validation rows 0 to {queries - 1}, in order", + "warmup_queries": WARMUP, + "intra_op_threads": threads, + "inter_op_threads": 1, + "percentiles": f"numpy.percentile, method={PERCENTILE_METHOD}", + "end_to_end": "tokenization + forward + argmax", + "forward": "forward pass only", + "weights": WEIGHTS_NOTE, + }, + "hardware": { + "cpu": cpu_name(), + "machine": platform.machine(), + "platform": platform.platform(), + }, + "versions": { + "python": platform.python_version(), + "torch": torch.__version__, + "transformers": transformers.__version__, + }, + "measured_at": utc_now(), + "models": models, + } + + +def write_json(path: Path, body: dict[str, object]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(body, indent=2) + "\n", encoding="utf-8") + + +def main(argv: list[str] | None = None) -> None: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("target", choices=("cpu", "haiku")) + parser.add_argument("--results-root", default="results") + parser.add_argument("--queries", type=int, default=DEFAULT_QUERIES) + parser.add_argument("--threads", type=int, default=DEFAULT_THREADS) + parser.add_argument( + "--out", default=None, help="output file (default under results/efficiency)" + ) + args = parser.parse_args(argv) + root = Path(args.results_root) + if args.target == "cpu": + body = run_cpu_benchmark(root, args.queries, args.threads) + out = root / "efficiency" / "cpu_latency.json" + else: + body = {"format_version": 1, **haiku_latency(root / "llm" / "haiku-8way.jsonl")} + out = root / "efficiency" / "haiku_latency.json" + out = Path(args.out) if args.out else out + write_json(out, body) + print(f"wrote {out}") + + +if __name__ == "__main__": + main() diff --git a/src/tinyrouter/report.py b/src/tinyrouter/report.py index cd7aa64..effa130 100644 --- a/src/tinyrouter/report.py +++ b/src/tinyrouter/report.py @@ -1,45 +1,534 @@ -"""Build results/summary.md from results/runs/*.json. Minimal for now. +"""``make report``: results/report.md and the README's generated block, from committed JSON. -Tables are generated, never typed by hand (docs/PLAN.md AC7). With no -results yet this says so and exits 0; the README section it will feed is -not wired up until the first real run exists. +No number in the README's first screen is typed by hand (docs/PLAN.md AC8). +Everything between ``BEGIN`` and ``END`` in README.md is rewritten from: +``results/analysis/{summary,haiku}.json``, ``results/llm/haiku-8way.json``, +``results/efficiency/*.json``, ``results/cost/cost.json``, the curve indexes, +the pilots and the run JSONs. ``--check`` rewrites nothing and exits 1 when +either file differs from what the JSON gives; tests/test_report.py runs the +same comparison in CI. """ from __future__ import annotations import argparse import json +import sys +from dataclasses import dataclass from pathlib import Path -COLUMNS = ("accuracy_8", "in_scope_accuracy_150", "oos_recall_151", "ece_151") +from tinyrouter.cost import training_runs +BEGIN = "" +END = "" +TARGET = "0.02" +MODEL_LABEL = {"bert": "BERT-base", "modernbert": "ModernBERT-base"} +FIGURES = ( + ( + "learning_curves.png", + "Learning curves", + "Test 8-way accuracy and OOS recall against labelled examples per intent (log scale), " + "small model alone, 8-way aggregation chosen on validation; bands are mean ± std over " + "three seeds. The dash-dot line is Claude Haiku 4.5 zero-shot on the same queries. " + "Shaded: k ≤ 10 trains for the fixed 400-step budget.", + ), + ( + "risk_coverage.png", + "Risk-coverage", + "Test selective risk (8-way error among the queries the small model keeps) against " + "coverage, for ModernBERT at k=10 and k=100, for each confidence signal; the band is " + "the std of the temperature-scaled MSP, the main signal. The four signals nearly " + "coincide.", + ), + ( + "routers.png", + "Router comparison", + "Test 8-way accuracy (a) and share of queries sent to Haiku (b) for LLM-only, " + "small-only, the hybrid (threshold chosen on validation for 2% selective risk) and the " + "oracle, which defers exactly the small model's errors. Panel (a) uses points, not " + "bars, and its axis starts at 75%, so vertical distance is the difference in accuracy " + "points and nothing is read from bar length; panel (b) starts at 0. Error bars are std " + "over three seeds.", + ), + ( + "threshold_transfer.png", + "Threshold transfer", + "Selective risk of the hybrid whose threshold was chosen on validation for 2% risk: on " + "validation, on test, and on test reweighted to the validation OOS share (a diagnostic, " + "not a method). Points appear where all three seeds had a feasible threshold.", + ), +) -def render(records: list[dict[str, object]]) -> str: - header = "| run | T | " + " | ".join(f"test {c}" for c in COLUMNS) + " |" - lines = [header, "|" + "---|" * (len(COLUMNS) + 2)] - for record in sorted(records, key=lambda r: str(r["run_name"])): - metrics = record["metrics"] - assert isinstance(metrics, dict) - calibrated = metrics["test"]["calibrated"] - cells = " | ".join(f"{calibrated[c]:.4f}" for c in COLUMNS) - lines.append(f"| {record['run_name']} | {metrics['temperature']:.3f} | {cells} |") - return "\n".join(lines) + "\n" + +@dataclass(frozen=True) +class Results: + summary: dict + haiku: dict + llm: dict + cpu: dict + haiku_latency: dict + cost: dict + indexes: dict[str, dict] + pilots: dict[str, dict] + k100_training: dict[str, list[dict]] + + @classmethod + def load(cls, root: Path) -> Results: + def read(name: str) -> dict: + return json.loads((root / name).read_text(encoding="utf-8")) + + return cls( + summary=read("analysis/summary.json"), + haiku=read("analysis/haiku.json"), + llm=read("llm/haiku-8way.json"), + cpu=read("efficiency/cpu_latency.json"), + haiku_latency=read("efficiency/haiku_latency.json"), + cost=read("cost/cost.json"), + indexes={m: read(f"curves/{m}.json") for m in MODEL_LABEL}, + pilots={p: read(f"pilots/{p}.json") for p in ("lr", "steps")}, + k100_training={m: training_runs(root, m)[100] for m in MODEL_LABEL}, + ) + + def group(self, name: str) -> dict: + return self.summary["groups"][name]["result"] + + def hybrid(self, name: str) -> dict: + return self.group(name)["final"]["fallback"][TARGET]["hybrid"] + + @property + def test_n(self) -> int: + return int(self.summary["llm_only_test"]["n"]) + + +def pct(value: float, digits: int = 1) -> str: + return f"{100 * value:.{digits}f}" + + +def pm(stat: dict, digits: int = 1, scale: float = 100.0) -> str: + """``mean ± std`` of a combined statistic; std shown only when there is one.""" + mean = f"{scale * stat['mean']:.{digits}f}" + if stat.get("std") is None: + return mean + return f"{mean} ± {scale * stat['std']:.{digits}f}" + + +def usd(value: float, digits: int = 3) -> str: + return f"US${value:,.{digits}f}" + + +def queries(value: float | None) -> str: + return "never" if value is None else f"{value:,.0f}" + + +def headline(r: Results) -> list[str]: + main = r.group("modernbert/k100") + small10 = r.group("modernbert/k10")["final"]["small_only"] + hybrid10 = r.hybrid("modernbert/k10")["test"] + diag = main["diagnostics"][TARGET] + haiku = r.summary["llm_only_test"] + target = pct(float(TARGET), 0) + transferred = ( + f"ModernBERT-base fine-tuned on all 100 examples per intent routes " + f"{pm(main['final']['small_only']['accuracy_8'])}% of test queries correctly in the " + f"8-way space (Claude Haiku 4.5 zero-shot: {pct(haiku['accuracy_8'])}%).", + f"With 10 examples per intent, deferring low-confidence queries to Haiku lifts 8-way " + f"accuracy from {pm(small10['accuracy_8'])}% (small model alone) to " + f"{pm(hybrid10['accuracy_8'])}%, with Haiku called on " + f"{pm(hybrid10['llm_call_rate'])}% of queries " + f"({call_counts(hybrid10['llm_call_rate'], r.test_n)} for seeds 42, 43, 44).", + ) + missed = ( + f"The deferral threshold chosen on validation for {target}% selective risk gives " + f"{pm(diag['test']['selective_risk'], 2)}% selective risk on test (ModernBERT k=100; " + f"validation: {pm(diag['validation']['selective_risk'], 2)}%).", + f"Validation is {pct(diag['validation_oos_share']['mean'])}% OOS and test " + f"{pct(diag['test_oos_share']['mean'])}%. In a reweighting diagnostic, weighting test to " + f"the validation OOS share closes about " + f"{round(100 * diag['share_of_gap_explained_by_oos_share']['mean'])}% of the gap between " + f"validation and test risk; the remaining difference is consistent with a higher " + f"conditional error rate " + f"on test OOS queries that the model keeps ({pm(diag['test']['kept_oos_error_rate'])}% " + f"vs {pm(diag['validation']['kept_oos_error_rate'])}% on validation).", + ) + return [ + "Test split, 8-way routing (7 agents + out-of-scope), mean ± std over seeds 42, 43, 44.", + "", + "| What transferred | What did not transfer |", + "|---|---|", + *(f"| {a} | {b} |" for a, b in zip(transferred, missed, strict=True)), + "", + "**Engineering conclusion.** A deferral threshold has to be calibrated on labelled data " + "that represents the traffic it will see. A benchmark's validation split is not a risk " + "guarantee for production.", + ] + + +def call_counts(rate: dict, n: int) -> str: + """Per-seed Haiku calls as absolute numbers, e.g. ``170, 9, 31 of 5,500``.""" + return ", ".join(f"{round(v * n):,}" for v in rate["values"]) + f" of {n:,}" + + +def hybrid_reading(r: Results) -> list[str]: + main = r.group("modernbert/k100")["final"] + hybrid = r.hybrid("modernbert/k100")["test"] + calls = call_counts(hybrid["llm_call_rate"], r.test_n) + return [ + "**How to read the hybrid.** With few labels the fallback is worth the most: at k=10 it " + "is the best evidence for the cascade (numbers above). With enough labels the local model " + f"covers almost all traffic: at k=100 the hybrid sends {pm(hybrid['llm_call_rate'])}% of " + f"test queries to Haiku ({calls} for seeds 42, 43, 44) and moves 8-way " + f"accuracy from {pm(main['small_only']['accuracy_8'])}% to {pm(hybrid['accuracy_8'])}%. " + "There the fallback is a small safety and diagnostic lever, not the main source of " + "accuracy.", + ] + + +def router_row(label: str, metrics: dict, llm: str | None = None) -> str: + misroute = metrics["high_conf_oos_misroute_rate"] + cells = [ + pm(metrics["accuracy_8"]), + pm(metrics["oos_recall"]), + "n/a" if misroute is None else pm(misroute), + pm(metrics["llm_call_rate"]), + llm if llm is not None else pm(metrics["llm_cost_usd_per_1k"], 3, 1.0), + ] + return f"| {label} | " + " | ".join(cells) + " |" + + +def routers_table(r: Results) -> list[str]: + haiku = r.summary["llm_only_test"] + rows = [ + "| router | 8-way acc (%) | OOS recall (%) | high-confidence OOS misroute (%) " + "| Haiku calls (%) | Haiku US$ per 1K queries |", + "|---|---|---|---|---|---|", + f"| LLM-only (Haiku 4.5) | {pct(haiku['accuracy_8'])} | {pct(haiku['oos']['recall'])} " + f"| n/a | 100.0 | {haiku['llm_cost_usd_per_1k']:.3f} |", + ] + for k in (10, 100): + final = r.group(f"modernbert/k{k}")["final"] + rows += [ + router_row(f"ModernBERT k={k}, small-only", final["small_only"]), + router_row( + f"ModernBERT k={k}, hybrid (target {pct(float(TARGET), 0)}%)", + r.hybrid(f"modernbert/k{k}")["test"], + ), + router_row(f"ModernBERT k={k}, oracle (upper bound)", final["oracle"]), + ] + return rows + + +def efficiency_cells(r: Results, model: str) -> list[str]: + result = r.group(f"{model}/k100") + runs = r.k100_training[model] + seconds = [b["train_wall_seconds"] for b in runs] + memory = [b["peak_memory"]["driver_allocated_bytes"] / 2**30 for b in runs] + ece = result["calibration"]["test_ece"] + latency = r.cpu["models"][model]["end_to_end"] + return [ + f"{runs[0]['parameters']['total'] / 1e6:.1f}M", + f"{stat_line(seconds, 0)} s", + f"{stat_line(memory, 2)} GiB", + pm(result["final"]["small_only"]["accuracy_8"]), + pm(result["final"]["small_only"]["oos_recall"]), + f"{pm(ece['uncalibrated']['ece_151'], 2)} → {pm(ece['calibrated']['ece_151'], 2)}", + f"{latency['p50_ms']:.1f} / {latency['p95_ms']:.1f} ms", + ] + + +def stat_line(values: list[float], digits: int) -> str: + n = len(values) + mean = sum(values) / n + std = (sum((v - mean) ** 2 for v in values) / (n - 1)) ** 0.5 + return f"{mean:,.{digits}f} ± {std:,.{digits}f}" + + +def efficiency_table(r: Results) -> list[str]: + bert, modern = efficiency_cells(r, "bert"), efficiency_cells(r, "modernbert") + haiku = r.summary["llm_only_test"] + lat = r.haiku_latency["splits"]["test"] + haiku_cells = [ + "n/a", + "none (zero-shot)", + "n/a", + pct(haiku["accuracy_8"]), + pct(haiku["oos"]["recall"]), + "n/a", + f"{lat['p50_ms']:.0f} / {lat['p95_ms']:.0f} ms (API, incl. network)", + ] + names = ( + "parameters", + "training time, k=100 (Apple M4, MPS)", + "training peak memory, k=100 (MPS driver, sampled)", + "8-way accuracy, test (%)", + "OOS recall, test (%)", + "ECE 151-way, test, before → after temperature (%)", + "latency p50 / p95, batch 1", + ) + cpu = r.cpu + return [ + "| | BERT-base (historical baseline) | ModernBERT-base (main model) | Claude Haiku 4.5 |", + "|---|---|---|---|", + *( + f"| {n} | {a} | {b} | {c} |" + for n, a, b, c in zip(names, bert, modern, haiku_cells, strict=True) + ), + "", + f"Encoder latency: {cpu['hardware']['cpu']} CPU, batch 1, " + f"{cpu['method']['intra_op_threads']} threads, torch {cpu['versions']['torch']}, " + f"`torch.inference_mode()`, {cpu['method']['warmup_queries']} warm-up queries, then " + f"{cpu['method']['queries']}; tokenization + forward + argmax. It is timed with the " + "same architecture, not the fine-tuned weights (deleted to save disk): the pinned " + "pretrained backbone, a 151-way head, the same tokenizer and max_length. Latency " + "depends on shapes, not weight values. The timed model is built by the training loader " + "with the model revision, max_length and torch and transformers versions of the trained " + "run, and its parameter count equals the trained run's." + f" Haiku: client-side time per call on the {lat['n']:,} test queries, including " + "the network round trip, with up to 6 calls in flight; it is not a like-for-like " + "comparison with the encoders (different machine, a network in between). Peak memory " + "samples " + "`torch.mps.driver_allocated_memory()` after each backward pass and optimizer step " + "(includes the allocator cache). Accuracy and OOS recall use the 8-way aggregation " + "chosen on validation; ECE is over the 151 intents. Values are mean ± std over seeds " + "42, 43, 44.", + ] + + +def cost_lines(r: Results) -> list[str]: + measured = r.cost["measured"] + rows = [x for x in r.cost["break_even"]["rows"] if x["label_usd_per_example"] == 0.0] + prices = r.cost["assumptions"]["accelerator_usd_per_hour"] + out = [ + f"Measured: Haiku costs {usd(measured['haiku']['cost_usd_per_1k_queries']['test'])} per " + f"1K test queries ({measured['haiku']['input_tokens_per_1k_queries'] / 1000:,.1f}K input " + f"and {measured['haiku']['output_tokens_per_1k_queries'] / 1000:,.2f}K output tokens per " + f"1K; the whole run of {measured['haiku']['calls']:,} calls cost " + f"{usd(measured['haiku']['cost_usd'], 2)}). One ModernBERT training run takes " + f"{measured['training']['modernbert']['10']['train_wall_seconds']['mean'] / 60:.1f} min " + f"at k=10 and " + f"{measured['training']['modernbert']['100']['train_wall_seconds']['mean'] / 60:.1f} min " + "at k=100 on an Apple M4.", + "", + "Break-even queries against LLM-only, for **assumed** accelerator prices (training only, " + "no labelling cost, local inference priced as below). A scenario sensitivity, not " + "a forecast:", + "", + "| point | router | " + " | ".join(f"at US${p:g}/h" for p in prices) + " |", + "|---|---|" + "---|" * len(prices), + ] + for point, router in dict.fromkeys((x["point"], x["router"]) for x in rows): + cells = [ + queries(x["break_even_queries"]) + for x in rows + if (x["point"], x["router"]) == (point, router) + ] + name = point.replace("modernbert/", "ModernBERT ").replace("k", "k=") + out.append(f"| {name} | {router.replace('_', '-')} | " + " | ".join(cells) + " |") + return out + label_lines(r) + + +def more_or_less(value: float, reference: float) -> str: + """How ``value`` compares with ``reference``, as a phrase that ends before the reference.""" + if value == reference: + return "as accurate as" + return "more accurate than" if value > reference else "less accurate than" + + +def label_lines(r: Results) -> list[str]: + assumptions = r.cost["assumptions"] + middle = assumptions["accelerator_usd_per_hour"][1] + labels = assumptions["label_usd_per_example"] + rows = [ + x + for x in r.cost["break_even"]["rows"] + if x["router"] == "hybrid" and x["accelerator_usd_per_hour"] == middle + ] + table = [ + f"| hybrid, US${middle:g}/h | labelled rows | " + + " | ".join(f"at US${p:g} per label" for p in labels) + + " |", + "|---|---|" + "---|" * len(labels), + ] + for point in dict.fromkeys(x["point"] for x in rows): + model, k = point.split("/") + cells = [ + queries(x["break_even_queries"]) + for x in rows + if x["point"] == point and x["label_usd_per_example"] > 0 + ] + n = r.cost["measured"]["training"][model][k[1:]]["train_rows"] + table.append( + f"| {MODEL_LABEL[model]} {k.replace('k', 'k=')} | {n:,} | " + " | ".join(cells) + " |" + ) + haiku = r.summary["llm_only_test"]["accuracy_8"] + k10 = r.group("modernbert/k10")["final"] + k10_small = more_or_less(k10["small_only"]["accuracy_8"]["mean"], haiku) + k10_hybrid = more_or_less(r.hybrid("modernbert/k10")["test"]["accuracy_8"]["mean"], haiku) + return [ + "", + f"Break-even compares cost only: at k=10 the small model alone is {k10_small} Haiku " + f"and the hybrid is {k10_hybrid} Haiku (router table).", + "", + "Training compute costs cents; labelling dominates once it has to be paid for. CLINC150 " + "is an existing dataset, so labelling cost is an assumed sensitivity only (break-even " + "queries, training plus labelling):", + "", + *table, + "", + f"Local inference is assumed to cost US${assumptions['vcpu_usd_per_hour']:g} per " + "vCPU-hour at full utilisation, times the measured p50 latency and thread count. " + "Full grid and every input: `results/cost/cost.json`.", + ] + + +def aggregation_line(r: Results) -> list[str]: + parts = [] + for model, label in MODEL_LABEL.items(): + picks = [] + for k in (1, 5, 10, 25, 50, 100): + chosen = r.group(f"{model}/k{k}")["selected_aggregation"] + text = chosen if isinstance(chosen, str) else f"per seed {'/'.join(chosen)}" + picks.append(f"k={k} {text}") + parts.append(f"{label}: " + ", ".join(picks)) + return [ + "**8-way aggregation.** The model predicts 151 intents. Two ways map that to 8 agents: " + "take the argmax intent and map it (`argmax`), or sum the probabilities per agent and " + "take the largest (`summed`). Each run picks one by validation 8-way accuracy only " + "(ties go to argmax). " + ". ".join(parts) + ".", + ] + + +def limitations(r: Results) -> list[str]: + lr, steps = r.pilots["lr"], r.pilots["steps"] + chosen_lr = {r.indexes[m]["learning_rate"] for m in MODEL_LABEL} + s_min = {r.indexes[m]["min_train_steps"] for m in MODEL_LABEL} + budget = sorted( + { + p["k"] + for m in MODEL_LABEL + for p in r.indexes[m]["points"] + if p["decided_by"] == "min_train_steps" + } + ) + at_top = [ + f"learning rate {'/'.join(f'{v:g}' for v in sorted(chosen_lr))} is the top of " + f"{{{', '.join(f'{v:g}' for v in lr['grid'])}}}" + if max(chosen_lr) == max(lr["grid"]) + else "", + f"S_min {'/'.join(str(v) for v in sorted(s_min))} is the top of " + f"{{{', '.join(str(v) for v in steps['grid'])}}}" + if max(s_min) == max(steps["grid"]) + else "", + ] + return [ + "## Limitations", + "", + "- **Hyperparameters at the edge of their grids.** The validation pilots chose values at " + f"the upper end: {'; '.join(t for t in at_top if t)}. A better value may lie beyond.", + f"- **Small k is a fixed step budget.** k in {{{', '.join(map(str, budget))}}} trains for " + f"S_min = {max(s_min)} steps (more than 5 epochs), so those points show performance at " + "that budget.", + "- **Latency uses the same architecture, not the trained weights** (see the efficiency " + "table note).", + "- **Validation thresholds do not transfer** to test (headline above).", + "- **One benchmark.** CLINC150, English, one domain mix; BANKING77-OOS is planned for " + "v0.2.", + "- Training on Apple MPS is not bit-for-bit deterministic, hence three seeds.", + ] + + +def figure_lines() -> list[str]: + lines: list[str] = [] + for name, alt, caption in FIGURES: + lines += [f"![{alt}](results/figures/{name})", "", f"*{caption}*", ""] + return lines[:-1] + + +def readme_block(r: Results) -> str: + sections = [ + headline(r), + hybrid_reading(r), + ["### Routers (test, 8-way)", "", *routers_table(r)], + ["### Efficiency (AC5)", "", *efficiency_table(r)], + ["### Cost (RQ5)", "", *cost_lines(r)], + aggregation_line(r), + ["### Figures", "", *figure_lines()], + limitations(r), + ] + body = "\n\n".join("\n".join(s) for s in sections) + return f"{BEGIN}\n{body}\n{END}" + + +def curve_table(r: Results) -> list[str]: + rows = [ + "| model | k | 8-way acc (%) | OOS recall (%) | hybrid acc (%) | hybrid Haiku calls (%) |", + "|---|---|---|---|---|---|", + ] + for model in ("modernbert", "bert", "tfidf-centroid", "majority"): + for k in (1, 5, 10, 25, 50, 100): + final = r.group(f"{model}/k{k}")["final"] + hybrid = final["fallback"][TARGET]["hybrid"] + tail = ( + ("n/a", "n/a") + if hybrid is None + else (pm(hybrid["test"]["accuracy_8"]), pm(hybrid["test"]["llm_call_rate"])) + ) + rows.append( + f"| {model} | {k} | {pm(final['small_only']['accuracy_8'])} | " + f"{pm(final['small_only']['oos_recall'])} | {tail[0]} | {tail[1]} |" + ) + return rows + + +def report_md(r: Results) -> str: + block = readme_block(r).replace("](results/figures/", "](figures/") + parts = [ + "# TinyRouter results", + "", + "Generated by `make report` from `results/*.json`. Do not edit by hand.", + "", + block, + "", + f"## Learning curves (test, target {pct(float(TARGET), 0)}% for the hybrid)", + "", + *curve_table(r), + ] + return "\n".join(parts) + "\n" + + +def replace_block(readme: str, block: str) -> str: + start, end = readme.find(BEGIN), readme.find(END) + if start < 0 or end < start: + raise ValueError("README.md has no generated block (BEGIN ... END markers)") + return readme[:start] + block + readme[end + len(END) :] + + +def outputs(root: Path, repo: Path) -> dict[Path, str]: + r = Results.load(root) + readme = repo / "README.md" + return { + root / "report.md": report_md(r), + readme: replace_block(readme.read_text(encoding="utf-8"), readme_block(r)), + } def main(argv: list[str] | None = None) -> None: - parser = argparse.ArgumentParser(description=__doc__) + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser.add_argument("--results-dir", default="results") + parser.add_argument("--repo", default=".") + parser.add_argument("--check", action="store_true", help="compare only; exit 1 if stale") args = parser.parse_args(argv) - results_dir = Path(args.results_dir) - runs_dir = results_dir / "runs" - paths = sorted(runs_dir.glob("*.json")) if runs_dir.is_dir() else [] - if not paths: - print(f"no results in {runs_dir}/ yet; run `make train evaluate` first") - return - records = [json.loads(p.read_text(encoding="utf-8")) for p in paths] - out = results_dir / "summary.md" - out.write_text(render(records), encoding="utf-8") - print(f"wrote {out} ({len(records)} runs)") + stale = [] + for path, text in outputs(Path(args.results_dir), Path(args.repo)).items(): + current = path.read_text(encoding="utf-8") if path.is_file() else None + if args.check: + if current != text: + stale.append(str(path)) + else: + path.write_text(text, encoding="utf-8") + print(f"wrote {path}") + if stale: + print(f"stale, run `make report`: {', '.join(stale)}", file=sys.stderr) + sys.exit(1) if __name__ == "__main__": diff --git a/tests/test_cost.py b/tests/test_cost.py new file mode 100644 index 0000000..55a88d8 --- /dev/null +++ b/tests/test_cost.py @@ -0,0 +1,79 @@ +import json +from pathlib import Path + +import pytest + +from tinyrouter import cost +from tinyrouter.cost import break_even_queries, local_inference_usd_per_query, training_cost_usd + +RESULTS = Path("results") + + +def test_training_cost_prices_wall_clock_hours(): + assert training_cost_usd(3600, 2.0) == pytest.approx(2.0) + assert training_cost_usd(1800, 0.5) == pytest.approx(0.25) + + +def test_local_inference_cost_scales_with_latency_threads_and_price(): + # 1 s on 4 vCPUs at US$0.05 per vCPU-hour + assert local_inference_usd_per_query(1000, 4, 0.05) == pytest.approx(4 * 0.05 / 3600) + + +def test_break_even_divides_one_time_cost_by_the_per_query_saving(): + assert break_even_queries(10.0, 0.001, 0.0005) == pytest.approx(20_000) + + +@pytest.mark.parametrize("router_cost", [0.001, 0.002]) +def test_break_even_is_none_when_the_router_saves_nothing(router_cost): + assert break_even_queries(10.0, 0.001, router_cost) is None + + +def test_committed_cost_json_is_what_make_cost_writes(): + committed = json.loads((RESULTS / "cost" / "cost.json").read_text()) + assert cost.build(RESULTS) == committed + + +def test_measured_and_assumed_numbers_live_in_separate_blocks(): + body = cost.build(RESULTS) + assert "accelerator_usd_per_hour" in body["assumptions"] + assert "usd_per_hour" not in json.dumps(body["measured"]) + assert body["assumptions"]["accelerator_usd_per_hour"] == [0.5, 1.0, 2.0] + assert body["assumptions"]["label_usd_per_example"] == [0.05, 0.2, 1.0] + assert "not a forecast" in body["break_even"]["label"] + + +def test_break_even_row_matches_a_hand_computation_from_the_raw_inputs(): + """ModernBERT k=10 hybrid at US$1/h with labels at US$0.2, from the files, not cost.py.""" + runs = [ + json.loads((RESULTS / "runs" / f"ModernBERT-base-k10-seed{s}.json").read_text()) + for s in (42, 43, 44) + ] + seconds = sum(r["training"]["train_wall_seconds"] for r in runs) / 3 + rows_labelled = runs[0]["training"]["train_rows"] + summary = json.loads((RESULTS / "analysis" / "summary.json").read_text()) + hybrid = summary["groups"]["modernbert/k10"]["result"]["final"]["fallback"]["0.02"] + api = hybrid["hybrid"]["test"]["llm_cost_usd_per_1k"]["mean"] / 1000 + haiku = json.loads((RESULTS / "analysis" / "haiku.json").read_text()) + llm_only = haiku["splits"]["test"]["cost_usd_per_1k_queries"] / 1000 + bench = json.loads((RESULTS / "efficiency" / "cpu_latency.json").read_text()) + p50 = bench["models"]["modernbert"]["end_to_end"]["p50_ms"] + local = p50 / 1000 / 3600 * bench["method"]["intra_op_threads"] * 0.05 + expected = (seconds / 3600 * 1.0 + 0.2 * rows_labelled) / (llm_only - api - local) + + row = next( + r + for r in cost.build(RESULTS)["break_even"]["rows"] + if (r["point"], r["router"], r["accelerator_usd_per_hour"], r["label_usd_per_example"]) + == ("modernbert/k10", "hybrid", 1.0, 0.2) + ) + assert row["break_even_queries"] == pytest.approx(expected, abs=0.1) + assert rows_labelled == 1525 + + +def test_every_break_even_scenario_is_present_once(): + rows = cost.build(RESULTS)["break_even"]["rows"] + keys = { + (r["point"], r["router"], r["accelerator_usd_per_hour"], r["label_usd_per_example"]) + for r in rows + } + assert len(keys) == len(rows) == 2 * 2 * 3 * 4 diff --git a/tests/test_figures.py b/tests/test_figures.py new file mode 100644 index 0000000..c0b70bd --- /dev/null +++ b/tests/test_figures.py @@ -0,0 +1,55 @@ +from pathlib import Path + +import pytest + +pytest.importorskip("matplotlib") + +from tinyrouter import figures # noqa: E402 + +EXPECTED = {"learning_curves.png", "risk_coverage.png", "routers.png", "threshold_transfer.png"} + + +def test_figures_are_drawn_from_json_and_are_byte_identical_on_rerun(tmp_path): + first = figures.build(Path("results"), tmp_path / "a") + second = figures.build(Path("results"), tmp_path / "b") + assert {p.name for p in first} == EXPECTED + for a, b in zip(first, second, strict=True): + assert a.read_bytes() == b.read_bytes(), a.name + assert b"Matplotlib version" not in a.read_bytes() + + +def test_risk_coverage_refuses_a_point_whose_seeds_chose_different_aggregations(): + summary = {"groups": {"x": {"result": {"selected_aggregation": ["argmax", "summed"]}}}} + with pytest.raises(ValueError, match="different aggregations"): + figures.selected_aggregation(summary, "x") + + +def test_threshold_transfer_skips_points_without_a_feasible_threshold_on_every_seed(): + summary = figures.load(Path("results"), "analysis/summary.json") + series = figures.transfer_series(summary, "modernbert") + ks = [k for k, _, _ in series["test"]] + assert 1 not in ks + assert ks == sorted(ks) + assert series["test"][-1][0] == 100 + + +def test_threshold_transfer_drops_a_point_where_only_some_seeds_were_feasible(): + """BERT k=1 has a feasible 2% threshold on one seed of three; a mean of one is not plotted.""" + summary = figures.load(Path("results"), "analysis/summary.json") + stat = summary["groups"]["bert/k1"]["result"]["diagnostics"]["0.02"]["test"]["selective_risk"] + assert stat["n"] == 1 + for name, points in figures.transfer_series(summary, "bert").items(): + assert 1 not in [k for k, _, _ in points], name + + +def test_router_accuracy_panel_uses_points_not_truncated_bars(): + """A bar starting at 75% would turn 91.9 vs 92.1 into a visible length gap (review R3).""" + plt = figures.pyplot() + summary = figures.load(Path("results"), "analysis/summary.json") + fig = figures.routers(summary, plt) + accuracy, calls = fig.axes[:2] + assert len(accuracy.patches) == 0 + assert "points" in accuracy.get_title() + assert len(calls.patches) > 0 + assert calls.get_ylim()[0] == 0 + plt.close(fig) diff --git a/tests/test_latency.py b/tests/test_latency.py new file mode 100644 index 0000000..e1074db --- /dev/null +++ b/tests/test_latency.py @@ -0,0 +1,130 @@ +import json + +import pytest +import torch + +from tinyrouter import latency +from tinyrouter.latency import ( + ArchitectureMismatchError, + haiku_latency, + latency_summary, + time_queries, + trained_parameter_count, +) + + +def test_latency_summary_uses_linear_percentiles(): + stats = latency_summary([1.0, 2.0, 3.0, 4.0]) + assert stats["n"] == 4 + assert stats["mean_ms"] == 2.5 + assert stats["p50_ms"] == 2.5 + # linear: rank 0.95 * 3 = 2.85 between 3 and 4 + assert stats["p95_ms"] == pytest.approx(3.85) + assert (stats["min_ms"], stats["max_ms"]) == (1.0, 4.0) + + +def test_latency_summary_p95_is_not_the_max_on_a_long_list(): + stats = latency_summary([float(v) for v in range(1, 101)]) + assert stats["p50_ms"] == pytest.approx(50.5) + assert stats["p95_ms"] == pytest.approx(95.05) + + +def test_latency_summary_refuses_an_empty_list(): + with pytest.raises(ValueError, match="non-empty"): + latency_summary([]) + + +def test_haiku_latency_splits_rows_and_counts_retries(tmp_path): + rows = [ + {"split": "validation", "latency_ms": 100, "attempts": 1}, + {"split": "validation", "latency_ms": 300, "attempts": 2}, + {"split": "test", "latency_ms": 200, "attempts": 1}, + ] + journal = tmp_path / "haiku.jsonl" + journal.write_text("\n".join(json.dumps(r) for r in rows) + "\n") + out = haiku_latency(journal) + assert out["calls_retried"] == 1 + assert out["all"]["n"] == 3 + assert out["all"]["mean_ms"] == 200.0 + assert out["splits"]["validation"]["p50_ms"] == 200.0 + assert out["splits"]["test"]["n"] == 1 + assert "network" in out["measures"] + + +def test_trained_parameter_count_reads_the_k100_seed42_run(): + assert trained_parameter_count(latency.Path("results"), "modernbert") == 149_720_983 + assert trained_parameter_count(latency.Path("results"), "bert") == 109_598_359 + + +class FakeTokenizer: + def __call__(self, texts, truncation, max_length, return_tensors): + assert truncation and return_tensors == "pt" + ids = torch.tensor([[len(t) % 7 for t in texts[0][:max_length]]]) + return {"input_ids": ids} + + +class FakeModel(torch.nn.Module): + def __init__(self): + super().__init__() + self.linear = torch.nn.Linear(1, 151) + + def forward(self, input_ids): + return type("Out", (), {"logits": self.linear(input_ids.float().mean(1, keepdim=True))}) + + +def test_time_queries_times_every_query_in_order_and_forward_is_not_longer(): + end_to_end, forward = time_queries(FakeModel(), FakeTokenizer(), ["ab", "cde", "f"], 8) + assert len(end_to_end) == len(forward) == 3 + assert all(e >= f > 0 for e, f in zip(end_to_end, forward, strict=True)) + + +def test_benchmark_stops_when_the_timed_model_is_not_the_trained_architecture(monkeypatch): + monkeypatch.setattr(latency, "load_timed_model", lambda config: (FakeModel(), FakeTokenizer())) + with pytest.raises(ArchitectureMismatchError, match="parameters"): + latency.benchmark_encoder("bert", latency.Path("results"), ["a"], threads=1) + + +def test_committed_cpu_latency_records_method_hardware_and_matching_parameters(): + body = json.loads(latency.Path("results/efficiency/cpu_latency.json").read_text()) + assert body["method"]["batch_size"] == 1 + assert body["method"]["device"] == "cpu" + assert body["hardware"]["cpu"] + for name in ("bert", "modernbert"): + model = body["models"][name] + assert model["parameters_match_trained_run"] is True + assert model["parameters_total"] == trained_parameter_count(latency.Path("results"), name) + assert model["end_to_end"]["n"] == 500 + + +def test_timed_setup_must_match_the_trained_run_versions_and_revision(): + run = latency.trained_run(latency.Path("results"), "modernbert") + installed = { + "torch": run["environment"]["torch"], + "transformers": run["environment"]["transformers"], + } + same = latency.check_same_setup("modernbert", run, installed) + assert same["model_revision"] == run["config"]["model_revision"] + with pytest.raises(ArchitectureMismatchError, match="transformers"): + latency.check_same_setup("modernbert", run, {**installed, "transformers": "0.0.1"}) + other = {**run, "config": {**run["config"], "model_revision": "0" * 40}} + with pytest.raises(ArchitectureMismatchError, match="model_revision"): + latency.check_same_setup("modernbert", other, installed) + + +def test_committed_benchmark_records_the_attention_implementation_and_the_loader(): + body = json.loads(latency.Path("results/efficiency/cpu_latency.json").read_text()) + for model in body["models"].values(): + assert model["attention_implementation"] + assert "train.load_model_and_tokenizer" in model["loader"] + assert model["matches_trained_run"]["transformers"] == body["versions"]["transformers"] + + +def test_a_local_version_label_is_not_a_different_torch(): + """CI installs the Linux CPU wheel, which reports torch as 2.14.0+cpu.""" + run = latency.trained_run(latency.Path("results"), "bert") + torch_version = run["environment"]["torch"] + installed = { + "torch": f"{torch_version}+cpu", + "transformers": run["environment"]["transformers"], + } + assert latency.check_same_setup("bert", run, installed)["torch"] == torch_version diff --git a/tests/test_report.py b/tests/test_report.py new file mode 100644 index 0000000..b954b23 --- /dev/null +++ b/tests/test_report.py @@ -0,0 +1,223 @@ +import copy +import json +import shutil +import subprocess +from pathlib import Path + +import pytest + +from tinyrouter import report +from tinyrouter.report import BEGIN, END, Results, pm, readme_block, replace_block + +RESULTS = Path("results") +EM_DASH = chr(0x2014) + + +@pytest.fixture(scope="module") +def results(): + return Results.load(RESULTS) + + +def generated_block(text: str) -> str: + return text[text.index(BEGIN) : text.index(END) + len(END)] + + +def test_readme_generated_block_matches_the_committed_json(results): + """AC8: every first-screen number comes from make report; run it if this fails.""" + readme = Path("README.md").read_text(encoding="utf-8") + assert generated_block(readme) == readme_block(results) + + +def test_results_report_md_matches_the_committed_json(results): + assert (RESULTS / "report.md").read_text(encoding="utf-8") == report.report_md(results) + + +def test_check_mode_exits_1_on_a_stale_readme_and_0_on_a_fresh_one(tmp_path, results): + shutil.copytree( + RESULTS, tmp_path / "results", ignore=shutil.ignore_patterns("logits", "*.jsonl") + ) + readme = tmp_path / "README.md" + readme.write_text(f"# x\n\n{BEGIN}\n91.9 typed by hand\n{END}\n", encoding="utf-8") + argv = ["--results-dir", str(tmp_path / "results"), "--repo", str(tmp_path), "--check"] + with pytest.raises(SystemExit) as stale: + report.main(argv) + assert stale.value.code == 1 + report.main(argv[:-1]) + report.main(argv) # no SystemExit: fresh + assert "typed by hand" not in readme.read_text() + assert readme.read_text().startswith("# x\n\n") + + +def test_a_changed_number_in_the_json_changes_the_block(results): + summary = copy.deepcopy(results.summary) + stat = summary["groups"]["modernbert/k100"]["result"]["final"]["small_only"]["accuracy_8"] + stat["mean"] = 0.5 + changed = Results(**{**results.__dict__, "summary": summary}) + assert "50.0 ± 0.1%" in readme_block(changed) + assert readme_block(changed) != readme_block(results) + + +def test_replace_block_refuses_a_readme_without_markers(): + with pytest.raises(ValueError, match="markers"): + replace_block("# no block here\n", f"{BEGIN}\n{END}") + + +def test_pm_shows_std_only_when_there_is_one(): + assert pm({"mean": 0.9187, "std": 0.0013}) == "91.9 ± 0.1" + assert pm({"mean": 0.9187, "std": None}) == "91.9" + assert pm({"mean": 0.07345, "std": 0.00857}, 2) == "7.34 ± 0.86" + + +def test_headline_numbers_and_wording(results): + block = readme_block(results) + diag = results.group("modernbert/k100")["diagnostics"]["0.02"] + assert f"{100 * diag['test']['selective_risk']['mean']:.2f}" in block + share = round(100 * diag["share_of_gap_explained_by_oos_share"]["mean"]) + assert f"closes about {share}% of the gap between validation and test risk" in block + assert "In a reweighting diagnostic" in block + for causal in ("caused", "causes", "because of the OOS"): + assert causal not in block + assert "not a risk guarantee for production" in block + + +def test_hybrid_call_counts_are_reported_per_seed_as_absolute_numbers(results): + hybrid = results.hybrid("modernbert/k100")["test"]["llm_call_rate"]["values"] + counts = ", ".join(str(round(v * results.test_n)) for v in hybrid) + assert f"({counts} of {results.test_n:,} for seeds 42, 43, 44)" in readme_block(results) + + +def test_generated_text_has_no_em_dash(results): + assert EM_DASH not in readme_block(results) + assert EM_DASH not in report.report_md(results) + + +def test_every_referenced_figure_is_committed(): + for name, _, _ in report.FIGURES: + assert (RESULTS / "figures" / name).is_file(), name + + +def test_efficiency_table_has_both_encoders_and_haiku_latency(results): + block = readme_block(results) + assert "BERT-base (historical baseline)" in block + assert "ModernBERT-base (main model)" in block + lat = results.haiku_latency["splits"]["test"] + assert f"{lat['p50_ms']:.0f} / {lat['p95_ms']:.0f} ms (API, incl. network)" in block + + +def test_limitations_name_the_grid_edges_and_the_step_budget(results): + block = readme_block(results) + assert "is the top of {1e-05, 2e-05, 5e-05}" in block + assert "S_min 400 is the top of {100, 200, 400}" in block + assert "k in {1, 5, 10} trains for S_min = 400 steps" in block + + +def test_every_file_report_reads_is_tracked_in_git(): + """The README check runs in CI from a plain checkout, so no input may be a Release download.""" + tracked = set( + subprocess.run( + ["git", "ls-files", "results"], capture_output=True, text=True, check=True + ).stdout.split() + ) + for name in ( + "analysis/summary.json", + "analysis/haiku.json", + "llm/haiku-8way.json", + "efficiency/cpu_latency.json", + "efficiency/haiku_latency.json", + "cost/cost.json", + "curves/bert.json", + "curves/modernbert.json", + "pilots/lr.json", + "pilots/steps.json", + "report.md", + ): + assert f"results/{name}" in tracked, name + + +SUMMARY = json.loads((RESULTS / "analysis" / "summary.json").read_text(encoding="utf-8")) +K100 = ("groups", "modernbert/k100", "result") +K10 = ("groups", "modernbert/k10", "result") +DIAG = (*K100, "diagnostics", "0.02") +K10_HYBRID = (*K10, "final", "fallback", "0.02", "hybrid", "test") +K100_HYBRID = (*K100, "final", "fallback", "0.02", "hybrid", "test") + + +def at(path: tuple[str, ...]): + node = SUMMARY + for key in path: + node = node[key] + return node + + +def one(path: tuple[str, ...], digits: int = 1) -> str: + """A fraction at ``path`` as a percentage, formatted independently of report.py.""" + value = at(path) + return f"{100 * (value['mean'] if isinstance(value, dict) else value):.{digits}f}" + + +def both(path: tuple[str, ...], digits: int = 1) -> str: + stat = at(path) + return f"{100 * stat['mean']:.{digits}f} ± {100 * stat['std']:.{digits}f}" + + +def counts(path: tuple[str, ...]) -> str: + n = at(("llm_only_test", "n")) + return ", ".join(f"{round(v * n):,}" for v in at(path)["values"]) + f" of {n:,}" + + +# Each first-screen number, the exact JSON field it must come from, and the text around it. +# A number rendered from the wrong field (another router, another split) fails here even +# after `make report` regenerates the README. +FIRST_SCREEN = { + "k=100 small-only accuracy": lambda: ( + f"routes {both((*K100, 'final', 'small_only', 'accuracy_8'))}% of test queries" + ), + "Haiku accuracy": lambda: ( + f"(Claude Haiku 4.5 zero-shot: {one(('llm_only_test', 'accuracy_8'))}%)" + ), + "k=10 small-only accuracy": lambda: ( + f"accuracy from {both((*K10, 'final', 'small_only', 'accuracy_8'))}% (small model alone)" + ), + "k=10 hybrid accuracy": lambda: f"to {both((*K10_HYBRID, 'accuracy_8'))}%, with Haiku called", + "k=10 hybrid call rate and counts": lambda: ( + f"called on {both((*K10_HYBRID, 'llm_call_rate'))}% of queries " + f"({counts((*K10_HYBRID, 'llm_call_rate'))} for seeds 42, 43, 44)" + ), + "k=100 test selective risk": lambda: ( + f"gives {both((*DIAG, 'test', 'selective_risk'), 2)}% selective risk on test" + ), + "k=100 validation selective risk": lambda: ( + f"(ModernBERT k=100; validation: {both((*DIAG, 'validation', 'selective_risk'), 2)}%)" + ), + "OOS shares": lambda: ( + f"Validation is {one((*DIAG, 'validation_oos_share'))}% OOS and test " + f"{one((*DIAG, 'test_oos_share'))}%." + ), + "share of gap": lambda: ( + f"closes about {round(100 * at((*DIAG, 'share_of_gap_explained_by_oos_share'))['mean'])}%" + " of the gap" + ), + "kept OOS error rates": lambda: ( + f"({both((*DIAG, 'test', 'kept_oos_error_rate'))}% vs " + f"{both((*DIAG, 'validation', 'kept_oos_error_rate'))}% on validation)" + ), + "k=100 hybrid call rate and counts": lambda: ( + f"sends {both((*K100_HYBRID, 'llm_call_rate'))}% of test queries to Haiku " + f"({counts((*K100_HYBRID, 'llm_call_rate'))} for seeds 42, 43, 44)" + ), + "k=100 hybrid accuracy": lambda: ( + f"from {both((*K100, 'final', 'small_only', 'accuracy_8'))}% to " + f"{both((*K100_HYBRID, 'accuracy_8'))}%. There the fallback" + ), +} + + +@pytest.mark.parametrize("name", sorted(FIRST_SCREEN)) +def test_each_first_screen_number_comes_from_its_own_json_field(name): + readme = Path("README.md").read_text(encoding="utf-8") + assert FIRST_SCREEN[name]() in generated_block(readme) + + +def test_k10_hybrid_counts_are_the_reviewed_absolute_numbers(): + assert counts((*K10_HYBRID, "llm_call_rate")) == "1,235, 1,157, 1,548 of 5,500" + assert counts((*K100_HYBRID, "llm_call_rate")) == "170, 9, 31 of 5,500" diff --git a/uv.lock b/uv.lock index 8136178..bbdf04d 100644 --- a/uv.lock +++ b/uv.lock @@ -220,6 +220,37 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, ] +[[package]] +name = "contourpy" +version = "1.4.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/83/5a/a55177dd22553a277388e8a1b3220e92de91bacb28356cdc73caa240121d/contourpy-1.4.0.tar.gz", hash = "sha256:20156f5a1ac4f8ce02656e39a61e82164a3d359796dc8026f75b062783d500e1", size = 13323726, upload-time = "2026-09-11T19:05:05.808Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/97/ba/01bde9753bdbba04a6da9d2bff3881f021bc708f654655e95fae26d3b3a3/contourpy-1.4.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:186ba929df36d61b6127da2e89cd1357e4cb79aca647381ab1a6feb0b152877b", size = 297302, upload-time = "2026-09-11T19:02:45.103Z" }, + { url = "https://files.pythonhosted.org/packages/dd/f5/c5bf67522d49a2222f3154fe46879451d26f0e35ece41770758a64ea78cb/contourpy-1.4.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:c76f3a5318164db7d9401132fc1b5364b784c613c93fa506e3b5e6d1bf353ec8", size = 284921, upload-time = "2026-09-11T19:02:48.049Z" }, + { url = "https://files.pythonhosted.org/packages/0f/cc/cb989599eec12fda312e127eb8e04a8b21a7a6c94bf7ff1bc68273334a42/contourpy-1.4.0-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:43c3ccbb32c6294b183dcc8e8c46dacf5ecef809497e3d48a5be298eef185ad0", size = 356332, upload-time = "2026-09-11T19:02:51.408Z" }, + { url = "https://files.pythonhosted.org/packages/47/fb/6f620d7602507817b0b4c21dd790ed68688f1f4ae9c29aacf21f08db5cf1/contourpy-1.4.0-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:86d05cec773c9507a3950122e0e40ce77c23c75ceeb2fc189514e71893cbb34b", size = 405878, upload-time = "2026-09-11T19:02:54.093Z" }, + { url = "https://files.pythonhosted.org/packages/32/4e/693c6d6bdece679f0953775eef3a1b66d56be55478d936e7deb1f3004422/contourpy-1.4.0-cp312-cp312-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2deb580178ca19437a84bd77e4bc2cd91a8ccad212413a83c273900868b5978c", size = 408296, upload-time = "2026-09-11T19:02:56.528Z" }, + { url = "https://files.pythonhosted.org/packages/78/f9/b6831508960d559581c448532ba4df312217072975486dbf2b24fcc5b76f/contourpy-1.4.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:875f42444c9cf48d56f724f2637e60d0f73b3b12c9041e1484580a233edf9591", size = 383243, upload-time = "2026-09-11T19:02:58.638Z" }, + { url = "https://files.pythonhosted.org/packages/f2/21/52903825a0ae7bb625e8bd30a09816d4c3c5d6174a469c10a616166cf780/contourpy-1.4.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:f863c6100bf926cf47d13f3cd75f9bb8ebd98aaab230eeb4468b91f98f39a6b3", size = 1357306, upload-time = "2026-09-11T19:03:01.036Z" }, + { url = "https://files.pythonhosted.org/packages/d8/79/d68b1ce8539e4071518fed9523f558398f34dcd078b8927b109c72dad2ef/contourpy-1.4.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3863ef2e2b13fe93f8c0ebb08ee400cb07153b8b0e91c5acb26e4537f283634f", size = 1427243, upload-time = "2026-09-11T19:03:03.866Z" }, + { url = "https://files.pythonhosted.org/packages/ac/f0/b75a10e9d0616b97a30de277b6dfe83f1879880854c7330337309530eb48/contourpy-1.4.0-cp312-cp312-win32.whl", hash = "sha256:5450f091ac1be0be3ad3a2a3b3f23b5e443e78c670ced4fd347d626f92a28fd2", size = 347345, upload-time = "2026-09-11T19:03:06.192Z" }, + { url = "https://files.pythonhosted.org/packages/50/a9/dab08786bb4d77ef9a046b3c1be923be4e73c5d07d91f5a54e8ea9b09e41/contourpy-1.4.0-cp312-cp312-win_amd64.whl", hash = "sha256:6e697d94e69f499ff6bebb899cae97a58d5d14f0e1fe9568b43a0248d2f9af8c", size = 234083, upload-time = "2026-09-11T19:03:08.122Z" }, + { url = "https://files.pythonhosted.org/packages/47/b9/3ba509755a970dbde5948e7142ac21f266be72f38017cc4087a05fdceee1/contourpy-1.4.0-cp312-cp312-win_arm64.whl", hash = "sha256:0c7a4c2716a4e98342221954416a836cca77c996c14ddbf22b5f01d5d93ca09c", size = 559521, upload-time = "2026-09-11T19:03:10.047Z" }, +] + +[[package]] +name = "cycler" +version = "0.12.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a9/95/a3dbbb5028f35eafb79008e7522a75244477d2838f38cbb722248dabc2a8/cycler-0.12.1.tar.gz", hash = "sha256:88bb128f02ba341da8ef447245a9e138fae777f6a23943da4540077d3601eb1c", size = 7615, upload-time = "2023-10-07T05:32:18.335Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl", hash = "sha256:85cef7cff222d8644161529808465972e51340599459b8ac3ccbac5a854e0d30", size = 8321, upload-time = "2023-10-07T05:32:16.783Z" }, +] + [[package]] name = "datasets" version = "5.0.1" @@ -280,6 +311,23 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e8/2d/d2a548598be01649e2d46231d151a6c56d10b964d94043a335ae56ea2d92/flatbuffers-25.12.19-py2.py3-none-any.whl", hash = "sha256:7634f50c427838bb021c2d66a3d1168e9d199b0607e6329399f04846d42e20b4", size = 26661, upload-time = "2025-12-19T23:16:13.622Z" }, ] +[[package]] +name = "fonttools" +version = "4.66.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a2/24/86f9930b930b97fc82266083320f3c34643ef67261c32651e893db525aca/fonttools-4.66.0.tar.gz", hash = "sha256:ef0610dfe7bb5bf574d9bdad6f597403ebc9807d124ac6f148d7604b2609be98", size = 3692007, upload-time = "2026-09-23T17:59:14.345Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e7/c1/25821e3bddd0644df229cfdc2e8b4101d28e4da15ffbe6e6b563ea6f3acd/fonttools-4.66.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:67ea5af3ca60e1e5c9b2841b1f6dba5ef16aab581bf463abb68515447154ab61", size = 3096788, upload-time = "2026-09-23T17:57:08.302Z" }, + { url = "https://files.pythonhosted.org/packages/0c/3d/d16b4cb63710abd91f1aaa482f2848d8b0c402502de5b8a72ba9d2613cb0/fonttools-4.66.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:dbee639a23c4e067aedfbcf84d2e466a71a1a1156ad04d6d09023791ed167495", size = 2586926, upload-time = "2026-09-23T17:57:10.678Z" }, + { url = "https://files.pythonhosted.org/packages/e2/f0/dffa4fa83ec6780464e8484fa2f30ccc252d480137c8e4d3fc6ac510b3eb/fonttools-4.66.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:35684b562df7154d7a0ebfa512db9199c7fb0a93b5abfb59dac9d022dbee7aaa", size = 5415821, upload-time = "2026-09-23T17:57:13.174Z" }, + { url = "https://files.pythonhosted.org/packages/6d/56/3c862bdd297a272a37e2ba1149da01446fd3f387ee80855699bb939bf5b2/fonttools-4.66.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:aea0cc5609a5f2a2500f91bd6e1989fdd22da0f225be00dc1c4cec775838a7d0", size = 5396689, upload-time = "2026-09-23T17:57:15.987Z" }, + { url = "https://files.pythonhosted.org/packages/93/d7/f4b7c81a7a6b316b7226d11f8e8659f023babf1e0957f8e7ddcc4e155a82/fonttools-4.66.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8af85dfc2ad564f95b3e650024578bb2268bc4d827d418aab9650bbde4b6926a", size = 5353218, upload-time = "2026-09-23T17:57:18.803Z" }, + { url = "https://files.pythonhosted.org/packages/70/1e/f12bc535f3da359a4d29d5e7442ab2d7d67579d1516b9c7e1a38890168bd/fonttools-4.66.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:2b278e2abe596872b3c9fe611f956ded85588537974e48e2adbe2d6ee159e099", size = 5516965, upload-time = "2026-09-23T17:57:21.077Z" }, + { url = "https://files.pythonhosted.org/packages/14/25/a1df66888c8b5f63c14e089400f2353da0c1ff9d09683cc0bf85afea348e/fonttools-4.66.0-cp312-cp312-win32.whl", hash = "sha256:ee7b9f6c835ea1a9beb5a35c8eb0c62b6a3290f8105d3d0c29a0a18c90f9b8ba", size = 2434425, upload-time = "2026-09-23T17:57:23.463Z" }, + { url = "https://files.pythonhosted.org/packages/b5/f2/fd490f1ddf2e430289c13059982771e6b02a2bf37504401d2a5c53307892/fonttools-4.66.0-cp312-cp312-win_amd64.whl", hash = "sha256:1d120ea0f5260b04e9b5ac0d9239efde4c23d990347d5d5889cdd37221726fec", size = 2486335, upload-time = "2026-09-23T17:57:25.561Z" }, + { url = "https://files.pythonhosted.org/packages/83/65/826290863c9df6041f2e36a5ae5d604cd8247fffc8ec7b45581fd2473e4d/fonttools-4.66.0-py3-none-any.whl", hash = "sha256:bc7b7ddc1a1f46c363354304e9a8dd93722e4a6a24f785015650898dacf40df9", size = 1201583, upload-time = "2026-09-23T17:59:12.104Z" }, +] + [[package]] name = "frozenlist" version = "1.8.0" @@ -499,6 +547,32 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/18/53/84099323c2ec4be98d935f63c033ac4151ee83836ca1050ede3b3aadf155/joblib-1.6.0-py3-none-any.whl", hash = "sha256:3dbbf9f6e4b592a2357b854608e980fe6390d131d7a82f011a377ef2ebef7aba", size = 306115, upload-time = "2026-08-31T09:39:02.298Z" }, ] +[[package]] +name = "kiwisolver" +version = "1.5.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ba/07/bd78e6a8fae171ea041ef5bba3ed21a003522fa088834b069b1909981f30/kiwisolver-1.5.1.tar.gz", hash = "sha256:f1303ef2eec81262a4b708c3e858afe58d7c75ad91c1c05266eda7673369859a", size = 104395, upload-time = "2026-08-28T10:28:27.153Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6b/9b/65b302742389c6f96f2956bef5decf26011309feb2fc5d79613af18adea4/kiwisolver-1.5.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:63fb7294b768f444eb4b068965f2662f28c2fd4161e23bd60fcf3ff27b74c046", size = 123876, upload-time = "2026-08-28T10:25:27.44Z" }, + { url = "https://files.pythonhosted.org/packages/71/74/c21f339956f6f691b2ed7e31d5f3ae767304df6c460192739fc830853051/kiwisolver-1.5.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:0ebdef3eae5336568147c39a55be6a2036ffde53faa9ca2d978989ae7c2da12c", size = 66487, upload-time = "2026-08-28T10:25:28.728Z" }, + { url = "https://files.pythonhosted.org/packages/84/e5/bdb34e21523e01dceda064d63713f3bdec91388af24fba1eca7ea5e85864/kiwisolver-1.5.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1798e83840c3f627246104c4d8a9639c60fa068adf9ce92b61791781fa8a68c1", size = 64660, upload-time = "2026-08-28T10:25:30.071Z" }, + { url = "https://files.pythonhosted.org/packages/fc/f4/dadfec469313c7f428efa7e84b4aba9732f813c13ea7131a24b7b008ef57/kiwisolver-1.5.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:34633ecf50d16187ab8e5528b7a2530f2feb4e23f300db4672538b51cfc5cd38", size = 1477929, upload-time = "2026-08-28T10:25:31.495Z" }, + { url = "https://files.pythonhosted.org/packages/6f/35/09c58daac34e6f6ea5c6dee0094b422118e5a7c265586008a95fd135ac5f/kiwisolver-1.5.1-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d27c2123977cb9269c30a49ba45f03a4323017ef693e19db4ec9dbe1299a3002", size = 1278499, upload-time = "2026-08-28T10:25:33.375Z" }, + { url = "https://files.pythonhosted.org/packages/19/32/739765e24fbad29d13f83e546ea4abc215a78cea9d677ca09025b027724d/kiwisolver-1.5.1-cp312-cp312-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6a797a1cefc8b9c93170db580337e1fe3d011ad18b1299943231279406342048", size = 1296677, upload-time = "2026-08-28T10:25:35.059Z" }, + { url = "https://files.pythonhosted.org/packages/df/32/03304d1010e2cc45e5b3b52cef7e43fed3a2a5cd6c87a89b4a88e1d85b5d/kiwisolver-1.5.1-cp312-cp312-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2551cf9917af48ee7c4b29cc82320489508cf96fd26a51f6fc124de661cd44c7", size = 1346037, upload-time = "2026-08-28T10:25:36.705Z" }, + { url = "https://files.pythonhosted.org/packages/3e/57/4c49377bfd274450dd72ecaa13eaac32ea804a03363e4d1db0c5aa999ceb/kiwisolver-1.5.1-cp312-cp312-manylinux_2_39_riscv64.whl", hash = "sha256:38f6e0deb4d0a4615efe0c4efc5990b06ae450ab50a0b321c0b078b6d238c083", size = 988248, upload-time = "2026-08-28T10:25:38.299Z" }, + { url = "https://files.pythonhosted.org/packages/cb/c3/38df144a08b6c5d75ca4504e5cc3141bb3bfef64c04f4ef48204f42711b6/kiwisolver-1.5.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:bfd1de989b3330420e29de39352f5c049905c9e3ee67233a50d550e3d652c148", size = 2228722, upload-time = "2026-08-28T10:25:40.038Z" }, + { url = "https://files.pythonhosted.org/packages/e7/11/3221838a89cd64d9b386353e000cd8a296069a20fbe3584507fdfd5bebae/kiwisolver-1.5.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:1209042a623ddfda5497e4066c7b77651dde8e1d3a9dd97599dc7e97f3b9b78c", size = 2325216, upload-time = "2026-08-28T10:25:41.699Z" }, + { url = "https://files.pythonhosted.org/packages/83/d4/075c219230697bb5db910d37262b9bacf880f92b4811a02ab81ed073a253/kiwisolver-1.5.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:26e8268480be5061d509e29669d59103c067a26377a56491630ece11762e3858", size = 1977689, upload-time = "2026-08-28T10:25:43.559Z" }, + { url = "https://files.pythonhosted.org/packages/bb/08/1d219c3c2dd960983d0d4da623d916e9de6385df2b0bab3d1af0e9b8fccc/kiwisolver-1.5.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:d79308fa689fac89cbcfbd4dbfc80b5f95c54c5a7fd4d194be221f9d33d026e6", size = 2491443, upload-time = "2026-08-28T10:25:45.242Z" }, + { url = "https://files.pythonhosted.org/packages/ba/d3/024208ec1079d273f1047468d1bdffbf38bb75b7b268090fd3a0301b9d9a/kiwisolver-1.5.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:b03af77d77e50edba2030fd5f7c352ff209314b09030a3cba7c14edf9a09a444", size = 2295200, upload-time = "2026-08-28T10:25:46.984Z" }, + { url = "https://files.pythonhosted.org/packages/6e/7c/7b210498f9f92e1cd7855f260fa69ef056881087b199ee20c208f0e4189a/kiwisolver-1.5.1-cp312-cp312-win_amd64.whl", hash = "sha256:06a6917674de9e0fe3f66f5430787f59a9f2ddb64af9b714eaec547e29ef5c19", size = 70748, upload-time = "2026-08-28T10:25:48.444Z" }, + { url = "https://files.pythonhosted.org/packages/94/61/ef0daa157c8bb23672f7423e0d14c39db1dc6ef8ed47e6bc54c9c1bef3bf/kiwisolver-1.5.1-cp312-cp312-win_arm64.whl", hash = "sha256:ad8b9671348d7c8716715652ae11f85ed0eb99e265a2df2ca490577d69860b2c", size = 68324, upload-time = "2026-08-28T10:25:49.81Z" }, + { url = "https://files.pythonhosted.org/packages/a9/c4/1407df7512a5b36cc79840e01710dc575733c461b13ab866cae77eaf87f3/kiwisolver-1.5.1-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:482676e5bd48d70ac99d9fc78863469845421e01184fa83f1f9366dc49f7e974", size = 134002, upload-time = "2026-08-28T10:28:08.881Z" }, + { url = "https://files.pythonhosted.org/packages/16/45/c37a21ad5c0ab581a93c55ad544721aaa1f0ae94edb29c6a678a23d013e6/kiwisolver-1.5.1-graalpy312-graalpy250_312_native-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:072bdb15a3c19a5b5dbc8f8fb1f4e1884bf4f3507eeb4cc6334401274d37a5c0", size = 194292, upload-time = "2026-08-28T10:28:11.06Z" }, + { url = "https://files.pythonhosted.org/packages/a6/c1/69f00d627949580e43d57af0aa465df46868d7c29801c137a55374101294/kiwisolver-1.5.1-graalpy312-graalpy250_312_native-win_amd64.whl", hash = "sha256:a5a00665d1a0e26763a7338d7e911d4598fbc1d50dd0d6b7919b7dc6c5d6569f", size = 73362, upload-time = "2026-08-28T10:28:12.449Z" }, +] + [[package]] name = "markdown-it-py" version = "4.2.0" @@ -530,6 +604,32 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e5/f1/216fc1bbfd74011693a4fd837e7026152e89c4bcf3e77b6692fba9923123/markupsafe-3.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:35add3b638a5d900e807944a078b51922212fb3dedb01633a8defc4b01a3c85f", size = 13906, upload-time = "2025-09-27T18:36:40.689Z" }, ] +[[package]] +name = "matplotlib" +version = "3.11.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "contourpy" }, + { name = "cycler" }, + { name = "fonttools" }, + { name = "kiwisolver" }, + { name = "numpy" }, + { name = "packaging" }, + { name = "pillow" }, + { name = "pyparsing" }, + { name = "python-dateutil" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/e7/c8/9aa712a0afb882649424dd8de8ad9aa6235e796e84c6052e8f6dc1598d0d/matplotlib-3.11.2.tar.gz", hash = "sha256:cec596316640f2b394b8f0daa0ea61a8eae82d017b620b9f202befb972a59ea4", size = 32660610, upload-time = "2026-09-11T19:05:31.214Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/ce/1bfcc4873b121597791ad74032943b123218c0613af0b97e8dd05e916fb1/matplotlib-3.11.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ef752769cd962f39ea0b6ffc82d1ea43a0012c5a6157c7a075212fa509cfcff2", size = 9476466, upload-time = "2026-09-11T19:03:19.45Z" }, + { url = "https://files.pythonhosted.org/packages/a6/c4/7f5f3601ee69baf072c0c7d3ce60c03e0618621c5a56c34a62a460e29d11/matplotlib-3.11.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ef31985c4dedb5f1424e1aec6849a47dd37689cb7fa3c20b1b82187f26806261", size = 9305583, upload-time = "2026-09-11T19:03:22.39Z" }, + { url = "https://files.pythonhosted.org/packages/b8/90/2b3fd67ee273163faeda6d514be70b5596eaa0fd77b60ffc294ad0b34f5f/matplotlib-3.11.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:df4f7784aca81a94f254c0a2767d592ee25f407e488f5fa7203e51093fb6ca27", size = 9860353, upload-time = "2026-09-11T19:03:25.049Z" }, + { url = "https://files.pythonhosted.org/packages/f4/84/32549e7a462dc311aed2ab62e5d2538028840b5c53a8be0195c789937c3d/matplotlib-3.11.2-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1b9a7ad579856284135e401ecc918c5f8a017ee30539298862a109f51b971710", size = 10670348, upload-time = "2026-09-11T19:03:28.108Z" }, + { url = "https://files.pythonhosted.org/packages/84/39/02e21b74f7439bd643d717ea846006d46e309e6d29b632b57751265311d6/matplotlib-3.11.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3aa4b8516fd26659e4363abbf317c703d9116496c5db2e9d0609a2866dd39dd2", size = 10810580, upload-time = "2026-09-11T19:03:31.402Z" }, + { url = "https://files.pythonhosted.org/packages/04/93/0d239bde12b308262265c2d98909b2f0ecad2b5deeae96241642e822e9c7/matplotlib-3.11.2-cp312-cp312-win_amd64.whl", hash = "sha256:c5c1c68ee401fc98271263410f0e5ce88285abacf7627132914e8adf3d70ff43", size = 9349409, upload-time = "2026-09-11T19:03:34.195Z" }, + { url = "https://files.pythonhosted.org/packages/49/a8/06baf901c02246c8a222b655cc4540ef9c15e1549a04e07927f9cde716c3/matplotlib-3.11.2-cp312-cp312-win_arm64.whl", hash = "sha256:643ff850d8e0f5b8319337f87ed3cb59506afb3df3cc48de777d85871233be7b", size = 9028084, upload-time = "2026-09-11T19:03:37.032Z" }, +] + [[package]] name = "mdurl" version = "0.1.2" @@ -748,6 +848,23 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/3b/11/3d580a604a1e35d69f6676847bd12db7d14344bc677b717f4413e79c5d0d/pandas-3.0.6-cp312-cp312-win_arm64.whl", hash = "sha256:f4e7c52eb108d752e7592268108fd3e98efd76d83a3125cdd06c621c2e44359b", size = 8959806, upload-time = "2026-09-17T23:21:13.851Z" }, ] +[[package]] +name = "pillow" +version = "12.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1c/3d/bb7fca845737cf9d7dbde16ed1843984665ff2e0a518f5db43e77ec540b9/pillow-12.3.0.tar.gz", hash = "sha256:3b8182a766685eaa002637e28b4ec8d6b18819a0c71f579bf0dbaa5830297cce", size = 47025035, upload-time = "2026-07-01T11:56:38.965Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/37/bf/fb3ebff8ddcb76aac5a01389251bbbb9519922a9b520d8247c1ca864a25d/pillow-12.3.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ba09209fbe443b4acccebe845d8a138b89a8f4fbaeedd44953490b5315d5e965", size = 5345969, upload-time = "2026-07-01T11:54:06.397Z" }, + { url = "https://files.pythonhosted.org/packages/d8/66/9a386a92561f402389a4fc70c18838bf6d35eb5eb5c6850b4b2dc64f5048/pillow-12.3.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ffd0c5368496f41b0944be820fcb7a838aa6e623d250b01acf2643939c3f99d7", size = 4780323, upload-time = "2026-07-01T11:54:09.351Z" }, + { url = "https://files.pythonhosted.org/packages/25/27/ac8f99618ffd3dde21db0f4d4b1d2ab00c0880595bfd17df103f7f39fd0c/pillow-12.3.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d9c7f76c0673154f044e9d78c8655fb4213f6ca31a836df48b40fe5d187717b9", size = 6266838, upload-time = "2026-07-01T11:54:11.71Z" }, + { url = "https://files.pythonhosted.org/packages/84/21/a35af28dcc61f37ed850a2d64c65c701321dfbf25085e469d5559360cbbf/pillow-12.3.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:78cb2c6865a35ab8ff8b75fd122f6033b92a62c82801110e48ddd6c936a45d91", size = 6940830, upload-time = "2026-07-01T11:54:13.732Z" }, + { url = "https://files.pythonhosted.org/packages/eb/51/8b08617af3ad95e33ce6d7dd2c99ed6c8298f7fb131636303956be022e25/pillow-12.3.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:e491916b378fba47242221bb9ead245211b70d504f495d105d17b14a24b4907c", size = 6344383, upload-time = "2026-07-01T11:54:15.756Z" }, + { url = "https://files.pythonhosted.org/packages/1d/72/cf78ac9780bb93c28328f408973845a309d4d145041665f734572ced1b52/pillow-12.3.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:0dd2064cbc55aaec028ef5fbb60fa47bb6c3e7918e07ff17935284b227a9d2df", size = 7052934, upload-time = "2026-07-01T11:54:17.721Z" }, + { url = "https://files.pythonhosted.org/packages/20/20/25e0f4dc178a6bc0696793720055519a0de89e7661dae886992decbd2f81/pillow-12.3.0-cp312-cp312-win32.whl", hash = "sha256:dbce0b29841537a2fa4a214c2bbf14de3587c9680caa9b4e217568472490b28f", size = 6472684, upload-time = "2026-07-01T11:54:19.839Z" }, + { url = "https://files.pythonhosted.org/packages/45/89/da2f7971a317f83d807fdd4065c0af40208e59e692cc43d315a71a0e96d1/pillow-12.3.0-cp312-cp312-win_amd64.whl", hash = "sha256:a2b55dd6b2a4c4b7d87ffa56bdb33fdc5fdb9a462173861a7bc097f17d91cb09", size = 7227137, upload-time = "2026-07-01T11:54:22.025Z" }, + { url = "https://files.pythonhosted.org/packages/de/47/4845a0a6c0dbf1db8456bd9fc791f13c5ced7ced20606d08a0aacfd25b49/pillow-12.3.0-cp312-cp312-win_arm64.whl", hash = "sha256:331b624368d4f1d069149002f25f44bc61c8919ce8ddb3c45bdad8f6e2d89510", size = 2568267, upload-time = "2026-07-01T11:54:24.051Z" }, +] + [[package]] name = "pluggy" version = "1.6.0" @@ -883,6 +1000,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/71/46/17f022dd3e953bf20a04a028a21ec746d942f8d2af30fa0f124fa0e6a684/pygments-2.21.0-py3-none-any.whl", hash = "sha256:2363c69b61c4a97c838da3b130dcd6468f4848992b21a82f2a63ec34377137d9", size = 1250147, upload-time = "2026-08-17T08:02:44.912Z" }, ] +[[package]] +name = "pyparsing" +version = "3.3.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e4/11/b213bebff182584360cb8d17c72c1677fec5c5c228de439e63bcf8ab1c8f/pyparsing-3.3.3.tar.gz", hash = "sha256:928ae7e20211f3b6f3915a72f06a0cfd29ab9d24279dd6346b6b1a7146397d36", size = 1050487, upload-time = "2026-09-20T20:59:05.609Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/38/bb/d215ee7c73b61497b28a5503f9f53523f294fcc936762b7caf90e0c1c2b5/pyparsing-3.3.3-py3-none-any.whl", hash = "sha256:ece8c00a69cf01b45d0b1dedabb469c90d8caf996d4fda40f147627a122849a4", size = 126420, upload-time = "2026-09-20T20:59:04.025Z" }, +] + [[package]] name = "pytest" version = "9.1.1" @@ -1152,6 +1278,9 @@ dev = [ { name = "ruff" }, { name = "scipy" }, ] +figures = [ + { name = "matplotlib" }, +] llm = [ { name = "anthropic" }, ] @@ -1181,6 +1310,7 @@ dev = [ { name = "ruff", specifier = "==0.16.8" }, { name = "scipy", specifier = "==1.18.1" }, ] +figures = [{ name = "matplotlib", specifier = "==3.11.2" }] llm = [{ name = "anthropic", specifier = "==1.8.0" }] onnx = [ { name = "onnx", specifier = "==1.23.0" },