diff --git a/bench/tokenaudit/README.md b/bench/tokenaudit/README.md new file mode 100644 index 000000000..21f55bf34 --- /dev/null +++ b/bench/tokenaudit/README.md @@ -0,0 +1,121 @@ +# tokenaudit — is `est_tokens=` true? + +`est_tokens=` is the number `--token-budget` gates on, the number `--max-tokens` fits to, and the number +the tool prints when it says what an answer cost. Until 2026-09-09 nothing in the tree compared it to a +real tokenizer. `test/tokenbudgetcheck.sh` validated its *properties* — present, positive, deterministic, +monotone under a tighter budget, bounded by an allowance derived from the estimate's own constants — and +its own header said the accuracy number "is REPORTED by the agent in the T1 write-up". That write-up was +2026-07. METHODOLOGY §9 principle 6 says measuring gets its own instrument; this directory is it. + +Two artifacts, deliberately split by whether they may need a Python package: + +| File | Needs tiktoken | What it is | +| --- | --- | --- | +| `sweep.py` | yes | the survey: ~25 invocations × N corpora, est vs real, per verb, plus the legend's share in real tokens | +| `pin.py` | yes | writes `test/estcalib.manifest` — real o200k/cl100k counts on the frozen fixture `test/estcalibfix` | +| `test/tokenbudgetcheck.sh` #18 | **no** | the gate: reads the manifest, holds every pin inside a band and the set's MAPE under a ceiling | + +The split is the point. G3 is one deterministic build step with nothing host-installed, so a gate that +imported tiktoken would be a dependency the build contract forbids. The tokenizer runs by hand and writes +numbers; the gate reads numbers. Same shape as `test/printf_parity.manifest`. + +The corpus **labels** are what the results JSON records — never a path, never a private tree's name. +`test/ripwirepubliccheck.sh` caught all three leak classes in this directory's first committed results +file; `sweep.py` now writes labels by construction so the gate has nothing to catch. + +```bash +python3 -m venv /tmp/tokvenv && /tmp/tokvenv/bin/pip install tiktoken +/tmp/tokvenv/bin/python bench/tokenaudit/sweep.py --bin build/ripwire \ + --corpus self=. --corpus other=/path/to/another/tree \ + --out bench/tokenaudit/results/tokenaudit-YYYY-MM-DD.json +/tmp/tokvenv/bin/python bench/tokenaudit/pin.py --bin build/ripwire # regenerates the manifest +python3 -c "import json,statistics as s; d=json.load(open('bench/tokenaudit/results/tokenaudit-2026-09-09.json'))" +``` + +## What the 2026-09-09 run measured + +`results/tokenaudit-2026-09-09.json`: 25 invocations on this repository and on a 1,500-file private C++ +tree, `ripwire 0.5.0 built_from=4c10be9d7`, tiktoken `o200k_base` (and `cl100k_base` beside it — the two +agree to within 1.4% on every row, which is the ≤4% spread `kTokenCalib`'s header claims, re-derived). +No Anthropic `count_tokens` arm: `ANTHROPIC_API_KEY` was not in the environment, so **Claude's own +tokenizer is unmeasured here** and o200k_base remains the public stand-in the table was calibrated against. + +**1. `est_tokens=` is printed by 9 of the 25 invocations.** The sixteen that print no price include every +navigation verb — `--callers`, `--callees`, `--impact`, `--uses`, `--affected`, `--edit-check`, `--grep`, +`--test-gate`, `--hotspots`, `--lint`, `--tree`, `--clones` — and both JSON dialects. These are exactly the +answers whose legend share is largest, so the price is missing where it is highest. A `--edit-check` on a +macro with thousands of call sites emitted **348,224 B / 99,006 real tokens in one answer**, priced at +nothing. + +**2. The signed error is not centred and is not one-directional.** Per-verb, against o200k: + +| verb | this repo | 1500-file C++ tree | fixture pin | +| --- | --- | --- | --- | +| map | +2.8% | +9.8% | +11.3% | +| map `--top-k=10` | +15.9% | +19.0% | +21.4% (`--top-k=5`) | +| `--metrics` | +4.4% | +7.2% | +5.4% | +| `--pack-signatures` | +17.0% | +18.5% | +17.5% | +| `--expand` | +1.9% | **−18.4%** | **−16.4%** | +| `--for` (named) | +26.1% | +35.6% | +33.1% | +| `--for` (conceptual) | +21.9% | +25.5% | — | +| `--pack-task` | +18.9% | +24.6% | +25.2% | +| `--around` | +14.7% | +13.7% | — | + +Median +15.9% / +18.5%; MAPE over the eight pins 21%. The mechanism is measured, not guessed: real +bytes-per-token ranges **2.44 (dense signature rows) to 4.66 (legend prose)** across these documents, +while the conversion uses one language-keyed rate near 2.5 for markup and 3.8 for bodies. The error is a +property of the **document shape**, not of the corpus language the rate is keyed on — which is why adding +a language row cannot fix it, and why the `--expand` body rate that is right on large C++ bodies +under-reads by 16–18% on short dense ones. + +**3. The legend's price, in the unit the owner mandated.** `--help` states the compact saving in bytes +("at least 50% of a small `--callers`/`--uses`/`--impact`/`--affected` answer") and +`test/legendcostcheck.sh` holds the binary to it in bytes. In **tokens** the same measurement on the +gate's own symbol (`lookupLang`) reads lower on all four, because the legend is prose (4.4 B/tok) and the +rows it is compared against are markup (2.7 B/tok): + +| verb | byte saving | token saving | gap | +| --- | --- | --- | --- | +| `--callers` | 70.9% | 60.5% | 10.4 pt | +| `--uses` | 65.8% | 55.2% | 10.6 pt | +| `--impact` | 51.9% | **39.4%** | 12.5 pt | +| `--affected` | 70.1% | 65.8% | 4.3 pt | + +`--impact` clears the published 50% floor in bytes and misses it in tokens. Across the whole sweep the +legend's token share ran 3.2% (whole map) to **79.8% (`--callees`)**, with `--edit-check` 62.9% and +`--test-gate` 52.4% on this repository — corroborating the magnitude of the outside evaluation that +started this (callstack/agent-device #2400, "a fixed per-call preamble, 62% of `--callers`' whole +response") while placing `--callers` itself at 42.9%/33.3% on the two corpora here. + +**4. `--legend=compact` is not a saving on `--for`.** Measured −0.8% (this repo) and −1.9% (the C++ tree) +at `4c10be9d`: the compact posture emitted *more* tokens, because `--for` is budget-shaped and the bytes the +legend frees are refilled from the trim ladder's tail. Re-measured after the commit that added this +directory, the same two arms read −0.1% and −1.9% — the magnitude moves with the corpus, the direction is +what to carry: a wash or a small loss, never the 39-66% token saving the navigation verbs show. `--help`'s +advice ("MAKING REPEATED CALLS? USE compact") is right for the navigation verbs and wrong-signed for the +bundle it also names as "a little". + +**5. What a `--token-budget=N` actually delivers.** Real o200k tokens as a fraction of the requested N, +and the reported `est_tokens` error at that budget: + +| verb | N=1500 | N=3000 | N=6000 | +| --- | --- | --- | --- | +| `--for` delivered | 76% / 75% | 74% / 67% | 54% / 48% | +| `--for` est error | +38% / +42% | +25% / +27% | +22% / +26% | +| `--pack-task` delivered | 82% / 81% | 52% / 61% | 58% / 57% | + +(this repo / the C++ tree). Part of the shortfall at a large N is content exhaustion — there is no more to +serve — but at the binding budgets the estimate over-reads by a quarter to two fifths, and the budget is a +hard ceiling on the estimate, so a caller asking for 3,000 tokens of context is handed about 2,000. + +## What was NOT changed, and why + +No constant in `kTokenCalib` moved. The error is per-document-shape and signed both ways: no single rate, +and no per-language row, corrects a +40% on a legend-heavy bundle and a −16% on a short body at the same +time. The change that would is a per-SPAN charge (prose bytes at a prose rate, the way `kBytesPerTokenBody` +already charges body bytes at a body rate) — a behaviour change to a number pinned by goldens, +`fornotesbudgetcheck`, `forbudgetmonotoncheck`, `packtaskquotacheck` and published budget figures. That is +a round, not a lane. What this round leaves behind is the instrument that makes the round's before/after +measurable, and one corrected sentence in `src/serialize.h` — "the number never systematically +under-reads" was false for `--expand` on two corpora and is now the measured range with the gate that +holds it. diff --git a/bench/tokenaudit/loop_ledger.py b/bench/tokenaudit/loop_ledger.py new file mode 100644 index 000000000..a79a1a68d --- /dev/null +++ b/bench/tokenaudit/loop_ledger.py @@ -0,0 +1,290 @@ +#!/usr/bin/env python3 +# loop_ledger.py — what a ripwire call costs across a WHOLE EDIT LOOP, from the agent's own transcript. +# +# THE GAP. Every number ripwire prints is per call: est_tokens on the answer it just wrote. The claim the +# tool makes is per LOOP — "the agent needs no further native read after the call" (METHODOLOGY §9 +# principle 1). Those are different quantities, and only the second one can go negative: an answer that +# costs 1,400 tokens and saves three 2,000-token file reads is a win; the same answer followed by the +# three reads anyway is a 1,400-token loss. Nothing in the tree measured the second quantity in TOKENS. +# `bench/substitution_report.py` counts the CALLS (the meter's terminality section, §5) and is the +# ancestor of this file; it deliberately prints no byte or token figure, so a verb could hold its +# terminality rate steady while its answers doubled in size and the report would not move. +# +# WHAT THIS READS, AND THE PRIVACY RULE. Claude Code writes one JSONL per session under +# ~/.claude/projects//. This script reads ONE project's directory, named on the command +# line, and prints AGGREGATES: per-session token counts and ratios, and distribution summaries over +# sessions. It never prints a prompt, a file path from a transcript, a tool argument, or a session id +# unless --sessions is passed (a local-only debugging aid). Nothing it prints is intended to be pasted +# anywhere that the transcripts themselves would not be. +# +# WHAT IT COUNTS. +# * TOOL-RESULT tokens, split ripwire / native / other. This is context an answer PUT INTO the window. +# ripwire = mcp__ripwire__* tools, plus Bash commands that invoke the binary (`ripwire`, +# `build/ripwire`, `./build/ripwire`, `asan/ripwire`). +# native = Read / Grep / Glob / NotebookRead, plus Bash commands whose head is a retrieval command +# (grep, rg, cat, head, sed -n, find, ls, awk-over-a-file). These are the calls ripwire is +# a substitute FOR — the same family split bench/substitution_report.py uses. +# other = everything else (edits, builds, git, gates, agent plumbing). Counted, never in a ratio. +# * NON-TERMINAL ripwire calls: a ripwire call with at least one native call in the NEXT 3 tool calls. +# Three is the substitution meter's own definition of a post-call sweep, kept identical on purpose so +# the two instruments are comparable; the meter's own window is 5 and is reported beside it. +# * PROVIDER-REPORTED session usage from `message.usage` (input / output / cache_creation / +# cache_read). This is the codeburn-style measurement: what the API actually billed, not an estimate. +# It is reported beside the tool-result totals so the tool-result share of a session is visible. +# +# TOKENIZER. tiktoken if importable (o200k_base), else bytes / 2.5 with `tokenizer=estimated` stamped on +# the output — never silently. The provider `usage` numbers need no tokenizer and are exact. +# +# Usage: +# python3 bench/tokenaudit/loop_ledger.py ~/.claude/projects/ [--json OUT] [--sessions] + +import argparse +import collections +import glob +import json +import os +import re +import statistics +import sys + +NATIVE_TOOLS = frozenset(("Read", "Grep", "Glob", "NotebookRead")) +SWEEP_LOOKAHEAD = 3 # the substitution meter's post-call sweep definition + +# ── THE BASH CLASSIFIER IS THE METER'S, DELIBERATELY ────────────────────────────────────────────── +# The first version of this file matched a retrieval command only at the START of the line, and on +# this operator's log that under-counted native retrieval by ~3x: almost every Bash line here begins +# with plumbing (`cd X && grep …`, `echo "=== x ==="; sed -n …`), which is the exact gap +# hooks/ripwire-nudge.sh recorded and fixed in its 2026-08-12 classifier-gap round. The rules below +# are that function's, ported: walk the SEQUENCED segments (`;`, `&&`, `||`) and stop at the first +# the head table decides; do NOT walk pipeline stages (a `| grep` filters the FIRST command's output +# and is not a second observation); `grep -c`/`-q` is a poll, not a search; `cat > f` is a write; +# `sed -n` is a whole-file read; `ls -R` is a walk. The parity is the point — the two instruments +# measure the same population differently ONLY where they are meant to, so a disagreement between +# this file and bench/substitution_report.py is a finding rather than a definition mismatch. +RIPWIRE_WORD = re.compile(r"(^|[\s;&|(`\"'])(\./)?(build/|asan/)?ripwire(?![\w/])") +SEQ_SPLIT = re.compile(r"(?:;|&&|\|\||\n)") +GREP_POLL = re.compile(r"(^|\s)(-[A-Za-z]*[cq][A-Za-z]*|--count|--quiet)(\s|$)") +SED_QUIET = re.compile(r"(^|\s)-[A-Za-z]*n(\s|$)") +LS_WALK = re.compile(r"(^|\s)-[A-Za-z]*R[A-Za-z]*(\s|$)|--recursive(\s|$)") +AWK_PATTERN = re.compile(r"['\"]/") + + +def classify_segment(seg): + words = seg.strip().split() + if not words: + return None + lead = words[0] + if lead == "sudo" and len(words) > 1: + words = words[1:] + lead = words[0] + sub = words[1] if len(words) > 1 else "" + base = lead.rsplit("/", 1)[-1] + if base == "ripwire": + return "ripwire" + if base in ("grep", "egrep", "fgrep", "zgrep", "rg", "ag", "ack", "ack-grep", "ugrep"): + return "other" if GREP_POLL.search(seg) else "native" + if base in ("ps", "pgrep"): + return "other" + if base in ("find", "fd", "fdfind"): + return "native" + if base in ("cat", "head", "tail", "less", "more", "bat", "nl", "tac"): + return "other" if sub.startswith(">") else "native" + if base == "ls": + return "native" if LS_WALK.search(seg) else None + if base in ("awk", "gawk", "mawk"): + return "native" if AWK_PATTERN.search(seg) else None + if base == "sed": + return "native" if SED_QUIET.search(seg) else None + if base == "git": + return "other" + return None + + +def classify(name, inp): + if isinstance(name, str) and name.startswith("mcp__ripwire__"): + return "ripwire" + if name in NATIVE_TOOLS: + return "native" + if name == "Bash": + cmd = inp.get("command") or "" if isinstance(inp, dict) else "" + # ripwire anywhere on the line wins before the walk, exactly as the meter's head table puts + # `ripwire` first: `ripwire . --for=x | head -40` is one ripwire call, not a `head`. + if RIPWIRE_WORD.search(cmd): + return "ripwire" + for seg in SEQ_SPLIT.split(cmd): + verdict = classify_segment(seg) + if verdict is not None: + return verdict + return "other" + + +def result_text(entry): + """The bytes a tool result actually put into the context window, as text.""" + out = [] + tur = entry.get("toolUseResult") + if isinstance(tur, str): + out.append(tur) + elif tur is not None: + out.append(json.dumps(tur, ensure_ascii=False)) + else: + msg = entry.get("message") or {} + content = msg.get("content") + if isinstance(content, list): + for b in content: + if isinstance(b, dict) and b.get("type") == "tool_result": + c = b.get("content") + out.append(c if isinstance(c, str) else json.dumps(c, ensure_ascii=False)) + return "".join(out) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("project_dir") + ap.add_argument("--json", default=None) + ap.add_argument("--sessions", action="store_true", help="print per-session rows (local debugging only)") + args = ap.parse_args() + + try: + import tiktoken + enc = tiktoken.get_encoding("o200k_base") + def tok(s): + return len(enc.encode(s, disallowed_special=())) + tokenizer = "o200k_base" + except Exception: + def tok(s): + return int(len(s.encode("utf-8", "replace")) / 2.5 + 0.5) + tokenizer = "estimated(bytes/2.5)" + + files = sorted(glob.glob(os.path.join(os.path.expanduser(args.project_dir), "*.jsonl"))) + sessions = {} + + for path in files: + # calls: ordered [(family, tool_use_id)]; results: tool_use_id -> tokens + calls = [] + results = {} + usage = collections.Counter() + sid = None + with open(path, errors="replace") as f: + for line in f: + line = line.strip() + if not line: + continue + try: + e = json.loads(line) + except Exception: + continue + sid = sid or e.get("sessionId") + msg = e.get("message") or {} + if e.get("type") == "assistant": + u = msg.get("usage") or {} + for k in ("input_tokens", "output_tokens", + "cache_creation_input_tokens", "cache_read_input_tokens"): + v = u.get(k) + if isinstance(v, int): + usage[k] += v + content = msg.get("content") + if isinstance(content, list): + for b in content: + if isinstance(b, dict) and b.get("type") == "tool_use": + calls.append((classify(b.get("name"), b.get("input")), b.get("id"))) + elif e.get("type") == "user": + ids = [] + content = msg.get("content") + if isinstance(content, list): + for b in content: + if isinstance(b, dict) and b.get("type") == "tool_result": + ids.append(b.get("tool_use_id")) + if ids: + t = tok(result_text(e)) + # a result block carries one id in practice; split evenly if ever more + for i in ids: + results[i] = results.get(i, 0) + t // max(1, len(ids)) + if not calls: + continue + + fam_tokens = collections.Counter() + fam_calls = collections.Counter() + for fam, cid in calls: + fam_calls[fam] += 1 + fam_tokens[fam] += results.get(cid, 0) + + nonterminal = 0 + rip_total = 0 + for i, (fam, _) in enumerate(calls): + if fam != "ripwire": + continue + rip_total += 1 + if any(calls[j][0] == "native" for j in range(i + 1, min(i + 1 + SWEEP_LOOKAHEAD, len(calls)))): + nonterminal += 1 + + sessions[sid or os.path.basename(path)] = { + "calls": dict(fam_calls), + "result_tokens": dict(fam_tokens), + "ripwire_calls": rip_total, + "nonterminal_ripwire_calls": nonterminal, + "provider_usage": dict(usage), + } + + # ── aggregates only ─────────────────────────────────────────────────────────────────────────── + tot = collections.Counter() + tot_calls = collections.Counter() + usage_tot = collections.Counter() + rip = nonterm = 0 + with_rip = [] + for sid, s in sessions.items(): + for k, v in s["result_tokens"].items(): + tot[k] += v + for k, v in s["calls"].items(): + tot_calls[k] += v + for k, v in s["provider_usage"].items(): + usage_tot[k] += v + rip += s["ripwire_calls"] + nonterm += s["nonterminal_ripwire_calls"] + if s["ripwire_calls"]: + with_rip.append(s) + + report = { + "schema": "ripwire.loopledger/v1", + "tokenizer": tokenizer, + "sweep_lookahead": SWEEP_LOOKAHEAD, + "sessions": len(sessions), + "sessions_with_a_ripwire_call": len(with_rip), + "tool_calls": dict(tot_calls), + "tool_result_tokens": dict(tot), + "ripwire_calls": rip, + "nonterminal_ripwire_calls": nonterm, + "provider_usage_tokens": dict(usage_tot), + } + if rip: + report["nonterminality_rate"] = round(nonterm / rip, 4) + n = tot["native"] + tot["ripwire"] + if n: + report["ripwire_share_of_retrieval_tokens"] = round(tot["ripwire"] / n, 4) + if tot_calls["native"] + tot_calls["ripwire"]: + report["ripwire_share_of_retrieval_calls"] = round( + tot_calls["ripwire"] / (tot_calls["native"] + tot_calls["ripwire"]), 4) + for fam in ("ripwire", "native"): + per = [s["result_tokens"].get(fam, 0) / s["calls"][fam] + for s in sessions.values() if s["calls"].get(fam)] + if per: + report["median_tokens_per_%s_call" % fam] = round(statistics.median(per), 1) + report["mean_tokens_per_%s_call" % fam] = round(statistics.mean(per), 1) + if usage_tot: + billed = sum(usage_tot.values()) + if billed: + report["tool_result_share_of_billed_tokens"] = round( + (tot["ripwire"] + tot["native"] + tot["other"]) / billed, 4) + + json.dump(report, sys.stdout, indent=1, sort_keys=True) + sys.stdout.write("\n") + if args.json: + with open(args.json, "w") as f: + json.dump(report, f, indent=1, sort_keys=True) + f.write("\n") + if args.sessions: + for sid, s in sorted(sessions.items()): + sys.stderr.write("%s %s\n" % (sid[:8], json.dumps(s, sort_keys=True))) + + +if __name__ == "__main__": + main() diff --git a/bench/tokenaudit/pin.py b/bench/tokenaudit/pin.py new file mode 100644 index 000000000..09f2a03ce --- /dev/null +++ b/bench/tokenaudit/pin.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +# pin.py — regenerate test/estcalib.manifest: the REAL o200k_base / cl100k_base token counts of +# ripwire's output on the frozen fixture test/estcalibfix, for the invocations that print est_tokens=. +# +# WHY A MANIFEST AND NOT A GATE THAT TOKENIZES. tiktoken is a Python package. G3 is one deterministic +# build step with nothing host-installed, and every gate in test/ runs on a machine that has bash, +# python3's STDLIB and the binary — nothing else. A gate that imported tiktoken would be a dependency +# the build contract forbids, and one that shelled out to a network-fetching encoder would be worse. +# So the tokenizer runs HERE, by hand, and writes numbers; the gate (test/tokenbudgetcheck.sh #18) +# reads those numbers and needs no package at all. Same shape as test/printf_parity.manifest. +# +# WHY THE PINS DO NOT ROT. The fixture is copied to a temp dir OUTSIDE any repository and crawled by a +# RELATIVE path, so the output carries no `at="+dirty"` stamp (nothing to stamp) and root="f" is +# one byte on every machine. The only thing that can move these numbers is the fixture (frozen) or the +# emitter (which is exactly what the gate is for). +# +# Usage: python3 bench/tokenaudit/pin.py --bin build/ripwire # needs tiktoken in the ambient python + +import argparse +import os +import shutil +import subprocess +import sys +import tempfile + +# The pinned invocations: label, then argv AFTER the root. No argument may contain a space — the +# manifest is space-separated and the gate splits on whitespace, exactly like printf_parity.manifest. +# Only verbs that PRINT est_tokens= belong here; a verb that prints none has nothing to calibrate +# (that absence is itself reported in bench/tokenaudit/README.md, not gated here). +PINNED = [ + ("map", []), + ("map-topk5", ["--top-k=5"]), + ("metrics", ["--metrics"]), + ("pack-signatures", ["--pack-signatures"]), + ("expand", ["--expand=billableTotal"]), + ("for-named", ["--for=billableTotal"]), + ("for-budgeted", ["--for=dedupe", "--token-budget=400"]), + ("pack-task", ["--pack-task=dedupe"]), +] + +EST = b'est_tokens="' + + +def est_of(out: bytes): + i = out.find(EST) + if i < 0: + return None + j = out.find(b'"', i + len(EST)) + return int(out[i + len(EST):j]) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--bin", default="build/ripwire") + ap.add_argument("--fixture", default="test/estcalibfix") + ap.add_argument("--out", default="test/estcalib.manifest") + args = ap.parse_args() + + import tiktoken + o200k = tiktoken.get_encoding("o200k_base") + cl100k = tiktoken.get_encoding("cl100k_base") + + binpath = os.path.abspath(args.bin) + fixture = os.path.abspath(args.fixture) + out_path = os.path.abspath(args.out) + + lines = [ + "# est_calib pins — REAL tokenizer counts of ripwire's output on test/estcalibfix.", + "# label o200k cl100k est_at_pin_time argv... (space-separated; no argv may contain a space)", + "# Regenerate: python3 bench/tokenaudit/pin.py --bin build/ripwire (needs tiktoken)", + "# Read by: test/tokenbudgetcheck.sh #18 — the calibration band. See bench/tokenaudit/README.md.", + ] + tmp = tempfile.mkdtemp() + try: + shutil.copytree(fixture, os.path.join(tmp, "f")) + for label, argv in PINNED: + p = subprocess.run([binpath, "f"] + argv, capture_output=True, cwd=tmp) + if p.returncode != 0: + print("pin.py: '%s' exited %d" % (label, p.returncode), file=sys.stderr) + return 1 + est = est_of(p.stdout) + if est is None: + print("pin.py: '%s' printed no est_tokens — remove it from PINNED" % label, file=sys.stderr) + return 1 + text = p.stdout.decode("utf-8", "replace") + lines.append("%s %d %d %d %s" % (label, len(o200k.encode(text)), + len(cl100k.encode(text)), est, " ".join(argv))) + finally: + shutil.rmtree(tmp, ignore_errors=True) + + with open(out_path, "w") as f: + f.write("\n".join(lines) + "\n") + print("wrote", out_path) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bench/tokenaudit/results/loopledger-2026-09-09.json b/bench/tokenaudit/results/loopledger-2026-09-09.json new file mode 100644 index 000000000..c13499089 --- /dev/null +++ b/bench/tokenaudit/results/loopledger-2026-09-09.json @@ -0,0 +1,33 @@ +{ + "mean_tokens_per_native_call": 5700.5, + "mean_tokens_per_ripwire_call": 406.7, + "median_tokens_per_native_call": 716.7, + "median_tokens_per_ripwire_call": 333.0, + "nonterminal_ripwire_calls": 1666, + "nonterminality_rate": 0.4821, + "provider_usage_tokens": { + "cache_creation_input_tokens": 221210191, + "cache_read_input_tokens": 12846299888, + "input_tokens": 286905, + "output_tokens": 43186745 + }, + "ripwire_calls": 3456, + "ripwire_share_of_retrieval_calls": 0.422, + "ripwire_share_of_retrieval_tokens": 0.0426, + "schema": "ripwire.loopledger/v1", + "sessions": 78, + "sessions_with_a_ripwire_call": 67, + "sweep_lookahead": 3, + "tokenizer": "o200k_base", + "tool_calls": { + "native": 4733, + "other": 8921, + "ripwire": 3456 + }, + "tool_result_share_of_billed_tokens": 0.0025, + "tool_result_tokens": { + "native": 26083509, + "other": 5095267, + "ripwire": 1159938 + } +} diff --git a/bench/tokenaudit/results/tokenaudit-2026-09-09.json b/bench/tokenaudit/results/tokenaudit-2026-09-09.json new file mode 100644 index 000000000..696bf3509 --- /dev/null +++ b/bench/tokenaudit/results/tokenaudit-2026-09-09.json @@ -0,0 +1,907 @@ +{ + "bin": "build/ripwire", + "corpora": { + "private-cpp-tree": { + "file": "math/doctest.h", + "root_label": "a private 1500-file C++ tree", + "rows": [ + { + "argv": [], + "bytes": 29123, + "bytes_per_o200k": 2.7159, + "cl100k": 10570, + "compact_bytes": 27881, + "compact_err_pct": 13.85, + "compact_est_tokens": 11773, + "compact_o200k": 10341, + "err_pct": 9.79, + "est_tokens": 11773, + "legend_bytes": 1242, + "legend_o200k": 382, + "legend_share_pct": 3.56, + "o200k": 10723, + "rc": 0, + "verb": "map" + }, + { + "argv": [ + "--top-k=10" + ], + "bytes": 2936, + "bytes_per_o200k": 2.9478, + "cl100k": 996, + "compact_bytes": 1694, + "compact_err_pct": 93.0, + "compact_est_tokens": 1185, + "compact_o200k": 614, + "err_pct": 18.98, + "est_tokens": 1185, + "legend_bytes": 1242, + "legend_o200k": 382, + "legend_share_pct": 38.35, + "o200k": 996, + "rc": 0, + "verb": "map-topk10" + }, + { + "argv": [ + "--json" + ], + "bytes": 29377, + "bytes_per_o200k": 2.9545, + "cl100k": 9942, + "est_tokens": null, + "o200k": 9943, + "rc": 0, + "verb": "map-json" + }, + { + "argv": [ + "--metrics", + "--top-k=40" + ], + "bytes": 14746, + "bytes_per_o200k": 2.6512, + "cl100k": 5471, + "compact_bytes": 12103, + "compact_err_pct": 24.89, + "compact_est_tokens": 5961, + "compact_o200k": 4773, + "err_pct": 7.17, + "est_tokens": 5961, + "legend_bytes": 2643, + "legend_o200k": 789, + "legend_share_pct": 14.19, + "o200k": 5562, + "rc": 0, + "verb": "metrics" + }, + { + "argv": [ + "--pack-signatures", + "--top-k=40" + ], + "bytes": 17065, + "bytes_per_o200k": 2.9478, + "cl100k": 5703, + "compact_bytes": 15790, + "compact_err_pct": 27.27, + "compact_est_tokens": 6860, + "compact_o200k": 5390, + "err_pct": 18.5, + "est_tokens": 6860, + "legend_bytes": 1275, + "legend_o200k": 399, + "legend_share_pct": 6.89, + "o200k": 5789, + "rc": 0, + "verb": "pack-signatures" + }, + { + "argv": [ + "--for=how are tokens counted and budgeted" + ], + "bytes": 9021, + "bytes_per_o200k": 3.1366, + "cl100k": 2904, + "compact_bytes": 8963, + "compact_err_pct": 22.35, + "compact_est_tokens": 3585, + "compact_o200k": 2930, + "err_pct": 25.45, + "est_tokens": 3608, + "legend_bytes": 58, + "legend_o200k": -54, + "legend_share_pct": -1.88, + "o200k": 2876, + "rc": 0, + "verb": "for-conceptual" + }, + { + "argv": [ + "--for=TEST_CASE" + ], + "bytes": 3318, + "bytes_per_o200k": 3.5186, + "cl100k": 935, + "compact_bytes": 2613, + "compact_err_pct": 26.36, + "compact_est_tokens": 997, + "compact_o200k": 789, + "err_pct": 35.63, + "est_tokens": 1279, + "legend_bytes": 705, + "legend_o200k": 154, + "legend_share_pct": 16.33, + "o200k": 943, + "rc": 0, + "verb": "for-named" + }, + { + "argv": [ + "--for=how are tokens counted and budgeted", + "--json" + ], + "bytes": 8274, + "bytes_per_o200k": 3.1, + "cl100k": 2747, + "est_tokens": null, + "o200k": 2669, + "rc": 0, + "verb": "for-json" + }, + { + "argv": [ + "--pack-task=fix the token estimate" + ], + "bytes": 11449, + "bytes_per_o200k": 3.3793, + "cl100k": 3423, + "compact_bytes": 10092, + "compact_err_pct": 38.76, + "compact_est_tokens": 4221, + "compact_o200k": 3042, + "err_pct": 24.59, + "est_tokens": 4221, + "legend_bytes": 1357, + "legend_o200k": 346, + "legend_share_pct": 10.21, + "o200k": 3388, + "rc": 0, + "verb": "pack-task" + }, + { + "argv": [ + "--expand=TEST_CASE" + ], + "bytes": 1695, + "bytes_per_o200k": 3.4173, + "cl100k": 496, + "compact_bytes": 1229, + "compact_err_pct": 2.27, + "compact_est_tokens": 405, + "compact_o200k": 396, + "err_pct": -18.35, + "est_tokens": 405, + "legend_bytes": 466, + "legend_o200k": 100, + "legend_share_pct": 20.16, + "o200k": 496, + "rc": 0, + "verb": "expand" + }, + { + "argv": [ + "--callers=TEST_CASE" + ], + "bytes": 8264, + "bytes_per_o200k": 3.7292, + "cl100k": 2182, + "compact_bytes": 4956, + "compact_est_tokens": null, + "compact_o200k": 1478, + "est_tokens": null, + "legend_bytes": 3308, + "legend_o200k": 738, + "legend_share_pct": 33.3, + "o200k": 2216, + "rc": 0, + "verb": "callers" + }, + { + "argv": [ + "--callees=TEST_CASE" + ], + "bytes": 3838, + "bytes_per_o200k": 4.1627, + "cl100k": 916, + "compact_bytes": 659, + "compact_est_tokens": null, + "compact_o200k": 209, + "est_tokens": null, + "legend_bytes": 3179, + "legend_o200k": 713, + "legend_share_pct": 77.33, + "o200k": 922, + "rc": 0, + "verb": "callees" + }, + { + "argv": [ + "--impact=TEST_CASE" + ], + "bytes": 10984, + "bytes_per_o200k": 3.5512, + "cl100k": 3063, + "compact_bytes": 7594, + "compact_est_tokens": null, + "compact_o200k": 2321, + "est_tokens": null, + "legend_bytes": 3390, + "legend_o200k": 772, + "legend_share_pct": 24.96, + "o200k": 3093, + "rc": 0, + "verb": "impact" + }, + { + "argv": [ + "--uses=TEST_CASE" + ], + "bytes": 10235, + "bytes_per_o200k": 3.5464, + "cl100k": 2785, + "compact_bytes": 6246, + "compact_est_tokens": null, + "compact_o200k": 1994, + "est_tokens": null, + "legend_bytes": 3989, + "legend_o200k": 892, + "legend_share_pct": 30.91, + "o200k": 2886, + "rc": 0, + "verb": "uses" + }, + { + "argv": [ + "--around=TEST_CASE" + ], + "bytes": 9035, + "bytes_per_o200k": 2.819, + "cl100k": 3047, + "compact_bytes": 7605, + "compact_err_pct": 30.95, + "compact_est_tokens": 3643, + "compact_o200k": 2782, + "err_pct": 13.67, + "est_tokens": 3643, + "legend_bytes": 1430, + "legend_o200k": 423, + "legend_share_pct": 13.2, + "o200k": 3205, + "rc": 0, + "verb": "around" + }, + { + "argv": [ + "--edit-check=TEST_CASE" + ], + "bytes": 348224, + "bytes_per_o200k": 3.5172, + "cl100k": 97575, + "compact_bytes": 342339, + "compact_est_tokens": null, + "compact_o200k": 97709, + "est_tokens": null, + "legend_bytes": 5885, + "legend_o200k": 1297, + "legend_share_pct": 1.31, + "o200k": 99006, + "rc": 0, + "verb": "edit-check" + }, + { + "argv": [ + "--grep=token" + ], + "bytes": 32149, + "bytes_per_o200k": 3.5809, + "cl100k": 8890, + "compact_bytes": 25881, + "compact_est_tokens": null, + "compact_o200k": 7564, + "est_tokens": null, + "legend_bytes": 6268, + "legend_o200k": 1414, + "legend_share_pct": 15.75, + "o200k": 8978, + "rc": 0, + "verb": "grep" + }, + { + "argv": [ + "--hotspots" + ], + "bytes": 6582, + "bytes_per_o200k": 2.9555, + "cl100k": 2222, + "compact_bytes": 5465, + "compact_est_tokens": null, + "compact_o200k": 1987, + "est_tokens": null, + "legend_bytes": 1117, + "legend_o200k": 240, + "legend_share_pct": 10.78, + "o200k": 2227, + "rc": 0, + "verb": "hotspots" + }, + { + "argv": [ + "--lint" + ], + "bytes": 56589, + "bytes_per_o200k": 3.0643, + "cl100k": 18607, + "compact_bytes": 53998, + "compact_est_tokens": null, + "compact_o200k": 17863, + "est_tokens": null, + "legend_bytes": 2591, + "legend_o200k": 604, + "legend_share_pct": 3.27, + "o200k": 18467, + "rc": 0, + "verb": "lint" + }, + { + "argv": [ + "--tree" + ], + "bytes": 13265, + "bytes_per_o200k": 2.9478, + "cl100k": 4369, + "compact_bytes": 12543, + "compact_est_tokens": null, + "compact_o200k": 4344, + "est_tokens": null, + "legend_bytes": 722, + "legend_o200k": 156, + "legend_share_pct": 3.47, + "o200k": 4500, + "rc": 0, + "verb": "tree" + }, + { + "argv": [ + "--clones" + ], + "bytes": 19145, + "bytes_per_o200k": 3.1432, + "cl100k": 6093, + "compact_bytes": 16115, + "compact_est_tokens": null, + "compact_o200k": 5424, + "est_tokens": null, + "legend_bytes": 3030, + "legend_o200k": 667, + "legend_share_pct": 10.95, + "o200k": 6091, + "rc": 0, + "verb": "clones" + }, + { + "argv": [ + "--situ" + ], + "bytes": 1826, + "bytes_per_o200k": 3.7572, + "cl100k": 486, + "est_tokens": null, + "o200k": 486, + "rc": 0, + "verb": "situ" + }, + { + "argv": [ + "--recall=how are tokens counted" + ], + "bytes": 11912, + "bytes_per_o200k": 3.6185, + "cl100k": 3328, + "est_tokens": null, + "o200k": 3292, + "rc": 0, + "verb": "recall" + }, + { + "argv": [ + "--test-gate" + ], + "bytes": 3232, + "bytes_per_o200k": 3.843, + "cl100k": 833, + "compact_refused": "", + "est_tokens": null, + "o200k": 841, + "rc": 4, + "verb": "test-gate" + }, + { + "argv": [ + "--affected=math/doctest.h" + ], + "bytes": 17006, + "bytes_per_o200k": 3.2849, + "cl100k": 5081, + "compact_bytes": 15537, + "compact_est_tokens": null, + "compact_o200k": 4844, + "est_tokens": null, + "legend_bytes": 1469, + "legend_o200k": 333, + "legend_share_pct": 6.43, + "o200k": 5177, + "rc": 0, + "verb": "affected" + } + ], + "symbol": "TEST_CASE" + }, + "self": { + "file": "src/infra/svector.h", + "root_label": "this repository", + "rows": [ + { + "argv": [], + "bytes": 24756, + "bytes_per_o200k": 2.5388, + "cl100k": 9561, + "compact_bytes": 23762, + "compact_err_pct": 6.15, + "compact_est_tokens": 10022, + "compact_o200k": 9441, + "err_pct": 2.78, + "est_tokens": 10022, + "legend_bytes": 994, + "legend_o200k": 310, + "legend_share_pct": 3.18, + "o200k": 9751, + "rc": 0, + "verb": "map" + }, + { + "argv": [ + "--top-k=10" + ], + "bytes": 2860, + "bytes_per_o200k": 2.8686, + "cl100k": 985, + "compact_bytes": 1866, + "compact_err_pct": 68.12, + "compact_est_tokens": 1155, + "compact_o200k": 687, + "err_pct": 15.85, + "est_tokens": 1155, + "legend_bytes": 994, + "legend_o200k": 310, + "legend_share_pct": 31.09, + "o200k": 997, + "rc": 0, + "verb": "map-topk10" + }, + { + "argv": [ + "--json" + ], + "bytes": 25352, + "bytes_per_o200k": 2.7521, + "cl100k": 9091, + "est_tokens": null, + "o200k": 9212, + "rc": 0, + "verb": "map-json" + }, + { + "argv": [ + "--metrics", + "--top-k=40" + ], + "bytes": 12016, + "bytes_per_o200k": 2.5824, + "cl100k": 4623, + "compact_bytes": 9621, + "compact_err_pct": 23.45, + "compact_est_tokens": 4859, + "compact_o200k": 3936, + "err_pct": 4.43, + "est_tokens": 4859, + "legend_bytes": 2395, + "legend_o200k": 717, + "legend_share_pct": 15.41, + "o200k": 4653, + "rc": 0, + "verb": "metrics" + }, + { + "argv": [ + "--pack-signatures", + "--top-k=40" + ], + "bytes": 15710, + "bytes_per_o200k": 2.913, + "cl100k": 5353, + "compact_bytes": 14683, + "compact_err_pct": 24.5, + "compact_est_tokens": 6307, + "compact_o200k": 5066, + "err_pct": 16.95, + "est_tokens": 6307, + "legend_bytes": 1027, + "legend_o200k": 327, + "legend_share_pct": 6.06, + "o200k": 5393, + "rc": 0, + "verb": "pack-signatures" + }, + { + "argv": [ + "--for=how are tokens counted and budgeted" + ], + "bytes": 9725, + "bytes_per_o200k": 3.0486, + "cl100k": 3160, + "compact_bytes": 9562, + "compact_err_pct": 18.94, + "compact_est_tokens": 3825, + "compact_o200k": 3216, + "err_pct": 21.94, + "est_tokens": 3890, + "legend_bytes": 163, + "legend_o200k": -26, + "legend_share_pct": -0.82, + "o200k": 3190, + "rc": 0, + "verb": "for-conceptual" + }, + { + "argv": [ + "--for=buf" + ], + "bytes": 3822, + "bytes_per_o200k": 3.2472, + "cl100k": 1173, + "compact_bytes": 3124, + "compact_err_pct": 17.56, + "compact_est_tokens": 1205, + "compact_o200k": 1025, + "err_pct": 26.08, + "est_tokens": 1484, + "legend_bytes": 698, + "legend_o200k": 152, + "legend_share_pct": 12.91, + "o200k": 1177, + "rc": 0, + "verb": "for-named" + }, + { + "argv": [ + "--for=how are tokens counted and budgeted", + "--json" + ], + "bytes": 7774, + "bytes_per_o200k": 3.051, + "cl100k": 2560, + "est_tokens": null, + "o200k": 2548, + "rc": 0, + "verb": "for-json" + }, + { + "argv": [ + "--pack-task=fix the token estimate" + ], + "bytes": 11671, + "bytes_per_o200k": 3.3673, + "cl100k": 3456, + "compact_bytes": 10410, + "compact_err_pct": 31.04, + "compact_est_tokens": 4120, + "compact_o200k": 3144, + "err_pct": 18.87, + "est_tokens": 4120, + "legend_bytes": 1261, + "legend_o200k": 322, + "legend_share_pct": 9.29, + "o200k": 3466, + "rc": 0, + "verb": "pack-task" + }, + { + "argv": [ + "--expand=buf" + ], + "bytes": 26539, + "bytes_per_o200k": 2.5814, + "cl100k": 10091, + "compact_bytes": 24634, + "compact_err_pct": 7.54, + "compact_est_tokens": 10474, + "compact_o200k": 9740, + "err_pct": 1.88, + "est_tokens": 10474, + "legend_bytes": 1905, + "legend_o200k": 541, + "legend_share_pct": 5.26, + "o200k": 10281, + "rc": 0, + "verb": "expand" + }, + { + "argv": [ + "--callers=buf" + ], + "bytes": 5373, + "bytes_per_o200k": 3.4487, + "cl100k": 1521, + "compact_bytes": 2396, + "compact_est_tokens": null, + "compact_o200k": 889, + "est_tokens": null, + "legend_bytes": 2977, + "legend_o200k": 669, + "legend_share_pct": 42.94, + "o200k": 1558, + "rc": 0, + "verb": "callers" + }, + { + "argv": [ + "--callees=buf" + ], + "bytes": 3750, + "bytes_per_o200k": 4.1993, + "cl100k": 887, + "compact_bytes": 571, + "compact_est_tokens": null, + "compact_o200k": 180, + "est_tokens": null, + "legend_bytes": 3179, + "legend_o200k": 713, + "legend_share_pct": 79.84, + "o200k": 893, + "rc": 0, + "verb": "callees" + }, + { + "argv": [ + "--impact=buf" + ], + "bytes": 6674, + "bytes_per_o200k": 3.3023, + "cl100k": 1969, + "compact_bytes": 3284, + "compact_est_tokens": null, + "compact_o200k": 1249, + "est_tokens": null, + "legend_bytes": 3390, + "legend_o200k": 772, + "legend_share_pct": 38.2, + "o200k": 2021, + "rc": 0, + "verb": "impact" + }, + { + "argv": [ + "--uses=buf" + ], + "bytes": 7917, + "bytes_per_o200k": 3.3964, + "cl100k": 2285, + "compact_bytes": 4259, + "compact_est_tokens": null, + "compact_o200k": 1508, + "est_tokens": null, + "legend_bytes": 3658, + "legend_o200k": 823, + "legend_share_pct": 35.31, + "o200k": 2331, + "rc": 0, + "verb": "uses" + }, + { + "argv": [ + "--around=buf" + ], + "bytes": 6033, + "bytes_per_o200k": 2.8404, + "cl100k": 2078, + "compact_bytes": 4851, + "compact_err_pct": 37.39, + "compact_est_tokens": 2436, + "compact_o200k": 1773, + "err_pct": 14.69, + "est_tokens": 2436, + "legend_bytes": 1182, + "legend_o200k": 351, + "legend_share_pct": 16.53, + "o200k": 2124, + "rc": 0, + "verb": "around" + }, + { + "argv": [ + "--edit-check=buf" + ], + "bytes": 7986, + "bytes_per_o200k": 3.8748, + "cl100k": 2013, + "compact_bytes": 2101, + "compact_est_tokens": null, + "compact_o200k": 764, + "est_tokens": null, + "legend_bytes": 5885, + "legend_o200k": 1297, + "legend_share_pct": 62.93, + "o200k": 2061, + "rc": 0, + "verb": "edit-check" + }, + { + "argv": [ + "--grep=token" + ], + "bytes": 68695, + "bytes_per_o200k": 3.1402, + "cl100k": 21522, + "compact_bytes": 62482, + "compact_est_tokens": null, + "compact_o200k": 20477, + "est_tokens": null, + "legend_bytes": 6213, + "legend_o200k": 1399, + "legend_share_pct": 6.4, + "o200k": 21876, + "rc": 0, + "verb": "grep" + }, + { + "argv": [ + "--hotspots" + ], + "bytes": 6002, + "bytes_per_o200k": 2.8311, + "cl100k": 2077, + "compact_bytes": 4885, + "compact_est_tokens": null, + "compact_o200k": 1880, + "est_tokens": null, + "legend_bytes": 1117, + "legend_o200k": 240, + "legend_share_pct": 11.32, + "o200k": 2120, + "rc": 0, + "verb": "hotspots" + }, + { + "argv": [ + "--lint" + ], + "bytes": 56147, + "bytes_per_o200k": 2.9386, + "cl100k": 18957, + "compact_bytes": 53556, + "compact_est_tokens": null, + "compact_o200k": 18503, + "est_tokens": null, + "legend_bytes": 2591, + "legend_o200k": 604, + "legend_share_pct": 3.16, + "o200k": 19107, + "rc": 0, + "verb": "lint" + }, + { + "argv": [ + "--tree" + ], + "bytes": 11755, + "bytes_per_o200k": 2.7574, + "cl100k": 4092, + "compact_bytes": 11033, + "compact_est_tokens": null, + "compact_o200k": 4107, + "est_tokens": null, + "legend_bytes": 722, + "legend_o200k": 156, + "legend_share_pct": 3.66, + "o200k": 4263, + "rc": 0, + "verb": "tree" + }, + { + "argv": [ + "--clones" + ], + "bytes": 18652, + "bytes_per_o200k": 2.9819, + "cl100k": 6128, + "compact_bytes": 15622, + "compact_est_tokens": null, + "compact_o200k": 5588, + "est_tokens": null, + "legend_bytes": 3030, + "legend_o200k": 667, + "legend_share_pct": 10.66, + "o200k": 6255, + "rc": 0, + "verb": "clones" + }, + { + "argv": [ + "--situ" + ], + "bytes": 210, + "bytes_per_o200k": 3.3333, + "cl100k": 64, + "est_tokens": null, + "o200k": 63, + "rc": 0, + "verb": "situ" + }, + { + "argv": [ + "--recall=how are tokens counted" + ], + "bytes": 15246, + "bytes_per_o200k": 3.7672, + "cl100k": 4050, + "est_tokens": null, + "o200k": 4047, + "rc": 0, + "verb": "recall" + }, + { + "argv": [ + "--test-gate" + ], + "bytes": 2078, + "bytes_per_o200k": 3.9134, + "cl100k": 526, + "compact_bytes": 805, + "compact_est_tokens": null, + "compact_o200k": 253, + "est_tokens": null, + "legend_bytes": 1273, + "legend_o200k": 278, + "legend_share_pct": 52.35, + "o200k": 531, + "rc": 0, + "verb": "test-gate" + }, + { + "argv": [ + "--affected=src/infra/svector.h" + ], + "bytes": 4267, + "bytes_per_o200k": 3.6039, + "cl100k": 1156, + "compact_bytes": 2798, + "compact_est_tokens": null, + "compact_o200k": 851, + "est_tokens": null, + "legend_bytes": 1469, + "legend_o200k": 333, + "legend_share_pct": 28.12, + "o200k": 1184, + "rc": 0, + "verb": "affected" + } + ], + "symbol": "buf" + } + }, + "redaction": "corpus roots and the binary path are recorded as LABELS, never as absolute paths, and the second corpus is named by its shape rather than its name: it is a private tree and this file is public. bench/tokenaudit/sweep.py writes it this way; nothing else was altered, every count below is the measurement as taken.", + "schema": "ripwire.tokenaudit/v1", + "version": "ripwire 0.5.0 (dev, AppleClang 21.0.0.21000101, emit=std::print, built_from=4c10be9d7)" +} diff --git a/bench/tokenaudit/sweep.py b/bench/tokenaudit/sweep.py new file mode 100644 index 000000000..287b8fdff --- /dev/null +++ b/bench/tokenaudit/sweep.py @@ -0,0 +1,203 @@ +#!/usr/bin/env python3 +# sweep.py — does ripwire's est_tokens= agree with a real tokenizer, per verb and per corpus? +# +# WHY THIS EXISTS. est_tokens is the number every --token-budget gates on and the number the owner +# named as the user-facing unit, and until this script it was validated only for its PROPERTIES +# (deterministic, present, monotone under a tighter budget — test/tokenbudgetcheck.sh) and by a +# MAPE the T1 author reported by hand in a write-up. Nothing in the tree re-derived the error against +# a tokenizer, and nothing measured it PER VERB. A single-corpus MAPE cannot see a per-verb bias: +# the divisor is a per-language content-byte model, and a --callers answer is ~90% markup while a +# --expand answer is ~70% body, so one rate serving both is a hypothesis, not a measurement. +# +# It also prices THE LEGEND. A third-party evaluation (callstack/agent-device #2400, 2026-09-08) +# measured ripwire spending ~10% MORE tokens than grep-and-read and traced it to the fixed per-call +# preamble. test/legendcostcheck.sh holds --help's BYTE claim to what the binary delivers; this +# script measures the same thing in the unit the claim is read in — real tokenizer tokens — by +# running each invocation twice, --legend=full and --legend=compact, on the same corpus. +# +# WHAT IS MEASURED, PER (corpus, invocation): +# bytes len(stdout) in bytes +# est_tokens the number ripwire printed, or null if the verb prints none <-- a finding in itself +# o200k / cl100k real token counts (tiktoken), the ground truth this file compares against +# err_pct 100 * (est - o200k) / o200k signed: negative = ripwire UNDER-reports its price +# legend_* the same numbers for --legend=compact, and the token delta = the legend's price +# +# WHAT IT IS NOT. Not a gate. tiktoken is not a build dependency and never will be (G3: one +# deterministic build step, nothing host-installed), and o200k_base is not Claude's tokenizer — +# it is the closest public stand-in, which is exactly why kTokenCalib was calibrated against it and +# why a gate that needs it at runtime would be a dependency problem. The gate arm this round adds +# (test/tokenbudgetcheck.sh #18 — absorbed into the gate that already owns est_tokens rather than added +# beside it) reads PINNED counts out of test/estcalib.manifest, which bench/tokenaudit/pin.py writes; +# regenerating that manifest is a deliberate manual step, never a build step. +# +# Usage: +# python3 bench/tokenaudit/sweep.py --bin build/ripwire --out bench/tokenaudit/results/x.json \ +# [--corpus NAME=PATH ...] [--python PYTHON_WITH_TIKTOKEN] +# +# Determinism: the invocation table below is fixed and ordered; symbol arguments are DERIVED from +# each corpus's own map (highest-ranked symbol with callers) so the table transfers to a corpus this +# file has never seen, and the derived symbol is recorded in the results so a rerun is auditable. + +import argparse +import json +import os +import re +import subprocess +import sys + +EST_RE = re.compile(rb'est_tokens="(\d+)"') +SYM_RE = re.compile(rb']*\bn="([^"]+)"') +FILE_RE = re.compile(rb' row of the map (rank 1) and the first file. Deterministic + because the map is; recorded in the output because a number is unauditable without its input. + """ + out, _, _ = run(binpath, root, ["--top-k=40"]) + syms = SYM_RE.findall(out) + files = FILE_RE.findall(out) + sym = syms[0].decode() if syms else "main" + fil = files[0].decode() if files else "" + # Prefer a symbol that actually has callers AND is unambiguous, else --callers/--impact answer + # nothing and --edit-check refuses — the legend share would then be measured on an empty or + # refused document (true, but not the case anyone runs). The FIRST candidate that satisfies both + # wins, so the choice stays deterministic and is recorded in the results. + for cand in syms[:20]: + c = cand.decode() + co, _, rc = run(binpath, root, ["--callers=" + c, "--legend=compact"]) + if rc != 0 or co.count(b" 0.0 ); diff --git a/test/estcalib.manifest b/test/estcalib.manifest new file mode 100644 index 000000000..f877c4333 --- /dev/null +++ b/test/estcalib.manifest @@ -0,0 +1,12 @@ +# est_calib pins — REAL tokenizer counts of ripwire's output on test/estcalibfix. +# label o200k cl100k est_at_pin_time argv... (space-separated; no argv may contain a space) +# Regenerate: python3 bench/tokenaudit/pin.py --bin build/ripwire (needs tiktoken) +# Read by: test/tokenbudgetcheck.sh #18 — the calibration band. See bench/tokenaudit/README.md. +map 1009 999 1123 +map-topk5 538 535 653 --top-k=5 +metrics 2381 2370 2509 --metrics +pack-signatures 1801 1784 2116 --pack-signatures +expand 543 544 454 --expand=billableTotal +for-named 789 785 1050 --for=billableTotal +for-budgeted 374 370 525 --for=dedupe --token-budget=400 +pack-task 882 862 1104 --pack-task=dedupe diff --git a/test/estcalibfix/ledger.cpp b/test/estcalibfix/ledger.cpp new file mode 100644 index 000000000..842a87e0b --- /dev/null +++ b/test/estcalibfix/ledger.cpp @@ -0,0 +1,86 @@ +// estcalibfix — a FROZEN corpus for the est_tokens calibration band (tokenbudgetcheck #18). +// Do not edit: test/estcalib.manifest pins real o200k_base token counts of ripwire's output on +// THIS tree, and any edit here invalidates every pin. Regenerate with bench/tokenaudit/pin.py. +#include +#include +#include + +namespace ledger +{ + +struct Entry +{ + std::string name; + std::size_t inputTokens; + std::size_t outputTokens; + std::size_t cacheReadTokens; +}; + +// Sum one class across the ledger. A zero here means "none found in these rows", never "none exists". +inline std::size_t sumInput( const std::vector& rows ) +{ + std::size_t total = 0; + for( const Entry& e : rows ) + { + total += e.inputTokens; + } + return total; +} + +inline std::size_t sumOutput( const std::vector& rows ) +{ + std::size_t total = 0; + for( const Entry& e : rows ) + { + total += e.outputTokens; + } + return total; +} + +// Cache reads are charged at a different rate than fresh input, so they are never folded into +// sumInput; a caller that wants one number asks for it and says which classes it merged. +inline std::size_t sumCacheRead( const std::vector& rows ) +{ + std::size_t total = 0; + for( const Entry& e : rows ) + { + total += e.cacheReadTokens; + } + return total; +} + +inline std::size_t billableTotal( const std::vector& rows ) +{ + return sumInput( rows ) + sumOutput( rows ) + sumCacheRead( rows ); +} + +inline bool isEmpty( const std::vector& rows ) +{ + return rows.empty(); +} + +class Report +{ +public: + explicit Report( std::vector rows ) : rows_( std::move( rows ) ) {} + + std::size_t total() const + { + return billableTotal( rows_ ); + } + + std::size_t rowCount() const + { + return rows_.size(); + } + + bool degenerate() const + { + return isEmpty( rows_ ) || total() == 0; + } + +private: + std::vector rows_; +}; + +} // namespace ledger diff --git a/test/estcalibfix/rates.py b/test/estcalibfix/rates.py new file mode 100644 index 000000000..cf793aec1 --- /dev/null +++ b/test/estcalibfix/rates.py @@ -0,0 +1,40 @@ +"""estcalibfix — FROZEN. See ledger.cpp's header; edits invalidate test/estcalib.manifest.""" + +DEFAULT_RATE = 2.5 +BODY_RATE = 3.8 + + +def bytes_per_token(language): + """Return the measured bytes-per-token rate for a language, or the mid-band default.""" + table = {"cpp": 2.46, "python": 2.36, "typescript": 2.59} + return table.get(language, DEFAULT_RATE) + + +def estimate_tokens(byte_count, language): + """Convert emitted bytes to a token estimate at the language's own rate.""" + rate = bytes_per_token(language) + if rate <= 0: + raise ValueError("a non-positive rate is a corrupt caller, never a runtime condition") + return int(byte_count / rate + 0.5) + + +def estimate_body_tokens(byte_count): + """Body text tokenizes leaner than signature markup; charge it at its own rate.""" + return int(byte_count / BODY_RATE + 0.5) + + +def signed_error_pct(estimate, real): + """Positive means the estimate over-reads the real count.""" + if real == 0: + return None + return 100.0 * (estimate - real) / real + + +def summarize(rows): + """rows: [(estimate, real)]. Returns (n, worst_over, worst_under) with None for an empty set.""" + if not rows: + return (0, None, None) + errs = [signed_error_pct(e, r) for e, r in rows if signed_error_pct(e, r) is not None] + if not errs: + return (len(rows), None, None) + return (len(rows), max(errs), min(errs)) diff --git a/test/estcalibfix/session.ts b/test/estcalibfix/session.ts new file mode 100644 index 000000000..a3187931a --- /dev/null +++ b/test/estcalibfix/session.ts @@ -0,0 +1,46 @@ +// estcalibfix — FROZEN. See ledger.cpp's header; edits invalidate test/estcalib.manifest. + +export interface UsageRecord { + sessionId: string; + model: string; + inputTokens: number; + outputTokens: number; + cacheReadTokens: number; + cacheWriteTokens: number; +} + +export function totalForRecord(record: UsageRecord): number { + return ( + record.inputTokens + + record.outputTokens + + record.cacheReadTokens + + record.cacheWriteTokens + ); +} + +export function groupBySession(records: UsageRecord[]): Map { + const out = new Map(); + for (const record of records) { + const prior = out.get(record.sessionId) ?? 0; + out.set(record.sessionId, prior + totalForRecord(record)); + } + return out; +} + +export function dedupe(records: UsageRecord[], keyOf: (r: UsageRecord) => string): UsageRecord[] { + const seen = new Set(); + const kept: UsageRecord[] = []; + for (const record of records) { + const key = keyOf(record); + if (seen.has(key)) { + continue; + } + seen.add(key); + kept.push(record); + } + return kept; +} + +export function isMeasured(record: UsageRecord): boolean { + return record.inputTokens > 0 || record.outputTokens > 0; +} diff --git a/test/tokenbudgetcheck.sh b/test/tokenbudgetcheck.sh index cd5ebd408..e3e0e5b44 100755 --- a/test/tokenbudgetcheck.sh +++ b/test/tokenbudgetcheck.sh @@ -318,5 +318,86 @@ else done fi +# ── #18: THE CALIBRATION BAND — est_tokens against a REAL tokenizer, pinned ──────────────────────────── +# THE GAP THIS CLOSES. Everything above measures the estimate's PROPERTIES: present, positive, +# deterministic, monotone under a tighter budget, and bounded by an allowance derived from the estimate's +# OWN constants. Not one arm asks whether the number is TRUE. This file's own header said so — "the +# MAPE-vs-tiktoken number is REPORTED by the agent in the T1 write-up (tiktoken isn't a build +# dependency)" — which makes the accuracy of the tool's most-quoted number a thing a human typed into a +# document once, in 2026-07, and nothing has re-derived since. METHODOLOGY §9 principle 6: measuring gets +# its own instrument. This is it. +# +# HOW IT AVOIDS THE DEPENDENCY. The tokenizer runs OUT OF BAND (bench/tokenaudit/pin.py, by hand, with +# tiktoken) and writes real o200k_base/cl100k_base counts into test/estcalib.manifest. This arm reads +# numbers. No package, no network, bash + python3's stdlib + the binary — the same contract as +# test/printf_parity.manifest. +# +# WHY THE PINS DO NOT ROT, and why the fixture is copied. test/estcalibfix is frozen, and the copy is +# crawled from a temp dir by a RELATIVE path: outside any repository there is no `at="+dirty"` stamp +# to change under a commit (or under a concurrent gate — the same shared-resource class this file's header +# records at #14), and root="f" is one byte on every machine, so a deep checkout path cannot move the +# count. Both artifacts that CAN move these numbers are the ones under test: the fixture and the emitter. +# +# THE BAND IS AN ENVELOPE, NOT A TARGET, and it is deliberately signed-asymmetric-free. Measured on three +# corpora (this fixture, this repository, a 1500-file private C++ tree; bench/tokenaudit/README.md) the +# signed error runs -18.4% to +41.7%: est_tokens OVER-reads the small legend-heavy bundles (the legend is +# English prose at ~4.4 B/tok charged at a ~2.5 B/tok signature rate) and UNDER-reads --expand's dense +# small bodies. So the band is +-, not a one-sided "never under-reads" floor: that claim was in +# src/serialize.h and this instrument is what disproved it. What the band DOES buy is that the error +# cannot silently double — a rate table edit, a new emitter charging bytes at the wrong rate, or a legend +# that grows into a document whose price is quoted at the markup rate all leave the band. +EST_MAN="$ROOT/test/estcalib.manifest" +EST_FIX="$ROOT/test/estcalibfix" +if [ ! -r "$EST_MAN" ] || [ ! -d "$EST_FIX" ]; then + no "#18: test/estcalib.manifest or test/estcalibfix missing — the calibration band cannot be measured" +else + EST_TMP="$( mktemp -d )" + cp -R "$EST_FIX" "$EST_TMP/f" + est_pins=0 + est_abs_sum=0 + while read -r label o200k cl100k pinned_est rest; do + case "$label" in ""|\#*) continue ;; esac + # shellcheck disable=SC2086 # $rest is the pinned argv: deliberately word-split, never quoted + EST_OUT="$( cd "$EST_TMP" && "$BIN" f $rest 2>/dev/null )" + EST_GOT="$( printf '%s' "$EST_OUT" | grep -oE 'est_tokens="[0-9]+"' | head -1 | grep -oE '[0-9]+' )" + if [ -z "$EST_GOT" ]; then + no "#18 $label: no est_tokens in the output — the pin says this verb prices itself" + continue + fi + est_pins=$(( est_pins + 1 )) + EST_ERR="$( python3 -c "print( round( 100.0 * ( $EST_GOT - $o200k ) / $o200k, 2 ) )" )" + EST_ABS="$( python3 -c "print( int( round( abs( 100.0 * ( $EST_GOT - $o200k ) / $o200k ) ) ) )" )" + est_abs_sum=$(( est_abs_sum + EST_ABS )) + EST_LO="$( python3 -c "print( int( $o200k * 0.75 ) )" )" + EST_HI="$( python3 -c "print( int( $o200k * 1.55 ) )" )" + if [ "$EST_GOT" -ge "$EST_LO" ] && [ "$EST_GOT" -le "$EST_HI" ]; then + ok "#18 $label: est=$EST_GOT vs o200k=$o200k (cl100k=$cl100k), ${EST_ERR}% — inside [$EST_LO,$EST_HI]" + else + no "#18 $label: est=$EST_GOT vs o200k=$o200k, ${EST_ERR}% — OUTSIDE the measured band [$EST_LO,$EST_HI]" + fi + done < "$EST_MAN" + rm -rf "$EST_TMP" + + # #18b AGGREGATE. A per-pin band tolerates one verb drifting to the edge; the MAPE is the number the + # T1 write-up quoted by hand and is what moves when the RATE TABLE is wrong rather than one emitter. + # 30 is the measured 21% plus headroom — tighten it when a calibration round earns the tightening. + if [ "$est_pins" -gt 0 ]; then + EST_MAPE=$(( est_abs_sum / est_pins )) + if [ "$EST_MAPE" -le 30 ]; then + ok "#18b est_tokens MAPE vs o200k = ${EST_MAPE}% over $est_pins pins (ceiling 30%)" + else + no "#18b est_tokens MAPE vs o200k = ${EST_MAPE}% over $est_pins pins — past the 30% ceiling" + fi + fi + + # #18c MUTATION CONTROL. Eight pins are committed; a loop that measured fewer (a manifest truncated by + # a merge, a verb that stopped printing est_tokens) asserted less than the PASS lines above suggest. + if [ "$est_pins" -ge 8 ]; then + ok "#18c $est_pins pins measured against the tokenizer manifest" + else + no "#18c only $est_pins pin(s) measured — the band arms above asserted almost nothing" + fi +fi + [ "$fail" = 0 ] && echo "ALL PASS" || echo "FAILURES ABOVE" exit $fail