feat(dashboard): estimate token count and cost from an uploaded file - #81
Open
FrancescoGnan wants to merge 8 commits into
Open
feat(dashboard): estimate token count and cost from an uploaded file#81FrancescoGnan wants to merge 8 commits into
FrancescoGnan wants to merge 8 commits into
Conversation
Covers the empty and whitespace cases, the ~4 chars/token rule of thumb on prose, the denser ratio on symbol-heavy text, CJK weighting, range and integer invariants, and monotonic growth. Red: src/lib/estimate.ts does not exist yet. Adds vitest as the dashboard's test runner, matching the TypeScript SDK. Refs Atena-IT#80
Pure function returning a central estimate, an uncertainty range and the method used. Adapts the characters-per-token ratio to the composition of the text: CJK characters count as roughly one token each, and symbol-dense text such as JSON or source code fragments into shorter tokens than prose. Deliberately an estimate and not a count: the libraries in this repository stay out of token counting because tokenizers differ across providers, and the same reasoning is why this reports a range rather than a single figure. Refs Atena-IT#80
Plain-text formats are read directly. PDF goes through pdfjs-dist, loaded with a dynamic import so the parser lands in its own chunk: the main bundle grows by 5 kB rather than by the 2.6 MB the parser and its worker weigh. Unsupported extensions fail with a message naming what can be read. Refs Atena-IT#80
The ratios and the uncertainty band were initially chosen by judgement. Measured against GPT-4o, GPT-4, Llama 3, Gemma and Claude over a nine-document corpus of English, Italian and Japanese prose plus markdown, Python, TSX and JSON taken from this repository, the worst-case error was 22.8%. Fitted constants bring it to 15.3%, median 10.1%. The band widens from 20% to 25% for the same reason: of the 45 reference counts, 37 fell inside 20% and 41 fall inside 25%. The measurement also retired the folkloric four-characters-per-token rule the test encoded, since the tokenizers measured between 3.5 (Italian) and 4.8 (English) on the same corpus. Worth recording on its own: the five tokenizers disagree with each other by a median of 20% and by up to 61% on Japanese. That is the empirical case for this repository's no-token-counting policy, and the reason the dashboard reports a range and never calls it a count. Refs Atena-IT#80
Upload a document, get an estimated token count, and have it fill the existing input-token field so the cost math already in place answers the question. The estimate is always shown with its range and its method. calculator.tsx changes by two lines: all new logic lives in new files. Refs Atena-IT#80
… builds for pdf.js 5 and later require Safari 17.4: they use Iterator helpers and async iteration over ReadableStream, and the 6.x legacy build still carries the second one. Vite builds this app for baseline-widely-available, which reaches further back, so the newer parser broke PDF reading on Safari 17.1 while every automated gate stayed green. Reproduced deterministically in Chromium with both features removed, verified against the same harness after the change, and confirmed by hand on Safari 17.1. The 4.x parser chunk is also smaller: 374 kB against 427. Refs Atena-IT#80
Records what the estimate is and is not, the calibration figures behind the ratios and the band, the reason pdfjs-dist is pinned to 4.x, and a note that typecheck skips the referenced projects so build is the real gate. Refs Atena-IT#80
…closure The full extension list crowded the panel. It now sits in a native details/summary element, which needs no extra dependency and is keyboard accessible on its own. The disclosure lives outside the header row so opening it does not reflow the button, and the webkit marker is hidden so Safari does not draw a second triangle next to the chevron. Also retitles the panel to Estimate tokens from a file. Refs Atena-IT#80
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #80
The Calculator prices a workload from token counts the user types in by hand, which leaves the hard part to the user: nobody knows upfront how many tokens a document is worth. This adds the missing step — upload a file, get an estimated token count, and the cost math already in place answers the question.
Why the dashboard and not the SDKs
libraries/python/AGENTS.md,libraries/typescript/AGENTS.md,AGENTS-docs/contribution-checklist.mdandskills/tokenpricing/SKILL.mdall place token counting out of scope for the libraries, because tokenizers differ across providers and estimates presented as counts are unreliable.This PR does not challenge that. The libraries are untouched. Every changed file is under
services/dashboard/, a separate experimental service whose Calculator is already labelled "Estimated cost", and the number is presented as an estimate with its range and its method wherever it appears.While building this I ended up measuring the premise, and it holds up (below).
The estimate is fitted, not guessed
The first version used a characters-per-token ratio and a ±20% band chosen by judgement. That seemed too weak to ship, so I built a bench: five tokenizer families — GPT-4o (o200k), GPT-4 (cl100k), Llama 3, Gemma, Claude — over a nine-document corpus of English, Italian and Japanese prose plus
README.md,docs/quickstart.md,core.py,calculator.tsxand two JSON files taken from this repository. 45 reference counts in total.Result 1 — the tokenizers disagree with each other.
core.pyprices.jsonMedian disagreement 20.2%, worst case 60.7%. This is the empirical case for the repository's no-token-counting policy, and the reason this feature reports a range and never calls it a count.
Result 2 — fitted constants. A grid search minimising worst-case error against the mean of the five tokenizers moved the ratios from 4.00 / 2.80 / 1.00 to 4.25 / 2.95 / 1.05, cutting worst-case error from 22.8% to 15.3% (median 10.1%).
Result 3 — the band width is a coverage decision. Of the 45 reference counts, ±20% contained 37 and ±25% contains 41 (91%). ±10% would contain 21 — reassuring and false. The band is ±25%.
The measurement also retired the folkloric "4 characters per token" rule my own test had encoded: modern tokenizers measured between 3.5 (Italian) and 4.8 (English) on this corpus. I updated the test rather than the finding.
Known limitation, stated in the code: a single global ratio cannot satisfy English at 4.8 characters per token and Italian at 3.5 simultaneously — it overestimates one and underestimates the other. Per-language ratios are the natural next step.
Browser compatibility
Every automated gate was green — 19 tests, typecheck, lint, build — while PDF reading was completely broken on Safari 17.1. Only manual testing on a real browser found it.
pdf.js 5 and later require Safari 17.4: they use Iterator helpers and async iteration over
ReadableStream, and the 6.x legacy build still carries the second one. Vite builds this app forbaseline-widely-available, which reaches further back, so the newer parser would silently break the browser support the rest of the dashboard already promises.pdfjs-distis therefore pinned to 4.x — coherence with the app's own target, not a workaround. The 4.x parser chunk is also smaller: 374 kB against 427.I reproduced the failure deterministically before fixing it, by removing
IteratorandReadableStream.prototype[Symbol.asyncIterator]from Chromium in a Playwright harness. The old code fails there exactly as it does on Safari 17.1; the new code passes. That harness is also what caught the first fix being wrong.Verified: Safari 17.1 (by hand), Chromium (headless), Chromium with both features removed. Firefox is the lowest-risk case, since pdf.js is Firefox's built-in PDF viewer, but I am not claiming it as tested.
Bundle impact
pdfjs-distis loaded through a dynamicimport()inside the PDF branch, so the parser lands in its own chunk and is only downloaded by users who open a PDF. The production bundle was already above Vite's 500 kB warning, so this mattered.How to review
The interesting file is
src/lib/estimate.ts— a pure function, no I/O, no React, with every constant named and its calibration documented at the top.src/lib/extract.tshandles reading.calculator.tsxchanges by two lines: all new logic lives in new files.19 tests. Note that
npm run typecheckis a weak gate in this workspace:tsconfig.jsonuses project references withfiles: [], sotsc --noEmitskips the referenced projects — it passed on code thattsc -brejected.npm run buildis the real gate, and I have added a note to the service README.Dependencies added
pdfjs-dist(runtime, 4.x pinned) — reading a PDF needs a real parser; this is the reference implementation and it costs the main bundle nothing. 3 packages, no new audit findings.vitest(dev) — this repository mandates TDD but the dashboard was the only area without a test runner. The TypeScript SDK already uses vitest.Out of scope
.docx,.epub, OCR of scanned PDFsHappy to open follow-up issues for per-language ratios and for
.docxsupport (a ZIP container, so it needs a small unzip dependency andword/document.xmlparsing).Unrelated observation
src/components/explorer.tsx:193carries a pre-existing lint warning (react-hooks/incompatible-library, from TanStack Table'suseReactTable). Present before this branch and untouched by it — flagging it in case it is not already known.