Summary
count_tokens treats any run of digits as groups of three (ceil(len/3)) regardless of context, and every symbol as its own token. For long structured payloads — base64 blobs, hex hashes, large JSON with many numeric fields, UUIDs — this can drift noticeably from how real BPE tokenizers merge those runs, and structured/tool-heavy prompts are exactly where budgeting matters most.
Repro
from tokenmeter.core import count_tokens
count_tokens("a"*64) # long alpha run
count_tokens("0123456789"*10) # long digit run -> groups of 3
count_tokens("550e8400-e29b-41d4-a716") # UUID-ish: hyphens each cost 1Ask
Not necessarily a bug to "fix" blindly — but we should (a) measure the error on a few structured fixtures (JSON, base64, hex, UUID) and (b) decide whether digit grouping / symbol handling needs tuning or at least a documented caveat in the README accuracy note. Pairs naturally with the calibration-test issue.
Summary
count_tokenstreats any run of digits as groups of three (ceil(len/3)) regardless of context, and every symbol as its own token. For long structured payloads — base64 blobs, hex hashes, large JSON with many numeric fields, UUIDs — this can drift noticeably from how real BPE tokenizers merge those runs, and structured/tool-heavy prompts are exactly where budgeting matters most.Repro
Ask
Not necessarily a bug to "fix" blindly — but we should (a) measure the error on a few structured fixtures (JSON, base64, hex, UUID) and (b) decide whether digit grouping / symbol handling needs tuning or at least a documented caveat in the README accuracy note. Pairs naturally with the calibration-test issue.