tokenizer

Text tokenization for Olive programs. Turn prompts into token ids for model calls, count tokens before you send them, and decode ids back to text. Covers every encoding tiktoken.get_encoding does (GPT-2, GPT-3, GPT-4, GPT-4o, and gpt-oss style) plus any HuggingFace tokenizer.json (Llama and similar). The heavy lifting runs in a Rust byte-level BPE / SentencePiece engine, so bulk encoding stays fast while your code stays plain Olive.

import tokenizer

let enc = tokenizer.get_encoding("cl100k_base")
let ids = enc.encode("hello world")
print(ids)             // [15339, 1917]
print(enc.decode(ids)) // hello world

cl100k_base, o200k_base and r50k_base build straight from data embedded in the engine, no file, no network, no setup beyond get_encoding. Rarer presets (p50k_base, p50k_edit, gpt2, o200k_harmony) fetch their source file over HTTPS on first use, verify it against a pinned hash, and cache it, so every call after the first is offline too (see Preset encodings).

Install

Add the pod to your project:

pit add tokenizer

No further setup. The pod ships its Rust engine sources and pit builds them on your machine at install time (needs a nightly Rust toolchain, one-time build of a few minutes), then links with a relocatable rpath, so compiled binaries run wherever their directory goes. When the registry carries a prebuilt library for your platform, pit downloads it instead and skips the build.

Quickstart

Encode, stay inside a token budget, decode back:

import tokenizer

let enc = tokenizer.get_encoding("cl100k_base")
print(enc.vocab_size()) // 100277

let ids = enc.encode("The verdict was unanimous.")
print(ids)

let budget = 100
if len(ids) > budget:
    print("prompt too long, trimming")
    let ids = ids[0:budget]
print(enc.decode(ids))

Picking an encoding by model name

encoding_for_model resolves a model name the same way tiktoken does, by exact name first, then by known prefix, so "gpt-4o-2024-05-13" and "gpt-4o" both land on o200k_base:

import tokenizer

let enc = tokenizer.encoding_for_model("gpt-4o")
print(enc.encode("hello world"))

print(tokenizer.encoding_name_for_model("gpt-4")) // cl100k_base

Both panic on an unrecognized model name; call get_encoding directly when you already know the encoding.

Using HuggingFace models

Any HuggingFace tokenizer.json loads directly, from a file or from contents you already hold in memory:

import tokenizer
import io

let enc = tokenizer.from_hf_file("gpt2_tokenizer.json")

let json = io.read_file("llama_tokenizer.json")
let enc2 = tokenizer.from_hf_json(json)

Both byte-level BPE and SentencePiece with byte fallback work, so the same two calls cover GPT-2 style and Llama style vocabularies.

Using rank files and special tokens

get_encoding and encoding_for_model need no file at all. If you already have a .tiktoken rank file on disk instead (offline builds, a mirrored copy, a vendored preset), from_preset_file takes the scheme and specials from the preset table so you only supply the path:

let enc = tokenizer.from_preset_file("cl100k_base", "cl100k_base.tiktoken")

For a rank file that is not one of the named presets, state the splitting scheme explicitly and register any special tokens yourself:

import tokenizer

let enc = tokenizer.from_tiktoken("my_ranks.tiktoken", "gpt2")
tokenizer.add_special(enc, "<|endoftext|>", 50257)
print(enc.encode("hello<|endoftext|>"))

The scheme names the pretokenizer family: "gpt2", "gpt4", "o200k", and similar. The pattern lives in code for the encoding, so a mismatched scheme quietly changes every result. Pick the scheme that matches the file. Special token ids must sit above the mergeable ranks, and their contents match atomically during encoding.

Counting and trimming prompts

encode returns a plain [int], so budgeting is ordinary list work. Trim from the end to keep the prompt head, or from the front to keep the recent context:

import tokenizer

let enc = tokenizer.get_encoding("cl100k_base")

fn within_budget(enc: tokenizer.Tokenizer, text: str, budget: int) -> [int]:
    let ids = enc.encode(text)
    if len(ids) > budget:
        return ids[0:budget]
    return ids

Long documents

For inputs far beyond a model's context, encode in word chunks and concatenate the id lists. Every chunk after the first keeps its leading space, so boundaries match a single pass over plain prose:

import tokenizer
import string

let enc = tokenizer.get_encoding("cl100k_base")

fn encode_long(enc: tokenizer.Tokenizer, text: str) -> [int]:
    let words = string.split(text, " ")
    let mut ids: [int] = []
    let mut i = 0
    while i < len(words):
        let mut j = i
        let mut chunk = ""
        while j < len(words) and j < i + 50:
            if j > i:
                chunk = chunk + " "
            chunk = chunk + words[j]
            j = j + 1
        if i > 0:
            chunk = " " + chunk
        let part = enc.encode(chunk)
        let mut k = 0
        while k < len(part):
            ids.append(part[k])
            k = k + 1
        i = j
    return ids

API reference

Every constructor returns a Tokenizer and panics with the engine's message on failure (missing file, corrupt data, unknown scheme). Handles free themselves through __drop__. Treat a Tokenizer as affine: do not copy live ones, and use one per thread for parallel encoding.

Constructors:

• get_encoding(name): preset encoding by name alone, tiktoken's get_encoding. No file, no setup; see Preset encodings.
• encoding_for_model(model_name): preset encoding for a model name, tiktoken's encoding_for_model.
• from_hf_file(path): HuggingFace tokenizer.json file, byte-level BPE or SentencePiece with byte fallback.
• from_hf_json(json): same, from the file's contents in memory.
• from_tiktoken(path, scheme): .tiktoken rank file with an explicit pretokenizer scheme and no registered special tokens.
• from_preset_file(name, path): preset rank file already on disk. Scheme and special tokens come from the preset table, so only the file path is needed. Every preset works here except gpt2, which has no single rank-file form (use get_encoding("gpt2")).
• add_special(tok, content, id): register one special token (content, id) on a .tiktoken loaded tokenizer.

Functions:

• encoding_name_for_model(model_name) -> str: the preset name a model uses, without building a tokenizer, tiktoken's encoding_name_for_model.

Methods:

• encode(text) -> [int]: token ids for one text.
• decode(ids) -> str: text for token ids. Decoding arbitrary ids can yield non-UTF-8 bytes; those surface as replacement characters, like tiktoken with errors="replace". A decoded NUL byte cannot cross the FFI boundary and fails instead of truncating.
• vocab_size() -> int: one more than the largest token id.

Helpers:

• version() -> str: engine version.
• preset_names() -> str: supported preset names, comma separated.

Preset encodings

Name	Scheme	Models	Source
r50k_base	GPT-2 family	davinci, ada, and similar (deprecated)	embedded
p50k_base	GPT-2 family	text-davinci-003, code-davinci-002 (deprecated)	fetched, cached
p50k_edit	GPT-2 family	text-davinci-edit-001 (deprecated)	fetched, cached
cl100k_base	GPT-4 family	gpt-4, gpt-3.5-turbo	embedded
o200k_base	GPT-4o family	gpt-4o, gpt-4.1, o1, o3	embedded
o200k_harmony	GPT-4o family	gpt-oss-*	fetched, cached
gpt2	GPT-2 family	gpt2	fetched, cached

get_encoding(name) builds any of these with no path to supply. Embedded presets never touch the disk or the network. Fetched presets download their real source file (p50k_base/p50k_edit share OpenAI's p50k_base.tiktoken, gpt2 builds from the original release's vocab.bpe) over HTTPS on first use, verify it against a hash pinned in the engine, and cache the verified bytes so later calls, including in other processes, are offline. TOKENIZER_CACHE_DIR overrides the cache location; it otherwise follows the platform's usual cache directory ($XDG_CACHE_HOME/tokenizer or ~/.cache/tokenizer on Linux, ~/Library/Caches/tokenizer on macOS, %LOCALAPPDATA%\tokenizer on Windows).

from_preset_file(name, path) works for every name above except gpt2, if you would rather supply a rank file you already downloaded than let the engine fetch one:

let enc = tokenizer.from_preset_file("o200k_base", "o200k_base.tiktoken")

Performance notes

• Reuse one Tokenizer per thread and encode in bulk. The engine memoizes merge results, so repeated and overlapping inputs encode from cache.
• encode hands ids back through one native call per id today. A bulk getter is planned; until then, tight loops over millions of ids pay crossing overhead per element.

Limits

• Inputs cannot contain NUL bytes; they terminate strings at the native boundary and the view truncates there.
• decode of arbitrary ids is lossy by design (replacement characters), matching tiktoken defaults. Only decode ids you encoded if you need an exact round trip.
• Tokenizers hold reusable native state. Construct one Tokenizer per thread rather than encoding from one handle concurrently.
• Fetched presets (p50k_base, p50k_edit, gpt2, o200k_harmony) need network access on first use per cache directory; a download that fails the pinned sha256 check is rejected and never cached.