rust-token-slim

rust-token-slim reduces the language-model token count of Rust source files. It outputs Rust code directly, with no intermediate format or decoder. The original source is always one of the candidates, so the output never uses more tokens according to the selected tokenizer.

cargo run --release -- path/to/file.rs > file.min.rs
cargo run --release -- --stats path/to/repository
cargo run --release -- --model cl100k_base path/to/file.rs

o200k_base is used by default. Select another tokenizer with --model:

• o200k_base
• o200k_harmony
• cl100k_base
• p50k_base
• p50k_edit
• r50k_base

The --stats command reads every .rs file in a repository without modifying it. It excludes .git and target, then reports analyzed and skipped files, token counts before and after transformation, and the percentage saved. Files that are not valid UTF-8 or cannot be parsed as standalone Rust files are skipped.

Results on public projects

Across ten public Rust repositories on GitHub, rust-token-slim reduced the combined token count by 13.86%, saving 700,062 tokens over 2,295 Rust files. Every repository showed a reduction.

Repository	Files	Tokens before	Tokens after	Saved
anyhow	37	58,598	52,658	10.14%
thiserror	69	36,363	30,341	16.56%
once_cell	20	32,000	27,882	12.87%
itertools	76	200,810	179,640	10.54%
rayon	190	321,337	264,786	17.60%
serde	237	419,369	354,305	15.51%
regex	227	1,558,308	1,336,159	14.26%
clap	337	642,651	559,338	12.96%
tokio	802	1,443,747	1,255,765	13.02%
axum	300	336,285	288,532	14.20%
Total	2,295	5,049,468	4,349,406	13.86%

See the full benchmark for pinned commits, methodology, limitations, and reproduction instructions.

How it works

The program compares several representations of the source: quote, prettyplease, lexical compaction, and item-by-item selection. It verifies that the canonical syntax tree and macro text remain identical, then selects the representation with the lowest o200k_base token count.

Files that use line! or column! are returned unchanged because these macros observe positions in the source file.

Limitations

Ordinary formatting and comments are not preserved. Syntax-tree equivalence does not guarantee that every transformed project will compile or behave identically: relative paths, source positions, and external tooling can depend on the original file layout. Review generated code before using it in a build.

Library usage

let compact = rust_token_slim::minify(source)?;

let compact = rust_token_slim::minify_with_tokenizer(
    source,
    rust_token_slim::Tokenizer::Cl100kBase,
)?;

Validation

cargo test --all-targets
cargo clippy --all-targets --all-features -- -D warnings
cargo fmt --check

Licensed under the MIT License.