perf: use stack merging for short encoded BPE pretokens

[perfloop-agent]

Summary

• Add a stack-resident linear BPE merger for generic encoded cache misses with at most 32 emitted symbols.
• Count byte-fallback expansion in that limit, then use the existing priority-queue merger for larger inputs. The raw fused path and cache behavior are unchanged.
• Preserve greedy rank and leftmost tie order, with differential coverage for Unicode, ByteLevel characters, fallback expansion, boundaries, ties, errors, and pre-populated output buffers.
• Add a focused benchmark for unique generic cache misses with elapsed and process CPU-time output.

Workload: forced generic encoded cache miss, merge-dense pretoken with 16 emitted symbols

Metric	Before	After	Change
ns/op	1356	1004	26% lower
cpu-ns/op	1325	980	26% lower

Workload: forced generic encoded cache miss, merge-dense pretoken with 32 emitted symbols

Metric	Before	After	Change
ns/op	2127	1528	28.2% lower
cpu-ns/op	2085	1495	28.3% lower
Checks: 10 passed.

Validation

• cargo fmt --all --check
• cargo clippy --locked -- -D warnings
• cargo test --locked --example encoded_merge_bench
• cargo test --locked --lib models::bpe::tests
• cargo test --locked --lib ordinary_tests
• cargo test --locked --features ureq/native-certs
• cargo publish -p fastokens --dry-run --locked
• Qwen Qwen3-0.6B smoke test and Transformers 4.57.1/5.3.0 patch and add-token tests

---

Generated by Perfloop. Human sponsor: Tomás Senart. Measurements and checks.