Add local json hf tokenization demos#798

Open

klei22 wants to merge 3 commits into

ReaLLMASIC:masterfrom

klei22:add-local-json-hf-tokenization-demos

Commits on Apr 15, 2026

Add demo wiring a local JSON vocab into the HF tokenizer pipeline
claude
committed
Use absolute tokenizer path in hf_local_json_byte_fallback demo
claude
committed
Add letters+punctuation premade vocab and a dialogsum HF demo
claude
committed