fix(pragent): bake the tiktoken encodings so a read-only install works (0.12.1)

[huhamhire]

Fixes #232 and prepares the 0.12.1 patch release.

The released 0.12.0 Windows build fails every /describe and /review with PermissionError under C:\Program Files, which makes the installed app unusable.

Cause

Not a permissions setting — a bad file inside the package.

litellm points TIKTOKEN_CACHE_DIR at its own litellm_core_utils/tokenizers/ and ships encoding files there so tiktoken works offline. But the files it ships fail the expected_hash pinned by the tiktoken it depends on — all three of them (litellm 1.99.0 / tiktoken 0.12.0). tiktoken therefore treats the cache as invalid on every run: it deletes the file and re-downloads it into site-packages. Writable while building, read-only once installed.

Not model specific: the reported trace is a CLI provider (o200k_base), but a gpt model resolves through encoding_for_model() to the same stale files.

Why it shipped

Worth stating, because it explains why the existing gates were useless here:

• A dev machine never shows it — vendor/ is writable, so tiktoken repairs itself on first run and stays fine forever.
• CI never showed it either — the build machine can write too, so the runtime assembled and smoke-tested clean. A stale file loads wherever it can be re-downloaded, so "CI passed" said nothing about the installed copy.

Fix

scripts/tiktoken-cache.py:

• prime (assemble time, before slimming) — repairs the bundled bytes so the cache is a hit and nothing is written at runtime, which also makes it work offline;
• verify (smoke test, after slimming) — compares the bundled bytes against tiktoken''s pinned hashes.

Two deliberate choices:

• verify compares hashes rather than loading an encoding. Loading is exactly the check that fails to discriminate here — it passes on any machine that can re-download. Comparing against the pin tests what actually decides fetch-or-reuse at runtime.
• Expected hashes are read from the installed tiktoken, never copied into our source. A hash pasted here would drift on the next tiktoken upgrade and quietly reintroduce the bug this exists to prevent.

verify runs after slimming, so a slim rule that deletes the cache is caught too.

Verification

Four steps, closing the loop in both directions:

Check	Result
Restore litellm''s stale copy, run verify	fails, naming the mismatch — the new check does catch the 0.12.0 build
Run prime	both encodings repaired, .tmp leftovers cleaned
Load encodings from a genuinely read-only dir (write denied via icacls)	succeeds
Put the stale file back in that same read-only dir	reproduces the reported PermissionError exactly

lint / typecheck / test / build pass.

Rule recorded

Same class as the pre-written .secrets.toml already handled in the assembler — a dependency writing under its own package at runtime. That was fixed as a one-off without becoming a rule, which is why this recurred. Now in AGENTS.md (engineering maintenance pitfalls) and docs/arch/02-agent/05-pragent-runtime.md: when adding or upgrading a Python dependency, check whether it writes under its own package at runtime; if so pre-populate it at assemble time and assert it, because the build machine can always write.

Release

This branch also carries the chore(release): 0.12.1 commit (version, both changelogs with compare links, lockfile). After merging to dev, the release flow is dev → master, tag v0.12.1, back-merge, bump.

🤖 Generated with Claude Code