openai runtime: prompt cache hits are bimodal — add per-session cache routing affinity

[Yevanchen]

Evidence (staging, 33 ping runs, gpt-5.5, identical per-session prompts)

Per-run cache_read_tokens from the mosoo usage ledger (runs 27–55s apart, well inside the OpenAI prefix-cache TTL):

• stack A: [0×5, 10112, 0×6, 12160, 0, 12160, 0] — 3/16 hits
• stack B: [0×5, 10112, 0×4, 12160, 0×6] — 2/17 hits
• When a hit occurs it is near-full (10,112–12,160 of ~12,400 input tokens) → the prompt prefix is byte-stable; prefix hygiene is not the problem.
• Aggregate hit rate is only 10–17%, versus 99.98% on the anthropic runtime (Agent SDK automatic caching) for the equivalent workload.

Hypothesis

Identical-prefix requests are being dispersed across OpenAI inference shards, so most requests miss a warm prefix cache that provably exists. OpenAI's automatic caching improves dramatically with a routing affinity hint (prompt_cache_key, previously the user field).

Ask

Give the openai runtime a stable per-session cache routing key:

1. Check whether the pinned app-server (OPENAI_RUNTIME_VERSION=0.144.0) exposes prompt_cache_key / user in its config or protocol — the driver's generated protocol types currently contain no cache-related field, so this may require a version bump or an upstream request.
2. If exposed: plumb a stable value (e.g. the mosoo session/thread id) from the boot payload into the app-server config; no prompt content changes.
3. Re-measure the per-run cache_read_tokens distribution: expectation is near-100% hits within TTL.

Value

Primarily cost: each ping run pays ~12.4k input tokens at full price on a miss (−50%+ on the cached share at scale). Latency impact on gpt-5.5 is secondary — measured first-token windows were already ~0.4–1.5s even on misses.

Refs: langgenius/mosoo#387 (direction 2), staging artifact warm-ping-openai-v31-16pairs-20260724.json.

[Yevanchen]

Attribution update (controlled experiment, 2026-07-24)

Corrected root cause: it is not a missing prompt_cache_key — codex sets one automatically. It is per-run thread churn making that key toxic.

Facts:

1. codex client.rs (build_responses_request) sets prompt_cache_key = thread_id unconditionally; no config override exists (upstream issues App-server sessions need stable prompt-cache key support for identical startup context openai/codex#21796, #26283 request exactly that; #26283 verified --config prompt_cache_key=... does not override).
2. OpenAI routing: requests are routed by a hash of the first ~256 prompt tokens combined with prompt_cache_key.
3. Controlled run (stage-b, one mosoo thread, 4 sequential identical pings 12–23s apart, input 12,366 tokens each): cache_read = [0, 0, 12160, 0]. Worker logs show each run spawns a fresh driver + fresh codex thread (nativeResumeRefPresent: false — cattle policy is nativeResume.persistence: "volatile").

Mechanism: identical prompts would route to the same shard by prefix hash alone — but a fresh thread per run gives each request a different prompt_cache_key, changing the routing hash and scattering byte-identical requests across shards. codex's auto-key + our stateless per-run threads actively causes the 10–17% hit rate. (anthropic is immune: content-addressed server-side cache, no routing component — hence 99.98%.)

Paths:

• A (correct, small, blocked upstream): client-supplied stable key (= mosoo sessionId) → needs App-server sessions need stable prompt-cache key support for identical startup context openai/codex#21796. We should add this evidence upstream.
• B (long-term, big): platform-persisted native thread resume for openai cattle — turns per-run full re-prefill (12.4k tokens) into append-only turns; fixes caching and removes most prefill entirely. Requires context-injection redesign (avoid double-history), thread expiry handling. This is the real prize but is a design project, not a patch.
• No interim driver-side hack exists (key not configurable).

Parking this issue on upstream #21796; evidence preserved in the perf ledger.