Skip to content

feat(serve): automatic shared-prefix write at the system/developer frontier - #152

Open
Astrangemaninhere wants to merge 1 commit into
Neroued:masterfrom
Astrangemaninhere:pr-issue142
Open

feat(serve): automatic shared-prefix write at the system/developer frontier#152
Astrangemaninhere wants to merge 1 commit into
Neroued:masterfrom
Astrangemaninhere:pr-issue142

Conversation

@Astrangemaninhere

@Astrangemaninhere Astrangemaninhere commented Sep 1, 2026

Copy link
Copy Markdown

Closes #142

What

Agent clients (Hermes, OpenCode, Pi, etc.) rarely send prompt_cache_breakpoint; on master they always miss the shared system/tools head. This adds one automatic write candidate at the leading system/developer frontier so the existing shared-prefix machinery can publish a prefix those clients actually share.

  • Keeps the OpenAI explicit-breakpoint contract and the existing last-content implicit candidate (both writes coexist).
  • Opt-out flag --no-auto-system-shared-prefix (extra write on by default) for the Responses path; the Chat path currently uses the default-on policy value (consistent with the issue discussion; wiring the flag into the Chat parse signature was deliberately left out to keep the diff minimal).
  • No second store, no new catalog, no engine changes - the planner's existing shared_stable_prefix path does the work.

Per-file rationale (8 files, +65/-7, serve layer only)

  1. src/serve/openai_common.h (+5) - OpenAIPromptCachePolicy gains auto_system_shared_prefix (default true). The policy is the single switch that reaches both Chat and Responses without threading a new parameter through every parser.
  2. src/serve/openai_common.cpp (+31/-1) - the actual feature:
    • Locates the end of the contiguous leading system/developer run (not the first turn): requests that share both a system and a developer turn reuse the whole instruction prefix.
    • Publishes the DefaultAutomatic candidate on that last leading turn when unmarked (the existing explicit-breakpoint contract is untouched).
    • Reserves one of the four frontend marker slots for the new candidate (frontend.cpp rejects >4 markers); the explicit-write selection shrinks by one when the candidate is distinct, so a request with 3 explicit breakpoints + both automatic writes stays at 4.
  3. src/serve/serve_options.h (+3) - ServeOptions gains auto_system_shared_prefix = true.
  4. src/serve/serve_options.cpp (+4/-1) - parses --no-auto-system-shared-prefix and documents it in the usage text.
  5. src/serve/openai_responses.h (+8/-1) - OpenAIResponsesPromptRequest carries the parsed cache_policy so the policy can be applied after prompt resolution (at parse time the leading turns are still in input_turns/instructions, not in generation.messages).
  6. src/serve/openai_responses_request.cpp (+11/-3) - stores the policy on the prompt instead of applying it to the empty message list; the parser signature gains the opt-out bool from the HTTP layer.
  7. src/serve/openai_responses_state.cpp (+7) - resolve_openai_responses_prompt applies the policy once generation.messages is assembled (instructions + input turns), which is the earliest point the leading-instruction candidate can be placed correctly. (Fixes the review finding that Responses requests saw no system/developer turns.)
  8. src/serve/openai_responses_http.cpp (+3/-1) - passes options_.auto_system_shared_prefix into the parser; without this one line the flag would exist but never reach Responses requests.

Verification

  • Build: clean on master da49c0d (WSL2, CUDA 13.3, sm_120a).
  • Serve smoke: sibling Chat requests sharing a long system head reuse the published prefix without any client breakpoint; Responses requests place the candidate after the resolved leading instructions.
  • --no-auto-system-shared-prefix restores strict OpenAI-implicit behavior on the Responses path.

Follow-ups agreed with the issue author (#142 discussion, not blockers)

  • Frontier: if the Qwen template renders tools before the system block, the candidate should sit after the combined tools+system prefix. Open to folding in.
  • Floor: skip the extra write below ~256 tokens of shared head.
  • @steve8697 offered to re-run the 16k sibling probe against this patch.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 58ded12138

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/serve/openai_responses_request.cpp Outdated
Comment thread src/serve/openai_common.cpp Outdated
Comment thread src/serve/openai_common.cpp Outdated
@Astrangemaninhere

Copy link
Copy Markdown
Author

这次的pr修改程度较小,希望能得到采纳;如果不能接受,希望能给出意见。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agent sibling sessions miss shared prefix without prompt_cache_breakpoint

1 participant