Add exact request budget preflight

[Baiju Meswani (baijumeswani)]

Summary

• Add an exact request preflight API for chat requests in C, C++, and JavaScript.
• Report prompt tokens, reserved output tokens, required tokens, the model context limit, whether the request fits, and the token deficit.
• Use the same prompt rendering, tools, tokenizer, and output limit resolution as generation.
• Keep the public Foundry Local API at version 2 for the next minor release.
• Run JavaScript preflight asynchronously without blocking the extension host.
• Capture request and session state before asynchronous work so later mutations do not change the result.
• Keep URI file access, media preparation, prompt rendering, and tokenization off the JavaScript thread.
• Use model.context_length as the context limit, with the existing legacy fallback.
• Apply bounded output defaults when clients omit a limit: 2,048 tokens for text and 3,072 tokens for media.

Why

Toolkit needs an exact token count before generation so it can compact long coding conversations and retry safely.

The same budget checks also prevent oversized requests and unbounded output when clients such as GitHub Copilot BYOK omit an output limit.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
foundry-local	Ready	Preview	Sep 19, 2026 8:39am UTC

[Copilot]

🟡 Changes recommended

The current implementation introduces an unsafe v2 ABI extension and a streaming option-update deadlock.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds exact chat request token-budget preflight across C, C++, and JavaScript while sharing generation preparation logic and preserving asynchronous state isolation.

Changes:

• Adds versioned preflight APIs and asynchronous JavaScript bindings.
• Unifies prompt preparation, media handling, context limits, and output-limit resolution.
• Adds comprehensive snapshot, parity, lifecycle, ABI, and backend tests.

File summaries

File	Description
sdk_v2/js/test/chat-session.test.ts	Tests asynchronous preflight behavior.
sdk_v2/js/src/session.ts	Exposes the preflight API and result.
sdk_v2/js/src/index.ts	Exports the preflight type.
sdk_v2/js/src/detail/native.ts	Defines native preflight bindings.
sdk_v2/js/README.md	Documents JavaScript preflight usage.
sdk_v2/js/native/src/session.h	Declares the native addon method.
sdk_v2/js/native/src/session.cc	Implements asynchronous native preflight.
sdk_v2/js/binding.gyp	Adds private native headers.
sdk_v2/cpp/test/internal_api/session_manager_test.cc	Tests synchronization and session validation.
sdk_v2/cpp/test/internal_api/item_test.cc	Tests request snapshot and URI behavior.
sdk_v2/cpp/test/internal_api/chat/search_options_test.cc	Tests output-limit resolution.
sdk_v2/cpp/test/internal_api/chat/request_budget_test.cc	Tests budget arithmetic and limits.
sdk_v2/cpp/test/internal_api/chat/dynamic_engine_chat_test.cc	Tests Engine preflight parity and state.
sdk_v2/cpp/test/internal_api/chat/chat_session_test.cc	Tests chat preflight behavior.
sdk_v2/cpp/test/internal_api/chat_completions_converter_test.cc	Tests OpenAI output-limit mapping.
sdk_v2/cpp/test/internal_api/c_api_test.cc	Tests ABI layout and private API.
sdk_v2/cpp/test/CMakeLists.txt	Registers request-budget tests.
sdk_v2/cpp/src/util/file_uri.h	Centralizes URI-backed file reads.
sdk_v2/cpp/src/service/responses_handler.cc	Atomically replaces request tools.
sdk_v2/cpp/src/node_private_api.h	Defines the Node preflight bridge.
sdk_v2/cpp/src/items/message_item.h	Removes the old clone helper.
sdk_v2/cpp/src/items/message_item.cc	Uses shared request-item cloning.
sdk_v2/cpp/src/items/item.h	Declares deep request-item cloning.
sdk_v2/cpp/src/items/item.cc	Implements cloning and audio reads.
sdk_v2/cpp/src/items/image_item.cc	Reuses common file reading.
sdk_v2/cpp/src/items/audio_item.h	Exposes audio byte loading.
sdk_v2/cpp/src/inferencing/session/tool_registry.h	Adds atomic tool replacement.
sdk_v2/cpp/src/inferencing/session/tool_registry.cc	Implements replacement and normalization.
sdk_v2/cpp/src/inferencing/session/session.h	Adds state locking and preflight operation support.
sdk_v2/cpp/src/inferencing/session/session.cc	Synchronizes session options and validation.
sdk_v2/cpp/src/inferencing/session/request.h	Adds immutable request snapshots.
sdk_v2/cpp/src/inferencing/generative/genai_config.h	Widens context limits to 64-bit.
sdk_v2/cpp/src/inferencing/generative/genai_config.cc	Parses 64-bit context limits.
sdk_v2/cpp/src/inferencing/generative/chat/search_options.h	Defines canonical output-limit resolution.
sdk_v2/cpp/src/inferencing/generative/chat/search_options.cc	Enforces bounded limits and overflow checks.
sdk_v2/cpp/src/inferencing/generative/chat/request_budget.h	Declares request-budget results.
sdk_v2/cpp/src/inferencing/generative/chat/request_budget.cc	Computes checked request budgets.
sdk_v2/cpp/src/inferencing/generative/chat/prepared_chat_prompt.h	Defines reusable prepared prompts.
sdk_v2/cpp/src/inferencing/generative/chat/prepared_chat_prompt.cc	Implements text and media preparation.
sdk_v2/cpp/src/inferencing/generative/chat/onnx_engine_chat_stream.h	Adds prepared-prompt Engine APIs.
sdk_v2/cpp/src/inferencing/generative/chat/onnx_engine_chat_stream.cc	Reuses prepared Engine prompts.
sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_generator.h	Adds prepared-prompt generator APIs.
sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_generator.cc	Reuses prepared generator inputs.
sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_engine.cc	Enforces resolved Engine output limits.
sdk_v2/cpp/src/inferencing/generative/chat/chat_session.h	Exposes native chat preflight.
sdk_v2/cpp/src/inferencing/generative/chat/chat_session.cc	Shares snapshot preparation with generation.
sdk_v2/cpp/src/inferencing/generative/chat/chat_generator.h	Adds prepared-prompt abstraction.
sdk_v2/cpp/src/exports.def	Exports the private Node getter.
sdk_v2/cpp/src/contracts/chat_completions_converter.h	Clarifies catalog-default behavior.
sdk_v2/cpp/src/contracts/chat_completions_converter.cc	Stops inferring output limits from catalog metadata.
sdk_v2/cpp/src/c_api.cc	Implements public and private preflight entry points.
sdk_v2/cpp/src/c_api_types.h	Adds preflight-version validation.
sdk_v2/cpp/include/foundry_local/foundry_local_cpp.inline.h	Implements the C++ wrapper call.
sdk_v2/cpp/include/foundry_local/foundry_local_cpp.h	Adds the C++ preflight surface.
sdk_v2/cpp/include/foundry_local/foundry_local_c.h	Adds the C ABI result and function.
sdk_v2/cpp/docs/CppPortGuide.md	Documents the private Node bridge.
sdk_v2/cpp/docs/CodingConventionsAndStandards.md	Updates export conventions.
sdk_v2/cpp/CMakeLists.txt	Builds the new prompt and budget sources.

Review details

• Files reviewed: 58/58 changed files
• Comments generated: 2
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.