Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/specs/thinking-budget.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,10 @@ ceiling (e.g. `effort: "max"` on a model whose card has no
There are two equivalent ways a client opts into the budget envelope.
Both unlock Level 1, Level 2, and `finish_details` emission.

By themselves, `chat_template_kwargs.thinking` and
`chat_template_kwargs.enable_thinking` only control prompt rendering. They do
not activate the budget envelope or `finish_details` emission.

### 4.1 Anthropic-style `thinking`

```json
Expand Down
20 changes: 20 additions & 0 deletions server/docs/DS4.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,26 @@ also accepts named bare-JSON fallbacks emitted by compatible checkpoints:
calls remain unambiguous when a request supplies more than one tool; ordinary
JSON that does not resolve to an allowed tool is preserved as assistant text.

## Reasoning effort

The native DeepSeek V4 renderer supports the official `low`, `high`, and `max`
reasoning encodings. `low` adds no prefix; `high` and `max` prepend their
official model-facing instruction before the system message. The server accepts
`reasoning.effort`, top-level `reasoning_effort`, and the official
`chat_template_kwargs` form:

```json
{"chat_template_kwargs":{"thinking":true,"reasoning_effort":"max"}}
```

For DeepSeek V4 Flash API compatibility, `medium` and `xhigh` map to `high`.
Lucebox's hyphenated `x-high` extension retains its own phase-1 budget tier
and uses the `max` model-facing encoding. Explicitly enabling thinking without
an effort uses DeepSeek's `high` default. Bare `chat_template_kwargs.thinking`
and `chat_template_kwargs.enable_thinking` toggles only control prompt rendering;
they do not activate the force-close budget envelope unless the request also
sets an explicit effort. Disabling thinking suppresses every effort prefix.

## Code Layout

| Area | Files |
Expand Down
12 changes: 11 additions & 1 deletion server/src/server/chat_template.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,8 @@ std::string render_chat_template(
ChatFormat format,
bool add_generation_prompt,
bool enable_thinking,
const std::string & tools_json)
const std::string & tools_json,
const std::string & reasoning_effort)
{
std::string result;
bool has_tools = !tools_json.empty() && tools_json != "[]" && tools_json != "null";
Expand Down Expand Up @@ -375,6 +376,15 @@ std::string render_chat_template(
}

result = "<|begin▁of▁sentence|>";
if (enable_thinking && reasoning_effort == "high") {
result += "Reasoning Effort: Absolute maximum with no shortcuts permitted.\n";
result += "You MUST be very thorough in your thinking and comprehensively decompose the problem to resolve the root cause, rigorously stress-testing your logic against all potential paths, edge cases, and adversarial scenarios.\n";
result += "Explicitly write out your entire deliberation process, documenting every intermediate step, considered alternative, and rejected hypothesis to ensure absolutely no assumption is left unchecked.\n\n";
} else if (enable_thinking && reasoning_effort == "max") {
result += "Reasoning Effort: Beyond maximum — exhaustive, relentless, and uncompromising.\n";
result += "You MUST reason with the utmost depth and rigor, leaving absolutely nothing to chance: exhaustively decompose the problem into its most fundamental components, trace every causal chain to its root, and resolve the underlying cause rather than any surface symptom.\n";
result += "Do not stop reasoning until you have independently verified the solution from multiple angles and are certain that no assumption remains unchecked and no error remains undiscovered.\n\n";
}
if (has_tools) {
result += "### Tools\n\n"
"You may call functions to assist with the user query. "
Expand Down
6 changes: 5 additions & 1 deletion server/src/server/chat_template.h
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,16 @@ enum class ChatFormat {
// `tools_json` is an optional JSON string containing the tool definitions
// array. When non-empty, the Qwen3/3.5 template injects a tool preamble
// into the system message instructing the model how to emit <tool_call> tags.
//
// `reasoning_effort` is the normalized model-facing effort. DeepSeek V4 uses
// low, high, and max; high and max prepend the official encoding prefixes.
std::string render_chat_template(
const std::vector<ChatMessage> & messages,
ChatFormat format,
bool add_generation_prompt = true,
bool enable_thinking = false,
const std::string & tools_json = "");
const std::string & tools_json = "",
const std::string & reasoning_effort = "");

// Detect the appropriate chat format for an architecture.
ChatFormat chat_format_for_arch(const std::string & arch);
Expand Down
144 changes: 92 additions & 52 deletions server/src/server/http_server.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -701,7 +701,8 @@ json build_props_body(const ServerConfig & config,
const ToolMemory & tool_memory) {
// arch-gated capabilities (mirrors Python _capabilities()).
const bool is_qwen = (config.arch.rfind("qwen", 0) == 0);
const bool reasoning_supported = is_qwen;
const bool is_deepseek4 = (config.arch == "deepseek4");
const bool reasoning_supported = is_qwen || is_deepseek4;
const bool speculative_supported = is_qwen;
const bool tools_supported = is_qwen || config.arch == "deepseek4";

Expand All @@ -725,12 +726,16 @@ json build_props_body(const ServerConfig & config,
// all activate the phase-1 envelope. Advertise the full set when the
// arch supports reasoning so clients can negotiate the higher tiers.
json reasoning_efforts = json::array();
if (reasoning_supported) {
if (is_qwen) {
reasoning_efforts.push_back("low");
reasoning_efforts.push_back("medium");
reasoning_efforts.push_back("high");
reasoning_efforts.push_back("x-high");
reasoning_efforts.push_back("max");
} else if (is_deepseek4) {
reasoning_efforts.push_back("low");
reasoning_efforts.push_back("high");
reasoning_efforts.push_back("max");
}

json server = {
Expand Down Expand Up @@ -1827,8 +1832,8 @@ bool HttpServer::parse_endpoint_request(
return false;
}

void HttpServer::apply_request_reasoning(
const json & body, ParsedRequest & req) {
void apply_request_reasoning(
const json & body, const ServerConfig & config, ParsedRequest & req) {
// Explicit thinking budgets override reasoning-effort tiers. Template
// kwargs can still override whether the rendered prompt enables thinking.
// Default: thinking OFF (Qwen3.6 thinking wrecks DFlash acceptance
Expand All @@ -1838,23 +1843,47 @@ void HttpServer::apply_request_reasoning(
int request_reply_budget = -1;
int effort_phase1_cap = -1;
bool effort_set = false;
std::string normalized_effort;

req.thinking_opt_in = false;
req.per_req_phase1_cap = -1;
req.per_req_reply_budget = -1;

auto apply_reasoning_effort = [&](const std::string & effort) {
if (effort == "none") {
enable_thinking = false;
normalized_effort.clear();
effort_set = true;
return;
}

// Five-tier vocabulary (spec §4.2). Unknown tier → high.
int tier_value = config_.effort_tiers.high;
int tier_value = config.effort_tiers.high;
if (effort == "minimal" || effort == "low") {
tier_value = config_.effort_tiers.low;
tier_value = config.effort_tiers.low;
normalized_effort = "low";
} else if (effort == "medium") {
tier_value = config_.effort_tiers.medium;
tier_value = config.effort_tiers.medium;
normalized_effort = config.arch == "deepseek4" ? "high" : "medium";
} else if (effort == "xhigh") {
// DeepSeek V4 Flash's OpenAI-compatible APIs map xhigh to high.
// Other architectures retain Lucebox's x-high tier alias.
if (config.arch == "deepseek4") {
tier_value = config.effort_tiers.high;
normalized_effort = "high";
} else {
tier_value = config.effort_tiers.x_high;
normalized_effort = "x-high";
}
} else if (effort == "x-high") {
tier_value = config_.effort_tiers.x_high;
// Hyphenated x-high is Lucebox's explicit five-tier extension.
tier_value = config.effort_tiers.x_high;
normalized_effort = config.arch == "deepseek4" ? "max" : "x-high";
} else if (effort == "max") {
tier_value = config_.effort_tiers.max;
tier_value = config.effort_tiers.max;
normalized_effort = "max";
} else {
normalized_effort = "high";
}

effort_phase1_cap = tier_value;
Expand All @@ -1864,7 +1893,7 @@ void HttpServer::apply_request_reasoning(
req.thinking_opt_in = true;
};

if (body.contains("reasoning")) {
if (body.contains("reasoning") && body["reasoning"].is_object()) {
const auto & reasoning = body["reasoning"];
if (reasoning.contains("effort")) {
apply_reasoning_effort(reasoning.value("effort", "high"));
Expand All @@ -1876,7 +1905,7 @@ void HttpServer::apply_request_reasoning(
body["reasoning_effort"].is_string()) {
apply_reasoning_effort(body["reasoning_effort"].get<std::string>());
}
if (body.contains("thinking")) {
if (body.contains("thinking") && body["thinking"].is_object()) {
const auto & thinking = body["thinking"];
if (thinking.contains("type")) {
const bool enabled = thinking.value("type", "") == "enabled";
Expand All @@ -1892,33 +1921,58 @@ void HttpServer::apply_request_reasoning(
request_reply_budget = thinking["reply_budget"].get<int>();
}
}
if (body.contains("chat_template_kwargs")) {
if (body.contains("chat_template_kwargs") &&
body["chat_template_kwargs"].is_object()) {
const auto & kwargs = body["chat_template_kwargs"];
if (!effort_set && kwargs.contains("reasoning_effort") &&
kwargs["reasoning_effort"].is_string()) {
apply_reasoning_effort(
kwargs["reasoning_effort"].get<std::string>());
}
if (kwargs.contains("thinking") && kwargs["thinking"].is_boolean()) {
enable_thinking = kwargs["thinking"].get<bool>();
}
if (kwargs.contains("enable_thinking")) {
enable_thinking = kwargs["enable_thinking"].get<bool>();
}
}
// DeepSeek uses high whenever thinking is enabled without an explicit
// model-facing effort. Only API-style thinking.type="enabled" also selects
// the high budget tier; bare template toggles affect rendering alone.
if (enable_thinking && config.arch == "deepseek4" &&
normalized_effort.empty()) {
normalized_effort = "high";
if (req.thinking_opt_in) {
effort_phase1_cap = config.effort_tiers.high;
effort_set = true;
}
}
if (!enable_thinking) {
normalized_effort.clear();
req.thinking_opt_in = false;
}
req.thinking_enabled = enable_thinking;
req.reasoning_effort = normalized_effort;

// Spec §4.3 combined precedence + §4.4 clamping: thinking.budget_tokens
// (if set) wins over reasoning.effort for the phase-1 cap; either is
// clamped to the server ceilings.
if (request_budget_tokens >= 0) {
if (req.thinking_opt_in && request_budget_tokens >= 0) {
req.per_req_phase1_cap =
(std::min)(request_budget_tokens, config_.think_max_tokens);
if (request_budget_tokens > config_.think_max_tokens) {
(std::min)(request_budget_tokens, config.think_max_tokens);
if (request_budget_tokens > config.think_max_tokens) {
std::fprintf(stderr,
"[server] thinking.budget_tokens=%d clamped to "
"think_max_tokens=%d\n",
request_budget_tokens, config_.think_max_tokens);
request_budget_tokens, config.think_max_tokens);
}
} else if (effort_set) {
} else if (req.thinking_opt_in && effort_set) {
// Spec §4.4: effective cap is min(tier value, max_tokens -
// hard_limit_reply_budget). Tier values can legitimately exceed
// default_max_tokens; clients that want the full tier budget must
// pass an explicit max_tokens. Otherwise we narrow silently to fit.
const int max_output_phase1_room = (std::max)(
0, req.max_output - config_.hard_limit_reply_budget);
0, req.max_output - config.hard_limit_reply_budget);
req.per_req_phase1_cap =
(std::min)(effort_phase1_cap, max_output_phase1_room);
if (effort_phase1_cap > max_output_phase1_room) {
Expand All @@ -1927,17 +1981,17 @@ void HttpServer::apply_request_reasoning(
"(max_tokens=%d - hard_limit_reply_budget=%d); "
"pass a larger max_tokens to use the full tier budget\n",
effort_phase1_cap, req.per_req_phase1_cap,
req.max_output, config_.hard_limit_reply_budget);
req.max_output, config.hard_limit_reply_budget);
}
}
if (request_reply_budget >= 0) {
if (req.thinking_opt_in && request_reply_budget >= 0) {
req.per_req_reply_budget =
(std::min)(request_reply_budget, config_.hard_limit_reply_budget);
if (request_reply_budget > config_.hard_limit_reply_budget) {
(std::min)(request_reply_budget, config.hard_limit_reply_budget);
if (request_reply_budget > config.hard_limit_reply_budget) {
std::fprintf(stderr,
"[server] thinking.reply_budget=%d clamped to "
"hard_limit_reply_budget=%d\n",
request_reply_budget, config_.hard_limit_reply_budget);
request_reply_budget, config.hard_limit_reply_budget);
}
}
// (The effort tier doesn't influence reply_budget — spec §4.2: the
Expand Down Expand Up @@ -1979,7 +2033,7 @@ bool HttpServer::render_messages_to_text(
} else {
rendered = render_chat_template(
chat_messages, chat_format_, add_generation_prompt,
req.thinking_enabled, tools_json);
req.thinking_enabled, tools_json, req.reasoning_effort);
}

return true;
Expand Down Expand Up @@ -2024,12 +2078,16 @@ bool HttpServer::validate_request_context(
void HttpServer::log_parsed_request(const ParsedRequest & req) const {
std::fprintf(stderr,
"[server] chat %s format=%s stream=%s msgs=%zu tools=%zu prompt_tokens=%zu "
"max_tokens=%d max_ctx=%d thinking=%s started_in_thinking=%s stops=%zu model=%s\n",
"max_tokens=%d max_ctx=%d thinking=%s reasoning_effort=%s "
"started_in_thinking=%s stops=%zu model=%s\n",
req.response_id.c_str(), api_format_name(req.format),
req.stream ? "true" : "false",
json_array_size(req.messages), json_array_size(req.tools),
req.prompt_tokens.size(), req.max_output, config_.max_ctx,
req.thinking_enabled ? "true" : "false",
!req.thinking_enabled ? "none" :
(req.reasoning_effort.empty() ? "default" :
req.reasoning_effort.c_str()),
req.started_in_thinking ? "true" : "false",
req.stop_sequences.size(), req.model.c_str());
}
Expand Down Expand Up @@ -2085,7 +2143,7 @@ bool HttpServer::route_request(SocketHandle fd, const HttpRequest & hr) {
normalize_chat_messages(req.messages, req.format, tool_memory_);
// Reasoning must be applied BEFORE rendering: the template injects
// the empty <think>\n\n</think>\n\n block when thinking is disabled.
apply_request_reasoning(body, req);
apply_request_reasoning(body, config_, req);
// Bandit: parse session_id from extra_body (opt-in adaptive keep_ratio).
req.session_id = parse_session_id_from_body(body);

Expand Down Expand Up @@ -2717,36 +2775,18 @@ void HttpServer::apply_flowkv_compression(
return;
}

std::string tools_json;
if (req.tools.is_array() && !req.tools.empty()) {
tools_json = req.tools.dump();
}
const std::vector<ChatMessage> chat_messages = normalize_chat_messages(
modified_messages, req.format, tool_memory_);

std::string rendered;
if (!config_.chat_template_src.empty()) {
const std::string & bos = tokenizer_.bos_id() >= 0
? tokenizer_.raw_token(tokenizer_.bos_id())
: std::string();
const std::string & eos = tokenizer_.eos_id() >= 0
? tokenizer_.raw_token(tokenizer_.eos_id())
: std::string();
try {
rendered = render_chat_template_jinja(
config_.chat_template_src, chat_messages, bos, eos,
/*add_generation_prompt=*/true,
req.thinking_enabled, tools_json);
} catch (const std::exception & error) {
std::fprintf(stderr,
"[flowkv] jinja re-render failed (%s) — skipping\n",
error.what());
return;
}
} else {
rendered = render_chat_template(
chat_messages, chat_format_, /*add_generation_prompt=*/true,
req.thinking_enabled, tools_json);
std::string render_error;
if (!render_messages_to_text(
chat_messages, req, /*add_generation_prompt=*/true,
rendered, render_error)) {
std::fprintf(stderr,
"[flowkv] re-render failed (%s) — skipping\n",
render_error.c_str());
return;
}

const int tokens_before = (int) prepared.tokens.size();
Expand Down
Loading
Loading