perf(turboquant): raise KV activation threshold 2048 → 100K

[dusterbloom]

What

Raises the default TurboQuant KV activation threshold from 2048 → 100K tokens, and adds a doctor warning for kv_cache = "turboquant".

Why

TurboQuant's custom Metal decode kernels (decode_scores / decode_weighted_values) are measurably slower than MLX's built-in dense SDPA. At the current 2048 default, any model configured with kv_cache = "turboquant" silently drops onto the slow path from ~2K tokens of context onward — exactly where you'd enable it.

Benchmarked with bench_decode on Qwen3.5-9B-4bit (M4 32 GB), same server, --kv-cache off vs turboquant (+ HIGGS_TURBOQUANT_MIN_TOKENS=0 to force activation):

context	dense decode	turboquant decode	Δ
~15 tok	15.6 tok/s	11.7 tok/s	−25%
~7K tok	12.5 tok/s	10.1 tok/s	−19%

Plus a multi-second first-token stall at 7K when the prefilled KV is bulk-quantized (TTFT 0.4 s → 6.4 s). TurboQuant never wins in the tested range.

Dense KV costs only ~10 KB/token — ~1 GB at 100K context — so TurboQuant's memory saving only pays for its decode tax near that scale. The new default keeps decode on the fast dense path; set HIGGS_TURBOQUANT_MIN_TOKENS to opt back in when context length genuinely threatens memory.

Changes

• cache.rs: DEFAULT_TURBOQUANT_ACTIVATE_AT 2048 → 100_000, with the bench data in the doc comment.
• doctor.rs: warn() when a model sets kv_cache = turboquant.

Follow-up

The real fix is a faster fused decode kernel — the values kernel is O(T)/thread with no SIMD-group reduction (flagged with a ponytail: marker in cache.rs). This PR just closes the footgun until then.

Test

• cargo fmt + cargo clippy -p higgs -p higgs-models clean
• cargo test -p higgs doctor -- --test-threads=1 → 42 passed

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added a warning in model checks when TurboQuant is enabled, highlighting expected performance behavior and the token-threshold override option.

• Bug Fixes

  • Increased the default TurboQuant activation threshold, so decoding stays in the standard mode for longer before switching behavior.
  • Kept existing validation behavior unchanged for invalid cache settings and unsupported batch/TurboQuant combinations.

[coderabbitai]

Warning

Review limit reached

@dusterbloom, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 35 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d85e2fa0-b834-4e80-b706-edea94c6a1e8

📥 Commits

Reviewing files that changed from the base of the PR and between c9f85b1 and 4a35164.

📒 Files selected for processing (2)

• crates/higgs-models/src/cache.rs
• crates/higgs/src/doctor.rs

📝 Walkthrough

Walkthrough

The default TurboQuant decode activation threshold constant was raised from 2048 to 100,000 with expanded documentation explaining the rationale. Additionally, the model doctor now emits a warning when a model's KV-cache config is set to turboquant, describing performance characteristics and the activation threshold override.

Changes

TurboQuant threshold and diagnostics

Layer / File(s)	Summary
Default threshold update and doctor warning crates/higgs-models/src/cache.rs, crates/higgs/src/doctor.rs	Increases the default TurboQuant activation threshold to 100,000 with added documentation, and adds a WARN check in the model doctor when kv_cache_config() is turboquant, referencing HIGGS_TURBOQUANT_MIN_TOKENS.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested reviewers: panbanda

Suggested labels: enhancement, documentation

Poem:
A rabbit tunes a threshold high,
From two-oh-four-eight to numbers sky,
The doctor checks each model's gear,
And warns when turboquant draws near,
Hop along, all clear! 🐇

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: increasing the TurboQuant KV activation threshold from 2,048 to 100K.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

crates/higgs-models/src/cache.rs (1)

21-41: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider making the constant visible to higgs to avoid literal drift.

doctor.rs hardcodes "100K" in its new warning text since DEFAULT_TURBOQUANT_ACTIVATE_AT is private to this crate. If this default changes again, the doctor message will silently go stale.

♻️ Suggested fix

-const DEFAULT_TURBOQUANT_ACTIVATE_AT: i32 = 100_000;
+pub const DEFAULT_TURBOQUANT_ACTIVATE_AT: i32 = 100_000;

Then reference it from doctor.rs instead of the literal 100K in the warning string.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/higgs-models/src/cache.rs` around lines 21 - 41, The warning text in
the doctor path hardcodes the TurboQuant threshold as “100K”, which can drift
from the real default. Expose the existing DEFAULT_TURBOQUANT_ACTIVATE_AT
constant from cache.rs to higgs so doctor.rs can reference it when building the
warning message instead of embedding a literal. Update the warning construction
in the doctor code to use that shared constant (or a shared formatter derived
from it) so future default changes stay in sync automatically.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/higgs-models/src/cache.rs`:
- Around line 21-41: The warning text in the doctor path hardcodes the
TurboQuant threshold as “100K”, which can drift from the real default. Expose
the existing DEFAULT_TURBOQUANT_ACTIVATE_AT constant from cache.rs to higgs so
doctor.rs can reference it when building the warning message instead of
embedding a literal. Update the warning construction in the doctor code to use
that shared constant (or a shared formatter derived from it) so future default
changes stay in sync automatically.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 64779fb2-7e6a-4878-ace9-8ca1c0313b17

📥 Commits

Reviewing files that changed from the base of the PR and between 9a7437e and c9f85b1.

📒 Files selected for processing (2)

• crates/higgs-models/src/cache.rs
• crates/higgs/src/doctor.rs

[dusterbloom]

Addressed the CodeRabbit nitpick on crates/higgs-models/src/cache.rs:

• Made DEFAULT_TURBOQUANT_ACTIVATE_AT pub and referenced it from doctor.rs (higgs_models::cache::DEFAULT_TURBOQUANT_ACTIVATE_AT) instead of the hardcoded 100K, so the doctor warning's threshold can't drift from the actual default. Mirrors the existing pub const DEFAULT_MIN_TOKENS_TO_PERSIST → higgs import pattern. (4a351647)

Verified locally: cargo fmt + cargo clippy -p higgs -p higgs-models clean, cargo check -p higgs -p higgs-models builds on the main base, cargo test -p higgs doctor -- --test-threads=1 → 44 pass.

On the red MSRV check — it's a pre-existing CI infra failure, not from this PR. The mlx-sys metallib build fails on the macos-26-arm64 runner with:

LLVM ERROR: Cannot link symbol 'air.normalize_function_constant_predicate.i8': Used in deployment target macosx15.0.0 but defined in deployment target macosx26.0.0

The identical failure appears on unrelated dependency-bump PRs since ~2026-06-24 (e.g. the renovate lockfile PRs). This change touches only a constant and a doctor warning string — it cannot affect Metal shader compilation. The fix belongs in CI config (bump MACOSX_DEPLOYMENT_TARGET for the metallib step, or pin the runner image), out of scope here.

🤖 Addressed by Claude Code

[dusterbloom]

The failing Build / Lint / Coverage / MSRV checks here are the known macOS-runner infra breakage, not anything in this PR (a constant + a doctor string can't affect Metal shader linking). This run hit both variants of it:

• Coverage/MSRV landed on macOS 26 / Xcode 26.5 → LLVM ERROR: Cannot link symbol ... macosx15.0.0 ... macosx26.0.0
• Build landed on Xcode 16.4 but reused cache-poisoned mlx-sys .air → air version set to 2.8.0 ... but expecting 2.7

Both are fixed in #211 (pin macos-15 + bust the poisoned rust-cache). Since this is a fork PR, its CI runs from the base repo's workflow definition, so once #211 merges these checks go green here on the next run with no change to this branch. The Test check already passes.