Lambda optimized/model core - #49
Conversation
Thread the contiguous f32 unary/binary elementwise ops (SiLU, SwiGLU mul, residual adds, scaling) across the barrier pool at the unary_impl/binary_impl dispatch point, split into one disjoint output range per worker. Bit-identical to the serial path; gated by a size threshold so small tensors stay serial. Recovers multi-thread prefill scaling where these single-threaded elementwise ops were an Amdahl bottleneck (matmuls scale ~Nx, these did not). Knobs: CANDLE_PAR_ELEMWISE=0 disables; CANDLE_PAR_ELEMWISE_MIN sets the element threshold (default 16384).
Add an f16 specialization of the interleaved-cache CPU flash path alongside the existing f32 one: RawInterleavedKvCacheF16 (head-major, stores K/V as f16 -> half the bytes streamed per token, the decode bandwidth bottleneck) and the matching causal_decode_f16kv_interleaved / causal_prefill_f16kv_headmajor kernels. The QK dot widens the f16 K row in-register against the f32 query; with the opt-in f16-attn-dot feature it uses a native f16.f16 dot instead. Purely additive on the upstream f32 interleaved infrastructure. Tests validate both kernels against an f32 reference in both feature configs.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 718c042e9d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| #[arg(long)] | ||
| output: String, | ||
| /// Comma-separated substrings; any tensor whose name contains one is requantized. | ||
| #[arg(long, default_value = "token_embd,output")] |
There was a problem hiding this comment.
With this default, the documented gguf-requant ... --pack invocation also matches token_embd/output in the requantization branch and converts those tensors to the default q4k before any packing happens. For a Q6_K model this silently changes the embeddings/lm head even when the user only asked to bake Q6_Kx8 weights, and it also prevents the tied output.weight Q6Kx8 emission because te.dtype() is no longer Q6K. Requantization should be opt-in, e.g. by defaulting this list to empty or requiring --tensors for the requant path.
Useful? React with 👍 / 👎.
|
@codex review |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Remove the CANDLE_* feature toggles that gated each optimization for same-binary A/B during review; the deploy branch runs every optimized path unconditionally (the toggles stay on the individual PR branches). Collapse the dual f32/f16 KV cache to f16-only. Pick up the odd-head-dim RoPE guard from the rope branch.
|
@codex review |
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
1 participant
No description provided.