Skip to content

[SM100] Add Kimi-K3 SiTU activation - #67

Merged
Fridge003 merged 1 commit into
devfrom
codex/add-kimi-k3-situ-activation
Jul 29, 2026
Merged

[SM100] Add Kimi-K3 SiTU activation#67
Fridge003 merged 1 commit into
devfrom
codex/add-kimi-k3-situ-activation

Conversation

@Fridge003

Copy link
Copy Markdown
Collaborator

Summary

  • add Kimi-K3 SiTU selection to the SM100 FP8/FP4 Mega-MoE L1 epilogue when kActivationClamp == 0.03125f
  • use the K3 constants beta = 4.0 and linear_beta = 25.0, bypassing the ordinary SwiGLU hard clamp for the SiTU path
  • preserve the existing limited and unlimited SwiGLU behavior for every other clamp value

Why

SGLang's Kimi-K3 integration currently applies this change at container startup through apply_deepgemm_situ_patch.py. Carrying the behavior in DeepGEMM removes the need to rewrite the installed JIT header at runtime. The sentinel clamp value also creates a distinct JIT template instantiation, leaving cached SwiGLU kernels unaffected.

Impact

Callers that pass the Kimi-K3 sentinel get beta * tanh(gate / beta) * sigmoid(gate) multiplied by linear_beta * tanh(up / linear_beta). Existing callers continue to use the original SwiGLU path.

Validation

  • compared the modified epilogue block byte-for-byte with the NEW block in the linked SGLang patch
  • verified the SiTU sentinel appears exactly once
  • git diff --check
  • SM100 JIT compilation and GPU numerical tests were not run locally because the current environment has no CUDA runtime or GPU

@Fridge003
Fridge003 marked this pull request as ready for review July 29, 2026 07:51
@Fridge003
Fridge003 merged commit 75f6062 into dev Jul 29, 2026
2 of 3 checks passed
Fridge003 added a commit that referenced this pull request Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant