Repository navigation
feat(jetspec): register training and wire Plan, trainer and checkpoint #54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
bb09634
e1ccf1b
335a1bb
00335cd
b70c244
220441c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| { | ||
| "architectures": ["JetSpecDraftModel"], | ||
| "model_type": "jetspec", | ||
| "hidden_size": 4096, | ||
| "intermediate_size": 12288, | ||
| "num_hidden_layers": 5, | ||
| "num_attention_heads": 32, | ||
| "num_key_value_heads": 8, | ||
| "vocab_size": 151936, | ||
| "rms_norm_eps": 1e-6, | ||
| "max_position_embeddings": 40960, | ||
| "rope_theta": 1000000.0, | ||
| "num_target_layers": 5, | ||
| "target_hidden_size": 4096, | ||
| "target_num_hidden_layers": 36, | ||
| "target_layer_ids": [1, 9, 17, 25, 33], | ||
| "mask_token_id": 151669, | ||
| "block_size": 16, | ||
| "is_causal": true, | ||
| "tie_word_embeddings": false | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1055,9 +1055,24 @@ def attention_observed(item): | |
| attention_policy_observed = bool(consumers) and all( | ||
| attention_observed(item) for item in consumers | ||
| ) | ||
| if ( | ||
| attention_decision = next( | ||
| ( | ||
| planned_attention_backend in {"fa4", "cudnn_flex_attention"} | ||
| item | ||
| for item in self._plan.capabilities | ||
| if item.name == "draft_attention_execution" | ||
| ), | ||
| None, | ||
| ) | ||
| attention_planned = ( | ||
| attention_decision is not None | ||
| and attention_decision.effective_state != "omitted" | ||
| ) | ||
| if ( | ||
| attention_planned | ||
| and ( | ||
| planned_attention_backend in { | ||
| "fa4", "cudnn_flex_attention", "flex_attention", | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The JetSpec case looks good now. One small thing I noticed with plain DSpark at CP=1 is that the plan marks |
||
| } | ||
| or selected_policy_requested | ||
| ) | ||
| and attention_policy_observed | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried the default
aspec exporton a JetSpec checkpoint. It succeeded and wroteJETSPECas the speculative algorithm in the manifest, but SGLang 0.5.21 doesn’t recognize that name. Could we have--format sglangreturn a clear error for JetSpec until that export path is supported?