Commit e0d596e
authored
Fix markdown generation crash on LLM special tokens (#4154)
The gpt-tokenizer encode() call rejects special tokens like
<|im_start|> that appear in documentation code examples.
Pass allowedSpecial: 'all' since these are content, not control tokens.1 parent aeafa91 commit e0d596e
1 file changed
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
226 | 226 | | |
227 | 227 | | |
228 | 228 | | |
229 | | - | |
| 229 | + | |
230 | 230 | | |
231 | 231 | | |
232 | 232 | | |
| |||
0 commit comments