Repository navigation
feat(gemma4e-dflash-flash): nemotron3.5-bolt-dflash-flash:e4b (single-turn flash + DFlash speculation) - #899
Merged
Merged
Conversation
…ed system prefix With think on, Gemma4e::insert() took a legacy checkpoint after every prefill. A flash model keeps its pinned system prefix in that one checkpoint, so the first request overwrote it with the whole first prompt and every later request restored into that stale context: repeated requests of gemma4e-flash gave different answers. Skip the checkpoint when the model is single-turn. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…-turn flash serving with DFlash speculation New family gemma4e-dflash-flash: the gemma4e-flash prefill (fused_prefill overlay, pinned 1k context, system prompt pinned and reused) together with the DFlash speculative decode of nemotron3.5-bolt-dflash:e4b. - Gemma4e_DFlash_Flash derives from Gemma4e_Flash; the speculative loop is now a member template of Gemma4e shared with Gemma4e_DFlash (single_turn: no end-of-turn checkpoint, no KV row for the stop token). The multi-slot prefix cache stays off (the pin owns the KV). - Catalog entry on the existing packages (base nemotron3.5-bolt:e4b + the DFlash draft package): no new download. - Engine header, registration (all_models, builtin_backends, CMake link), Inno and WiX installer entries, CI maps, test rig and a catalog test. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Card with measured numbers on the DFlash page, the Flash page, the flash example and request-parameter tables in the server docs, prefix-cache note, and rows on the flash benchmark page. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…efill.xclbin from IRON CI run 41132815890 Engine: src/lib/xrt (linux-libs + windows-libs) and src/lib/hrx (HRX-linux-libs + HRX-windows-libs): libgemma4e_dflash_flash.so, gemma4e_dflash_flash.dll, gemma4e_dflash_flash.lib. Kernel: src/xclbins/Nemotron-3.5-Bolt-E4B-DFlash-NPU2/fused_prefill.xclbin (xclbins-Gemma4_fused_prefill). IRON PR AMD-ROCm-Internal/FastFlowLM_IRON#129, PR Check run 41132815890. sha256 match the run's artifacts; verified on Linux/XRT with the CI .so (byte-identical to the local build) and the full CI xclbin set. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…he 70-token image budget) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
tawei-amd
force-pushed
the
feat/bolt-dflash-flash
branch
from
October 31, 2026 01:04
06b0a27 to
10c968b
Compare
tawei-amd
marked this pull request as ready for review
October 31, 2026 01:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
nemotron3.5-bolt-dflash-flash:e4b: single-turn flash serving (1k context, system prompt pinned and reused) with DFlash speculativedecoding. It is
nemotron3.5-bolt-dflash:e4bwith the prefill ofgemma4e-flash:e4b. IRON PR: AMD-ROCm-Internal/FastFlowLM_IRON#129.No new download. The catalog entry uses the two packages that already exist: base
nemotron3.5-bolt:e4band the draft packageNemotron-3.5-Bolt-E4B-DFlash-NPU2(base_model). Only one new xclbin is needed in that package's xclbin folder:fused_prefill.xclbin.Changes (4 commits)
fix(gemma4e): withthinkon,Gemma4e::insertre-checkpointed after prefill and overwrote the pinned system prefix of flash models, sorepeated requests to the released
gemma4e-flashtags gave different answers (shown with a system prompt + request sequence A, B, A, B, A,think on). Skipped for single-turn models. Also fixes the released flash tags.
feat(gemma4e-dflash-flash): familygemma4e-dflash-flash,Gemma4e_DFlash_Flash : Gemma4e_Flash; the speculative loop is now a membertemplate of
Gemma4eshared byGemma4e_DFlash(unchanged behaviour:single_turn=false) and the new class; catalog +model_info.jsonentry; engine header, registration, CMake link; Inno + WiX entries; CI maps; test rig; catalog test.
docs: model card (section on the DFlash page), Flash page, server docs lists, flash benchmark rows.build: the vendored CI binaries (below).Vendored binaries (CI-built, verified locally)
IRON run: 41132815890 (PR Check,
feat/bolt-dflash-flash, head 2f4b93405).src/lib/hrx/libgemma4e_dflash_flash.soHRX-linux-libs3af34dbc5fa7651bd8584b762234c42f2fba0459a9c3e0571be6188186e10770src/lib/hrx/gemma4e_dflash_flash.dllHRX-windows-libs3c216f3827aad9ffc53cba8e1577912f87e5369f744f3b643a6a299545d17af6src/lib/hrx/gemma4e_dflash_flash.libHRX-windows-libsf2c1cb262af189fea27ade1c02966f872439f5682242e541e54d81749060e107src/lib/xrt/libgemma4e_dflash_flash.solinux-libs1464d7009dd4bd99fb729f5d8ef9882b3754da06e7fa215a5de36d9638ba3599src/lib/xrt/gemma4e_dflash_flash.dllwindows-libs72649b114766c149f4fc59aa8492b0628a5b7a44a9a66f077a414fd507c2ffaasrc/lib/xrt/gemma4e_dflash_flash.libwindows-libsb9e9517a018c0a8239b8e36ca66277d84385d1fca5575723e867aaec9a892bb7src/xclbins/Nemotron-3.5-Bolt-E4B-DFlash-NPU2/fused_prefill.xclbinxclbins0e1d502f6fa04e8ab0db5b7d156311a0a03f547a054733daf2f320a9c18dfaa6Verified on Linux/XRT with these exact files: contract test (20 checks), pin test think on/off, warm-restore A/B, goldens compare with my local build, perf and soak. HRX run from the HRX artifacts of this PR's CI.
Tests
src/test/phi4_corecatalog test builds and passes with the new assertions (shared base and package, family, 1k context,single-turnlabel, notool-calling)..github/script/ci/test_changes.py: same single failure as on untouchedmain(minicpm-v-4.7-moenot named in the tables), nothing new.gemma4e-flash:e4b), decode 33-40 tok/s on code, 19-21 on prose, speculation off == flash.Not covered here
Windows runtime test (CI builds the MSI/portable; a manual run is needed before release).
flm_min_versionstays1.0.12(version of the package); the card says 1.0.13 or later: to be set by the release.🤖 Generated with Claude Code