Add experimental Qwen3.8 packed hybrid export for ORT GenAI Engine - #738
Draft
titaiwangms with Copilot wants to merge 7 commits into
Draft
titaiwangms with Copilot wants to merge 7 commits into
titaiwangms with Copilot wants to merge 7 commits into
Conversation
|
|
12 tasks
Emit native SEPARATE PagedAttention and varlen DeltaNet state, with distinct Engine and workflow state groups. Preserve dense inference and record runtime pins. Add structural, CPU projection parity, and opt-in CUDA state-carry/Engine tests; full CUDA qualification remains outstanding. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: titaiwangms <18010845+titaiwangms@users.noreply.github.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Declare native output shapes before downstream construction, reject incompatible direct tasks and RoPE configurations, and verify the 64-layer state manifest. Exercise both CLI runtime packagers and strengthen page-reuse coverage. Focused checks: 20 passed, 3 CUDA-dependent skips. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: titaiwangms <18010845+titaiwangms@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Add Qwen3.8 packed hybrid PagedAttention export for ORT GenAI Engine
Add experimental Qwen3.8 packed hybrid export for ORT GenAI Engine
Sep 16, 2026
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Performance Comparison
|
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The registered paged task cannot currently enable prefix pruning by task name, and an existing unit-test assertion is broken.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds an experimental packed hybrid Qwen3.8 export path for ORT GenAI continuous batching.
Changes:
- Adds paged attention and native packed GatedDeltaNet execution.
- Generates hybrid state-group and dynamic-batching metadata.
- Adds structural, parity, metadata, and Engine tests.
File summaries
| File | Description |
|---|---|
README.md |
Documents experimental serving workflow. |
src/mobius/__main__.py |
Routes the paged-attention feature. |
src/mobius/_builder.py |
Validates and configures packed tasks. |
src/mobius/components/__init__.py |
Exports packed context types. |
src/mobius/components/_attention.py |
Adds SEPARATE paged Qwen attention. |
src/mobius/components/_gated_deltanet.py |
Adds native packed recurrence. |
src/mobius/components/_paged_attention.py |
Defines native operator adapters. |
src/mobius/components/_rotary_embedding.py |
Supports packed MRoPE positions. |
src/mobius/integrations/_paged_hybrid.py |
Verifies packed graph ABI. |
src/mobius/integrations/onnx_genai/_schema/inference_metadata.schema.json |
Adds paged-scatter metadata schema. |
src/mobius/integrations/onnx_genai/paged_hybrid_metadata_test.py |
Tests ABI and metadata generation. |
src/mobius/integrations/onnx_genai/workflow_metadata.py |
Publishes externally scheduled workflows. |
src/mobius/integrations/ort_genai/auto_export.py |
Introspects hybrid state groups. |
src/mobius/integrations/ort_genai/genai_config.py |
Emits Engine configuration. |
src/mobius/integrations/ort_genai/genai_config_test.py |
Tests generated Engine fields. |
src/mobius/integrations/transformers/_builder.py |
Selects the packed Qwen task. |
src/mobius/models/paged_qwen35_export_test.py |
Tests packed graph structure. |
src/mobius/models/qwen35.py |
Wires packed execution through Qwen. |
src/mobius/tasks/__init__.py |
Registers the new task. |
src/mobius/tasks/_causal_lm.py |
Implements the packed decoder ABI. |
tests/integration/paged_hybrid_test.py |
Tests CUDA parity and Engine scheduling. |
Review details
- Files reviewed: 21/21 changed files
- Comments generated: 2
- Review effort level: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
|
@copilot Please fix advanced security, CIs, code-quality and reviews. |
Enable prefix pruning when the paged hybrid task is selected by its registered name and cover that dispatch path. Correct the packed MRoPE contract documentation and apply the CI lint fixes. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: titaiwangms <18010845+titaiwangms@users.noreply.github.com>
Match Ruff's formatting for the packed prefix-pruning gather after removing the unused paged-attention import. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: titaiwangms <18010845+titaiwangms@users.noreply.github.com>
Contributor
Author
Fixed in |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The documented Qwen3.8 command resolves the composite qwen3_5 type and bypasses the new paged-hybrid dispatch.
Get a fresh assessment by requesting another Copilot review.
Review details
- Files reviewed: 22/22 changed files
- Comments generated: 1
- Review effort level: Balanced
| paged_task_placeholder = isinstance(task, CausalLMTask) and getattr( | ||
| task, "_paged_cache", False | ||
| ) | ||
| if model_type == "qwen3_5_text": |
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The documented Qwen3.8 checkpoint resolves through the composite model type and bypasses the new packed-hybrid task.
Review details
Suppressed comments (1)
src/mobius/integrations/transformers/_builder.py:642
- The documented
Qwen/Qwen3.8-27Bcommand does not select this branch. That checkpoint has top-levelmodel_type="qwen3_5"with a nestedqwen3_5_textconfig;_select_primary_config()unwraps the text config but retains the top-level type, so module/task resolution has already selected the VL path before thisqwen3_5_textcheck. Route this composite checkpoint to its text backbone before_resolve_module_class()when paged attention is requested, and cover the documented command in a builder/CLI regression test.
if model_type == "qwen3_5_text":
- Files reviewed: 22/22 changed files
- Comments generated: 0 new
- Review effort level: Balanced
Verify native packed-state dataflow, operand layouts, and Phase 1 attributes before emitting runtime metadata. Route paged context explicitly, expand malformed-graph coverage, and fix the CI dependency and stale VibeVoice streaming denylist regressions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: titaiwang <titaiwang@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add an opt-in continuous-batching ABI for Qwen3.8 (
qwen3_5_text), combining paged full attention with fixed GatedDeltaNet state. Existing dense Qwen and LATENT MLA paths remain unchanged.Packed decoder
PagedHybridCausalLMTaskwith[N]tokens and[3, N]positions; no attention mask or slot mapping.Recurrent execution
[B, H_v, D_v, D_k].Runtime packaging
paged_kv,fixed_conv, andfixed_recurrentgroups, explicit state mappings, and dynamic-batching configuration.Qualification: Includes structural, numerical/state-carry, and pinned Engine probes. Native CUDA/Engine execution and full 27B hardware qualification remain outstanding; support is explicitly experimental.