feat(connector): support Gemma4 hybrid KV cache

[GentleCold]

Kaon-v3/Gemma4 uses dense attention and sliding-window KV groups with different logical block sizes. The connector now plans hashes, storage queries, GPU destinations, and asynchronous save/load for each group's cadence when vLLM's Hybrid KV Cache Manager is enabled.

Sliding-window membership queries verify the retained window at the selected dense hit boundary. Partial hits shrink to a boundary supported by every group, and groups sharing a query share one lease/load entry. Sliding saves pin their source blocks until all workers report completion; null block 0 is filtered together with its hash. Full32 + Sliding16 is supported; layouts whose sliding block size does not divide the dense block size are rejected during initialization.

The implementation uses one block-size value per group, reuses the existing request-hash validation, and carries the final contiguous load window as a block range. It removes redundant per-layer size state, duplicate load-destination construction, and unreachable heterogeneous tail-save branches. Both dense and sliding layouts use the asynchronous load path.

Validation on the refactored implementation (c5dee4f):

• Python default gate with installed dependencies: 336 passed, 13 deselected. The isolated source-only gate also passed all 336 tests.
• Ruff lint/format and git diff --check passed.
• Qwen3.5-4B linear-hybrid attention E2E: 5 passed, including output comparison and cache/RPC failure checks.
• Kaon-v3 on RTX 5090, vLLM release 0.29.0, CUDA graphs enabled, vLLM prefix caching disabled, asynchronous load, GPU1/NUMA0, 256 GiB L1, L2 disabled.
• Identical 200-request cold/warm payloads, concurrency 8, 9,993 input tokens and 1 output token: 200/200 succeeded in each phase. Mean TTFT was 2,733.22 ms cold and 240.65 ms warm; p99 was 2,904.66 ms and 273.68 ms respectively.
• Warm metrics: 62,400 RAM hit blocks, 200 successful load RPCs, 41,418,752,000 bytes loaded, and no additional evictions. Resident cache used 224,919,552,000 bytes.
• Strict cold/warm output equality was 182/200. The remaining 18 differences are unresolved; this run does not establish full numerical correctness for Kaon-v3.

Benchmark scripts, launch files, logs, and results are excluded from the commits. Remote evidence: /root/kexi/kaon_validation/pr455-refactor-20260916/ on 192.168.172.86.

Related: AIINF-294.

[Copilot]

🟡 Changes recommended

Unresolved critical and moderate issues affect heterogeneous layout validation, namespace safety, lease cleanup, and save/load correctness.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds Gemma4 hybrid KV-cache support with per-group mappings, sliding-window leases, synchronous loading, and related tests/documentation.

Changes:

• Adds hybrid cache-group layout and namespace handling.
• Implements per-group query, save/load, and lease management.
• Adds synchronous-load safeguards and regression coverage.

File summaries

File	Description
python/tests/test_hybrid_group_mapping.py	Tests hybrid mapping and lease behavior.
python/tests/test_connector_fault_tolerance.py	Tests synchronous-load failures.
python/tests/test_combine_hashes.py	Covers asynchronous-mode behavior.
python/tests/test_cache_group_layout.py	Tests cache-layout validation.
python/README.md	Documents synchronous loading.
python/pegaflow/connector/worker.py	Handles per-group registration and loading.
python/pegaflow/connector/tp_shards.py	Extends query results with group metadata.
python/pegaflow/connector/scheduler.py	Plans hybrid queries, saves, loads, and leases.
python/pegaflow/connector/common.py	Defines cache layouts and intent metadata.
python/pegaflow/connector/__init__.py	Configures hybrid layouts and load scheduling.

Review details

• Files reviewed: 10/10 changed files
• Comments generated: 10
• Review effort level: Lite

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.