Hybrid recurrent × M-RoPE vision: recurrent memory keys cells on positions an image repeats — warn storm at pin and master; upstream crash class (llama.cpp#27931)

[lloyal-research]

Summary

Every image prefill on a hybrid attention+recurrent model (Qwen3.5: full-attention layers + Gated DeltaNet layers) triggers a warning storm from llama_memory_recurrent::find_slot, at our pin and at current llama.cpp master. The mechanism: M-RoPE assigns image rows a frozen temporal position and resumes text at base + max(nx, ny), while the recurrent memory keys its per-sequence cell bookkeeping on consecutive positions. Upstream tracks the same signature escalating to hard crashes on hybrid+vision models (ggml-org/llama.cpp#27931, open), has fixed the mechanism only for the newer qwen4exp arch in a new memory class, and left llama-memory-recurrent — the class Qwen3.5 uses — untouched. Nothing in liblloyal causes this (our position geometry is byte-identical to upstream's own mtmd helper), but every liblloyal deployment running Qwen3.5 with images sits on this seam. This issue is the reference record: exact scenario, call stacks at both revisions, mechanism, consequence spectrum, and options.

Observed scenario (2026-09-01, first-hand)

Qwen3.5-4B (Q4_K_M) + mmproj-F16 (qwen3vl_merger), Metal, macOS, 16 GB unified memory. One user turn with one attached image entering via the trunk multimodal prefill at session start (templated prompt ~444 tokens + ~580 image rows; n_batch 512). Server stderr, verbatim:

find_slot: non-consecutive token position 9 after 8 for sequence 0 with 512 new tokens
find_slot: non-consecutive token position 9 after 8 for sequence 0 with 512 new tokens
find_slot: non-consecutive token position 9 after 9 for sequence 0 with 512 new tokens
find_slot: non-consecutive token position 9 after 9 for sequence 0 with 512 new tokens
find_slot: non-consecutive token position 48 after 9 for sequence 0 with 8 new tokens
find_slot: non-consecutive token position 48 after 9 for sequence 0 with 8 new tokens

Read the three shapes against the mechanism below: image-row ubatches whose rows all carry temporal position 9 (9 after 9), the first of them following text that ended at position 8 (9 after 8 — the expectation is cell.pos + n_seq_tokens, i.e. 8 + 512, not 9), and text resuming at 9 + max(nx,ny) = 48 where the recurrent cache expected 9 + 8 = 17.

The run itself proceeded correctly for ~150 s (11 grounded research turns). It then died at ~288 s to a Metal kIOGPUCommandBufferCallbackErrorOutOfMemory → sticky backend error state ("backend is in error state from a previous command buffer failure - recreate the backend to recover") → llama_decode: failed to decode, ret = -3 on every subsequent call until process restart. Attribution of that crash is dual-hypothesis and unresolved: (a) concurrent heavy processes on the same unified-memory box (timing correlates exactly), or (b) #27931's long-session mode, where hybrid+vision sessions "eventually crash the same way" after warn storms. Discriminator: re-run an image session on an idle box.

Call stack at our pin (b9581 / llama.cpp d6d0ce821)

Branch.prefillMultimodal / BranchStore.prefillMultimodal        (sdk)
  → _storePrefillMultimodal                                     (lloyal.node binding)
    → StorePrefillMultimodalWorker::Execute                     SessionContext.cpp:737
      → BranchStore::decode_segments                            liblloyal branch.hpp:1296
        → TEXT segments: decode::scatter / decode::many         decode.hpp (llama_decode at :193/:260)
        → IMAGE segments: decode::embd                          decode.hpp:625
          (positions from MtmdSource::positions(), mtmd.hpp:234
           → mtmd_helper_image_get_decoder_pos, section-major t/y/x/z)
          → llama_decode
            → llama_context::decode → memory->init_batch        llama-context.cpp:1735
              → llama_memory_hybrid::init_batch                 llama-memory-hybrid.cpp:67
                → mem_recr->prepare                             llama-memory-hybrid.cpp:105
                  → llama_memory_recurrent::prepare             llama-memory-recurrent.cpp:463
                    → llama_memory_recurrent::find_slot         llama-memory-recurrent.cpp:487
                      → WARN at :644, then cell.pos = last_pos  (tolerate-and-overwrite)

The warn site, verbatim (identical at pin and master):

if (cell.pos >= 0 && last_pos != cell.pos + (llama_pos) n_seq_tokens) {
    // What should happen when the pos backtracks or skips a value?
    // Clearing the state mid-batch would require special-casing which isn't done.
    LLAMA_LOG_WARN("%s: non-consecutive token position %d after %d for sequence %d with %u new tokens\n",
        __func__, last_pos, cell.pos, ubatch.seq_id[i][0], n_seq_tokens);
}
cell.pos = last_pos;

Same stack at master (9d817213a, 1160 commits past the pin)

Byte-identical warn block at llama-memory-recurrent.cpp:659-664. Qwen3.5 still constructs plain llama_memory_hybrid (llama-model.cpp:2531; the comment at :2459 is explicit: "only the sparse-attention architectures use llama_memory_hybrid_idx"). Nothing on master changes this path for Qwen3.5.

Mechanism

M-RoPE gives image rows 4-section positions where the temporal section is frozen across the image and text resumes at base + max(nx, ny) (mtmd_image_tokens_get_n_pos, mtmd.cpp:1726 at master — returns max(nx, ny) for MTMD_POS_TYPE_MROPE). liblloyal reproduces this exactly: MtmdSource::positions() (mtmd.hpp:234) calls the same mtmd_helper_image_get_decoder_pos and emits the same section-major layout as upstream's set_position_mrope_2d (mtmd-helper.cpp:172). Any hybrid+vision decode — llama-server included — feeds the recurrent memory these positions.

The recurrent memory's find_slot expects last_pos == cell.pos + n_seq_tokens per sequence. Image geometry violates that three ways (the three log shapes above). The code warns and overwrites; upstream's own comment admits the unresolved design question.

Why generation still works (and where the real risk sits)

• The DeltaNet recurrence never reads positions: src/models/qwen35.cpp:167-173 — the recurrent branch is build_layer_attn_linear(inp->get_recr(), cur, il) with no inp_pos; positions + M-RoPE sections feed only the full-attention branch. State evolves in batch order regardless of pos labels.
• The bookkeeping converges: after the final text ubatch, cell.pos holds the true last position; subsequent decode is linear from there.
• seq_cp (fork) aliases the tail cell wholesale and ignores the position-range arguments — so branching does not pass through the broken bookkeeping.
• Empirical: liblloyal's committed multimodal_integration_test.cpp:358-470 runs this exact warn-generating geometry (image → fork ×4 → generate) and asserts zero-cell forking plus vl_strict grounded answers — green on the Qwen3.5-4B tier.

The risk is therefore not the math but the cell bookkeeping under stress: upstream #27931 reports the identical warn signature escalating to STACK_OVERFLOW / ACCESS_VIOLATION inside ggml-base on hybrid+mmproj (Ornith-1.5-35B, qwen3_5_moe arch), 100 % reproducible with -ub 1024 on the first image turn, surviving with -ub 128 short-term but "eventually crashing the same way" in long alternating text/image sessions. Reproduced by the reporter on current master. Related: #27408 (mtmd image chunks leave positional holes; "nothing here is claimed as fixed upstream"), #28007 (open workaround: full re-process when recurrent rollback fails), #25717 (closed not_planned).

Upstream fix landscape

The mechanism has been fixed upstream once, for the wrong arch: 36b101543 (#27941, qwen4exp) reworks cell selection in the new llama-memory-hybrid-idx class, and names the mechanism in its own diff:

"mrope repeats one position across an image, so rank cells instead of using the position"

llama-memory-recurrent did not receive that treatment. A pin bump does not fix this — it buys the mtmd batching API and recurrent-rollback infrastructure (ggml_ssm_scan rollback #26623, n_keep_tail ubatch splitting #25278), which are wins, but the Qwen3.5 hybrid×vision seam is identical at master.

Reproduction

1. Any liblloyal-based host: Qwen3.5-4B + its mmproj, submit one turn with one image through the multimodal prefill. The warn storm is deterministic; count scales with image ubatches (2 per 512-row ubatch here) + 1 per resuming text segment.
2. No liblloyal code needed: #27931's repro-server-crash.sh against stock llama-server reproduces both the storm (soak) and the crash (killer, -ub 1024).

Options

1. Wait/track: upstream migrates qwen35-class archs to hybrid-idx (the qwen4exp precedent suggests this trajectory), or patches llama-memory-recurrent. Watch #27931 at every pin bump.
2. Vendored patch: port the "rank cells, not positions" selection into llama-memory-recurrent::find_slot — surgical, design proven in-tree upstream.
3. Mitigations meanwhile: keep n_batch ≤ 512 for image decodes (#27931's matrix: crash at 1024, survival at 128); treat long multi-image alternating text/image sessions as at-risk; on Metal, remember the backend error state is sticky — a host that logged the OOM must be restarted, every later decode fails with ret -3.

Cross-references

• Pin: llama.cpp d6d0ce821 (b9581) · Master surveyed: 9d817213a (2026-09-01)
• Upstream: ggml-org/llama.cpp #27931, #27408, #28007, #25717, fix-for-qwen4exp #27941 (36b101543)
• liblloyal: mtmd.hpp:234 (positions()), branch.hpp:1296 (decode_segments), tests/integration/multimodal_integration_test.cpp:358 (fork-after-image, green with the storm live)

Tracker

• Upstream ggml-org/llama.cpp#28166 triaged / acknowledged
• Upstream ggml-org/llama.cpp#27931 (crash class) resolved
• Discriminator run: one image session on an idle box, to attribute the 2026-09-01 crash (memory pressure vs #27931 long-session mode)
• Decision: wait for upstream vs vendored rank-cells patch in llama_memory_recurrent::find_slot
• Fork-fidelity suite gains the hybrid-recurrent cases: fork-after-image vs unforked baseline; text-after-image equivalence (rides the held suite, lands with the pin bump)
• At next pin bump: re-check the warn region, qwen35 memory-class routing, and whether qwen35-class archs migrated to hybrid-idx

[lloyal-research]

Filed upstream as ggml-org/llama.cpp#28166 (warning mechanism confirmed backend-, frontend- and variant-independent; diagnostics offered).