Skip to content

api: send the image pixels with every n>1 choice (vision-fidelity multi-choice-image red on main since #174) - #201

Merged
AzeezIsh merged 1 commit into
mainfrom
fix/multi-choice-image-pixels
Oct 21, 2026
Merged

AzeezIsh merged 1 commit into
mainfrom
fix/multi-choice-image-pixels

Conversation

@AzeezIsh

Copy link
Copy Markdown
Contributor

Summary

vision-fidelity has been red on main: integrity 9/10 on multi-choice-image, "choice(s) [1] did not answer about the image". It fails on every vision model I ran: Qwen3.6-35B-A3B-FP8 (the gate subject), Qwen3.8-Flash-Next NVFP4 and FP4-Local. Main's last committed record (.benchmarks/vision-fidelity/2026-09-29-d5472821c8.json) passed 10/10.

Bisect (sparky GB10, spark benchmark run vision-fidelity --pull-request-gate, first-parent, 2026-10-21):

Mechanism:

  • chat_blocking::run_blocking_path sends one scheduler request per choice and gave the image pixels to choice 0 only. Choices 1.. carried no media and silently prefilled their image pad tokens from choice 0's leftover encoder rows.
  • qwen3.8-flash-next: two-GB10 TP=EP=2 (Enntity series + RC16) with RDMA GID and cuBLASLt startup fixes #174's ep_broadcast_vision_state_for_seq(enabled = false) clears that per-request vision state, in single-rank serving too, for any request without media. That clearing is the right guard, since a request without media should never splice a stale image, so choice 1 now gets nothing.
  • The server logs show it: on f3ea6f0bb choice 1's first token equals choice 0's (13602); on b9a7edad8 it does not (8725).

Fix: every choice carries its own pixels and is encoded on its own (crates/spark-server/src/api/chat_blocking.rs). The benchmark's doc comment (media_integrity::multi_choice) is updated to match. n=1 is unchanged.

Evidence

  • sparky, vision-fidelity --pull-request-gate, default subject Qwen/Qwen3.6-35B-A3B-FP8, alternating, 180 s cooldown.
  • main 6b09b40aa (binary sha256 94a1b8c0a3043dfe) vs this branch (4c7fd94187268ca3).
run record integrity multi-choice-image verdict
main #1 run-1792575629041714855 9/10 FAILED FAIL
fix #1 run-1792575886530913767 10/10 both choices answered about the image PASS (14 geometry cells, 3/3 probes, control held)
main #2 run-1792576144127996550 9/10 FAILED FAIL
fix #2 run-1792576398694970202 10/10 both choices answered about the image PASS

Test plan

  • cargo fmt --all -- --check (1.93.1), cargo clippy -p spark-server --tests
  • vision-fidelity on main vs fix, ×2 each (above)
  • Required-gate campaign at the final tip

Generated with Devin

vision-fidelity `multi-choice-image` has failed on main since #174
(bisected 2026-10-21 on sparky: f3ea6f0 passes 10/10, b9a7eda is the
first bad commit, 9/10 on Qwen/Qwen3.6-35B-A3B-FP8 and on every other
vision model tried). The blocking path runs one scheduler request per
choice and gave the pixels to choice 0 only; the later choices had no
media of their own and silently reused choice 0's leftover encoder rows.
#174 made a request without media clear that per-request vision state in
single-rank serving, which is the right guard against splicing a stale
image, so choice 1 now prefills its image pad tokens with nothing and
answers as if no picture were attached. Each choice now carries its own
pixels and is encoded on its own.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant