geniex-bench: honour --prompt-file in the VLM run loop - #1278
Merged
Conversation
Paul Zhu (vinovo)
force-pushed
the
fix/vlm-bench-prompt-file
branch
3 times, most recently
from
August 4, 2026 00:16
12cd393 to
4233bb6
Compare
The VLM run loop previously ignored --prompt-file and always fed the
hardcoded VLM_DEFAULT_PROMPT ("Describe the image.") through the
bundle's chat template, regardless of what text was supplied on the
command line. This broke the accuracy-eval path: the device script
calls geniex-bench --accuracy --prompt-file <prompt_NNN.txt> once
per dataset prompt, but every invocation produced a response to the
same fixed string instead of the actual dataset question.
Fix: mirror the LLM loop's prompt_buf handling in run_vlm. When
--prompt-file is supplied the file content (a pre-templated string,
chat template applied host-side) is fed verbatim as prompt_utf8,
bypassing build_vlm_prompt entirely. "---"-delimited multi-segment
files are supported the same way as in the LLM loop. When no
--prompt-file is given the existing behaviour is preserved: the fixed
default text is run through build_vlm_prompt so image tokens are
placed correctly by the bundle's chat template.
The VLM reset-between-runs logic is updated to also reset at the start
of each new prompt segment, matching the LLM loop's behaviour.
Signed-off-by: Paul Zhu <paulzhuzx@gmail.com>
Paul Zhu (vinovo)
force-pushed
the
fix/vlm-bench-prompt-file
branch
from
August 4, 2026 00:18
4233bb6 to
49467ea
Compare
Paul Zhu (vinovo)
marked this pull request as ready for review
August 4, 2026 00:28
Mengsheng Wu (mengshengwu)
approved these changes
Aug 4, 2026
RemiliaForever (RemiliaForever)
approved these changes
Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
geniex-bench --accuracy --prompt-filewas silently ignoring the supplied prompt file for VLM models. The VLM run loop always calledbuild_vlm_promptwith the hardcodedDescribe the image.string, so every invocation produced a response to that fixed text regardless of what the file contained.Root cause
run_vlmhad no handling foro->prompt_buf(the--prompt-filecontent). The LLM loop has full support forprompt_buf-- splitting on---separators, feeding each segment asprompt_utf8-- but that logic was never ported to the VLM loop.Fix
Mirror the LLM loop's
prompt_bufhandling inrun_vlm:--prompt-fileis supplied, the file content (a pre-templated string, with the chat template applied by the caller) is fed verbatim asprompt_utf8, bypassingbuild_vlm_promptentirely.----delimited multi-segment files are supported the same way as in the LLM loop.--prompt-fileis given, the existing behaviour is preserved: the fixedVLM_DEFAULT_PROMPTis run throughbuild_vlm_promptso image tokens are placed correctly by the bundle's chat template.The VLM reset-between-runs logic is also updated to reset at the start of each new prompt segment, matching the LLM loop.
Testing
--prompt-file): behaviour unchanged.--accuracy --prompt-file: each invocation now responds to the actual prompt in the file rather thanDescribe the image..--prompt-file(separated by---): each segment runs as its own prompt with a KV-cache reset between them.