Some issues encountered during inference

[cnyvfang]

Hi, thank you for your excellent work!

I am trying to inspect the intermediate process for each case, but I noticed some behavior that I am not sure is expected. (The final result does not seem to have reached expectations.). I am not sure whether this is caused by my configuration or usage.

For example, in the case shown in the screenshot, the main model seems to output a tool call directly without first reasoning about the input query. Also, the sub-agent does not seem to analyze or summarize the cropped content, and instead outputs unrelated or meaningless text.

Could you please help check this issue? It would also be very helpful if you could provide a script or option to save the intermediate chat content, including the main model outputs, tool calls, sub-agent responses, and final answers.

Thank you again for your great work!

[mwxely]

Thanks for the detailed report and the kind words!

TL;DR: the model does reason — the released eval driver just does not save the intermediate turns. I have opened a Pull Request #2 to add a --save_trajectory flag that dumps the full transcript so you can see exactly what happens on your runs.

Where the content goes, and where it was dropped

• The agentic loop in paravt/eval/driver.py (eval_one) builds a multi-turn messages list and sends it in full each turn. The model's reply — including its <think>…</think> reasoning and <tool_call> blocks — is read back from message.content and appended to messages. We do not launch vLLM with a reasoning parser, so <think> stays inline in .content and is not split off into a separate reasoning_content field. So the reasoning is generated and held in memory during inference/eval — nothing about it is special-cased or stripped.
• The loss happens only at save time: _make_result kept the scalar fields (pred, score, tool_calls, and full_response = final[:500] — the last turn truncated to 500 chars; plus pred_text for non-MCQ) but discarded the whole multi-turn messages transcript. The intermediate <think> turns, the <tool_call> arguments, and the <tool_response> crop turns were simply never serialized.
• This is also the likely source of "the main model outputs a tool call without first reasoning": when a sample is still calling tools at the second-to-last turn, the loop appends a forced closer — the prompt is literally "...answer with ONLY the letter." with max_tokens=64 — and that short closer becomes the saved full_response, so it looks reasoning-free even though the model reasoned in the earlier (discarded) turns. (If the model instead stops calling tools and answers on its own before the cap, that self-generated answer turn — reasoning included — is what gets saved.)

About the "sub-agent"

In this eval driver there is no second model instance. The parallel sub-agent dispatch from the method is realized by feeding each crop_video result back to the same model (same weights) as the next <tool_response> user turn — so the "sub-agent analysis of a crop" is just the model's next turn over the returned frames. If that turn looks off, it's the same model reasoning over the frames that went back, and the new trajectory dump lets you read exactly which frames went back and what the model said about them.

How to use it

After the PR lands (or from the branch):

python -m paravt.eval.driver \
    --model_path ParaVT/ParaVT-8B \
    --datasets videomme --output_dir ./eval-results/paravt \
    --prompt_mode agentic_general --save_trajectory --limit 5

Each result row then gains a trajectory field: an ordered list of {role, content} turns covering every main-model turn (<think>/<tool_call>), every <tool_response> crop turn, and the final answer. Base64 frames are collapsed to <N frame(s) omitted> markers so the JSON stays small. The flag is off by default and purely additive — it does not change pred/score/tool_calls/full_response, so reported numbers are unaffected.