Why the result is not ideal?

[cjbrain]

The Craft and smelt tasks's result is shown as blew , and the agent did not move .
https://github.com/user-attachments/assets/9737c623-bded-4adf-a502-5fd76c174c9c

The mine task just going straight:
https://github.com/user-attachments/assets/39d38bb4-05d6-4e65-975f-2fed499c181e

I run this evaluate.py shown as,, is there anything wrong?
if name == "main":
parser = argparse.ArgumentParser()
parser.add_argument('--workers', type=int, default=1)
parser.add_argument('--split-number', type=int, default=6)
parser.add_argument('--env-config',"-e", type=str, default='smelt/iron_ingot') #kill/kill_zombine craft/craft_crafting_table mine/mine_stone smelt/iron_ingot
parser.add_argument('--max-frames', type=int, default=300) #vpt/test_vpt
parser.add_argument('--verbos', type=bool, default=False)
parser.add_argument('--checkpoints', type=str, default="/root/autodl-tmp/JarvisVLA-Qwen2-VL-7B")
parser.add_argument('--device',type=str,default="cuda:1")

parser.add_argument('--base-url',type=str,default ="http://localhost:8000/v1")
parser.add_argument('--video-main-fold',type=str)

parser.add_argument('--instruction-type',type=str,default='normal')
parser.add_argument('--temperature','-t',type=float,default=0.8)
parser.add_argument('--history-num',type=int,default=0)
parser.add_argument('--action-chunk-len',type=int,default=1)

args = parser.parse_args()

model_config = dict(
    temperature=args.temperature,
    history_num = args.history_num,
    instruction_type = args.instruction_type,
    action_chunk_len = args.action_chunk_len,
)
environment_config = dict(
    env_config = args.env_config,
    max_frames = args.max_frames,
    verbos = args.verbos,
)
if not args.base_url:
    args.base_url=None

if args.workers==0:
    environment_config["verbos"] = True
    video_path = f"{args.checkpoints.split('/')[-1]}-{args.env_config.split('/')[-1]}.mp4"
    evaluate(video_path=video_path,checkpoints = args.checkpoints,environment_config = environment_config,device=args.device,model_config=model_config)
elif args.workers==1:
    video_path = f"{args.checkpoints.split('/')[-1]}-{args.env_config.split('/')[-1]}.mp4"
    evaluate(video_path=f"{args.checkpoints.split('/')[-1]}-{args.env_config.split('/')[-1]}.mp4",checkpoints = args.checkpoints,environment_config = environment_config,base_url=args.base_url,model_config=model_config)
elif args.workers>1:
    multi_evaluate(args)

[MuyaoLi-jimo]

The behavior you are observing is most likely due to changes in the new version of vLLM regarding sampling: it now loads the sampling configuration provided by the model by default. The JarvisVLA‑Qwen2‑VL‑7B model adheres to the sampling settings of Qwen2‑VL, where both top‑p and top‑k are effectively configured in a greedy manner—meaning the model consistently selects the highest‑probability token, which can lead to incorrect or overly deterministic outputs.

We have already adjusted these two parameters in the code to reintroduce more randomness, you can now retry reproducing the issue.

[wuxiongbin123]

The behavior you are observing is most likely due to changes in the new version of vLLM regarding sampling: it now loads the sampling configuration provided by the model by default. The JarvisVLA‑Qwen2‑VL‑7B model adheres to the sampling settings of Qwen2‑VL, where both top‑p and top‑k are effectively configured in a greedy manner—meaning the model consistently selects the highest‑probability token, which can lead to incorrect or overly deterministic outputs.

We have already adjusted these two parameters in the code to reintroduce more randomness, you can now retry reproducing the issue.

Hello! I met the same problem where the checkpoint downloaded from huggingface cannot move or carry out any task. Could you please share us with the exact hyper parameters to fix this problem? Thank you!

[MuyaoLi-jimo]

The behavior you are observing is most likely due to changes in the new version of vLLM regarding sampling: it now loads the sampling configuration provided by the model by default. The JarvisVLA‑Qwen2‑VL‑7B model adheres to the sampling settings of Qwen2‑VL, where both top‑p and top‑k are effectively configured in a greedy manner—meaning the model consistently selects the highest‑probability token, which can lead to incorrect or overly deterministic outputs.
We have already adjusted these two parameters in the code to reintroduce more randomness, you can now retry reproducing the issue.

Hello! I met the same problem where the checkpoint downloaded from huggingface cannot move or carry out any task. Could you please share us with the exact hyper parameters to fix this problem? Thank you!

You can try using the script scripts/evaluate/rollout-kill.sh, which has already been verified to work correctly. Here are the recommended hyperparameters:

temperature = 0.7 # please higher than that
top_p = 0.99
top_k = -1
history_num = 2  # values between [2, 4] are proper

[wuxiongbin123]

The behavior you are observing is most likely due to changes in the new version of vLLM regarding sampling: it now loads the sampling configuration provided by the model by default. The JarvisVLA‑Qwen2‑VL‑7B model adheres to the sampling settings of Qwen2‑VL, where both top‑p and top‑k are effectively configured in a greedy manner—meaning the model consistently selects the highest‑probability token, which can lead to incorrect or overly deterministic outputs.
We have already adjusted these two parameters in the code to reintroduce more randomness, you can now retry reproducing the issue.

Hello! I met the same problem where the checkpoint downloaded from huggingface cannot move or carry out any task. Could you please share us with the exact hyper parameters to fix this problem? Thank you!

You can try using the script scripts/evaluate/rollout-kill.sh, which has already been verified to work correctly. Here are the recommended hyperparameters:

temperature = 0.7 # please higher than that
top_p = 0.99
top_k = -1
history_num = 2  # values between [2, 4] are proper

Thanks for your rapid reply! I`ll try it!