Reproduction failed.

[Dragonborn1117]

Hi there,

I encountered the following issue when using CHATAFL:

json_object_array_get_idx: Assertion `json_object_get_type(jso) == json_type_array' failed.

After debugging, I found that the problem occurs because the LLM grammar parser often fails — the LLM’s output is more flexible and complex currently. When I run CHATAFL multiple times, it fails most of the time (around 9 out of 10 runs).

How can I fix this problem without modifying CHATAFL’s source code?

[Marti2203]

Hi,
Which model are you using for this? Gpt-3.5 or Gpt-4?

[Dragonborn1117]

We are using the open-source Qwen series models, which theoretically have surpassed the capabilities of the GPT-3.5 used in the paper at that time.

[Marti2203]

I agree that the Qwen models should have better responses than GPT-3.5 when it comes to some evaluations. This does not ensure that the model will be able to respond correctly to all tasks. Therefore, it may require a modification of the prompts to ensure a higher probability of getting correct responses.

[HantaoGG]

Have you solved the API issue with qwen3?

We are using the open-source Qwen series models, which theoretically have surpassed the capabilities of the GPT-3.5 used in the paper at that time.我们使用的是开源的 Qwen 系列模型，理论上这些模型已经超越了当时论文中使用的 GPT-3.5 的能力。

[Ryenum]

Hello, I encountered the same problem as you in the review process. I also used Qwen's model, but it would give an error. Later, I found that after modifying the model, not only the interaction part with the model should be modified, but also the part of the Prompt construction should be modified, because the format of Qwen and GPT are different. The specific changes are in the two functions in the chat-llm.c file, as shown in the figure below.