Skip to content

feat: the trigger test runs Qwen Code, with a model on any OpenAI-compatible server - #105

Merged
alkhatibdev merged 1 commit into
mainfrom
trigger-qwen-code
Nov 17, 2026
Merged

alkhatibdev merged 1 commit into
mainfrom
trigger-qwen-code

Conversation

@alkhatibdev

Copy link
Copy Markdown
Owner

The trigger test runs Qwen Code

The trigger test measured one AI assistant, Claude Code. This pull request adds Qwen Code 0.27.4,
talking to any server with an OpenAI-compatible API, so the same held-out tasks can show whether
a model on your own machine looks the law up before it touches personal data.

What changes

  • --client qwen-code, with --model and --base-url naming the model and the server that
    holds it, and the server's key in OPENAI_API_KEY. --sampling is sent with every request,
    and --context-window tells Qwen Code how much context the server gives the model. Both are
    kept with the batch, and a resumed batch has to match them.
  • The run container is one Dockerfile with a target for each AI assistant, both built on the
    same Larder stage. Claude Code's two scripts move to container/claude-code/, unchanged, and
    Qwen Code's sit in container/qwen-code/.
  • Qwen Code's profile is its settings file: the model at its server, DPOLens as the entry
    qwen mcp add writes, and nothing that reports usage, fetches a model catalogue or updates
    Qwen Code.
  • Qwen Code's stream is read for its version, its edit tools (edit, write_file,
    notebook_edit), and its turn limit, which it prints on stderr with no final event. It keeps
    MCP tools out of the prompt until the model looks one up, so a search can arrive through its
    tool_call tool, and counts the same.
  • --stop-at-edit ends a run at its first edit, when whether it searched before editing is
    known, which keeps runs short on a slow model. --write refuses such a batch, and any batch
    that is not Claude Code's, since evals/results/trigger.json is Claude Code's number.
  • Batches recorded before this read as Claude Code's.

Tests

  • A search through Qwen Code's tool_call is a call, and looking a tool up is not.
  • A name given to tool_call resolves the way Qwen Code resolves it.
  • Qwen Code's own edit tools are edits.
  • A run Qwen Code ended at the turn limit is scored as capped.
  • A Qwen Code run whose model request failed is not scored.
  • --stop-at-edit ends a run at its first edit, and without it the run goes on to its search.
  • Qwen Code's settings name the server, the model, the sampling and DPOLens.
  • Options that cannot make a batch start nothing.
  • --write refuses a batch stopped at its first edit and a Qwen Code batch.
  • Every AI assistant has a target in the Dockerfile and both of its scripts.

Each new behaviour, taken out in turn, fails its test.

Checked by running it

  • Two batches recorded before this change (90 and 72 runs of Claude Code) report the same
    numbers with it, and --reread changes no field of any of their outcomes.
  • End to end, with Qwen Code and Qwen3.5 4B (8-bit MLX) served locally, three held-out tasks
    with --stop-at-edit, all scored: the question searched first and its answer came back, with
    one row in the query log for its one call, and the two coding tasks edited without searching.
    Each run took 8 to 33 seconds, and the batch recorded the server, the sampling and the
    context window.
  • The Claude Code target holds what main builds: the same Claude Code, user, environment,
    global packages, app files, Larder history (the same three commits) and node_modules, and
    its two scripts byte for byte. prepare-profile.sh mcp-only runs in it. No Claude Code run was
    made with this change.

…patible server

Signed-off-by: alkhatibdev <me@alkhatib.dev>
@alkhatibdev
alkhatibdev merged commit 97a056d into main Nov 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant