Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 40 additions & 14 deletions evals/trigger/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

The retrieval evals measure whether search finds the right clause once an AI assistant asks. This
measures whether it asks: how often Claude Code calls DPOLens on its own when the work in front
of it touches personal data, and how often it calls when the work does not.
of it touches personal data, and how often it calls when the work does not. It also runs Qwen Code,
with any model served through an OpenAI-compatible API, such as a small model on your own machine.

## What is here

Expand All @@ -11,7 +12,7 @@ of it touches personal data, and how often it calls when the work does not.
| `tuning.yaml` | 20 tasks used while improving the guidance DPOLens gives an AI assistant |
| `held_out.yaml` | 30 tasks that produce the published number and are never used to change anything |
| `app/` | Larder, the small TypeScript shop every task runs against |
| `container/` | The container one run happens in |
| `container/` | The container one run happens in, one for each AI assistant |
| `../../scripts/run_trigger_test.py` | The runner |

## A task
Expand Down Expand Up @@ -48,26 +49,33 @@ this compliant" would trigger any AI assistant and the number would mean nothing

## What a run is

Every run is a new container: Claude Code at a pinned version, a fresh checkout of Larder with
its dependencies installed, and an empty profile with DPOLens installed the way
Every run is a new container: the AI assistant at a pinned version (Claude Code 2.1.288 or
Qwen Code 0.27.4), a fresh checkout of Larder with its dependencies installed, and an empty
profile with DPOLens installed. Claude Code gets it the way
[Get started](https://dpolens.com/docs/get-started) installs it, from the plugin marketplace,
with `claude plugin configure`. Nothing else is in the
profile: no `CLAUDE.md`, no memory, no other plugins. The runs talk to a throwaway DPOLens
instance built from your checkout, with every pack in it loaded, which is removed afterwards.
with `claude plugin configure`. Qwen Code gets the MCP server on its own, as the entry
`qwen mcp add` writes, beside settings that name the model and its server. Nothing else is in the
profile: no `CLAUDE.md` or `QWEN.md`, no memory, no other plugins. The runs talk to a throwaway
DPOLens instance built from your checkout, with every pack in it loaded, which is removed
afterwards.

A run counts as a call when the AI assistant calls `search_policies` or `get_clause`, the two
tools that return clause text. It counts as a call before editing when that happens before its
first `Edit`, `Write`, `MultiEdit` or `NotebookEdit`. A run stops as soon as the first call
comes back, since the outcome is known by then. Otherwise it goes until the AI assistant finishes,
25 turns or 10 minutes.
first edit: `Edit`, `Write`, `MultiEdit` or `NotebookEdit` in Claude Code, and `edit`,
`write_file` or `notebook_edit` in Qwen Code. Qwen Code keeps MCP tools out of the prompt until
the model looks one up, so a search can also come through its `tool_call` tool, and counts the
same. A run stops as soon as the first call comes back, since the outcome is known by then.
Otherwise it goes until the AI assistant finishes, 25 turns or 10 minutes. With
`--stop-at-edit`, a run also stops at its first edit, when whether it searched before editing is
known: runs stay short on a slow model, and the batch cannot say what a run would have done after.

A run that could not have called DPOLens is retried rather than scored: the plugin not loading,
the MCP server not connecting, or the request to the model failing. Each run's calls are also
counted in the instance's query log, and the report names any run where the two counts differ.

## Running it

You need Docker and either a Claude subscription or an API key.
For Claude Code you need Docker and either a Claude subscription or an API key.

```bash
claude setup-token # prints a one-year token for your subscription
Expand All @@ -93,9 +101,10 @@ of every run, and the full stream of events from each one, which is where to loo
did something unexpected. A batch stopped by a usage limit resumes with
`--batch <batch>`, and `--report <batch>` scores one again without running anything.

`--write held_out-plugin,held_out-mcp-only` writes `evals/results/trigger.json` from held-out
batches, and refuses one that is incomplete, ran on an unpinned model, or disagrees with the
others about the model or the version of Claude Code.
`--write held_out-plugin,held_out-mcp-only` writes `evals/results/trigger.json`, Claude Code's
number, from held-out batches, and refuses one that is incomplete, ran on an unpinned model,
stopped its runs at their first edit, or disagrees with the others about the model or the version
of Claude Code.

Building the instance asks the image registries about base images, so a network that cannot
reach them stops a batch before it starts. `--reuse-images` starts from the images already
Expand All @@ -111,6 +120,23 @@ the rule as it now stands to each run's saved stream of events. What was recorde
in `outcomes.recorded.jsonl`, and any run that should not have been scored is run again by the
next resume.

### With Qwen Code

Qwen Code talks to any server with an OpenAI-compatible API whose model can call tools, such as
llama.cpp, LM Studio, Ollama, oMLX or vLLM. Name the model as the server lists it, and the server
as the container reaches it: your own machine is `host.docker.internal`.

```bash
export OPENAI_API_KEY=<the server's key> # any value when it needs none
uv run python scripts/run_trigger_test.py --client qwen-code --set held_out --max-runs 30 \
--model <id> --base-url http://host.docker.internal:<port>/v1 \
--sampling temperature=0.6,top_p=0.95,top_k=20 --context-window 262144 --stop-at-edit
```

`--sampling` is sent with every request in place of the server's own choice, so give the values
the model's card recommends. `--context-window` tells Qwen Code how much context the server gives
the model. Both are kept with the batch, and resuming it takes the same.

## Reading the number

The report gives two rates: the share of runs that searched on tasks that should, and on tasks
Expand Down
63 changes: 46 additions & 17 deletions evals/trigger/container/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,32 +1,24 @@
# Where one trigger run happens: Claude Code at a pinned version, as an
# ordinary user, in a checkout of the Larder app with its dependencies
# Where one trigger run happens: an AI coding assistant at a pinned version, as
# an ordinary user, in a checkout of the Larder app with its dependencies
# installed, the way a developer's machine would have it.
#
# Built by scripts/run_trigger_test.py with evals/trigger as the context.
# Built by scripts/run_trigger_test.py with evals/trigger as the context, one
# target for each AI assistant: claude-code or qwen-code. Both start from the
# same checkout.

FROM node:24-bookworm-slim

ARG CLAUDE_CODE_VERSION=2.1.288
FROM node:24-bookworm-slim AS larder

RUN set -eux; \
apt-get update; \
apt-get install -y --no-install-recommends ca-certificates git openssl; \
rm -rf /var/lib/apt/lists/*; \
npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}"; \
claude --version

COPY --chmod=755 container/prepare-profile.sh container/run-task.sh /usr/local/bin/

# The pinned version is the one measured, so nothing may update it mid-batch.
ENV DISABLE_AUTOUPDATER=1 \
CLAUDE_CONFIG_DIR=/home/node/.claude
rm -rf /var/lib/apt/lists/*

USER node
WORKDIR /home/node/larder

# A named volume mounted here starts out owned by this user, which is how the
# prepared profile reaches every run read-only.
RUN mkdir -p /home/node/.claude /home/node/profile
RUN mkdir -p /home/node/profile

COPY --chown=node:node app/ ./

Expand All @@ -37,7 +29,7 @@ RUN set -eux; \
npx prisma generate; \
npm run typecheck

# A short history, because a real checkout has one and Claude Code reads it.
# A short history, because a real checkout has one and the AI assistant reads it.
RUN set -eux; \
git init -q -b main; \
git config user.name "Sam Okafor"; \
Expand All @@ -56,3 +48,40 @@ RUN set -eux; \
git add -A; \
commit "2026-09-19T11:42:00Z" "Orders, with order_placed sent to Segment"; \
test -z "$(git status --porcelain)"

FROM larder AS claude-code

ARG CLAUDE_CODE_VERSION=2.1.288

USER root
RUN set -eux; \
npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}"; \
claude --version

COPY --chmod=755 container/claude-code/ /usr/local/bin/

# The pinned version is the one measured, so nothing may update it mid-batch.
ENV DISABLE_AUTOUPDATER=1 \
CLAUDE_CONFIG_DIR=/home/node/.claude

USER node
RUN mkdir -p /home/node/.claude

FROM larder AS qwen-code

ARG QWEN_CODE_VERSION=0.27.4

USER root
RUN set -eux; \
npm install -g "@qwen-code/qwen-code@${QWEN_CODE_VERSION}"; \
qwen --version

COPY --chmod=755 container/qwen-code/ /usr/local/bin/

# Every run approves every tool call, so the notice saying so would only repeat
# itself in each transcript. The pinned version is the one measured: the
# settings each run starts from keep it from updating itself.
ENV QWEN_CODE_SUPPRESS_YOLO_WARNING=1

USER node
RUN mkdir -p /home/node/.qwen
19 changes: 19 additions & 0 deletions evals/trigger/container/qwen-code/prepare-profile.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#!/usr/bin/env bash
# Write Qwen Code's settings into an empty profile: the model server it talks
# to, and DPOLens as an MCP server on its own, the entry `qwen mcp add` writes.
#
# prepare-profile.sh mcp-only
#
# The settings arrive on stdin as JSON, so the token is in no argument list.
set -euo pipefail

setup="${1:?which setup: mcp-only}"
if [[ "$setup" != mcp-only ]]; then
echo "Qwen Code has no DPOLens plugin; the setup is mcp-only" >&2
exit 2
fi

mkdir -p "$HOME/.qwen"
cat > "$HOME/.qwen/settings.json"

cp -a "$HOME/.qwen"/. /home/node/profile/
14 changes: 14 additions & 0 deletions evals/trigger/container/qwen-code/run-task.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/usr/bin/env bash
# One trigger run in Qwen Code: a fresh copy of the prepared profile, then one prompt.
#
# The prompt and the turn limit arrive as TASK_* variables, and the model is
# named in the settings. The stream of events Qwen Code prints is the whole
# record of the run.
set -euo pipefail

cp -a /home/node/profile/. "$HOME/.qwen"/

exec qwen "${TASK_PROMPT:?}" \
--output-format stream-json \
--approval-mode yolo \
--max-session-turns "${TASK_MAX_TURNS:-25}"
Loading
Loading