Skip to content

feat: the trigger test measured again with the narrowed skill, none of 45 - #108

Merged
alkhatibdev merged 1 commit into
mainfrom
trigger-measured-again
Nov 19, 2026
Merged

alkhatibdev merged 1 commit into
mainfrom
trigger-measured-again

Conversation

@alkhatibdev

Copy link
Copy Markdown
Owner

The trigger test, measured again with the narrowed skill

  • Both published arms were run again from bf5b595, with the skill as fix: the skill says personal data for what is sent to another service #107 left it ("when
    sending personal data to another service") and the instance's law chosen as fix: the trigger test's instance chooses the law its searches cover #104 does it.
    Claude Code 2.1.288 with claude-sonnet-5-5, the 30 held-out tasks, three runs each, on 18
    and 19 November.

    Group With the plugin MCP server on its own
    Coding tasks that touch personal data 36 of 36, all before the first edit 36 of 36, all before the first edit
    Coding tasks that do not 0 of 45 (October: 3 of 45) 2 of 45 (October: 0 of 45)
    Questions about the rules 9 of 9 9 of 9
  • Both arms, not only the plugin's: the October server batch measured id-check-on-wine, which
    feat: dpolens pack scaffold starts a pack, and CI checks every pack #75 reworded to id-check-on-delivery, so it cannot stand beside a batch run on today's tasks.

  • evals/results/trigger.json is written from the two batches by --write. The README's line
    and evals/trigger-test.md say what they show: the published result, what changed since
    October (the two ways in, the fix, and an earlier rerun of the same change that had one search
    it did not need, so "rarely, not never"), and the server arm now searching
    product-viewed-event in 2 of 3 runs, which the page reports without guessing at a cause: the
    instructions the server sends are the same text as in October.

A search that asks nothing is the AI assistant's, not the instance's

In the server arm, one run called search_policies with no arguments at all, before loading the
tool's description, and the server's schema refused it, as it would on any instance. October's
page already counts this as a search. Since #104, though, any errored deciding search marked the
batch as one whose instance could not answer, and --write refused it.

  • The runner now notes whether the deciding call carried its question (query for
    search_policies, key for get_clause). A call without it counts as a search, is reported
    on its own line ("a search asked no question in"), and does not stop a batch from being
    published. An error on a call that did ask something still does.
  • Calls through Qwen Code's tool_call bridge carry their arguments in their own shape, so they
    are taken as asked, as before.
  • The two batches were scored again with --reread before --write.

Checked

  • Two new tests, each failing on main: a question-less search is scored, counts, and is not a
    failure; a batch of them is published. The tests that send a failing search now give it a
    question, so they still test an instance that cannot answer.
  • ruff, ruff format --check, mypy, the import contracts, scripts/client_text.py, the API
    contract, and the whole test suite with the integration tests (1503 passed) in a clean copy,
    with caches off and GITHUB_ACTIONS=true.
  • The query log agreed with the runs' records in all but two: the question-less call (refused,
    so no row) and one plugin run whose log holds a second question that its record, cut at the
    first answer, never shows. Both are in the page.

…f 45

Signed-off-by: alkhatibdev <me@alkhatib.dev>
@alkhatibdev
alkhatibdev merged commit ccf1f39 into main Nov 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant