Repository navigation
feat: the trigger test measured again with the narrowed skill, none of 45 - #108
Merged
Merged
Conversation
…f 45 Signed-off-by: alkhatibdev <me@alkhatib.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The trigger test, measured again with the narrowed skill
Both published arms were run again from
bf5b595, with the skill as fix: the skill says personal data for what is sent to another service #107 left it ("whensending personal data to another service") and the instance's law chosen as fix: the trigger test's instance chooses the law its searches cover #104 does it.
Claude Code 2.1.288 with
claude-sonnet-5-5, the 30 held-out tasks, three runs each, on 18and 19 November.
Both arms, not only the plugin's: the October server batch measured
id-check-on-wine, whichfeat: dpolens pack scaffold starts a pack, and CI checks every pack #75 reworded to
id-check-on-delivery, so it cannot stand beside a batch run on today's tasks.evals/results/trigger.jsonis written from the two batches by--write. The README's lineand
evals/trigger-test.mdsay what they show: the published result, what changed sinceOctober (the two ways in, the fix, and an earlier rerun of the same change that had one search
it did not need, so "rarely, not never"), and the server arm now searching
product-viewed-eventin 2 of 3 runs, which the page reports without guessing at a cause: theinstructions the server sends are the same text as in October.
A search that asks nothing is the AI assistant's, not the instance's
In the server arm, one run called
search_policieswith no arguments at all, before loading thetool's description, and the server's schema refused it, as it would on any instance. October's
page already counts this as a search. Since #104, though, any errored deciding search marked the
batch as one whose instance could not answer, and
--writerefused it.queryforsearch_policies,keyforget_clause). A call without it counts as a search, is reportedon its own line ("a search asked no question in"), and does not stop a batch from being
published. An error on a call that did ask something still does.
tool_callbridge carry their arguments in their own shape, so theyare taken as asked, as before.
--rereadbefore--write.Checked
main: a question-less search is scored, counts, and is not afailure; a batch of them is published. The tests that send a failing search now give it a
question, so they still test an instance that cannot answer.
ruff format --check, mypy, the import contracts,scripts/client_text.py, the APIcontract, and the whole test suite with the integration tests (1503 passed) in a clean copy,
with caches off and
GITHUB_ACTIONS=true.so no row) and one plugin run whose log holds a second question that its record, cut at the
first answer, never shows. Both are in the page.