Repository navigation
test: a practice set in the words AI assistants send, 98 parts of 64 queries - #100
Merged
Merged
Conversation
…queries Signed-off-by: alkhatibdev <me@alkhatib.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A practice set in the words AI assistants send
The tuning questions are what a person types. An AI assistant at work on code sends something
else: often a description, not a question, and often two things at once. "IP address logging
legal basis and retention" asks whether the logging is allowed and how long the logs may be
kept, and two different clauses answer it. Nothing measured that. This pull request adds a set
that does, and the little the harness needs to score it.
The set
evals/gdpr/assistant.en.yamlholds 64 queries that Claude Code typed while working throughthe coding tasks of the trigger test. They ask 98 things. It is used while improving retrieval,
as the tuning sets are, and produces no published score.
that asks two things is two entries.
topic: the group of queries and the part. A group keeps up to twowordings, the shortest and the longest that were sent, and one part asked in two wordings is
one topic.
quotegives theguideline's words that name the article.
scripts/check_eval_sources.pyreads eachguideline and finds all 98 quotes: 98 cited, none justified, none rejected.
notice states how long data is kept, and it is not a label of a part that asks how long data
may be kept.
Article 17(1)(a) for how long data may be kept.
The harness
topic.evals run --outcomeswrites it, soevals comparemovesone part asked in two wordings as one topic. A question that names none keeps the topic its
id gives, as before.
recall@5resamples topics, so two wordings ofone part do not count as two.
answered, in the top five and in the top ten.
--set assistantnames the set, as--set tuningdoes.Search itself does not change, and no published figure moves.
Tests
real set writes each entry's own topic and prints the count of whole queries.
query answered only when each part is, one label for a topic in every set that names topics,
and every set of a pack expecting only clauses the pack has.
evals/README.mddescribes the set under "What AI assistants send" and names it among thesets a change is judged on.