Skip to content

test: a practice set in the words AI assistants send, 98 parts of 64 queries - #100

Merged
alkhatibdev merged 1 commit into
mainfrom
assistant-set
Nov 14, 2026
Merged

alkhatibdev merged 1 commit into
mainfrom
assistant-set

Conversation

@alkhatibdev

Copy link
Copy Markdown
Owner

A practice set in the words AI assistants send

The tuning questions are what a person types. An AI assistant at work on code sends something
else: often a description, not a question, and often two things at once. "IP address logging
legal basis and retention" asks whether the logging is allowed and how long the logs may be
kept, and two different clauses answer it. Nothing measured that. This pull request adds a set
that does, and the little the harness needs to score it.

The set

evals/gdpr/assistant.en.yaml holds 64 queries that Claude Code typed while working through
the coding tasks of the trigger test. They ask 98 things. It is used while improving retrieval,
as the tuning sets are, and produces no published score.

  • Each thing a query asks is a part, with an entry of its own under the same words, so a query
    that asks two things is two entries.
  • Each entry names its topic: the group of queries and the part. A group keeps up to two
    wordings, the shortest and the longest that were sent, and one part asked in two wordings is
    one topic.
  • Every label follows what an EDPB guideline says of the nearest case, and quote gives the
    guideline's words that name the article. scripts/check_eval_sources.py reads each
    guideline and finds all 98 quotes: 98 cited, none justified, none rejected.
  • A clause counts only when reading it alone answers the part. Article 13(2)(a) says a privacy
    notice states how long data is kept, and it is not a label of a part that asks how long data
    may be kept.
  • Where two clauses state one rule, both are listed and either counts: Article 5(1)(e) and
    Article 17(1)(a) for how long data may be kept.

The harness

  • A question may name its topic. evals run --outcomes writes it, so evals compare moves
    one part asked in two wordings as one topic. A question that names none keeps the topic its
    id gives, as before.
  • In a set that names topics, the interval of recall@5 resamples topics, so two wordings of
    one part do not count as two.
  • For a set that asks a query in parts, the run also prints how many queries had every part
    answered, in the top five and in the top ten.
  • --set assistant names the set, as --set tuning does.

Search itself does not change, and no published figure moves.

Tests

  • With a database: a topic asked twice is one item in the interval, and the command run on the
    real set writes each entry's own topic and prints the count of whole queries.
  • Without one: the topic a question names, the judge counting two wordings as one topic, a
    query answered only when each part is, one label for a topic in every set that names topics,
    and every set of a pack expecting only clauses the pack has.

evals/README.md describes the set under "What AI assistants send" and names it among the
sets a change is judged on.

…queries

Signed-off-by: alkhatibdev <me@alkhatib.dev>
@alkhatibdev
alkhatibdev merged commit f94acc8 into main Nov 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant