Skip to content

fix: the trigger test's instance chooses the law its searches cover - #104

Merged
alkhatibdev merged 1 commit into
mainfrom
trigger-chooses-law
Nov 17, 2026
Merged

alkhatibdev merged 1 commit into
mainfrom
trigger-chooses-law

Conversation

@alkhatibdev

Copy link
Copy Markdown
Owner

The trigger test's instance chooses the law its searches cover

Since #42 an instance searches nothing until somebody chooses the laws that apply to the
organisation, and refuses every search until then. The trigger test's throwaway instance never
chose any, so every search_policies call in a run came back as an error: "nobody has chosen
which laws apply to this organisation yet, so nothing is searched." Nothing in the report showed
it: a run stops at its first answer, and an error is an answer.

What a run measures still held. It records the decision to search, and that decision comes
before the answer: in the batches run since #42, the only DPOLens call before the first search
was list_documents, which never failed. The published numbers were measured on 8 and 9
October, before #42.

It also explains the query log disagreeing with the stream in those batches: a refused search is
refused before it is logged, by design.

What changes

  • The instance chooses GDPR when it starts (dpolens laws select gdpr), as the published runs
    searched it.
  • A run whose deciding search came back as an error says so: search_failed in its outcome, and
    the report names it.
  • --write refuses a batch with such a run, since its instance could not answer.
  • evals/trigger/README.md says both.

Batches recorded before this read as having no failed search.

Tests

  • A search that came back as an error still counts as the decision to search, and the score
    names the run.
  • The instance chooses the law, read from the commands it sends when it starts.
  • --write refuses a batch whose searches came back as errors.

Each fails without its change.

Signed-off-by: alkhatibdev <me@alkhatib.dev>
@alkhatibdev
alkhatibdev merged commit 934c7da into main Nov 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant