Skip to content

feat: the evals measure a law beside the others, as an organisation that chose several searches - #101

Merged
alkhatibdev merged 1 commit into
mainfrom
evals-every-law
Nov 15, 2026
Merged

alkhatibdev merged 1 commit into
mainfrom
evals-every-law

Conversation

@alkhatibdev

Copy link
Copy Markdown
Owner

The evals measure a law beside the others

An eval run searches the law its pack holds, alone, so a score means that law. An organisation
that selects several laws searches them together, and nothing measured that. This pull request
adds the measure.

What changes

  • dpolens evals run --every-law searches every law loaded, as an organisation that selected
    them all does. The pack's questions still expect the pack's own clauses, so the score says
    how well that law is found beside the others.
  • The set is named with the laws beside it, such as tuning.ar beside uae-pdpl, in what the
    run prints and in the file --outcomes writes. evals compare therefore takes a law alone
    and a law beside the others as two sets, one folder can hold both, and a topic asked in both
    is still one topic.
  • Such a run is never written to evals/results: --write with --every-law is refused,
    since the published scores are of a law searched alone.
  • Where the pack's law is the only one loaded, the run is refused in one line, naming it.
  • --together scores sets as one only when they were searched beside the same laws.

A run without the flag searches exactly as before, and no published figure moves.

Tests

  • With a database holding both laws: a run for a pack reaches its law alone, a run with
    --every-law reaches both and says which, and the command keeps such a run as a set of its
    own that evals compare reads beside the run of the law alone.
  • With a database holding one law: --every-law is refused, naming the law.
  • Without one: how a run beside other laws is named, that sets scored together keep saying
    which laws were beside them, that a law alone and a law beside others are not scored
    together, and that --write is refused.

evals/README.md describes the measure under "Scoring" and "Judging a change".

…hat chose several searches

Signed-off-by: alkhatibdev <me@alkhatib.dev>
@alkhatibdev
alkhatibdev merged commit 1acbf05 into main Nov 15, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant