Skip to content

feat: the evals keep each question's outcome, and compare judges a change - #98

Merged
alkhatibdev merged 1 commit into
mainfrom
evals-compare
Nov 14, 2026
Merged

alkhatibdev merged 1 commit into
mainfrom
evals-compare

Conversation

@alkhatibdev

Copy link
Copy Markdown
Owner

The evals keep each question's outcome, and compare judges a change

An eval run printed recall and the questions it missed, so two runs could only be compared by
which questions crossed the fifth place. This pull request keeps every question's outcome and
adds the rule a change to search is judged by.

What changes

  • dpolens evals run --outcomes <file> writes, for every question of the configuration being
    scored, the clauses returned and the place of the first that answers, with the pack, the
    configuration, the model, the recipe and the commit.
  • dpolens evals compare <before> <after> reads two such files, or two folders with a file for
    each law, and prints each set before and after, then whether the change is kept.
  • The rule is in engine/evals/judge.py and written out in evals/README.md. A question
    scores by the place of its right clause: 1 for first, a half for second, down to a tenth, and
    nothing below the tenth. So a clause that climbs from ninth to sixth counts, which hits at
    five do not see. A topic asked in several languages, or again as chat, is one topic. A change
    is kept when chance alone would give a gain as large less than 5% of the time, found by
    giving each topic's change a random sign ten thousand times, and when no set is worse. A set
    is worse when its score falls by more than chance explains, with the 5% shared between the
    sets.
  • Search itself does not change, and no published figure moves.

On real data

The tuning sets of both laws, 216 questions on 79 topics, with multilingual-e5-small as
before and multilingual-e5-base as after, both on exact search:

gdpr tuning.ar: 50 questions   hits at five 28 to 30   place score 0.417 to 0.457
gdpr tuning.en: 50 questions   hits at five 31 to 32   place score 0.402 to 0.438
uae-pdpl tuning.ar: 58 questions   hits at five 40 to 44   place score 0.507 to 0.544
uae-pdpl tuning.en: 58 questions   hits at five 48 to 49   place score 0.570 to 0.584

All: 216 questions on 79 topics   hits at five 147 to 155   place score 0.479 to 0.510
By topic at five: 12 found, 5 lost

Chance alone gives a gain this large 1.69% of the time, and a change is kept under 5%.
Kept.

A run compared with itself prints "No gain" and "Not kept", and two runs that do not ask the
same questions are refused, naming three of the questions.

Checked

On 9fe00ea with this change, with GITHUB_ACTIONS=true, CI=true and every cache off:

  • ruff check, ruff format --check, mypy, lint-imports, scripts/api_contract.py,
    scripts/client_text.py, docker compose config
  • pytest: 1454 passed. The rule on hand-built runs: nothing changing, a clear gain, a gain on
    three topics (one time in eight by chance), a topic asked four ways counted once, a climb
    inside the top five, a set that loses more than chance explains, a set that slips by one
    question, a question held twice. At the database: a run over GDPR's tuning set writes every
    question with its ten clauses and the place of the first that answers, and comparing that
    run with itself is not kept

…ange

Signed-off-by: alkhatibdev <me@alkhatib.dev>
@alkhatibdev
alkhatibdev merged commit 934e3e1 into main Nov 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant