Repository navigation
feat: the evals keep each question's outcome, and compare judges a change - #98
Merged
Merged
Conversation
…ange Signed-off-by: alkhatibdev <me@alkhatib.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The evals keep each question's outcome, and
comparejudges a changeAn eval run printed recall and the questions it missed, so two runs could only be compared by
which questions crossed the fifth place. This pull request keeps every question's outcome and
adds the rule a change to search is judged by.
What changes
dpolens evals run --outcomes <file>writes, for every question of the configuration beingscored, the clauses returned and the place of the first that answers, with the pack, the
configuration, the model, the recipe and the commit.
dpolens evals compare <before> <after>reads two such files, or two folders with a file foreach law, and prints each set before and after, then whether the change is kept.
engine/evals/judge.pyand written out inevals/README.md. A questionscores by the place of its right clause: 1 for first, a half for second, down to a tenth, and
nothing below the tenth. So a clause that climbs from ninth to sixth counts, which hits at
five do not see. A topic asked in several languages, or again as chat, is one topic. A change
is kept when chance alone would give a gain as large less than 5% of the time, found by
giving each topic's change a random sign ten thousand times, and when no set is worse. A set
is worse when its score falls by more than chance explains, with the 5% shared between the
sets.
On real data
The tuning sets of both laws, 216 questions on 79 topics, with
multilingual-e5-smallasbefore and
multilingual-e5-baseas after, both on exact search:A run compared with itself prints "No gain" and "Not kept", and two runs that do not ask the
same questions are refused, naming three of the questions.
Checked
On
9fe00eawith this change, withGITHUB_ACTIONS=true,CI=trueand every cache off:ruff check,ruff format --check,mypy,lint-imports,scripts/api_contract.py,scripts/client_text.py,docker compose configpytest: 1454 passed. The rule on hand-built runs: nothing changing, a clear gain, a gain onthree topics (one time in eight by chance), a topic asked four ways counted once, a climb
inside the top five, a set that loses more than chance explains, a set that slips by one
question, a question held twice. At the database: a run over GDPR's tuning set writes every
question with its ten clauses and the place of the first that answers, and comparing that
run with itself is not kept