Repository navigation
feat: the evals measure a law beside the others, as an organisation that chose several searches - #101
Merged
Merged
Conversation
…hat chose several searches Signed-off-by: alkhatibdev <me@alkhatib.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The evals measure a law beside the others
An eval run searches the law its pack holds, alone, so a score means that law. An organisation
that selects several laws searches them together, and nothing measured that. This pull request
adds the measure.
What changes
dpolens evals run --every-lawsearches every law loaded, as an organisation that selectedthem all does. The pack's questions still expect the pack's own clauses, so the score says
how well that law is found beside the others.
tuning.ar beside uae-pdpl, in what therun prints and in the file
--outcomeswrites.evals comparetherefore takes a law aloneand a law beside the others as two sets, one folder can hold both, and a topic asked in both
is still one topic.
evals/results:--writewith--every-lawis refused,since the published scores are of a law searched alone.
--togetherscores sets as one only when they were searched beside the same laws.A run without the flag searches exactly as before, and no published figure moves.
Tests
--every-lawreaches both and says which, and the command keeps such a run as a set of itsown that
evals comparereads beside the run of the law alone.--every-lawis refused, naming the law.which laws were beside them, that a law alone and a law beside others are not scored
together, and that
--writeis refused.evals/README.mddescribes the measure under "Scoring" and "Judging a change".