leaderboard - #45
Open
KensingtonOscupant wants to merge 28 commits into
Open
Conversation
Author
|
fyi @neelguha |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi there,
Recently, I wanted to submit some results and checked for a LegalBench leaderboard that would allow for that, but couldn’t find one - HELM and vals.ai don't seem to be intended for this. From what I understand, they also focus more on comparing performance of foundation models using a "hardened" configuration rather than allowing for competition on which individual model configuration (choice of foundation model, prompt, other hyperparameters, dedicated pre- and postprocessing etc.) works best for a given task.
So I thought I’d enjoy contributing one and implemented a minimal working version in Weights & Biases. I have evaluated a few models for illustration purposes. It is very lightweight (see setup script here); most of the code in this PR is a standalone refactor of
evaluation.py. I have picked one task of each of the metric types for demonstration, so extending it to all LegalBench tasks (see below regarding manual evaluation) can be achieved by simply importing theEXACT_MATCH_BALANCED_ACClist of tasks fromevaluation.pyinto the setup script.Participation is very easy (feel free to give it a try!), I have put together a brief notebook on how to create a submission. The current version of
weavehas an issue preventing leaderboards with custom Scorer classes like this one from working out of the box, but they will resolve it in the next release (~1-2 weeks) and I have shipped a preliminary fix along with this PR, so the leaderboards created with this setup work already.Licenses of the task datasets are attached to the respective dataset object's description.
Some more features: row-level results, model comparisons, traces, dataset versioning.
Manual evaluation is possible, but not implemented yet - I didn't want to overshoot, so I created a minimal version first to see if this would help you. Feel free to reach out with any questions or input you may have.
Cheers and thank you for your work - it has been a while, but I enjoyed it a lot to read the paper!