Repository navigation
feat(experiments): local LLM judge for re-ranking, stopped before a result - #33
Merged
Merged
Conversation
…en before the run
…rned out to be this Mac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Step 3b of the local re-ranker work (#31): a local LLM (gemma-4-12b-qat, via LM Studio) as a pointwise judge, with the score taken as P(yes) from the first token's log-probabilities. Two wordings of the same question: A "does the passage state the answer" (the Jev wording) and B "is the passage relevant" (topical, what bge is trained for), to test whether the wording matters as much as the model.
Everything is under
experiments/local_rerank/, plus one.gitignoreline. Nothing inquarq/changes.Outcome: not completed, no result
The run was stopped by hand after 26 of the 150 requests the hard set needs, because the Mac overheated. No Hit@1 was produced, so the wording hypothesis is untested. Nothing in this PR says a local LLM judge works or fails.
What the PR does record:
--burst/--coolduty-cycle pacing, apmsetthrottling tripwire, and--thermal-logto read thepowermetricspressure level before each request (stops if the log goes stale). These are tripwires, not a thermometer; the mapping from pressure level to degrees is unknown.Why merge an unfinished experiment
So the attempt, its bands and its failure mode are on record, and the run can be repeated on a machine that can take the load, or with a smaller model after committing new bands. The conclusion from #31 stands: bge on the union pool is about 5 questions behind Jev on the 38.
Notes
lmstudio.url); nothing goes to a cloud API.experiments/local_rerank/.cache/(gitignored).