Skip to content

feat(experiments): local LLM judge for re-ranking, stopped before a result - #33

Merged
yodablocks merged 4 commits into
masterfrom
exp/llm-judge
Oct 4, 2026
Merged

yodablocks merged 4 commits into
masterfrom
exp/llm-judge

Conversation

@yodablocks

Copy link
Copy Markdown
Owner

What

Step 3b of the local re-ranker work (#31): a local LLM (gemma-4-12b-qat, via LM Studio) as a pointwise judge, with the score taken as P(yes) from the first token's log-probabilities. Two wordings of the same question: A "does the passage state the answer" (the Jev wording) and B "is the passage relevant" (topical, what bge is trained for), to test whether the wording matters as much as the model.

Everything is under experiments/local_rerank/, plus one .gitignore line. Nothing in quarq/ changes.

Outcome: not completed, no result

The run was stopped by hand after 26 of the 150 requests the hard set needs, because the Mac overheated. No Hit@1 was produced, so the wording hypothesis is untested. Nothing in this PR says a local LLM judge works or fails.

What the PR does record:

  • The bands were committed before the run (see the README).
  • Timing test: 6.4 s per request (max 7.7 s), 10 requests.
  • LM Studio at 192.168.1.107 is the same machine that runs everything else, so local inference heats it. A continuous run reached the temperature the user reported (about 110 C within a minute); even a 4-request burst raised macOS thermal pressure to Moderate.
  • Safety tooling added after that: --burst/--cool duty-cycle pacing, a pmset throttling tripwire, and --thermal-log to read the powermetrics pressure level before each request (stops if the log goes stale). These are tripwires, not a thermometer; the mapping from pressure level to degrees is unknown.

Why merge an unfinished experiment

So the attempt, its bands and its failure mode are on record, and the run can be repeated on a machine that can take the load, or with a smaller model after committing new bands. The conclusion from #31 stands: bge on the union pool is about 5 questions behind Jev on the 38.

Notes

  • Requests go only to the LM Studio host from the quarq config (lmstudio.url); nothing goes to a cloud API.
  • Cached scores live in experiments/local_rerank/.cache/ (gitignored).

@yodablocks
yodablocks merged commit 346a39c into master Oct 4, 2026
4 checks passed
@yodablocks
yodablocks deleted the exp/llm-judge branch October 4, 2026 06:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant