Repository navigation
Try the pipeline on a new local model and report quality #2
Description
Model backends implement a tiny protocol — complete(system, user) -> str — in core/src/scorekeeper/backends/. Local open-source models are first-class: openai_compat.py already talks to any OpenAI-compatible endpoint (Ollama, LM Studio, vLLM). Extraction/detection quality is meant to be measured per backend, not assumed.
Task: run scorekeeper's extractor + Tier-1 detector against a local model of your choice and report how it does.
How:
- point the pipeline at your endpoint:
export SCOREKEEPER_MODEL_URL=http://localhost:11434/v1(Ollama example) andSCOREKEEPER_MODEL=<model> - run the live smoke tests:
cd core && uv run pytest -q tests/test_extract_live.py tests/test_detect_live.py(they skip without a backend) - try the demo scenarios or your own turn text
Acceptance: an experience report issue with: which model, JSON-schema compliance (did it need the repair retry?), any false conflicts, rough latency. Bonus: a PR note in docs/ recording the result.
Great way to learn the pipeline end-to-end. See CONTRIBUTING.md → "Add a model backend".
Activity
HarperZ9 commentedon Jul 17, 2026
I can run one public local model through the existing OpenAI-compatible backend and post an experience report with the exact model and quant, endpoint and runtime versions, live smoke-test output, repair-retry count, false-conflict notes, and rough latency. I would keep the inputs synthetic, record skipped cases, and separate observed behavior from any broader model-quality claim.
Is an issue-only experience report the preferred first step, with a small docs PR only if the result is reusable?
michalstrnadel commentedon Jul 19, 2026
Hi HarperZ9, welcome, and thank you for picking this up. You are the first person to take on an issue in this repo, so this genuinely made my day.
Go ahead with both. Post the experience report as an issue using the template, and open the docs PR either way, no need to wait and see if the result looks reusable. A report saying "this model kept breaking schema compliance" is just as useful as a success story, because the whole point is to measure quality per backend instead of assuming it.
Suggested spot for the PR:
docs/model-reports/<model-name>.md. Since you are first, define the format however feels natural to you and I will use it as the reference for everyone who comes after.Your methodology sounds great, especially keeping inputs synthetic and separating observed behavior from broader model quality claims. That is exactly the right instinct. One tiny ask: include the repair retry count even when it is zero, since "no retries needed" is a useful data point on its own.
If anything in the setup gets rough (env vars, the live smoke tests, anything else), just comment here and I will get back to you quickly. Looking forward to seeing which model you go with.
docs: English spec translation, API reference, model-reports dir
docs: triage QUESTIONS.md — Q4/Q9 resolved, Q5/Q6 status, Q7/Q8 standing
docs(research): map the 2026 self-improvement landscape onto scorekeeper
HarperZ9 commentedon Jul 20, 2026
Followed through with both artifacts: opened the experience-report issue at #8 and the durable docs/compatibility PR at #7. The result is intentionally negative for \ elos-coder-14b:latest: extractor recall 3/12, Tier-1 accuracy 8/10 with 40% FPR, and 1 extractor repair retry across 14 extraction golden cases. The PR also includes the small Windows locking/UTF-8 fix needed to make the local Ollama smoke path run from this Windows checkout.
michalstrnadel commentedon Jul 21, 2026
Delivered exactly as promised: the experience report in #8 and the docs PR in #7. The format of that report is now officially the template, so if anyone picks up another backend, copy that shape. Full review is on #7. Thank you HarperZ9.