Repository navigation
docs: replace pip-install quickstart with source-install + add Positioning section - #1
Conversation
![devloai[bot]](https://avatars.githubusercontent.com/in/886113?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR Summary:
- Replaces PyPI
pip install eval-harnessquickstart with a source-install flow (git clone → venv → pip install -e '.[dev]'). - Adds a note that PyPI publishing is intentionally disabled until the package name is finalized.
- Adds a new "Positioning" section right after Quickstart clarifying the tool is not a hosted dashboard/leaderboard.
- No changes to
pyproject.toml, package name, or CLI entry point.
Review Summary:
Docs-only change, verified against the full README, pyproject.toml, PROJECT.md, and AGENTS.md for consistency. Two issues found: the quickstart's git clone uses a personal SSH URL that will fail for anyone without SSH keys configured (whereas pyproject.toml already publishes the HTTPS URL), and the new "Positioning" section duplicates messaging already present in the existing "Why This Exists" and "When to Use This (and When Not To)" sections. Overall the source-install rationale is sound and well-verified per the PR description.
Suggestions
| pip install eval-harness | ||
| eval-harness run --config configs/smoke_mock.yaml | ||
| ``` | ||
| git clone git@github.com:Revanth9727/LLM-Evaluation-Harness.git |
There was a problem hiding this comment.
The clone command uses an SSH URL (git@github.com:...), which will fail with a permission error for anyone who hasn't configured SSH keys for GitHub — likely most first-time users following this quickstart. The project metadata in pyproject.toml already uses the HTTPS URL for Homepage/Repository; consider matching that here so git clone works out of the box for everyone:
| git clone git@github.com:Revanth9727/LLM-Evaluation-Harness.git | |
| git clone https://github.com/Revanth9727/LLM-Evaluation-Harness.git |
actions
Quick Fix: Apply a small focused fix for this specific issue (takes < 1min).
Feedback: Rate this comment to help me improve future code reviews:
- 👍 Good - Helpful and accurate
- 👎 Poor - Wrong, unclear, or unhelpful
- Skip if you don't have any strong opinions either way.
| ## Positioning | ||
|
|
||
| When developing from a local checkout, use `pip install .[dev]`. The Makefile is a contributor convenience; the installed user interface is the `eval-harness` CLI. | ||
| This is not a hosted observability dashboard and not a public benchmark leaderboard. Use it to answer one production question: "I changed a prompt, retriever, model, or config. Did I regress my document-grounded assistant?" The harness is batch-oriented and CI-friendly: deterministic checks first, LLM judging second, an artifact trail always. |
There was a problem hiding this comment.
This new "Positioning" paragraph largely restates content already covered by the existing "When to Use This (and When Not To)" section further down (lines 134–138), which likewise says "not a hosted observability dashboard" / "not a benchmark leaderboard" with links to Langfuse/Braintrust/lm-evaluation-harness. Combined with the similar framing in "Why This Exists" (line 7), the README now makes the same "not a dashboard, not a leaderboard" point in three separate places. Consider merging "Positioning" into the existing "When to Use This" section (or removing one) to avoid redundancy and keep the doc tight.
actions
Feedback: Rate this comment to help me improve future code reviews:
- 👍 Good - Helpful and accurate
- 👎 Poor - Wrong, unclear, or unhelpful
- Skip if you don't have any strong opinions either way.
Changes
pip install eval-harness(PyPI) with a full source-install flow (git clone → venv → pip install -e '.[dev]'). Adds a note that PyPI publishing is intentionally disabled until the package name is finalized.Verified
pip install -e '.[dev]'clean install ✅pytest— 54 passed ✅eval-harness run --config configs/smoke_mock.yaml— 40 cases, exit 0 ✅pyproject.tomluntouched — package name and CLI entry point unchanged ✅