Following the quick-taste local run docs, running evalhub eval run immediately hits a connection error because the docs don't make it clear that a local EvalHub service must already be running on http://localhost:8080 before invoking the CLI.
Steps to reproduce
- Follow the quick-taste local run example verbatim
- Run:
evalhub eval run \
--name my-first-eval \
--model-url http://localhost:1234/v1 \
--model-name qwen2.5-1.5b-instruct-mlx \
--provider lm_evaluation_harness \
--benchmark mmlu \
--wait
Error
Connection error to http://localhost:8080/api/v1/evaluations/jobs after 3 retries: [Errno 61] Connection refused
Error: Connection error: [Errno 61] Connection refused
Expected behaviour
The docs should include a prerequisite step showing how to start the EvalHub service locally before running evalhub eval run, or the CLI should surface a clearer message explaining that no local service was found and pointing to the local-run setup docs.
Following the quick-taste local run docs, running
evalhub eval runimmediately hits a connection error because the docs don't make it clear that a local EvalHub service must already be running onhttp://localhost:8080before invoking the CLI.Steps to reproduce
Error
Expected behaviour
The docs should include a prerequisite step showing how to start the EvalHub service locally before running
evalhub eval run, or the CLI should surface a clearer message explaining that no local service was found and pointing to the local-run setup docs.