Skip to content

fix(local-run): quick-taste example fails with connection refused when evalhub service is not running #69

Description

@ruivieira

Following the quick-taste local run docs, running evalhub eval run immediately hits a connection error because the docs don't make it clear that a local EvalHub service must already be running on http://localhost:8080 before invoking the CLI.

Steps to reproduce

  1. Follow the quick-taste local run example verbatim
  2. Run:
    evalhub eval run \
      --name my-first-eval \
      --model-url http://localhost:1234/v1 \
      --model-name qwen2.5-1.5b-instruct-mlx \
      --provider lm_evaluation_harness \
      --benchmark mmlu \
      --wait
    

Error

Connection error to http://localhost:8080/api/v1/evaluations/jobs after 3 retries: [Errno 61] Connection refused
Error: Connection error: [Errno 61] Connection refused

Expected behaviour

The docs should include a prerequisite step showing how to start the EvalHub service locally before running evalhub eval run, or the CLI should surface a clearer message explaining that no local service was found and pointing to the local-run setup docs.

Metadata

Metadata

Assignees

No one assigned

    Type

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions