feat(evaluation) 10/15: the EvalRunner Ray actor and its dispatcher

[Ahmath-Gadji]

Part 10 of 15 of the split of #811. Targets eval/09-run-lifecycle (#820). This is where a run actually happens.

What

EvalRunner.run()
  1. update status → INDEXING
  2. upload + time each corpus file over the real HTTP API   → indexing metrics
  3. update status → EVALUATING
  4. render both promptfoo configs, exec them                → raw results.json
  5. summarize(), persist, drop the throwaway partition

Plus RayEvaluationRunner, the adapter binding the actor to the part-8 port, and the container wiring that hands it to EvaluationService.

Notable

• The runner drives OpenRAG through its own HTTP API, not in-process calls. Two reasons: it is the path a real user's documents take, so the indexing timings mean something; and it is the same surface promptfoo talks to, so an eval can never pass against a code path the API does not expose.
• The actor handle resolves on first use, not in __init__. EvalRunner is a detached actor, so merely building the adapter must not be what spawns it — listing datasets should not start a worker process. The container therefore builds EvaluationService eagerly and the worker lazily.
• run() never raises. The caller dispatched it fire-and-forget and has nobody to catch for, so cancellation and failure are both written to the run row. The finally drops the partition whatever happened.
• One bad corpus file is a failed sample, not a dead run — it still contributes to files_failed. Every file failing is an error worth surfacing, since the retrieval phase would otherwise grade an empty index.
• promptfoo exits non-zero when assertions fail, which is a result, not an error. The presence of the output file is what decides. Both streams are captured for the failure message because promptfoo reports config errors on stdout, not stderr.
• Each run gets its own PROMPTFOO_CONFIG_DIR. promptfoo keeps a SQLite eval history under it, defaulting to $HOME/.promptfoo, which is not guaranteed writable in a container; a per-run temp dir is also never contended. WAL mode is disabled because some filesystems don't support it.
• max_concurrency=4 so cancel() and is_busy() still land while run() holds a slot — otherwise cancellation would deadlock behind the run it is trying to cancel.

Testing

ruff, format check, the layer-import guard, and the full unit suite (2270 passed) including the DI wiring test, which now covers evaluation_service. The actor itself is exercised end to end on a real deployment rather than in unit tests — it is a thin shell over HTTP and a subprocess, both mocked out of meaning. The pure logic it calls is tested in parts 3–5.

The one failure, test_content_deduplication_can_be_disabled_by_env, reproduces on develop.

Deployment note. The images that install promptfoo land in part 11. On a separate Ray cluster (not compose, which runs Ray inside the API container), <data_dir> must be on shared storage — the runner reads dataset files by path.

[coderabbitai]

Warning

Review limit reached

@EnjoyBacon7, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 59 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: b85a19ac-7a48-4657-911f-43beeb24c192

📥 Commits

Reviewing files that changed from the base of the PR and between cc3263c and 6497847.

📒 Files selected for processing (5)

• openrag/di/container.py
• openrag/di/providers.py
• openrag/services/workers/eval_dispatcher.py
• openrag/services/workers/eval_runner.py
• tests/unit/di/test_container.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch eval/10-ray-worker

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Ahmath-Gadji]

@coderabbitai full review

[coderabbitai]

✅ Action performed

Full review finished.

---

Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 56 minutes.