feat(evaluation) 1/15: config surface and domain models

conf/config.yaml

@@ -122,9 +122,14 @@ verbose:
   level: INFO
 
 # --- Server ---
-# Env: PREFERRED_URL_SCHEME
+# Env: PREFERRED_URL_SCHEME, OPENRAG_INTERNAL_URL
 server:
   preferred_url_scheme: null
+  # internal_url — how out-of-process workers (e.g. the evaluation runner) reach
+  # the API from inside the deployment. Left unset here so it defaults to
+  # http://openrag:$APP_iPORT, following the port uvicorn actually binds. Set
+  # OPENRAG_INTERNAL_URL if the API is not reachable under the compose service
+  # name.
 
 # --- LLM Context ---
 # Env: MAX_LLM_CONTEXT_SIZE, MAX_OUTPUT_TOKENS
@@ -363,3 +368,19 @@ mcp:
   similarity_threshold: 0.8
   download_timeout: 30.0
   max_download_bytes: 104857600  # 100 MiB
+
+# --- Evaluation ---
+# Env: PROMPTFOO_BIN, EVAL_MAX_CORPUS_MB, EVAL_MAX_TESTSET_MB,
+#      EVAL_MAX_TESTSET_ROWS, EVAL_TOP_K, EVAL_TASK_TIMEOUT,
+#      EVAL_TASK_POLL_INTERVAL, EVAL_HTTP_TIMEOUT, EVAL_PROMPTFOO_TIMEOUT
+# The runner reaches the API via server.internal_url.
+evaluation:
+  promptfoo_bin: promptfoo
+  max_corpus_mb: 512
+  max_testset_mb: 5
+  max_testset_rows: 500
+  top_k: 5
+  task_timeout_seconds: 1800.0
+  task_poll_seconds: 1.0
+  http_timeout_seconds: 300.0
+  promptfoo_timeout_seconds: 3600.0

docs/content/docs/documentation/env_vars.md

@@ -572,6 +572,7 @@ The following environment variables configure the FastAPI server and control acc
 | `DEFAULT_FILE_QUOTA` | `int` | `-1` | Default per-user file quota. `<0` disables quotas globally; `>=0` sets the default limit when a user has no explicit quota. |
 | `PREFERRED_URL_SCHEME` | `string` | `null` | URL scheme (`http` or `https`) used when generating URLs in API responses (e.g., `task_status_url`). When running behind a reverse proxy that terminates SSL, set this to `https` to ensure generated URLs use the correct scheme. If unset, the scheme from the incoming request is used. |
 | `CORS_EXTRA_ORIGINS` | `string` | _(unset)_ | Semicolon-separated list of additional origins allowed by CORS (e.g. `https://app.example.com;https://other.example.com`). Extends the default list without replacing it. |
+| `OPENRAG_INTERNAL_URL` | `string` | `http://openrag:$APP_iPORT` | Overrides `server.internal_url`: the base URL out-of-process workers use to reach the API from inside the deployment. Used by the evaluation runner, which uploads the corpus and drives promptfoo over HTTP from the Ray container. The default already follows `APP_iPORT`, the container-internal port uvicorn binds; set this when the API is reachable under a different host than the compose service name. The bundled [Helm chart](/openrag/documentation/kubernetes/) sets it for you, to `http://<release>-openrag:<openrag.service.port>`. |
 | `UVICORN_FORWARDED_ALLOW_IPS` | `string` | `127.0.0.1` | Comma-separated CIDRs/IPs (or `*`) whose `X-Forwarded-*` headers uvicorn trusts. **Required when OpenRAG runs behind a reverse proxy that lives outside loopback** (typical docker-compose / k8s — including the bundled admin-ui proxy). Otherwise `X-Forwarded-Proto` is dropped and OIDC cookies ship with `Secure=False` even over HTTPS, and `X-Forwarded-For` is dropped so per-user rate limits collapse onto the proxy's single IP. **Set this to your proxy's subnet, not `*`** — see the proxy-trust caution under [Rate Limiting](#rate-limiting) for why `*` can be spoofed. |
 | `MAX_UPLOAD_SIZE_MB` | `int` | `1024` | Maximum accepted upload size, in MB. `0` or a negative value means unlimited. |
 | `MAX_PARTITIONS_PER_USER` | `int` | `100` | Maximum number of partitions a non-admin user may own. `-1` disables the cap (unlimited). Admin users always bypass it. |
@@ -673,6 +674,24 @@ OpenRAG ships a standalone [Model Context Protocol](https://modelcontextprotocol
 | `OPENRAG_MCP_DOWNLOAD_TIMEOUT` | `float` | `30.0` | Timeout (seconds) for the server-side `index_url` fetch (SSRF/DoS hardening). |
 | `OPENRAG_MCP_MAX_DOWNLOAD_BYTES` | `int` | `104857600` | Maximum bytes downloaded by an `index_url` fetch. Default is 100 MiB. |
 
+### Evaluation
+
+On-demand benchmarking from the admin **System → Evaluation** tab. A run indexes a stored corpus into a throwaway partition, replays the test set through [promptfoo](https://promptfoo.dev/), and reports indexing, retrieval and answer metrics.
+
+These are operational limits — every value is validated as strictly positive at config load, so a typo fails at startup rather than at run time. The reserved partition prefix, the test-set CSV column names and the `file_id` alphabet are deliberately *not* configurable: they are contracts with datasets already on disk. The base URL the runner calls back on is [`OPENRAG_INTERNAL_URL`](#fastapi--access-control), a server setting rather than an evaluation one.
+
+| Variable | Type | Default | Description |
+|----------|------|---------|-------------|
+| `PROMPTFOO_BIN` | `string` | `promptfoo` | Executable the evaluation runner shells out to. The API and Ray images both install a pinned promptfoo on `PATH`; set this only for a custom location. |
+| `EVAL_MAX_CORPUS_MB` | `int` | `512` | Maximum total size of one dataset's corpus upload, in MB. Enforced while streaming to disk, so an inflated `Content-Length` cannot get past it. A dataset is re-indexed on every run, so an oversized corpus costs far more than the upload itself. Also bounded by the global [`MAX_UPLOAD_SIZE_MB`](#fastapi--access-control) — raise both to go past 1024. |
+| `EVAL_MAX_TESTSET_MB` | `int` | `5` | Maximum size of the test-set CSV upload, in MB. |
+| `EVAL_MAX_TESTSET_ROWS` | `int` | `500` | Maximum number of questions in a test set. Each row costs one retrieval call plus one LLM-graded generation per run. |
+| `EVAL_TOP_K` | `int` | `5` | Chunks retrieved per question when measuring retrieval quality. Raising it makes hit rate and recall more forgiving, so compare runs only at a fixed value. |
+| `EVAL_TASK_TIMEOUT` | `float` | `1800` | Seconds to wait for one corpus file's indexing task before the run gives up on it. Raise it for slow parsers (large scanned PDFs through Marker). |
+| `EVAL_TASK_POLL_INTERVAL` | `float` | `1.0` | Seconds between polls of a file's indexing task status. |
+| `EVAL_HTTP_TIMEOUT` | `float` | `300` | Per-request timeout for the runner's own HTTP calls to the API. |
+| `EVAL_PROMPTFOO_TIMEOUT` | `float` | `3600` | Seconds allowed for one `promptfoo eval` invocation. Every row is graded by an LLM, so raise it for a slow grader or a large test set. |
+
 ### Advanced & Legacy Variables
 
 #### Model-endpoint seed overrides (legacy aliases)

docs/content/docs/documentation/kubernetes.md

@@ -55,6 +55,8 @@ This guide explains how to deploy the **OpenRAG** stack on a Kubernetes cluster
 
 - Ensure your GPU nodes have the correct NVIDIA drivers and `nvidia` `RuntimeClass` configured.
 
+- `OPENRAG_INTERNAL_URL` is set for you, to `http://<release>-openrag:<openrag.service.port>`. Out-of-process workers — the evaluation runner, which drives its corpus upload and promptfoo over HTTP — run in the RayCluster pod and use it to reach the API. The built-in default is the compose service name and does not resolve here. Override it only if you front the API with a different in-cluster Service.
+
 ## Managed PostgreSQL
 
 The chart can run against a database that is provisioned outside OpenRAG, which is the recommended setup on OpenShift or cloud-managed PostgreSQL.

infra/charts/openrag-stack/values.yaml

@@ -334,6 +334,12 @@ env:
     ENABLE_RAY_SERVE: "true"
     RAY_SERVE_NUM_REPLICAS: "4"
     RAY_SERVE_PORT: "80"
+    # How out-of-process workers reach the API. The built-in default is
+    # http://openrag:$APP_iPORT, which is the compose service name and does
+    # not resolve here — this chart's Service is <release>-openrag, and the
+    # RayCluster that runs those workers is a separate pod. Follows
+    # openrag.service.port, the same value that opens the container port.
+    OPENRAG_INTERNAL_URL: "http://{{ .Release.Name }}-openrag:{{ .Values.openrag.service.port }}"
 
     WITH_CHAINLIT_UI: "false"
     SAVE_UPLOADED_FILES: "false"

openrag/api/routers/admin/partitions.py

@@ -72,7 +72,7 @@ async def list_existant_partitions(
     # partitions_with_details. Gate the all-expansion on the caller actually being
     # an admin, so a (legacy) partition literally named ``all`` owned by a regular
     # user cannot leak every partition. New ``all`` partitions are already rejected
-    # at creation (_RESERVED_PARTITION_NAMES).
+    # at creation (``RESERVED_PARTITION_NAMES``, core.models.partition).
     is_admin = bool(request.state.user.get("is_admin"))
     summaries = await service.list_partition_summaries()
     if is_admin and len(partitions) == 1 and partitions[0]["partition"] == "all":

openrag/core/config/evaluation.py

@@ -0,0 +1,60 @@
+"""Configuration for the admin evaluation feature.
+
+Operational limits live here rather than as constants in the code that uses
+them, so a deployment can be retuned without a rebuild.
+
+Domain contracts stay out of this file: the reserved partition prefix, the CSV
+column names and the ``file_id`` alphabet are not settings, and changing them
+would invalidate stored datasets.
+"""
+
+from __future__ import annotations
+
+from pydantic import Field
+
+from .base import ConfigMixin
+
+
+class EvaluationConfig(ConfigMixin):
+    """Limits and timeouts for evaluation datasets and runs.
+
+    Every field is bounded: these are all reachable from the environment, and a
+    non-positive limit does not degrade gracefully — it reaches the runner as a
+    cap that rejects every upload, or as a timeout that expires instantly. A
+    typo should fail at config load, where the message names the field.
+    """
+
+    #: Executable the runner shells out to; the images install it on PATH.
+    promptfoo_bin: str = Field(default="promptfoo", min_length=1)
+
+    #: Upload caps. A dataset is re-indexed on every run, so an oversized
+    #: corpus costs far more than the upload itself.
+    max_corpus_mb: int = Field(default=512, gt=0)
+    max_testset_mb: int = Field(default=5, gt=0)
+    #: Each test-set row costs one retrieval call plus one graded generation.
+    max_testset_rows: int = Field(default=500, gt=0)
+
+    #: Chunks retrieved per question by the retrieval config. Bounded like the
+    #: retrieval pipeline's own ``top_k``.
+    top_k: int = Field(default=5, gt=0, le=1000)
+
+    #: How long to wait for one file's indexing task, and how often to poll it.
+    task_timeout_seconds: float = Field(default=1800.0, gt=0)
+    task_poll_seconds: float = Field(default=1.0, gt=0)
+
+    #: Per-request timeout for the runner's own HTTP calls.
+    http_timeout_seconds: float = Field(default=300.0, gt=0)
+
+    #: promptfoo grades every row with an LLM, so allow for a slow grader.
+    promptfoo_timeout_seconds: float = Field(default=3600.0, gt=0)
+
+    @property
+    def max_corpus_bytes(self) -> int:
+        return self.max_corpus_mb * 1024 * 1024
+
+    @property
+    def max_testset_bytes(self) -> int:
+        return self.max_testset_mb * 1024 * 1024
+
+
+__all__ = ["EvaluationConfig"]

openrag/core/config/infrastructure.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import os
 from pathlib import Path
 
 from pydantic import Field
@@ -96,8 +97,27 @@ class PathsConfig(ConfigMixin):
 # ---------------------------------------------------------------------------
 
 
+def _default_internal_url() -> str:
+    """Base URL under which the API reaches itself from inside the deployment.
+
+    Used by out-of-process workers (e.g. the evaluation runner, which uploads a
+    corpus and drives promptfoo over HTTP): they run in their own container and
+    cannot reuse whatever host the admin's browser happened to use.
+
+    The port follows ``APP_iPORT``, the container-internal port uvicorn binds
+    (``infra/scripts/entrypoint.sh``), so moving it does not silently leave
+    workers calling 8080. ``OPENRAG_INTERNAL_URL`` overrides the whole URL.
+
+    ``or`` rather than a ``get`` default, to match the ``${APP_iPORT:-8080}``
+    in entrypoint.sh and docker-compose.yaml: a bare ``APP_iPORT=`` line in an
+    env file is empty, not absent, and would otherwise yield ``http://openrag:``.
+    """
+    return f"http://openrag:{os.environ.get('APP_iPORT') or '8080'}"
+
+
 class ServerConfig(ConfigMixin):
     preferred_url_scheme: str | None = None
+    internal_url: str = Field(default_factory=_default_internal_url)
 
 
 # ---------------------------------------------------------------------------

openrag/core/config/loader.py

@@ -86,6 +86,7 @@
     ("LOG_LEVEL", "verbose.level", str),
     # Server
     ("PREFERRED_URL_SCHEME", "server.preferred_url_scheme", str),
+    ("OPENRAG_INTERNAL_URL", "server.internal_url", str),
     # LLM Context
     ("MAX_LLM_CONTEXT_SIZE", "llm_context.max_llm_context_size", int),
     ("MAX_OUTPUT_TOKENS", "llm_context.max_output_tokens", int),
@@ -183,6 +184,16 @@
     ("OPENRAG_MCP_SIMILARITY_THRESHOLD", "mcp.similarity_threshold", float),
     ("OPENRAG_MCP_DOWNLOAD_TIMEOUT", "mcp.download_timeout", float),
     ("OPENRAG_MCP_MAX_DOWNLOAD_BYTES", "mcp.max_download_bytes", int),
+    # Evaluation
+    ("PROMPTFOO_BIN", "evaluation.promptfoo_bin", str),
+    ("EVAL_MAX_CORPUS_MB", "evaluation.max_corpus_mb", int),
+    ("EVAL_MAX_TESTSET_MB", "evaluation.max_testset_mb", int),
+    ("EVAL_MAX_TESTSET_ROWS", "evaluation.max_testset_rows", int),
+    ("EVAL_TOP_K", "evaluation.top_k", int),
+    ("EVAL_TASK_TIMEOUT", "evaluation.task_timeout_seconds", float),
+    ("EVAL_TASK_POLL_INTERVAL", "evaluation.task_poll_seconds", float),
+    ("EVAL_HTTP_TIMEOUT", "evaluation.http_timeout_seconds", float),
+    ("EVAL_PROMPTFOO_TIMEOUT", "evaluation.promptfoo_timeout_seconds", float),
 ]
 
 _AUDIO_EXTENSIONS = ("mp3", "flac", "ogg", "aac", "flv", "wma", "mp4")

openrag/core/config/root.py

@@ -14,6 +14,7 @@
     SemaphoreConfig,
     VLMConfig,
 )
+from .evaluation import EvaluationConfig
 from .indexation import LoaderConfig
 from .infrastructure import (
     PathsConfig,
@@ -66,6 +67,22 @@ class Settings(ConfigMixin):
     rag: RAGConfig = Field(default_factory=RAGConfig)
     websearch: WebSearchConfig = Field(default_factory=StaanWebSearchConfig)
     mcp: MCPServerConfig = Field(default_factory=MCPServerConfig)
+    evaluation: EvaluationConfig = Field(default_factory=EvaluationConfig)
     models: ModelsConfig = Field(default_factory=ModelsConfig)
     presets: PresetsConfig = Field(default_factory=PresetsConfig)
     partitions: dict[str, PartitionConfig] = Field(default_factory=dict)
+
+    def resolved_rdb(self) -> RDBConfig:
+        """``rdb`` with its database name filled in.
+
+        ``rdb.database`` is optional: historically the name is derived from the
+        Milvus collection. Any process opening its own Postgres connection —
+        the API's catalog store, or a Ray worker such as ``EvalRunner`` — must
+        resolve it the same way, so the derivation lives here rather than in
+        the callers.
+        """
+        if self.rdb.database is not None:
+            return self.rdb
+        return self.rdb.model_copy(
+            update={"database": f"partitions_for_collection_{self.vectordb.collection_name}"},
+        )

openrag/core/models/__init__.py

@@ -1,4 +1,9 @@
-"""Domain models — pure Pydantic, no infrastructure imports."""
+"""Domain models — plain Pydantic models or dataclasses, no infrastructure imports.
+
+Types that are validated at a boundary (parsed input, stored rows) are Pydantic;
+purely internal value objects may be dataclasses. Either way nothing here may
+import from ``services`` or ``api``.
+"""
 
 from .catalog import TERMINAL_TASK_STATES, DocumentRecord, DocumentStatus, IndexationJob, JobStatus
 from .chunk import Chunk, ChunkType