feat: add stable evaluator error codes

CHANGELOG.md

@@ -4,6 +4,13 @@ All notable changes to SkillEvaluator are documented in this file.
 
 ## Unreleased
 
+### Added
+
+- Added an append-only, vendor-neutral error-code registry for evaluator
+  execution failures. Terminal provider and runtime-preflight results now
+  include a structured `error_code` while preserving `execution_errors` and
+  existing lowercase provider failure subtypes for compatibility.
+
 ### Fixed
 
 - Tier 3 accuracy and custom goal judges now retry one malformed (including

docs/README.md

@@ -17,6 +17,7 @@ contributors making non-trivial edits.
 - `tier1-validation.mdx` — checks, flags, reports, CI recipe
 - `tier2-deduplication.mdx` — dedup commands and thresholds
 - `tier3-live-evaluation.mdx` — skill evaluation with live agents
+- `error-codes.mdx` — stable execution failure identifiers
 - `developer-guide.mdx` — contributor setup
 
 Navigation order and slugs are defined in [`../fern/docs.yml`](../fern/docs.yml).

docs/error-codes.mdx

@@ -0,0 +1,83 @@
+---
+title: "Error Code Reference"
+description: "Stable, vendor-neutral error codes for SkillEvaluator execution failures."
+---
+
+SkillEvaluator emits `error_code` as the stable support-facing identifier for a
+terminal evaluator execution failure. Automation should use this field instead
+of matching human-readable messages.
+
+```json title="Failed evaluation result"
+{
+  "execution_status": "failed",
+  "error_code": "SKILLEVALUATOR-AUTH-002",
+  "execution_errors": ["Provider verification failed."]
+}
+```
+
+`execution_errors` remains available for backward compatibility and human
+diagnosis. Lowercase fields such as `failure_kind`, when present on a provider
+probe, are diagnostic subtypes rather than stable support identifiers. Their
+values may be more implementation-specific than `error_code`.
+
+## Contract
+
+Codes match this uppercase ASCII schema:
+
+```text
+^SKILLEVALUATOR-[A-Z][A-Z0-9]*-[0-9]{3}$
+```
+
+The in-package registry is authoritative and append-only. A released code is
+never renamed, reused, or assigned a different meaning. A string that matches
+the schema but is absent from the registry is not a valid SkillEvaluator error
+code.
+
+Python integrations can obtain a serialization-ready JSON Schema with
+`skillevaluator.error_codes.error_code_schema()`. Each call returns a fresh
+dictionary; changing it does not modify the authoritative registry.
+
+When several failures end one evaluation, `error_code` keeps their code only
+when every classified failure agrees. Conflicting or absent classifications
+produce `SKILLEVALUATOR-UNKNOWN-001`. `execution_errors` retains the available
+human-readable details. Older result files may omit `error_code`; readers
+continue to accept those files for compatibility.
+
+## Registered codes
+
+| Code | Meaning |
+| --- | --- |
+| `SKILLEVALUATOR-AUTH-002` | Provider authentication failed |
+| `SKILLEVALUATOR-AUTH-003` | Provider authorization failed |
+| `SKILLEVALUATOR-CONFIG-001` | Evaluator configuration was invalid |
+| `SKILLEVALUATOR-DEPENDENCY-001` | A required dependency was unavailable |
+| `SKILLEVALUATOR-DEPENDENCY-003` | The configured model was not found |
+| `SKILLEVALUATOR-DEPENDENCY-005` | A required dependency timed out |
+| `SKILLEVALUATOR-DEPENDENCY-006` | A required dependency rate-limited the evaluator |
+| `SKILLEVALUATOR-DEPENDENCY-007` | The selected dependency operation is unsupported |
+| `SKILLEVALUATOR-DEPENDENCY-008` | A required dependency returned an invalid response |
+| `SKILLEVALUATOR-DEPENDENCY-009` | A required dependency returned another HTTP failure |
+| `SKILLEVALUATOR-UNKNOWN-001` | The evaluator could not classify the failure |
+| `SKILLEVALUATOR-RUNTIME-005` | The evaluator could not start a runtime process |
+| `SKILLEVALUATOR-RUNTIME-007` | Evaluator runtime execution timed out |
+| `SKILLEVALUATOR-RUNTIME-008` | An evaluator runtime process exited unsuccessfully |
+| `SKILLEVALUATOR-CONTRACT-007` | The evaluator runtime produced an invalid job result |
+
+Provider and runtime preflight classifications use structured failure metadata
+such as the failure subtype and HTTP status. They never infer a code from error
+message text. In particular, an unavailable provider response maps as follows:
+
+| Structured signal | Code |
+| --- | --- |
+| HTTP 429 | `SKILLEVALUATOR-DEPENDENCY-006` |
+| HTTP 408 or an explicit timeout | `SKILLEVALUATOR-DEPENDENCY-005` |
+| HTTP 5xx or no more specific signal | `SKILLEVALUATOR-DEPENDENCY-001` |
+
+Local runtime-preflight failures use the `RUNTIME` domain. A process spawn
+failure, execution timeout, or unsuccessful exit is not classified as a
+dependency failure. This includes local credential-helper process spawn and
+timeout failures. A missing or invalid runtime job result uses the `CONTRACT`
+domain because the process ran but did not satisfy the result contract.
+
+See [Reports & Results](reports.mdx#machine-readable-contract) for the result
+artifacts that carry this field.

docs/reports.mdx

@@ -311,6 +311,9 @@ For CI and tooling, two JSON entry points matter:
 - **Standalone `tier3 evaluate` runs** — parse `result.json` in the run
   directory (or follow `latest`). It carries per-agent scores, dimensions,
   lift, pass@k, trial counts, the attempt policy, and execution status.
+  Failed execution results also carry the stable `error_code`; use the
+  [Error Code Reference](error-codes.mdx) instead of matching prose in
+  `execution_errors`.
   Unscored metrics stay unscored rather than defaulting to 0.0, so
   `overall_score` can be `null`.
 - **`validate --tier3` runs** — the combined

fern/docs.yml

@@ -65,6 +65,9 @@ navigation:
       - page: Reports & Results
         path: ../docs/reports.mdx
         slug: reports
+      - page: Error Code Reference
+        path: ../docs/error-codes.mdx
+        slug: error-codes
       - page: "BENCHMARK.md Rollout and Backfill"
         path: ../docs/benchmark-rollout.mdx
         slug: benchmark-rollout

src/skillevaluator/error_codes.py

@@ -0,0 +1,212 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Stable, vendor-neutral error codes for evaluator execution failures."""
+
+from __future__ import annotations
+
+import re
+from collections.abc import Iterable, Mapping
+from dataclasses import dataclass
+from enum import StrEnum
+from types import MappingProxyType
+
+
+class ErrorDomain(StrEnum):
+    """Factual subsystem that produced an evaluator execution failure."""
+
+    AUTH = "AUTH"
+    CONFIG = "CONFIG"
+    DEPENDENCY = "DEPENDENCY"
+    UNKNOWN = "UNKNOWN"
+    RUNTIME = "RUNTIME"
+    CONTRACT = "CONTRACT"
+
+
+class EvaluatorErrorCode(StrEnum):
+    """Stable support-facing identifiers emitted by SkillEvaluator.
+
+    Allocations are append-only. Existing values must never be renamed, reused,
+    or assigned a different meaning.
+    """
+
+    AUTHENTICATION = "SKILLEVALUATOR-AUTH-002"
+    AUTHORIZATION = "SKILLEVALUATOR-AUTH-003"
+    INVALID_CONFIGURATION = "SKILLEVALUATOR-CONFIG-001"
+    DEPENDENCY_UNAVAILABLE = "SKILLEVALUATOR-DEPENDENCY-001"
+    MODEL_NOT_FOUND = "SKILLEVALUATOR-DEPENDENCY-003"
+    DEPENDENCY_TIMEOUT = "SKILLEVALUATOR-DEPENDENCY-005"
+    RATE_LIMITED = "SKILLEVALUATOR-DEPENDENCY-006"
+    UNSUPPORTED = "SKILLEVALUATOR-DEPENDENCY-007"
+    INVALID_RESPONSE = "SKILLEVALUATOR-DEPENDENCY-008"
+    OTHER_HTTP = "SKILLEVALUATOR-DEPENDENCY-009"
+    UNKNOWN = "SKILLEVALUATOR-UNKNOWN-001"
+    PROCESS_SPAWN_FAILED = "SKILLEVALUATOR-RUNTIME-005"
+    EXECUTION_TIMEOUT = "SKILLEVALUATOR-RUNTIME-007"
+    PROCESS_EXITED = "SKILLEVALUATOR-RUNTIME-008"
+    JOB_RESULT_INVALID = "SKILLEVALUATOR-CONTRACT-007"
+
+
+@dataclass(frozen=True, slots=True)
+class ErrorCodeDefinition:
+    """Immutable public metadata for one registered error code."""
+
+    domain: ErrorDomain
+    summary: str
+
+
+# Registry insertion order is allocation order. Add new entries at the end.
+ERROR_CODE_REGISTRY: Mapping[str, ErrorCodeDefinition] = MappingProxyType(
+    {
+        EvaluatorErrorCode.AUTHENTICATION.value: ErrorCodeDefinition(
+            ErrorDomain.AUTH,
+            "Provider authentication failed.",
+        ),
+        EvaluatorErrorCode.AUTHORIZATION.value: ErrorCodeDefinition(
+            ErrorDomain.AUTH,
+            "Provider authorization failed.",
+        ),
+        EvaluatorErrorCode.INVALID_CONFIGURATION.value: ErrorCodeDefinition(
+            ErrorDomain.CONFIG,
+            "Evaluator configuration was invalid.",
+        ),
+        EvaluatorErrorCode.DEPENDENCY_UNAVAILABLE.value: ErrorCodeDefinition(
+            ErrorDomain.DEPENDENCY,
+            "A required dependency was unavailable.",
+        ),
+        EvaluatorErrorCode.MODEL_NOT_FOUND.value: ErrorCodeDefinition(
+            ErrorDomain.DEPENDENCY,
+            "The configured model was not found.",
+        ),
+        EvaluatorErrorCode.DEPENDENCY_TIMEOUT.value: ErrorCodeDefinition(
+            ErrorDomain.DEPENDENCY,
+            "A required dependency timed out.",
+        ),
+        EvaluatorErrorCode.RATE_LIMITED.value: ErrorCodeDefinition(
+            ErrorDomain.DEPENDENCY,
+            "A required dependency rate-limited the evaluator.",
+        ),
+        EvaluatorErrorCode.UNSUPPORTED.value: ErrorCodeDefinition(
+            ErrorDomain.DEPENDENCY,
+            "The selected dependency operation is unsupported.",
+        ),
+        EvaluatorErrorCode.INVALID_RESPONSE.value: ErrorCodeDefinition(
+            ErrorDomain.DEPENDENCY,
+            "A required dependency returned an invalid response.",
+        ),
+        EvaluatorErrorCode.OTHER_HTTP.value: ErrorCodeDefinition(
+            ErrorDomain.DEPENDENCY,
+            "A required dependency returned another HTTP failure.",
+        ),
+        EvaluatorErrorCode.UNKNOWN.value: ErrorCodeDefinition(
+            ErrorDomain.UNKNOWN,
+            "The evaluator could not classify the failure.",
+        ),
+        EvaluatorErrorCode.PROCESS_SPAWN_FAILED.value: ErrorCodeDefinition(
+            ErrorDomain.RUNTIME,
+            "The evaluator could not start a runtime process.",
+        ),
+        EvaluatorErrorCode.EXECUTION_TIMEOUT.value: ErrorCodeDefinition(
+            ErrorDomain.RUNTIME,
+            "Evaluator runtime execution timed out.",
+        ),
+        EvaluatorErrorCode.PROCESS_EXITED.value: ErrorCodeDefinition(
+            ErrorDomain.RUNTIME,
+            "An evaluator runtime process exited unsuccessfully.",
+        ),
+        EvaluatorErrorCode.JOB_RESULT_INVALID.value: ErrorCodeDefinition(
+            ErrorDomain.CONTRACT,
+            "The evaluator runtime produced an invalid job result.",
+        ),
+    }
+)
+
+ERROR_CODE_PATTERN_TEXT = r"^SKILLEVALUATOR-[A-Z][A-Z0-9]*-[0-9]{3}$"
+ERROR_CODE_PATTERN = re.compile(ERROR_CODE_PATTERN_TEXT, flags=re.ASCII)
+
+
+def error_code_schema() -> dict[str, object]:
+    """Return a serialization-ready JSON Schema for registered error codes."""
+    return {
+        "$schema": "https://json-schema.org/draft/2020-12/schema",
+        "title": "SkillEvaluator error code",
+        "type": "string",
+        "pattern": ERROR_CODE_PATTERN_TEXT,
+        "enum": list(ERROR_CODE_REGISTRY),
+    }
+
+
+def _validate_registry() -> None:
+    enum_values = {member.value for member in EvaluatorErrorCode}
+    if set(ERROR_CODE_REGISTRY) != enum_values:
+        raise RuntimeError("error code registry and enum are inconsistent")
+    for code, definition in ERROR_CODE_REGISTRY.items():
+        if not code.isascii() or ERROR_CODE_PATTERN.fullmatch(code) is None:
+            raise RuntimeError(f"invalid registered error code: {code!r}")
+        if f"-{definition.domain.value}-" not in code:
+            raise RuntimeError(f"error code domain does not match its registry entry: {code}")
+
+
+_validate_registry()
+
+
+def is_registered_error_code(value: object) -> bool:
+    """Return whether *value* is an exact registered error code."""
+    return isinstance(value, str) and value.isascii() and value in ERROR_CODE_REGISTRY
+
+
+def validate_error_code(value: object) -> str:
+    """Return a valid registered code or raise ``ValueError``."""
+    if not is_registered_error_code(value):
+        raise ValueError(f"unregistered SkillEvaluator error code: {value!r}")
+    return str(value)
+
+
+def provider_failure_error_code(
+    failure_kind: object,
+    http_status: object = None,
+    *,
+    timed_out: bool = False,
+) -> EvaluatorErrorCode:
+    """Map structured provider failure metadata without inspecting message text."""
+    kind = str(failure_kind) if failure_kind is not None else "unknown"
+    status = http_status if isinstance(http_status, int) and not isinstance(http_status, bool) else None
+    if kind == "local_process":
+        return EvaluatorErrorCode.EXECUTION_TIMEOUT if timed_out else EvaluatorErrorCode.PROCESS_SPAWN_FAILED
+    if kind == "unavailable":
+        if status == 429:
+            return EvaluatorErrorCode.RATE_LIMITED
+        if timed_out or status == 408:
+            return EvaluatorErrorCode.DEPENDENCY_TIMEOUT
+        return EvaluatorErrorCode.DEPENDENCY_UNAVAILABLE
+    return {
+        "authentication": EvaluatorErrorCode.AUTHENTICATION,
+        "authorization": EvaluatorErrorCode.AUTHORIZATION,
+        "invalid_configuration": EvaluatorErrorCode.INVALID_CONFIGURATION,
+        "model_not_found": EvaluatorErrorCode.MODEL_NOT_FOUND,
+        "unsupported": EvaluatorErrorCode.UNSUPPORTED,
+        "invalid_response": EvaluatorErrorCode.INVALID_RESPONSE,
+        "other_http": EvaluatorErrorCode.OTHER_HTTP,
+        "unknown": EvaluatorErrorCode.UNKNOWN,
+    }.get(kind, EvaluatorErrorCode.UNKNOWN)
+
+
+def primary_error_code(codes: Iterable[object]) -> str:
+    """Return a consensus code, or UNKNOWN when failures conflict or are absent."""
+    registered = {str(code) for code in codes if is_registered_error_code(code)}
+    return registered.pop() if len(registered) == 1 else EvaluatorErrorCode.UNKNOWN.value
+
+
+__all__ = (
+    "ERROR_CODE_PATTERN",
+    "ERROR_CODE_PATTERN_TEXT",
+    "ERROR_CODE_REGISTRY",
+    "ErrorCodeDefinition",
+    "ErrorDomain",
+    "EvaluatorErrorCode",
+    "error_code_schema",
+    "is_registered_error_code",
+    "primary_error_code",
+    "provider_failure_error_code",
+    "validate_error_code",
+)