diff --git a/docs/change/requests/CR-118-local-backend-telemetry-schema.md b/docs/change/requests/CR-118-local-backend-telemetry-schema.md new file mode 100644 index 0000000..c88c578 --- /dev/null +++ b/docs/change/requests/CR-118-local-backend-telemetry-schema.md @@ -0,0 +1,54 @@ +# CR-118: Local Backend Telemetry Record Schema + +## Status + +Implemented (contract-only) + +## Summary + +Harden the existing local backend probe's serialized record contract before +any further probe or routing work. This slice adds a JSON Schema artifact and +a pure strict mapping validator for `local_backend_probe_record.v1` records. +It does not execute probes, call models, write the ledger, change CLI +behavior, or wire telemetry into routing. + +## Scope + +- Add `schemas/local_backend_probe_record.schema.json` for the v1 record + shape emitted by `LocalBackendProbeRecord.to_dict()`. +- Add `local_backend_probe_record_from_mapping(...)` as a pure validator and + mapper for already-serialized telemetry records. +- Keep `source_type` closed to `ollama`, `lm_studio`, `llama_cpp`, and the + normalized `unsupported` sentinel. +- Require `source_type: "unsupported"` when `error_category: + "unsupported_backend"`; do not persist arbitrary requested backend names. +- Preserve metadata-only privacy rules: no prompts, completions, raw context, + credentials, private paths, or path-like model identifiers. +- Preserve deterministic fixture rules: `synthetic_fixture` records carry no + `observed_at` timestamp. + +## Non-Goals + +- No new probe execution path. +- No endpoint calls in new tests. +- No routing integration, `ResilienceRouteInput` population, circuit + breakers, degraded modes, or daily-driver enforcement. +- No ledger writes. +- No model, completion, chat, or embedding calls. +- No CLI behavior changes. + +## Validation + +- `tests/test_cr_118_local_backend_probe_contract.py` covers synthetic + fixture round-trip validation, unknown-field rejection, persistent privacy + invariant enforcement, closed `source_type`, the `unsupported` sentinel + relationship, path-like model identifier rejection, base URL normalization, + and timestamp exclusion for fixture-tier records. +- Existing local backend probe tests continue to run offline; the unsupported + backend record now serializes as `source_type: "unsupported"`. + +## Notes + +This slice intentionally hardens the evidence shape before expanding +behavior. CR-119+ remains the lane for any future probe-behavior changes, +routing use, or operational telemetry integration. diff --git a/docs/operations/local-backend-telemetry.md b/docs/operations/local-backend-telemetry.md index 8dc66ad..93ce063 100644 --- a/docs/operations/local-backend-telemetry.md +++ b/docs/operations/local-backend-telemetry.md @@ -20,6 +20,9 @@ Partially implemented. **CR-114 implements the read-only metadata probe slice** only: the `triage_core/local_backend_probe.py` module and the `tc probe` command produce metadata-only records against the endpoints below, with the closed failure vocabulary, tiered evidence, and `base_url` redaction described here. +**CR-118 pins the serialized record contract** with +`schemas/local_backend_probe_record.schema.json` and a pure strict mapping +validator for synthetic fixtures and recorded metadata. Routing wiring, route-input population (`ResilienceRouteInput`), circuit breakers, degraded modes, and any daily-driver enforcement **remain future @@ -64,7 +67,7 @@ Candidate record fields, all metadata-only: | Field | Design intent | Privacy consideration | |---|---|---| -| `source_type` | Closed vocabulary above. | None; closed vocabulary. | +| `source_type` | Closed serialized vocabulary: `ollama`, `lm_studio`, `llama_cpp`, or the normalized `unsupported` sentinel. | None; arbitrary backend identifiers are not serialized. `unsupported_backend` records must use `source_type: "unsupported"` rather than preserving the raw requested backend. | | `base_url` | The operator-supplied endpoint, stored for display only after a redaction check. | Store scheme, host, and port only. Reject or redact userinfo (`user:pass@`), query strings, and path segments beyond the documented endpoint path, so an endpoint URL cannot smuggle a token or private path into evidence. | | `reachable` | `true`/`false` for the single metadata probe. | None; boolean. | | `model_count` | Number of models the backend reported. | Preferred default: a count leaks less than names. | @@ -74,10 +77,10 @@ Candidate record fields, all metadata-only: | `error_category` | Closed failure vocabulary below; present only when the probe did not succeed. | None; closed vocabulary, no raw error text with embedded paths or hosts. | | `evidence_tier` | One of `synthetic_fixture`, `local_metadata_probe`, `operator_recorded`. | Keeps deterministic fixtures, actual probe output, and operator-typed claims distinguishable in every downstream report. | -Exact schema, field names, and validation rules are decided by the -implementation CR, which should route the record through the same strict -mapping/validation pattern as runtime strategy evidence (unknown fields -rejected, persistent privacy invariant enforced). +The CR-118 schema and pure validator define the exact v1 serialized contract: +unknown fields are rejected, the persistent privacy invariant is enforced, +`synthetic_fixture` records carry no timestamp, and `source_type: +"unsupported"` is reserved for `error_category: "unsupported_backend"`. ## Failure Categories @@ -89,7 +92,7 @@ error text: | `endpoint_unreachable` | Connection refused or no route to the endpoint. | | `timeout` | The metadata request exceeded its bounded timeout. | | `malformed_response` | The endpoint answered but the body did not match the expected metadata shape. | -| `unsupported_backend` | The requested `source_type` is outside the closed vocabulary. | +| `unsupported_backend` | The requested backend is outside the supported probe vocabulary; serialized records use `source_type: "unsupported"` and do not persist the raw requested backend identifier. | | `permission_or_policy_blocked` | A local permission or policy prevented the probe (e.g. an application-control or firewall block). | | `probe_disabled` | Probing is disabled in the current configuration; the default posture is disabled until explicitly invoked. | @@ -127,8 +130,8 @@ privacy invariant as every other persisted evidence record. What a future reviewer should be able to verify without running a model: - **Record shape.** Deterministic `synthetic_fixture`-tier examples validate - through the strict mapping path and render through whatever report surface - the implementation CR adds, with no backend present at all. + through the strict mapping path and schema contract, with no backend + present at all. - **Fail-closed behavior.** Each failure category is reachable in tests without a live backend (unsupported source type, disabled probe, and malformed fixture responses need no network; unreachable/timeout can use a diff --git a/schemas/local_backend_probe_record.schema.json b/schemas/local_backend_probe_record.schema.json new file mode 100644 index 0000000..5c98fea --- /dev/null +++ b/schemas/local_backend_probe_record.schema.json @@ -0,0 +1,94 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://triagecore.local/schemas/local_backend_probe_record.schema.json", + "title": "TriageCore Local Backend Probe Record", + "type": "object", + "required": [ + "schema_version", + "source_type", + "base_url", + "reachable", + "evidence_tier", + "model_count", + "observed_models", + "response_latency_ms", + "error_category", + "observed_at" + ], + "additionalProperties": false, + "properties": { + "schema_version": {"const": "local_backend_probe_record.v1"}, + "source_type": { + "enum": ["ollama", "lm_studio", "llama_cpp", "unsupported"] + }, + "base_url": { + "anyOf": [ + {"type": "null"}, + { + "type": "string", + "pattern": "^[A-Za-z][A-Za-z0-9+.-]*://[^/?#@]+(:[0-9]+)?$" + } + ] + }, + "reachable": {"type": "boolean"}, + "evidence_tier": { + "enum": ["synthetic_fixture", "local_metadata_probe", "operator_recorded"] + }, + "model_count": {"type": ["integer", "null"], "minimum": 0}, + "observed_models": { + "anyOf": [ + {"type": "null"}, + { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "not": {"pattern": "[\\\\/]"} + } + } + ] + }, + "response_latency_ms": {"type": ["integer", "null"], "minimum": 0}, + "error_category": { + "enum": [ + null, + "endpoint_unreachable", + "timeout", + "malformed_response", + "unsupported_backend", + "permission_or_policy_blocked", + "probe_disabled" + ] + }, + "observed_at": {"type": ["string", "null"], "minLength": 1} + }, + "allOf": [ + { + "if": { + "properties": {"error_category": {"const": "unsupported_backend"}}, + "required": ["error_category"] + }, + "then": { + "properties": {"source_type": {"const": "unsupported"}} + } + }, + { + "if": { + "properties": {"source_type": {"const": "unsupported"}}, + "required": ["source_type"] + }, + "then": { + "properties": {"error_category": {"const": "unsupported_backend"}} + } + }, + { + "if": { + "properties": {"evidence_tier": {"const": "synthetic_fixture"}}, + "required": ["evidence_tier"] + }, + "then": { + "properties": {"observed_at": {"type": "null"}} + } + } + ] +} diff --git a/tests/test_cr_118_local_backend_probe_contract.py b/tests/test_cr_118_local_backend_probe_contract.py new file mode 100644 index 0000000..db15065 --- /dev/null +++ b/tests/test_cr_118_local_backend_probe_contract.py @@ -0,0 +1,158 @@ +import json +from pathlib import Path + +import pytest + +from triage_core.local_backend_probe import ( + SCHEMA_VERSION, + LocalBackendProbeRecord, + local_backend_probe_record_from_mapping, +) + + +SCHEMA_PATH = Path("schemas/local_backend_probe_record.schema.json") + + +def _synthetic_record(**overrides): + record = { + "schema_version": SCHEMA_VERSION, + "source_type": "ollama", + "base_url": "http://localhost:11434", + "reachable": True, + "evidence_tier": "synthetic_fixture", + "model_count": 2, + "observed_models": None, + "response_latency_ms": None, + "error_category": None, + "observed_at": None, + } + record.update(overrides) + return record + + +def test_synthetic_fixture_contract_round_trips_without_probe_execution(): + payload = _synthetic_record() + + loaded = local_backend_probe_record_from_mapping(payload) + + assert loaded.to_dict() == payload + + +def test_contract_rejects_unknown_fields(): + payload = _synthetic_record(extra_field="not allowed") + + with pytest.raises(ValueError, match="unknown field: extra_field"): + local_backend_probe_record_from_mapping(payload) + + +def test_contract_rejects_missing_fields(): + payload = _synthetic_record() + payload.pop("observed_models") + + with pytest.raises(ValueError, match="missing field: observed_models"): + local_backend_probe_record_from_mapping(payload) + + +def test_contract_rejects_raw_persistent_content_keys(): + payload = _synthetic_record(raw_prompt="do not persist") + + with pytest.raises(ValueError, match="raw_prompt|forbidden"): + local_backend_probe_record_from_mapping(payload) + + +def test_contract_rejects_arbitrary_source_type(): + payload = _synthetic_record(source_type="vllm") + + with pytest.raises(ValueError, match="invalid source_type"): + local_backend_probe_record_from_mapping(payload) + + +def test_unsupported_backend_requires_unsupported_sentinel(): + payload = _synthetic_record( + source_type="ollama", + reachable=False, + error_category="unsupported_backend", + ) + + with pytest.raises(ValueError, match="source_type=unsupported"): + local_backend_probe_record_from_mapping(payload) + + +def test_unsupported_sentinel_is_reserved_for_unsupported_backend(): + payload = _synthetic_record( + source_type="unsupported", + reachable=False, + error_category="endpoint_unreachable", + ) + + with pytest.raises(ValueError, match="reserved for unsupported_backend"): + local_backend_probe_record_from_mapping(payload) + + +def test_valid_unsupported_backend_record_uses_sentinel_only(): + payload = _synthetic_record( + source_type="unsupported", + reachable=False, + model_count=None, + error_category="unsupported_backend", + ) + + loaded = local_backend_probe_record_from_mapping(payload) + + assert loaded.source_type == "unsupported" + assert loaded.error_category == "unsupported_backend" + + +def test_contract_rejects_path_like_model_identifiers(): + payload = _synthetic_record(observed_models=["qwen2.5", "C:\\models\\private.gguf"]) + + with pytest.raises(ValueError, match="path-like"): + local_backend_probe_record_from_mapping(payload) + + +def test_contract_rejects_secret_bearing_or_unredacted_base_url(): + with pytest.raises(ValueError, match="userinfo"): + local_backend_probe_record_from_mapping( + _synthetic_record(base_url="http://user:secret@localhost:11434") + ) + with pytest.raises(ValueError, match="query string"): + local_backend_probe_record_from_mapping( + _synthetic_record(base_url="http://localhost:11434/v1/models?token=abc") + ) + with pytest.raises(ValueError, match="normalized"): + local_backend_probe_record_from_mapping( + _synthetic_record(base_url="http://localhost:11434/v1/models") + ) + + +def test_synthetic_fixture_records_forbid_timestamps(): + payload = _synthetic_record(observed_at="2026-07-09T00:00:00+00:00") + + with pytest.raises(ValueError, match="synthetic_fixture"): + local_backend_probe_record_from_mapping(payload) + + +def test_schema_artifact_pins_closed_source_type_and_unsupported_relation(): + schema = json.loads(SCHEMA_PATH.read_text(encoding="utf-8")) + + assert schema["properties"]["source_type"]["enum"] == [ + "ollama", + "lm_studio", + "llama_cpp", + "unsupported", + ] + assert schema["additionalProperties"] is False + assert { + "properties": {"error_category": {"const": "unsupported_backend"}}, + "required": ["error_category"], + } in [rule["if"] for rule in schema["allOf"]] + + +def test_dataclass_rejects_invalid_contract_without_mapping_loader(): + with pytest.raises(ValueError, match="unsupported_backend"): + LocalBackendProbeRecord( + source_type="unsupported", + base_url="http://localhost:11434", + reachable=False, + evidence_tier="synthetic_fixture", + ) diff --git a/tests/test_local_backend_probe.py b/tests/test_local_backend_probe.py index 1c2a1d1..4b0cf71 100644 --- a/tests/test_local_backend_probe.py +++ b/tests/test_local_backend_probe.py @@ -102,6 +102,7 @@ def test_unsupported_backend_is_a_record_not_an_error(): transport=_openai_ok, ) assert rec.reachable is False + assert rec.source_type == "unsupported" assert rec.error_category == "unsupported_backend" diff --git a/triage_core/local_backend_probe.py b/triage_core/local_backend_probe.py index 79ccc37..7c47ec1 100644 --- a/triage_core/local_backend_probe.py +++ b/triage_core/local_backend_probe.py @@ -20,7 +20,7 @@ import time from dataclasses import dataclass from datetime import datetime, timezone -from typing import Any, Callable, Dict, List, Optional, Tuple +from typing import Any, Callable, Dict, List, Mapping, Optional, Sequence, Tuple from urllib.parse import urlsplit from .privacy_invariants import assert_persistent_privacy_safe @@ -28,6 +28,7 @@ SCHEMA_VERSION = "local_backend_probe_record.v1" SOURCE_TYPES = frozenset({"ollama", "lm_studio", "llama_cpp"}) +SERIALIZED_SOURCE_TYPES = SOURCE_TYPES | frozenset({"unsupported"}) EVIDENCE_TIERS = frozenset( {"synthetic_fixture", "local_metadata_probe", "operator_recorded"} ) @@ -49,6 +50,21 @@ "llama_cpp": "/v1/models", } +RECORD_FIELDS = frozenset( + { + "schema_version", + "source_type", + "base_url", + "reachable", + "evidence_tier", + "model_count", + "observed_models", + "response_latency_ms", + "error_category", + "observed_at", + } +) + DEFAULT_TIMEOUT_SECONDS = 3.0 # Model identifiers that look like filesystem paths (may embed private paths). @@ -81,6 +97,10 @@ class LocalBackendProbeRecord: schema_version: str = SCHEMA_VERSION def __post_init__(self) -> None: + if self.schema_version != SCHEMA_VERSION: + raise ValueError(f"schema_version must be {SCHEMA_VERSION}") + if self.source_type not in SERIALIZED_SOURCE_TYPES: + raise ValueError(f"invalid source_type: {self.source_type}") if self.evidence_tier not in EVIDENCE_TIERS: raise ValueError(f"invalid evidence_tier: {self.evidence_tier}") if ( @@ -88,11 +108,37 @@ def __post_init__(self) -> None: and self.error_category not in ERROR_CATEGORIES ): raise ValueError(f"invalid error_category: {self.error_category}") + if ( + self.error_category == "unsupported_backend" + and self.source_type != "unsupported" + ): + raise ValueError( + "unsupported_backend records must use source_type=unsupported" + ) + if ( + self.source_type == "unsupported" + and self.error_category != "unsupported_backend" + ): + raise ValueError( + "source_type=unsupported is reserved for unsupported_backend records" + ) # Determinism: fixture-tier records must carry no timestamp. if self.evidence_tier == "synthetic_fixture" and self.observed_at is not None: raise ValueError("synthetic_fixture records must not carry observed_at") if self.model_count is not None and self.model_count < 0: raise ValueError("model_count must be non-negative") + if self.response_latency_ms is not None and self.response_latency_ms < 0: + raise ValueError("response_latency_ms must be non-negative") + if self.observed_models is not None: + if not isinstance(self.observed_models, Sequence) or isinstance( + self.observed_models, (str, bytes, bytearray) + ): + raise ValueError("observed_models must be a list of strings") + for model_id in self.observed_models: + if not isinstance(model_id, str) or not model_id.strip(): + raise ValueError("observed_models must be a list of strings") + if _PATH_LIKE.search(model_id): + raise ValueError("observed_models must not contain path-like ids") def to_dict(self) -> Dict[str, Any]: record = { @@ -162,6 +208,101 @@ def _extract_models(source_type: str, payload: Any) -> Optional[List[str]]: return [str(n) for n in names if n is not None] +def local_backend_probe_record_from_mapping( + payload: Mapping[str, Any], +) -> LocalBackendProbeRecord: + """Load a serialized local backend probe record through the strict contract. + + This is a pure validator/mapper for already-recorded metadata. It does not + probe endpoints, write artifacts, route tasks, or call a model. + """ + assert_persistent_privacy_safe( + dict(payload), + artifact_name="local backend probe record", + ) + _reject_unknown_record_fields(payload) + + return LocalBackendProbeRecord( + schema_version=_mapping_text(payload, "schema_version"), + source_type=_mapping_text(payload, "source_type"), + base_url=_mapping_optional_base_url(payload, "base_url"), + reachable=_mapping_bool(payload, "reachable"), + evidence_tier=_mapping_text(payload, "evidence_tier"), + model_count=_mapping_optional_int(payload, "model_count"), + observed_models=_mapping_optional_string_list(payload, "observed_models"), + response_latency_ms=_mapping_optional_int(payload, "response_latency_ms"), + error_category=_mapping_optional_text(payload, "error_category"), + observed_at=_mapping_optional_text(payload, "observed_at"), + ) + + +def _reject_unknown_record_fields(payload: Mapping[str, Any]) -> None: + for key in payload: + key_text = str(key) + if key_text not in RECORD_FIELDS: + raise ValueError(f"unknown field: {key_text}") + for key in sorted(RECORD_FIELDS): + if key not in payload: + raise ValueError(f"missing field: {key}") + + +def _mapping_text(payload: Mapping[str, Any], key: str) -> str: + value = payload.get(key) + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"{key} must be a non-empty string") + return value + + +def _mapping_optional_text(payload: Mapping[str, Any], key: str) -> Optional[str]: + value = payload.get(key) + if value is None: + return None + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"{key} must be null or a non-empty string") + return value + + +def _mapping_bool(payload: Mapping[str, Any], key: str) -> bool: + value = payload.get(key) + if not isinstance(value, bool): + raise ValueError(f"{key} must be a boolean") + return value + + +def _mapping_optional_int(payload: Mapping[str, Any], key: str) -> Optional[int]: + value = payload.get(key) + if value is None: + return None + if not isinstance(value, int) or isinstance(value, bool): + raise ValueError(f"{key} must be null or an integer") + return value + + +def _mapping_optional_string_list( + payload: Mapping[str, Any], key: str +) -> Optional[List[str]]: + value = payload.get(key) + if value is None: + return None + if not isinstance(value, Sequence) or isinstance(value, (str, bytes, bytearray)): + raise ValueError(f"{key} must be null or a list of strings") + result: List[str] = [] + for item in value: + if not isinstance(item, str) or not item.strip(): + raise ValueError(f"{key} must be null or a list of strings") + result.append(item) + return result + + +def _mapping_optional_base_url(payload: Mapping[str, Any], key: str) -> Optional[str]: + value = _mapping_optional_text(payload, key) + if value is None: + return None + if redact_base_url(value) != value: + raise ValueError("base_url must be normalized as scheme://host[:port]") + return value + + def _default_transport(url: str, timeout: float) -> Tuple[int, Any]: import requests @@ -215,20 +356,20 @@ def probe_local_backend( # Validate/redact the URL first so a bad URL never reaches the network. redacted = redact_base_url(base_url) - if not enabled: + if source_type not in SOURCE_TYPES: return _record( - source_type=source_type, + source_type="unsupported", base_url=redacted, reachable=False, - error_category="probe_disabled", + error_category="unsupported_backend", ) - if source_type not in SOURCE_TYPES: + if not enabled: return _record( source_type=source_type, base_url=redacted, reachable=False, - error_category="unsupported_backend", + error_category="probe_disabled", ) url = redacted.rstrip("/") + METADATA_PATHS[source_type]