Implement CR-118 telemetry record schema

docs/change/requests/CR-118-local-backend-telemetry-schema.md

@@ -0,0 +1,54 @@
+# CR-118: Local Backend Telemetry Record Schema
+
+## Status
+
+Implemented (contract-only)
+
+## Summary
+
+Harden the existing local backend probe's serialized record contract before
+any further probe or routing work. This slice adds a JSON Schema artifact and
+a pure strict mapping validator for `local_backend_probe_record.v1` records.
+It does not execute probes, call models, write the ledger, change CLI
+behavior, or wire telemetry into routing.
+
+## Scope
+
+- Add `schemas/local_backend_probe_record.schema.json` for the v1 record
+  shape emitted by `LocalBackendProbeRecord.to_dict()`.
+- Add `local_backend_probe_record_from_mapping(...)` as a pure validator and
+  mapper for already-serialized telemetry records.
+- Keep `source_type` closed to `ollama`, `lm_studio`, `llama_cpp`, and the
+  normalized `unsupported` sentinel.
+- Require `source_type: "unsupported"` when `error_category:
+  "unsupported_backend"`; do not persist arbitrary requested backend names.
+- Preserve metadata-only privacy rules: no prompts, completions, raw context,
+  credentials, private paths, or path-like model identifiers.
+- Preserve deterministic fixture rules: `synthetic_fixture` records carry no
+  `observed_at` timestamp.
+
+## Non-Goals
+
+- No new probe execution path.
+- No endpoint calls in new tests.
+- No routing integration, `ResilienceRouteInput` population, circuit
+  breakers, degraded modes, or daily-driver enforcement.
+- No ledger writes.
+- No model, completion, chat, or embedding calls.
+- No CLI behavior changes.
+
+## Validation
+
+- `tests/test_cr_118_local_backend_probe_contract.py` covers synthetic
+  fixture round-trip validation, unknown-field rejection, persistent privacy
+  invariant enforcement, closed `source_type`, the `unsupported` sentinel
+  relationship, path-like model identifier rejection, base URL normalization,
+  and timestamp exclusion for fixture-tier records.
+- Existing local backend probe tests continue to run offline; the unsupported
+  backend record now serializes as `source_type: "unsupported"`.
+
+## Notes
+
+This slice intentionally hardens the evidence shape before expanding
+behavior. CR-119+ remains the lane for any future probe-behavior changes,
+routing use, or operational telemetry integration.

docs/operations/local-backend-telemetry.md

@@ -20,6 +20,9 @@ Partially implemented. **CR-114 implements the read-only metadata probe slice**
 only: the `triage_core/local_backend_probe.py` module and the `tc probe` command
 produce metadata-only records against the endpoints below, with the closed
 failure vocabulary, tiered evidence, and `base_url` redaction described here.
+**CR-118 pins the serialized record contract** with
+`schemas/local_backend_probe_record.schema.json` and a pure strict mapping
+validator for synthetic fixtures and recorded metadata.
 
 Routing wiring, route-input population (`ResilienceRouteInput`), circuit
 breakers, degraded modes, and any daily-driver enforcement **remain future
@@ -64,7 +67,7 @@ Candidate record fields, all metadata-only:
 
 | Field | Design intent | Privacy consideration |
 |---|---|---|
-| `source_type` | Closed vocabulary above. | None; closed vocabulary. |
+| `source_type` | Closed serialized vocabulary: `ollama`, `lm_studio`, `llama_cpp`, or the normalized `unsupported` sentinel. | None; arbitrary backend identifiers are not serialized. `unsupported_backend` records must use `source_type: "unsupported"` rather than preserving the raw requested backend. |
 | `base_url` | The operator-supplied endpoint, stored for display only after a redaction check. | Store scheme, host, and port only. Reject or redact userinfo (`user:pass@`), query strings, and path segments beyond the documented endpoint path, so an endpoint URL cannot smuggle a token or private path into evidence. |
 | `reachable` | `true`/`false` for the single metadata probe. | None; boolean. |
 | `model_count` | Number of models the backend reported. | Preferred default: a count leaks less than names. |
@@ -74,10 +77,10 @@ Candidate record fields, all metadata-only:
 | `error_category` | Closed failure vocabulary below; present only when the probe did not succeed. | None; closed vocabulary, no raw error text with embedded paths or hosts. |
 | `evidence_tier` | One of `synthetic_fixture`, `local_metadata_probe`, `operator_recorded`. | Keeps deterministic fixtures, actual probe output, and operator-typed claims distinguishable in every downstream report. |
 
-Exact schema, field names, and validation rules are decided by the
-implementation CR, which should route the record through the same strict
-mapping/validation pattern as runtime strategy evidence (unknown fields
-rejected, persistent privacy invariant enforced).
+The CR-118 schema and pure validator define the exact v1 serialized contract:
+unknown fields are rejected, the persistent privacy invariant is enforced,
+`synthetic_fixture` records carry no timestamp, and `source_type:
+"unsupported"` is reserved for `error_category: "unsupported_backend"`.
 
 ## Failure Categories
 
@@ -89,7 +92,7 @@ error text:
 | `endpoint_unreachable` | Connection refused or no route to the endpoint. |
 | `timeout` | The metadata request exceeded its bounded timeout. |
 | `malformed_response` | The endpoint answered but the body did not match the expected metadata shape. |
-| `unsupported_backend` | The requested `source_type` is outside the closed vocabulary. |
+| `unsupported_backend` | The requested backend is outside the supported probe vocabulary; serialized records use `source_type: "unsupported"` and do not persist the raw requested backend identifier. |
 | `permission_or_policy_blocked` | A local permission or policy prevented the probe (e.g. an application-control or firewall block). |
 | `probe_disabled` | Probing is disabled in the current configuration; the default posture is disabled until explicitly invoked. |
 
@@ -127,8 +130,8 @@ privacy invariant as every other persisted evidence record.
 What a future reviewer should be able to verify without running a model:
 
 - **Record shape.** Deterministic `synthetic_fixture`-tier examples validate
-  through the strict mapping path and render through whatever report surface
-  the implementation CR adds, with no backend present at all.
+  through the strict mapping path and schema contract, with no backend
+  present at all.
 - **Fail-closed behavior.** Each failure category is reachable in tests
   without a live backend (unsupported source type, disabled probe, and
   malformed fixture responses need no network; unreachable/timeout can use a

schemas/local_backend_probe_record.schema.json

@@ -0,0 +1,94 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://triagecore.local/schemas/local_backend_probe_record.schema.json",
+  "title": "TriageCore Local Backend Probe Record",
+  "type": "object",
+  "required": [
+    "schema_version",
+    "source_type",
+    "base_url",
+    "reachable",
+    "evidence_tier",
+    "model_count",
+    "observed_models",
+    "response_latency_ms",
+    "error_category",
+    "observed_at"
+  ],
+  "additionalProperties": false,
+  "properties": {
+    "schema_version": {"const": "local_backend_probe_record.v1"},
+    "source_type": {
+      "enum": ["ollama", "lm_studio", "llama_cpp", "unsupported"]
+    },
+    "base_url": {
+      "anyOf": [
+        {"type": "null"},
+        {
+          "type": "string",
+          "pattern": "^[A-Za-z][A-Za-z0-9+.-]*://[^/?#@]+(:[0-9]+)?$"
+        }
+      ]
+    },
+    "reachable": {"type": "boolean"},
+    "evidence_tier": {
+      "enum": ["synthetic_fixture", "local_metadata_probe", "operator_recorded"]
+    },
+    "model_count": {"type": ["integer", "null"], "minimum": 0},
+    "observed_models": {
+      "anyOf": [
+        {"type": "null"},
+        {
+          "type": "array",
+          "items": {
+            "type": "string",
+            "minLength": 1,
+            "not": {"pattern": "[\\\\/]"}
+          }
+        }
+      ]
+    },
+    "response_latency_ms": {"type": ["integer", "null"], "minimum": 0},
+    "error_category": {
+      "enum": [
+        null,
+        "endpoint_unreachable",
+        "timeout",
+        "malformed_response",
+        "unsupported_backend",
+        "permission_or_policy_blocked",
+        "probe_disabled"
+      ]
+    },
+    "observed_at": {"type": ["string", "null"], "minLength": 1}
+  },
+  "allOf": [
+    {
+      "if": {
+        "properties": {"error_category": {"const": "unsupported_backend"}},
+        "required": ["error_category"]
+      },
+      "then": {
+        "properties": {"source_type": {"const": "unsupported"}}
+      }
+    },
+    {
+      "if": {
+        "properties": {"source_type": {"const": "unsupported"}},
+        "required": ["source_type"]
+      },
+      "then": {
+        "properties": {"error_category": {"const": "unsupported_backend"}}
+      }
+    },
+    {
+      "if": {
+        "properties": {"evidence_tier": {"const": "synthetic_fixture"}},
+        "required": ["evidence_tier"]
+      },
+      "then": {
+        "properties": {"observed_at": {"type": "null"}}
+      }
+    }
+  ]
+}

tests/test_cr_118_local_backend_probe_contract.py

@@ -0,0 +1,158 @@
+import json
+from pathlib import Path
+
+import pytest
+
+from triage_core.local_backend_probe import (
+    SCHEMA_VERSION,
+    LocalBackendProbeRecord,
+    local_backend_probe_record_from_mapping,
+)
+
+
+SCHEMA_PATH = Path("schemas/local_backend_probe_record.schema.json")
+
+
+def _synthetic_record(**overrides):
+    record = {
+        "schema_version": SCHEMA_VERSION,
+        "source_type": "ollama",
+        "base_url": "http://localhost:11434",
+        "reachable": True,
+        "evidence_tier": "synthetic_fixture",
+        "model_count": 2,
+        "observed_models": None,
+        "response_latency_ms": None,
+        "error_category": None,
+        "observed_at": None,
+    }
+    record.update(overrides)
+    return record
+
+
+def test_synthetic_fixture_contract_round_trips_without_probe_execution():
+    payload = _synthetic_record()
+
+    loaded = local_backend_probe_record_from_mapping(payload)
+
+    assert loaded.to_dict() == payload
+
+
+def test_contract_rejects_unknown_fields():
+    payload = _synthetic_record(extra_field="not allowed")
+
+    with pytest.raises(ValueError, match="unknown field: extra_field"):
+        local_backend_probe_record_from_mapping(payload)
+
+
+def test_contract_rejects_missing_fields():
+    payload = _synthetic_record()
+    payload.pop("observed_models")
+
+    with pytest.raises(ValueError, match="missing field: observed_models"):
+        local_backend_probe_record_from_mapping(payload)
+
+
+def test_contract_rejects_raw_persistent_content_keys():
+    payload = _synthetic_record(raw_prompt="do not persist")
+
+    with pytest.raises(ValueError, match="raw_prompt|forbidden"):
+        local_backend_probe_record_from_mapping(payload)
+
+
+def test_contract_rejects_arbitrary_source_type():
+    payload = _synthetic_record(source_type="vllm")
+
+    with pytest.raises(ValueError, match="invalid source_type"):
+        local_backend_probe_record_from_mapping(payload)
+
+
+def test_unsupported_backend_requires_unsupported_sentinel():
+    payload = _synthetic_record(
+        source_type="ollama",
+        reachable=False,
+        error_category="unsupported_backend",
+    )
+
+    with pytest.raises(ValueError, match="source_type=unsupported"):
+        local_backend_probe_record_from_mapping(payload)
+
+
+def test_unsupported_sentinel_is_reserved_for_unsupported_backend():
+    payload = _synthetic_record(
+        source_type="unsupported",
+        reachable=False,
+        error_category="endpoint_unreachable",
+    )
+
+    with pytest.raises(ValueError, match="reserved for unsupported_backend"):
+        local_backend_probe_record_from_mapping(payload)
+
+
+def test_valid_unsupported_backend_record_uses_sentinel_only():
+    payload = _synthetic_record(
+        source_type="unsupported",
+        reachable=False,
+        model_count=None,
+        error_category="unsupported_backend",
+    )
+
+    loaded = local_backend_probe_record_from_mapping(payload)
+
+    assert loaded.source_type == "unsupported"
+    assert loaded.error_category == "unsupported_backend"
+
+
+def test_contract_rejects_path_like_model_identifiers():
+    payload = _synthetic_record(observed_models=["qwen2.5", "C:\\models\\private.gguf"])
+
+    with pytest.raises(ValueError, match="path-like"):
+        local_backend_probe_record_from_mapping(payload)
+
+
+def test_contract_rejects_secret_bearing_or_unredacted_base_url():
+    with pytest.raises(ValueError, match="userinfo"):
+        local_backend_probe_record_from_mapping(
+            _synthetic_record(base_url="http://user:secret@localhost:11434")
+        )
+    with pytest.raises(ValueError, match="query string"):
+        local_backend_probe_record_from_mapping(
+            _synthetic_record(base_url="http://localhost:11434/v1/models?token=abc")
+        )
+    with pytest.raises(ValueError, match="normalized"):
+        local_backend_probe_record_from_mapping(
+            _synthetic_record(base_url="http://localhost:11434/v1/models")
+        )
+
+
+def test_synthetic_fixture_records_forbid_timestamps():
+    payload = _synthetic_record(observed_at="2026-07-09T00:00:00+00:00")
+
+    with pytest.raises(ValueError, match="synthetic_fixture"):
+        local_backend_probe_record_from_mapping(payload)
+
+
+def test_schema_artifact_pins_closed_source_type_and_unsupported_relation():
+    schema = json.loads(SCHEMA_PATH.read_text(encoding="utf-8"))
+
+    assert schema["properties"]["source_type"]["enum"] == [
+        "ollama",
+        "lm_studio",
+        "llama_cpp",
+        "unsupported",
+    ]
+    assert schema["additionalProperties"] is False
+    assert {
+        "properties": {"error_category": {"const": "unsupported_backend"}},
+        "required": ["error_category"],
+    } in [rule["if"] for rule in schema["allOf"]]
+
+
+def test_dataclass_rejects_invalid_contract_without_mapping_loader():
+    with pytest.raises(ValueError, match="unsupported_backend"):
+        LocalBackendProbeRecord(
+            source_type="unsupported",
+            base_url="http://localhost:11434",
+            reachable=False,
+            evidence_tier="synthetic_fixture",
+        )

tests/test_local_backend_probe.py

@@ -102,6 +102,7 @@ def test_unsupported_backend_is_a_record_not_an_error():
         transport=_openai_ok,
     )
     assert rec.reachable is False
+    assert rec.source_type == "unsupported"
     assert rec.error_category == "unsupported_backend"