feat(monitoring): health check, alerts, backend-requested logging; v0.1.0 verification

CHANGELOG.md

@@ -7,6 +7,16 @@ changes, new features. **Patch:** fixes. Images are published to
 a release is validated. Release process: [runbook](docs/operate/runbook.md#release-a-new-version). Gateway/CLI-only releases
 (`make release-gateway`) reuse the previous serving images.
 
+## v0.1.2 (unreleased)
+
+Gateway and monitoring; serving images unchanged (v0.1.0).
+
+- Monitoring: scheduled health check (`scripts/probe.py`, `scripts/deploy_probe.sh`: Cloud Run jobs + Scheduler),
+  alert policies and log-based metrics (`scripts/setup_alerts.py`), guide `docs/operate/monitoring.md`.
+- Gateway decision logs record the requested backend (`dgem_backend_requested`) next to the one that answered, and
+  failed requests are logged with backend `none` instead of `cloudrun`, so failover can be measured.
+- `scripts/template_sweep.py`: run every policy template through a gateway with its sample variables.
+
 ## v0.1.1 (2026-09-28)
 
 Gateway, MCP and CLI only. **Serving images are unchanged**: the latest serving image is still `v0.1.0`

benchmarks/runs/20260925-serving-speed/deployments.md

@@ -29,6 +29,10 @@
 
 | 09-28 ~21:00 | Cloud Run `dgemma-gateway` → revision `00057` | **v0.1.1** (gateway/CLI-only release; serving stays v0.1.0): PRs #19, #22, #23 on top of #18; admin API off | `/health` version v0.1.1; all backends route; admin POSTs 403; unknown backend / foreign `vertex_url` 400; MCP tool schemas list `[vertex_first, vertex, cloudrun]` and reject unknown backends at validation; IAP custom domain; `dgem mcp --remote` | Revision `00055` |
 
+| 09-28 22:39–23:05 | Verification + rollback drill ([`20260928-v010-verification`](../20260928-v010-verification/README.md)) | All sample suites on v0.1.0; rollback model `504638d` redeployed on the endpoint and given 100% traffic for ~30 s, then traffic restored and the drill copy undeployed | All pass (two pre-existing harness issues noted) | — |
+| 09-28 23:10 | Cloud Run `dgemma`, `dgemma-gateway`; Vertex model registry | Pruned to n and n-1: 81 revisions deleted (kept gateway `00057`, `00055`; serving `00034`, `00032`); 3 older model resources deleted (kept v0.1.0 and `504638d`) | — | — |
+| 09-28 22:50–23:01 | Monitoring | Health check `dgem-probe-hourly` / `-daily` (Cloud Run jobs + Scheduler, SA `dgem-probe-sa`), 4 log-based metrics, 7 `dgem:` alert policies (email) | First runs: Vertex ok 78 ms, gateway ok 192 ms, Cloud Run ok 82 ms | — |
+
 Follow-ups:
 - ~~Rotate the Hugging Face token~~ done (version 2).
 - ~~Cloud Run health lacks `warmed`~~ not a bug: the earlier check ran before warmup finished; later checks show

benchmarks/runs/20260928-v010-verification/README.md

@@ -0,0 +1,40 @@
+# Run 20260928-v010-verification
+
+Before pruning old revisions, every sample suite was run against production **v0.1.0** (Vertex G4 endpoint, via
+`--vertex-url`) in one session, and the retained rollback targets (n-1) were exercised. Numbers compare with the
+committed receipts in `benchmarks/` (earlier images and hardware) and with the image-parity run.
+
+## Version n: v0.1.0 on Vertex G4
+
+| Suite | Items | Result | Reference | Verdict |
+| :--- | :---: | :--- | :--- | :--- |
+| `bench-jev` | 231 | 187 correct (81.0%) | Band 182–192 (image parity) | Pass |
+| `bench-calibration` | 50 | 46 (92%) | 43–46 (image parity) | Pass |
+| `bench` (triage, code review, security) | 30 | 22 (73.3%) | 23–24 on L4 receipts | Pass (±2 items is noise on 30) |
+| `bench-intents` clinc150 (small set) | 30 | 29 (96.7%) after tie-break; domain 28/30 | Committed receipt 96.7% | Pass |
+| `bench-intents` banking77 (small set) | 30 | 0: every request rejected, `at most 26 alternatives` | Committed receipt (Sep 20, older server) | **Harness issue**, see below |
+| `bench-bbox` | 12 | mIoU 0.406 argmax / 0.422 expectation | 0.290 / 0.377 (Cloud Run L4) | Pass |
+| `bench-rerank` | 30 queries | nDCG@10 0.824, 0.856, 0.856 (3 runs); 0% ties, 100% poison quarantine | 0.9265 Cloud Run L4, 0.8502 Vertex L4 | Pass (in the range of earlier receipts) |
+| `bench-permutation` | 16 | 100% accuracy, 6.25% flip rate | EXP-13 | Pass |
+| `bench-decision-index` panel | 22 | 22/22 supported, index 98.89 | 98.89 | Pass |
+| Every policy template through the gateway (`scripts/template_sweep.py`) | 28 | 23 pass; 5 need harness-supplied variables (below) | — | Pass |
+| `bench-ecotone` | — | Not run (needs the C++ ecotone sidecar) | — | Skipped |
+
+**Two findings, both pre-existing, not regressions:**
+- `bench-intents --dataset banking77` sends all 30 candidate intents in one `choice` question. The server has
+  rejected more than 26 options since before v0.1.0 (`ab208dd` had the same check); the committed receipt predates
+  it. The harness should split into brackets (as `dgem systemone serve` does) or use the 26-option slice.
+- `intent_banking77`, `intent_clinc150`, `intent_tiebreak`, `jevbench_generic` and `tn_disambiguation` are harness
+  templates: they take option lists from `bench-intents`, `bench-jev` or `bench-ecotone`, so the gateway's sample
+  variables leave them with fewer than two options (400). The harnesses themselves pass (above).
+
+## Rollback targets (n-1)
+
+| Target | Check | Result |
+| :--- | :--- | :--- |
+| Vertex model `504638d` (redeployed on the production endpoint, then 100% of traffic for the drill, then back) | Contract (`contract_rollback_vertex.json`), calibration | Contract identical (the digit-string `samples` case returns 400, confirming the older model served); 44/50 |
+| Cloud Run revision on `504638d` (tagged URL) | Contract, calibration | Contract identical; 43/50 |
+| Gateway revision on `v0.1.0-5-g3216438` (tagged URL) | Routing, admin 403s, unknown backend / foreign `vertex_url` 400s | All pass |
+
+After the checks, older Cloud Run revisions (81) and three older Vertex model resources were deleted; the drill copy
+of the rollback model was undeployed (its model resource is kept).