Repository navigation
feat(monitoring): health check, alerts, backend-requested logging; v0.1.0 verification - #28
Merged
Merged
Conversation
…uested logging; v0.1.0 verification run Monitoring - scripts/probe.py: health check per target (API contract cases with expected statuses, latency sample, gateway decisions must be answered by Vertex); one JSON log line per target (probe_event=dgem.probe). Works with ADC locally and the metadata server on Cloud Run; IAP gateways via PROBE_GATEWAY_AUDIENCE. - scripts/deploy_probe.sh + deploy/probe/Dockerfile: least-privilege service account, two Cloud Run jobs (hourly Vertex + gateway, daily Cloud Run) and Cloud Scheduler triggers. - scripts/setup_alerts.py: log-based metrics (decisions by backend, decision wall time, probe results and latency), email channel, seven 'dgem:' alert policies (PromQL), every threshold overridable; idempotent, --dry-run. - docs/operate/monitoring.md: what each alert means and what to do, how to change thresholds, deploy and remove; linked from the runbook and journey. Gateway logging - dgem_backend_requested logged next to dgem_backend so a vertex_first failover is distinguishable from an explicit Cloud Run request; failed requests log backend 'none' instead of 'cloudrun'. Verification (benchmarks/runs/20260928-v010-verification) - All sample suites against production v0.1.0 pass; two pre-existing harness issues documented (bench-intents banking77 > 26 options; harness-only templates). Rollback targets exercised, including a traffic drill on the redeployed Vertex rollback model. scripts/template_sweep.py added. CHANGELOG v0.1.2 (unreleased); deployment log updated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Monitoring
statuses, latency sample, gateway decisions must be answered by Vertex); one
JSON log line per target (probe_event=dgem.probe). Works with ADC locally and
the metadata server on Cloud Run; IAP gateways via PROBE_GATEWAY_AUDIENCE.
account, two Cloud Run jobs (hourly Vertex + gateway, daily Cloud Run) and
Cloud Scheduler triggers.
wall time, probe results and latency), email channel, seven 'dgem:' alert
policies (PromQL), every threshold overridable; idempotent, --dry-run.
thresholds, deploy and remove; linked from the runbook and journey.
Gateway logging
is distinguishable from an explicit Cloud Run request; failed requests log
backend 'none' instead of 'cloudrun'.
Verification (benchmarks/runs/20260928-v010-verification)
issues documented (bench-intents banking77 > 26 options; harness-only
templates). Rollback targets exercised, including a traffic drill on the
redeployed Vertex rollback model. scripts/template_sweep.py added.
CHANGELOG v0.1.2 (unreleased); deployment log updated.
Already applied to the hosting project (alerts, metrics, probe jobs). The logging change needs a gateway redeploy (v0.1.2) for the failover and p95 alerts to see data.