Swap Prometheus for Cortex, wire otel-demo metrics, and pre-can alerting

[lezzago]

Summary

• Replace vanilla Prometheus with Cortex (kept as service name prometheus for backward compat); add Alertmanager as a sibling service that always runs.
• Wire otel-demo metrics end-to-end: enable the demo by default, set OTEL_METRICS_EXPORTER=otlp globally, add envoy + collector self-scrape, and strip high-cardinality randomKey on the Cortex branch of the service-map pipeline.
• Surface the Alert Manager UI in OpenSearch Dashboards and point its Prometheus datasource at Cortex's query, Ruler, and Alertmanager URIs.
• Pre-can alerting: 4 stack-health rules (loaded always), 9 otel-demo RED rules on span-derived latency (loaded with overlay), 1 cluster-health monitor in OS, 5 demo trace/log monitors, and Alertmanager routing with an OpenSearch webhook receiver.

Alert Manager UI in observability stack

Notable design calls

• cortex.yaml raises max_label_names_per_series to 50 because resource_to_telemetry_conversion: true pushes JVM/.NET/Node.js series past the default 30-label cap.
• otel-demo-alerts.yml uses span-derived latency_seconds_*{namespace=\"span_derived\"} instead of per-service rpc_server_* / http_server_* — the demo services don't emit RED metrics uniformly, so span-derived gives full coverage.
• cortex-rules-init always upserts via POST /api/v1/rules/{namespace} so rule edits take effect on re-run. Adds a Cortex-readiness retry budget and exits non-zero on any POST failure.
• OS monitor init containers now have required: true on opensearch — the scripts hardcode https://opensearch:9200 so the old required: false with an infinite retry loop was a latent footgun.

Stability + upgrade-path fixes (commit d69b8e6)

Four issues were surfaced during a stability test pass (scenarios: in-place upgrade from pre-PR main, with and without otel-demo overlay, re-deploy idempotency). Details and raw verification output in PR-226-TEST-RESULTS.md / PR-226-FIX-REPORT.md (kept local, not committed).

• P0.1 — stack-monitor init race with the OpenSearch alerting plugin. /_cluster/health green is not sufficient — the plugin allocates its own internal indices after cluster green, and POST /_plugins/_alerting/monitors returns 500 "all shards failed" until they settle. init-stack-monitors.py / init-otel-demo-monitors.py now retry up to 12× / 5s on 5xx. Without this, fresh installs silently ended up with 5/6 monitors.
• P0.2 — in-place upgrades left the ObservabilityStack_Prometheus datasource with only {prometheus.uri, prometheus.auth.*}, so OSD Alert Manager silently surfaced zero alerts. The init now reads authoritative properties via GET /api/dataconnections, and on mismatch migrates via DELETE + POST (SQL plugin doesn't expose a working PUT/PATCH). The DELETE+POST changes the saved-object id, so the migration also cleans up the orphaned pre-PR data-connection saved-object and any correlations whose references still point at it. Reruns are idempotent.
• P1.3 — cortex-rules-init / cortex-rules-init-otel-demo had no healthcheck, so docker compose up -d --wait returned while rules were still loading. Rule counts at --wait return were (0, 0) for ~30s. init-cortex-rules.py now writes /tmp/rules-loaded on clean completion and both services test for that file. Counts at --wait return are now (1, 3) as expected.
• P1.4 — in-place upgrades left stale vanilla-Prometheus TSDB dirs (chunks_head, wal, wbl, lock, queries.active) dangling under /data alongside Cortex's own layout (/data/tsdb, /data/ruler-storage). Cortex entrypoint now cleans these up on first boot only — gated on /data/tsdb being absent AND /data/chunks_head being present, so fresh deploys and subsequent restarts are untouched.

Known follow-ups (not blocking)

• Residual Data Prepper duplicate-sample rejects (~0.15/s) for latency_seconds_* within a single sdk.language. Root cause is otel_apm_service_map's emit cadence within a 10s window; the proper fix is migrating RED metrics to the OTel Collector's spanmetrics connector. Tracked separately.
• opensearch-dashboards-init has no healthcheck, so docker compose up -d --wait returns ~70s before it completes. Not a regression (same behavior as pre-PR main); CI/scripts that query OSD state immediately after --wait may see stale state during that window. Same sentinel-file + healthcheck pattern as P1.3 would fix it.
• Cortex rejects many otel-demo runtime metrics (.NET, JVM, Node.js, Kafka, etc.) with "invalid temporality and type combination". Reproduces on fresh deploy — not an upgrade regression — but reduces the surface of series available to rules. Fix is either a cumulativetodelta processor in the collector or per-SDK OTEL_METRIC_EXPORT_TEMPORALITY_PREFERENCE.

Test plan

• promtool check rules for both rule files → SUCCESS (4 stack, 9 otel-demo)
• amtool check-config for the Alertmanager template → SUCCESS (7 receivers, 1 inhibit rule)
• python3 -m py_compile on the init scripts → OK
• docker compose config renders in both demo-on and demo-off modes, both compose v5.0.2 local and v2.38.2 CI-parity → exit 0
• Full Scenario 2b cold bring-up verified against final commit (d69b8e6):
  • docker compose up -d --wait exits 0 in 35s
  • All 6 OpenSearch monitors created at --wait return (P0.1)
  • Cortex rules loaded at --wait return: 1 stack + 3 otel_demo groups (P1.3)
  • Datasource has all 3 required URI properties after simulated upgrade (P0.2)
  • Exactly 1 data-connection saved-object — no orphans — after upgrade (P0.2 follow-up)
  • Correlations reference the current datasource id (P0.2 follow-up)
  • Cortex volume contains only tsdb/, ruler-storage/, compactor/ after simulated upgrade; fresh deploy unaffected (P1.4)
  • 21 Cortex / 20 Alertmanager alerts firing after 5-min demo soak
• Live verification with the otel-demo overlay:
  • Cortex up{job=\"otel-collector\"} and up{job=\"envoy-frontend-proxy\"} both report 1
  • OSD "Alert Manager" UI renders both OS-monitor alerts and Cortex alerts when both datasources are selected
  • Label-cap rejects: 16/5m → 0/5m after max_label_names_per_series: 50
  • Duplicate-sample rejects: 65/5m → 46/5m (partial — residual tracked as follow-up)
  • Dead otel-demo rules: 5 of 9 → 0 of 9
  • Rule-loader upsert verified by re-running cortex-rules-init after an edit — log shows loaded: 4, failed: 0

🤖 Generated with Claude Code

[lezzago]

Mend finding is a false-positive attribution, not a regression from this PR

The Mend gate flags CVE-2026-6321 in fast-uri@3.1.0 under aws/cdk/package.json → aws-cdk-lib@2.248.0 → table → ajv → fast-uri. This PR does not touch the CDK dependency tree:

$ git diff --name-only origin/main HEAD | grep -iE "cdk|package\.json"
(no matches)

aws/cdk/package.json was last modified in #159 (and added in #153), both well before this branch. The CVE is "new" to Mend's database, not new to this branch — the same scan run against current main would report the same vuln. Mend's base-branch commit in the report (1a8c22a3…) predates when the CVE was published to their feed.

Fix belongs in a separate CDK dependency-bump PR (upgrade aws-cdk-lib or add an npm overrides forcing fast-uri@>=3.1.1). I'll leave that out of this PR to keep the scope clean — happy to file a follow-up issue if useful.

[ps48]

@lezzago why is alert manager not coming up in the left nav?

[lezzago]

@lezzago why is alert manager not coming up in the left nav?

There is a bug in alert manager UI I think with the new side nav changes that were coming in.
I plan to take a look at the issue separately since it is in a different repo and this will pull in that change after its fixed.
For now to view the page, you need the end URI to be: app/observability-alerting

[kylehounslow]

If demo-only, then should move to another compose file e.g. docker-compose.otel-demo.yml

[kylehounslow]

use CORTEX_VERSION env var

[kylehounslow]

How are we handling data retention? Removing this option I am concerned metrics data storage will grow infinitely

[kylehounslow]

what is users don't deploy otel demo?

[lezzago]

Verified silent (no logs, no refused-metrics) when demo is off

[kylehounslow]

Is this init idempotent? I.e. can you run it multiple times and it will work all the same and not duplicate any resources?

[lezzago]

Verified idempotent (5 monitors before/after rerun)

[kylehounslow]

Below mentions alertmanager config only applies with otel-demo enabled. Is this ui-only and not applicable to same logic?

[lezzago]

Yea sadly the name of the UI makes it confusing, but this is concerning to enable the UI page or not.

[kylehounslow]

routes: [service_processed_metrics]

Why remove?

[lezzago]

routes: [service_processed_metrics] still on line 116, moved to sub-pipeline

[kylehounslow]

Is this idempotent?

[lezzago]

Verified idempotent (rule counts match before/after rerun; unconditional upsert)

[kylehounslow]

I am concerned how this may affect existing deployments. Some areas to test:

1. vanilla Prometheus running initially and then making this update to swap to cortex
2. running with/without otel-demo
3. re-deploying multiple times and ensure duplicate resources aren't created (and won't error out if resources already exist)

[joshuali925]

Code review

Found 1 issue:

1. Stale comment claims alertmanager lives in docker-compose.otel-demo.yml and the URI "dangles when the demo is off", but this PR defines alertmanager unconditionally in the main docker-compose.yml — which the adjacent comment a few lines above correctly describes ("Runs whether or not the otel-demo is enabled"). The misleading comment will confuse future readers about when the service runs.

observability-stack/docker-compose.yml

Lines 274 to 279 in d93fd52

	- PROMETHEUS_PORT=${PROMETHEUS_PORT}
	# alertmanager.uri is set on the Prometheus datasource unconditionally.
	# The service itself only starts with the otel-demo flag (alertmanager
	# lives in docker-compose.otel-demo.yml), so this URI dangles when the
	# demo is off — harmless, since no alerts are firing to route anyway.
	- ALERTMANAGER_HOST=alertmanager

Compare with the accurate description at

observability-stack/docker-compose.yml

Lines 145 to 152 in d93fd52

	# Prometheus Alertmanager - Alert routing, grouping, deduplication, and silencing.
	# Runs whether or not the otel-demo is enabled: the base stack rules (collector
	# health, scrape-target health) alert into it, and demo rules alert in when the
	# demo overlay is enabled too. The OSD Prometheus datasource's alertmanager.uri
	# points at this service's HTTP API.
	alertmanager:
	image: prom/alertmanager:${ALERTMANAGER_VERSION}
	container_name: alertmanager

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[lezzago]

I am concerned how this may affect existing deployments. Some areas to test:

1. vanilla Prometheus running initially and then making this update to swap to cortex

2. running with/without otel-demo

3. re-deploying multiple times and ensure duplicate resources aren't created (and won't error out if resources already exist)

Good call — I ran a full stability test pass against those three scenarios and surfaced four real issues. All fixed now.

1. Vanilla Prometheus → Cortex upgrade

• Datasource not migrated. Pre-PR datasource had only prometheus.uri; init short-circuited on name match, so upgraded deployments silently lost the new alertmanager.uri / prometheus.ruler.uri and OSD Alert Manager UI showed zero Cortex alerts. Init now diffs against /api/dataconnections and reconciles via DELETE + POST (PUT/PATCH aren't exposed); also cleans up the orphaned saved-object wrapper and dangling correlations so the graph stays consistent. Reruns are idempotent.
• Stale TSDB dirs. Cortex writes /data/tsdb + /data/ruler-storage alongside leftover chunks_head/wal/wbl. Entrypoint now cleans them up on first boot only, gated on /data/tsdb absent AND /data/chunks_head present.
• Historical metrics aren't migrated — Cortex can't read vanilla-Prometheus blocks. New OTLP writes work immediately but pre-upgrade data is gone. Worth a release note.

2. With / without otel-demo

• Base only: 15 containers, 1 stack rule group, AM empty, OSD reachable.
• With demo: 38 containers, 1 stack + 3 otel_demo rule groups, 6 monitors, 21 Cortex / 20 AM alerts firing after 5-min soak.
• Compose renders exit 0 on both compose v5.0.2 and v2.38.2 (CI parity).

Race surfaced: stack-monitor init vs. OpenSearch alerting plugin readiness. /_cluster/health green isn't enough — the plugin's indices still need to allocate.
Fresh installs silently ended up with 5/6 monitors. Monitor-create now retries on 5xx / "all shards failed" (up to 12× / 5s).

3. Re-deploy idempotency

Tested three teardown cycles + force-recreate of every init container:

• Repeat up: exit 0, IDs stable, zero error log lines.
• Rule loaders: loaded: N, failed: 0 on rerun; live-edit round-trip works.
• Monitor loaders: all log Monitor already exists; count stable.
• OSD init: saved-object counts stable across reruns (no duplicates).

Related bug fixed: cortex-rules-init had no healthcheck, so --wait returned while rules were still loading (counts (0, 0) for ~30s). Now writes /tmp/rules-loaded on clean completion; --wait blocks properly and counts are (1, 3) at return.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 55.62%. Comparing base (15f21e2) to head (033a0d2).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #226   +/-   ##
=======================================
  Coverage   55.62%   55.62%           
=======================================
  Files           4        4           
  Lines         169      169           
  Branches       48       48           
=======================================
  Hits           94       94           
  Misses         74       74           
  Partials        1        1           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.