Add focused alerting and durable incident evidence

[biancarosa]

Problem and behavior

Traffic investigation is useful while captures remain in memory, but sustained failures need an external signal and evidence that survives buffer eviction. This adds opt-in focused alerting through Prometheus and Alertmanager, plus a separate netkit alerts receiver that preserves bounded incident metadata and links back to the matching Traffic scope.

• Example rules distinguish upstream HTTP 5xx, transport failures, scrape unavailability and insufficient samples. Latency alerting requires an explicitly configured ordinary-workload profile; mixed traffic remains unclassified by default.
• Alertmanager examples cover grouping, repeats, silences, inhibition and recovery. No real notification destination is configured.
• The receiver accepts authenticated webhooks for one configured instance, deduplicates retries and preserves the first snapshot through recovery. Snapshots exclude URLs, headers, bodies and arbitrary alert text; default retention is 100 incidents / seven days with private files.
• Signal and time filters connect incidents to current Traffic. Missing or expired evidence never implies health.

Validation

• Go race-enabled unit tests and fresh-binary end-to-end suite; golangci-lint.
• Dashboard unit tests, TypeScript, ESLint and production build.
• Official promtool rule scenarios: sustained failures, brief spikes, sample gates, no data, downtime/recovery, cancellation exclusions and latency-profile gating.
• Real Alertmanager integration: grouping, repeat delivery, silence suppression, recovery and metadata-only durable deduplication.
• Browser-verified investigation links and saved evidence surviving live-history clearing.
• New CI job validates rules and the real notification lifecycle using checksum-verified pinned monitoring tools.

Operational boundaries

Rules are instance-scoped because Netkit does not export per-host/route metrics. Prometheus and Alertmanager own evaluation and delivery; this is not a native monitor editor. The incident UI defaults to loopback and requires authenticated private ingress before exposure. Deploy the receiver independently of the monitored proxy for availability incidents. Setup and limitations are documented in examples/alerting/README.md.

No production deployment or external notifications were performed.