cherry-pick: aggregator-head-lag downtime pills → main - #1248
Merged
Conversation
Short-window changes() probe per provider surfaces silent live feeds that the 24h quantile aggregate hides (Codex WS blackout 08:23-09:04 UTC today moved the p50 by nothing but the whole feed was mute for 40min). Bench-level probe_ok gate suppresses badges when our harness side is broken so a Railway crash cannot fake-flag every provider. Co-authored-by: Florent Tapponnier <contact@mobula.io>
…ges==0) (#1195) Companion to feed-down banner: makes the 40min Codex blackout from this morning (08:23-09:04 UTC) visible on the 24h chart instead of a flat line covering the outage. Per-series match preserves the aggregate line as long as one chain still receives events. Co-authored-by: Florent Tapponnier <contact@mobula.io>
Companion to #1195 (series null gaps). The gaps themselves render as line breaks but a break can read as 'the chart missed a sample'. A soft red column behind the break spans the exact outage window and makes the reader understand it was a real feed-side silence. Per line + per contiguous gap range, extended by one step so 1-bucket outages stay visible instead of collapsing to a zero-width sliver. Hidden for excluded (legend-toggled) providers. Co-authored-by: Florent Tapponnier <contact@mobula.io>
…ers (#1210) Feedback on #1203: the neutral red band read as 'something happened here' without telling the reader WHO. Tint each band in the provider's own color, add dashed edges at the outage boundaries, and label 'PROVIDER · SILENT' above the band when it fits (with a caret that survives on narrow single-bucket bands). Works uniformly for Mobula/Codex/Gecko or any future provider with a live_activity query. Co-authored-by: Florent Tapponnier <contact@mobula.io>
Screenshot review: the small text-only label above the band was still too subtle to read as 'this provider is down', and 3 simultaneous provider outages produced overlapping unreadable labels stacked on the same y. Replace with high-contrast rounded pills (white text on provider color) that stack vertically when their X-ranges overlap. Renders as '● NAME DOWN' in bold small caps so a reader gets the event and the provider identity at a glance. Co-authored-by: Florent Tapponnier <contact@mobula.io>
Co-authored-by: Florent Tapponnier <contact@mobula.io>
Two fixes based on 30D view review: - Leading gaps (Prom retention shorter than the visible window, harness started mid-range, provider added recently) rendered as a chart-wide band claiming every provider was down for the missing period. Skip runs that started before the first observed sample. - Rename 'DOWN' to 'DATA MISSING'. A silent series can be the provider going down, our harness losing its WebSocket, or a Prom scrape failure — the pill stays neutral instead of blaming the provider. Co-authored-by: Florent Tapponnier <contact@mobula.io>
#1224) Follow-up to #1221. Dropping the guard entirely restored the 30D range but silently hid today's Codex morning blackout on the 24H view (frozen gauge value = flat line, indistinguishable from a healthy quiet period). Bring the guard back, gated on count_over_time > 5 samples in the window. Modern data at 15s cadence carries ~60 samples per 15 min so the gate fires the moment a WS goes silent. Backfilled data compacted to ~1 sample/hour has count <= 2 in a 15m window so the gate stays open and every point Prom kept is served — no more fake wall-to-wall gap at the start of the 30D range. Co-authored-by: Florent Tapponnier <contact@mobula.io>
Fine-grained outages (24h step = 20 min buckets) rarely align with coarser evaluation timestamps (7d step = 2h, 30d step = 12h) so the Prom query returns a healthy average and the reader sees the pill on 24H but nothing on 7D/30D. Post-process: for every contiguous null run detected on s24, translate the time-ago range to bucket indices on s7 and s30 and null every coarse bucket the run straddles. Contiguity preserved so a single short outage doesn't split into two adjacent pills after propagation. Same treatment applied to per-region series. Co-authored-by: Florent Tapponnier <contact@mobula.io>
…tion (#1230) Bug: the previous math scaled by the fine window fraction, which projected a 7h-ago outage down to 40h-ago on the 7d grid (the pill landed at -1.7d instead of -6h). Both grids share 'now' as their right anchor and only the step sizes differ — take the fine idx's absolute time-ago and divide by the coarse step to find the covering bucket. Co-authored-by: Florent Tapponnier <contact@mobula.io>
Screenshot review: rpc-reliability rendered a 'LAVANODIES DATA MISSING' pill even though its spec never declared live_activity — natural nulls in its series24h (sparse scrapes, backfill edges) triggered the DowntimeBands render. Gate the render on benchmark.results.some(r => r.liveStatus != null), which is only set when the spec declared live_activity. Same gate on propagateNullsToCoarser so other benches' 7d/30d series stay untouched. Co-authored-by: Florent Tapponnier <contact@mobula.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picks the full downtime-detection feature stack from dev, scoped to bench 001 aggregator-head-lag only.
What ships
prometheus.probe_ok+ per-providerqueries.live_activityin its YAML. Other benches unchanged.PRs squashed in
#1193 #1195 #1203 #1210 #1211 #1213 #1215 #1224 #1229 #1230 #1235
After merge
vercel --prodto deploy the site