diff --git a/benchmarks/hyperlane-message-latency.yml b/benchmarks/hyperlane-message-latency.yml new file mode 100644 index 00000000..2360e540 --- /dev/null +++ b/benchmarks/hyperlane-message-latency.yml @@ -0,0 +1,178 @@ +# OpenChainBench. Bench № 105 + +slug: hyperlane-message-latency +number: "105" +title: Fastest Hyperlane source chain by end-to-end message delivery +seo_title: "Fastest Hyperlane source chain 2026" +seo_description: "{{best_name}} leads Hyperlane end-to-end delivery at {{best_p50}} (source send to destination delivery p50, 24h). Observed, not simulated." +subtitle: Wall-clock seconds from source-chain send to destination-chain delivery for a Hyperlane message, measured across Hyperlane-deployed source chains via the public Hyperlane Explorer GraphQL API. + +category: Bridges +status: draft +metric: Hyperlane delivery +unit: ms +higher_is_better: false + +seo_intro: | + Hyperlane is a permissionless cross-chain messaging network. Unlike + Wormhole (fixed Guardian set), Chainlink CCIP (fixed DON) or LayerZero + (application-selected DVN stack), Hyperlane lets anyone run a Mailbox + contract on any chain and lets anyone relay messages between them. + Application security is expressed via an Interchain Security Module + (ISM) the developer configures per lane. This benchmark measures + end-to-end message delivery latency by source chain: wall-clock + seconds from Hyperlane Mailbox `Dispatch` event on source to + `Process` event on destination, for every delivered message the + Hyperlane Explorer indexes. + Every value on this page is an observation of real user traffic , + never a canary transaction. Latency reflects whichever relayer + actually delivered the message, so the reading is practical UX + rather than a theoretical protocol floor. Source-chain block time + sets the observation floor; relayer coverage and ISM verification + drive the tail. + +abstract: | + For each Hyperlane-deployed source chain with mainnet traffic, we + measure the wall-clock latency between the Hyperlane Mailbox + `Dispatch` event on the source chain and the `Process` event on the + destination chain. Data source: the public + `https://api.hyperlane.xyz/v1/graphql` endpoint (Hasura), queried + every 60 seconds for the 100 most recent messages ordered by id + DESC. Each `is_delivered=true` message's + `delivery_occurred_at - send_occurred_at` is one observation of + end-to-end delivery latency, recorded in a per-source-chain + histogram. The p50/p90/p99 shown here are computed via Prometheus + `histogram_quantile` over the last 24 hours. + +methodology: + - "Data source: `https://api.hyperlane.xyz/v1/graphql` (Hasura), the public Hyperlane Explorer feed. Polled every 60 seconds, 100 messages per poll ordered by id DESC." + - "Latency definition: `delivery_occurred_at - send_occurred_at` from a message row with `is_delivered: true`. Both timestamps come from the Mailbox contract events on the source and destination chains." + - "Filter: only delivered messages contribute to latency. Undelivered messages (`is_delivered: false`) are counted separately in `hyperlane_message_undelivered_total` so relayer coverage gaps surface as a distinct signal rather than polluting the latency histogram." + - "Dedupe: last 10 000 msg_ids kept in memory (rolling FIFO). Hyperlane sustained rate is ~150 messages/hour network-wide, so the cache holds ~66 hours of history." + - "Bucketing: histogram with buckets 2, 5, 10, 15, 20, 30, 45, 60, 90, 120, 180, 300, 600, 900, 1200, 1800 seconds (30 min ceiling)." + - "Guard: samples with delta `< 0` or `> 30 min` are dropped (clock skew or stuck messages re-delivered later)." + - "No canary transactions: this bench observes organic Hyperlane traffic. No wallet is funded, no gas is paid." + - "Scope: Hyperlane-deployed source chains with mainnet traffic in the observation window. Non-EVM domains (Solana) map to their own row when Hyperlane exposes them under a numeric domain id." + - "Relayer caveat: Hyperlane is permissionless; the observed latency is whichever relayer actually delivered the message. Applications choosing an under-served or misconfigured Interchain Security Module (ISM) will see structurally slower delivery. The bench reads practical UX, not a theoretical protocol floor." + +findings: + - "{{best_name}} currently leads Hyperlane end-to-end delivery at {{best_p50}} (source send to destination delivery p50, 24h) across measured source chains." + - "Sub-second BFT source chains produce the lowest delivery floors; the tail is dominated by relayer coverage rather than protocol overhead." + - "Permissionless relayer model means chains with fewer active relayers see wider p99 spread than chains with well-resourced relayer sets." + +faq: + - q: "What is Hyperlane?" + a: "Hyperlane is a permissionless cross-chain messaging network. Anyone can deploy the Mailbox contract on a new chain and anyone can relay messages between chains. Application security is configured per lane via an Interchain Security Module (ISM), which is the developer's choice, not a global consensus." + - q: "What does this benchmark measure?" + a: "Wall-clock seconds from a Hyperlane Mailbox `Dispatch` event on the source chain to the `Process` event on the destination chain, for every message the Hyperlane Explorer indexes as delivered. It is a passive measurement of real user traffic." + - q: "Why do some source chains have huge tails?" + a: "Because Hyperlane relayers are permissionless. If an application chooses an under-served or misconfigured ISM, delivery will be slow regardless of Hyperlane's own protocol speed. The tail on this bench reflects practical UX rather than a theoretical protocol floor." + - q: "How does the data get sourced?" + a: "The harness POSTs a GraphQL query to `https://api.hyperlane.xyz/v1/graphql` every 60 seconds and records `delivery_occurred_at - send_occurred_at` for each new delivered message. The endpoint is Hyperlane's own Explorer feed , no auth required. No canary transactions are sent, no user funds are used." + - q: "Which source chain is fastest for Hyperlane today?" + a: "{{best_name}} currently leads at {{best_p50}} (p50 over the last 24 hours). The leaderboard re-sorts continuously against fresh Prometheus samples; the ranking on this page is the ranking right now." + - q: "Can I compare this to the LayerZero, CCIP, or Wormhole bench?" + a: "Only carefully. Each protocol's security model changes what a fair comparison looks like. Hyperlane observed latency is whichever relayer actually delivered, LayerZero is DVN + Executor, CCIP is DON + finality wait + execution, Wormhole is Guardian quorum. The cross-chain messaging comparison meta-bench exposes a shared corridor view so cell values are directly comparable." + +source: https://github.com/ChainBench/OpenChainBench/tree/main/harnesses/hyperlane-message-latency + +prometheus: + window: 24h + freshness_metric: hyperlane_message_seen_total + +providers: + - slug: ethereum + name: Ethereum + tag: Ethereum → any chain, Mailbox Dispatch → destination Process delivery + formula: "50th percentile over 24h of Hyperlane end-to-end delivery latency for messages sourced from Ethereum." + queries: + p50: histogram_quantile(0.50, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="ethereum"}[24h]))) + p90: histogram_quantile(0.90, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="ethereum"}[24h]))) + p99: histogram_quantile(0.99, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="ethereum"}[24h]))) + mean: sum(rate(hyperlane_message_latency_milliseconds_sum{source_chain="ethereum"}[24h])) / sum(rate(hyperlane_message_latency_milliseconds_count{source_chain="ethereum"}[24h])) + sample_size: sum(increase(hyperlane_message_seen_total{source_chain="ethereum"}[24h])) + series: histogram_quantile(0.50, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="ethereum"}[1h]))) + + - slug: bnb + name: BNB Chain + tag: BNB Chain → any chain, ~3s block + Mailbox delivery + formula: "50th percentile over 24h of Hyperlane end-to-end delivery latency for messages sourced from BNB Chain." + queries: + p50: histogram_quantile(0.50, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="bnb"}[24h]))) + p90: histogram_quantile(0.90, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="bnb"}[24h]))) + p99: histogram_quantile(0.99, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="bnb"}[24h]))) + mean: sum(rate(hyperlane_message_latency_milliseconds_sum{source_chain="bnb"}[24h])) / sum(rate(hyperlane_message_latency_milliseconds_count{source_chain="bnb"}[24h])) + sample_size: sum(increase(hyperlane_message_seen_total{source_chain="bnb"}[24h])) + series: histogram_quantile(0.50, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="bnb"}[1h]))) + + - slug: polygon + name: Polygon + tag: Polygon PoS → any chain, Bor block + Mailbox delivery + formula: "50th percentile over 24h of Hyperlane end-to-end delivery latency for messages sourced from Polygon." + queries: + p50: histogram_quantile(0.50, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="polygon"}[24h]))) + p90: histogram_quantile(0.90, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="polygon"}[24h]))) + p99: histogram_quantile(0.99, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="polygon"}[24h]))) + mean: sum(rate(hyperlane_message_latency_milliseconds_sum{source_chain="polygon"}[24h])) / sum(rate(hyperlane_message_latency_milliseconds_count{source_chain="polygon"}[24h])) + sample_size: sum(increase(hyperlane_message_seen_total{source_chain="polygon"}[24h])) + series: histogram_quantile(0.50, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="polygon"}[1h]))) + + - slug: arbitrum + name: Arbitrum + tag: Arbitrum One → any chain, sub-second L2 block + Mailbox delivery + formula: "50th percentile over 24h of Hyperlane end-to-end delivery latency for messages sourced from Arbitrum." + queries: + p50: histogram_quantile(0.50, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="arbitrum"}[24h]))) + p90: histogram_quantile(0.90, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="arbitrum"}[24h]))) + p99: histogram_quantile(0.99, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="arbitrum"}[24h]))) + mean: sum(rate(hyperlane_message_latency_milliseconds_sum{source_chain="arbitrum"}[24h])) / sum(rate(hyperlane_message_latency_milliseconds_count{source_chain="arbitrum"}[24h])) + sample_size: sum(increase(hyperlane_message_seen_total{source_chain="arbitrum"}[24h])) + series: histogram_quantile(0.50, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="arbitrum"}[1h]))) + + - slug: base + name: Base + tag: Base → any chain, 2s block + Mailbox delivery + formula: "50th percentile over 24h of Hyperlane end-to-end delivery latency for messages sourced from Base." + queries: + p50: histogram_quantile(0.50, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="base"}[24h]))) + p90: histogram_quantile(0.90, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="base"}[24h]))) + p99: histogram_quantile(0.99, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="base"}[24h]))) + mean: sum(rate(hyperlane_message_latency_milliseconds_sum{source_chain="base"}[24h])) / sum(rate(hyperlane_message_latency_milliseconds_count{source_chain="base"}[24h])) + sample_size: sum(increase(hyperlane_message_seen_total{source_chain="base"}[24h])) + series: histogram_quantile(0.50, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="base"}[1h]))) + + - slug: optimism + name: Optimism + tag: Optimism → any chain, 2s block + Mailbox delivery + formula: "50th percentile over 24h of Hyperlane end-to-end delivery latency for messages sourced from Optimism." + queries: + p50: histogram_quantile(0.50, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="optimism"}[24h]))) + p90: histogram_quantile(0.90, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="optimism"}[24h]))) + p99: histogram_quantile(0.99, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="optimism"}[24h]))) + mean: sum(rate(hyperlane_message_latency_milliseconds_sum{source_chain="optimism"}[24h])) / sum(rate(hyperlane_message_latency_milliseconds_count{source_chain="optimism"}[24h])) + sample_size: sum(increase(hyperlane_message_seen_total{source_chain="optimism"}[24h])) + series: histogram_quantile(0.50, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="optimism"}[1h]))) + + - slug: avalanche + name: Avalanche + tag: Avalanche C-Chain → any chain, Snowman++ finality + Mailbox delivery + formula: "50th percentile over 24h of Hyperlane end-to-end delivery latency for messages sourced from Avalanche." + queries: + p50: histogram_quantile(0.50, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="avalanche"}[24h]))) + p90: histogram_quantile(0.90, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="avalanche"}[24h]))) + p99: histogram_quantile(0.99, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="avalanche"}[24h]))) + mean: sum(rate(hyperlane_message_latency_milliseconds_sum{source_chain="avalanche"}[24h])) / sum(rate(hyperlane_message_latency_milliseconds_count{source_chain="avalanche"}[24h])) + sample_size: sum(increase(hyperlane_message_seen_total{source_chain="avalanche"}[24h])) + series: histogram_quantile(0.50, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="avalanche"}[1h]))) + + - slug: solana + name: Solana + tag: Solana → any chain, sub-second slot + Mailbox delivery + formula: "50th percentile over 24h of Hyperlane end-to-end delivery latency for messages sourced from Solana." + queries: + p50: histogram_quantile(0.50, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="solana"}[24h]))) + p90: histogram_quantile(0.90, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="solana"}[24h]))) + p99: histogram_quantile(0.99, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="solana"}[24h]))) + mean: sum(rate(hyperlane_message_latency_milliseconds_sum{source_chain="solana"}[24h])) / sum(rate(hyperlane_message_latency_milliseconds_count{source_chain="solana"}[24h])) + sample_size: sum(increase(hyperlane_message_seen_total{source_chain="solana"}[24h])) + series: histogram_quantile(0.50, sum by(le) (rate(hyperlane_message_latency_milliseconds_bucket{source_chain="solana"}[1h]))) diff --git a/harnesses/hyperlane-message-latency/Dockerfile b/harnesses/hyperlane-message-latency/Dockerfile new file mode 100644 index 00000000..270b0b69 --- /dev/null +++ b/harnesses/hyperlane-message-latency/Dockerfile @@ -0,0 +1,22 @@ +FROM golang:1.24-alpine AS builder + +WORKDIR /app +RUN apk add --no-cache git + +COPY go.mod ./ +RUN go mod download || true + +COPY . . + +RUN CGO_ENABLED=0 GOOS=linux go build -o /app/monitor ./cmd/script + +FROM debian:bookworm-slim + +WORKDIR /app +RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/* + +COPY --from=builder /app/monitor /app/monitor + +EXPOSE 2112 + +CMD ["/app/monitor"] diff --git a/harnesses/hyperlane-message-latency/cmd/script/chains.go b/harnesses/hyperlane-message-latency/cmd/script/chains.go new file mode 100644 index 00000000..c38b4a79 --- /dev/null +++ b/harnesses/hyperlane-message-latency/cmd/script/chains.go @@ -0,0 +1,62 @@ +package main + +// hyperlaneChainSlug maps Hyperlane's `origin_chain_id` / `destination_chain_id` +// (numeric) to the OCB canonical chain slug. +// +// For EVM chains, the id equals the standard EVM chainId; for non-EVM +// chains, Hyperlane assigns arbitrary domain IDs (Solana = +// 1264145989, Sui = 101, etc.). We enumerate all mainnet Hyperlane +// deployments here. +// +// Unknown ids fall through to a synthetic `chain-` slug so we +// never drop data — new deployments show up as their own row until +// we add the mapping. +var hyperlaneChainSlug = map[int]string{ + // EVM L1s (standard chainIds) + 1: "ethereum", + 56: "bnb", + 137: "polygon", + 43114: "avalanche", + 100: "gnosis", + 42161: "arbitrum", + 10: "optimism", + 8453: "base", + 59144: "linea", + 324: "zksync", + 534352: "scroll", + 81457: "blast", + 5000: "mantle", + 42220: "celo", + 1101: "polygon-zkevm", + 7777777: "zora", + 480: "world-chain", + 130: "unichain", + 60808: "bob", + 57073: "ink", + 1868: "soneium", + 146: "sonic", + 80094: "berachain", + 33139: "apechain", + 21000000: "corn", + // Ronin + 2020: "ronin", + // Fraxtal + 252: "fraxtal", + // Moonbeam + 1284: "moonbeam", + // Astar EVM + 592: "astar", + // Metis + 1088: "metis", + // Manta Pacific + 169: "manta", + // Kaia (ex-Klaytn) + 8217: "kaia", + + // Non-EVM (Hyperlane domain IDs, verified live via API sample) + 1264145989: "solana", + 1399811149: "solana", // Alt Solana domain id (Wormhole convention) seen in some rows + // Rome / Roman-Storage chain + 173: "chain-rome", + // Sui, Aptos, etc. Add as they gain traction. +} diff --git a/harnesses/hyperlane-message-latency/cmd/script/main.go b/harnesses/hyperlane-message-latency/cmd/script/main.go new file mode 100644 index 00000000..83f8eda6 --- /dev/null +++ b/harnesses/hyperlane-message-latency/cmd/script/main.go @@ -0,0 +1,227 @@ +// hyperlane-message-latency: passive Hyperlane delivery-latency bench. +// +// Every 60s, POSTs a GraphQL query to https://api.hyperlane.xyz/v1/graphql +// (Hasura) for the most recent 100 messages ordered by id DESC. For +// each message the monitor has not seen before AND that has +// is_delivered=true, computes `delivery_occurred_at - send_occurred_at` +// and records it in a Prometheus histogram bucketed by source + +// destination chain. +// +// Undelivered messages are counted separately (permissionless relayers +// mean some routes go under-served; a spike surfaces stuck relayer +// coverage rather than protocol latency). +// +// Design notes: +// - MUST use `order_by: {id: desc}`. Ordering by `send_occurred_at` +// hits Hasura's 5s query timeout (agent-verified). +// - `delivery_latency` is a Postgres interval string; we ignore it and +// compute the delta from the two timestamps directly (equivalent, +// avoids interval parsing). +// - Timestamps are ISO 8601 without timezone (`2026-07-25T23:36:42`). +// Parse as UTC. +// - Sustained volume ~150/hr → poll 100 rows/60s always covers the +// interval with margin; no pagination required. +package main + +import ( + "bytes" + "container/list" + "context" + "encoding/json" + "fmt" + "io" + "log" + "net/http" + "os" + "os/signal" + "strconv" + "sync" + "syscall" + "time" + + "github.com/prometheus/client_golang/prometheus/promhttp" +) + +const ( + hyperlaneGraphQLURL = "https://api.hyperlane.xyz/v1/graphql" + hyperlaneQuery = `{"query":"{ message_view(limit: 100, order_by: {id: desc}) { msg_id origin_chain_id destination_chain_id is_delivered send_occurred_at delivery_occurred_at } }"}` + pollInterval = 60 * time.Second + requestTimeout = 15 * time.Second + dedupeCacheMaxLen = 10_000 + metricsListenAddr = ":2112" + // 30 min hard ceiling — Hyperlane relayers are typically < 2 min + // end-to-end; > 30 min is either a stuck message or clock skew. + maxLatencyMs = 1_800_000 +) + +type hyperlaneMessage struct { + MsgID string `json:"msg_id"` + OriginChainID int `json:"origin_chain_id"` + DestinationChainID int `json:"destination_chain_id"` + IsDelivered bool `json:"is_delivered"` + SendOccurredAt string `json:"send_occurred_at"` + DeliveryOccurredAt *string `json:"delivery_occurred_at"` +} + +type hyperlaneResponse struct { + Data struct { + MessageView []hyperlaneMessage `json:"message_view"` + } `json:"data"` + Errors []map[string]interface{} `json:"errors,omitempty"` +} + +// Timestamps come back without timezone info; force UTC on parse. +var hyperlaneTimeLayout = "2006-01-02T15:04:05" + +type lruSet struct { + max int + order *list.List + set map[string]*list.Element +} + +func newLRUSet(max int) *lruSet { + return &lruSet{max: max, order: list.New(), set: make(map[string]*list.Element, max)} +} + +func (l *lruSet) contains(k string) bool { _, ok := l.set[k]; return ok } + +func (l *lruSet) add(k string) { + if _, ok := l.set[k]; ok { + return + } + el := l.order.PushBack(k) + l.set[k] = el + for l.order.Len() > l.max { + front := l.order.Front() + if front == nil { + return + } + l.order.Remove(front) + delete(l.set, front.Value.(string)) + } +} + +func (l *lruSet) size() int { return l.order.Len() } + +func chainSlug(id int) string { + if s, ok := hyperlaneChainSlug[id]; ok { + return s + } + return "chain-" + strconv.Itoa(id) +} + +func poll(ctx context.Context, client *http.Client, seen *lruSet) error { + req, err := http.NewRequestWithContext(ctx, "POST", hyperlaneGraphQLURL, bytes.NewReader([]byte(hyperlaneQuery))) + if err != nil { + return fmt.Errorf("request: %w", err) + } + req.Header.Set("Content-Type", "application/json") + req.Header.Set("Accept", "application/json") + req.Header.Set("User-Agent", "OpenChainBench/hyperlane-message-latency") + resp, err := client.Do(req) + if err != nil { + return fmt.Errorf("http: %w", err) + } + defer resp.Body.Close() + if resp.StatusCode != 200 { + body, _ := io.ReadAll(io.LimitReader(resp.Body, 200)) + return fmt.Errorf("status %d: %s", resp.StatusCode, string(body)) + } + var parsed hyperlaneResponse + if err := json.NewDecoder(resp.Body).Decode(&parsed); err != nil { + return fmt.Errorf("decode: %w", err) + } + if len(parsed.Errors) > 0 { + return fmt.Errorf("graphql errors: %v", parsed.Errors) + } + fresh := 0 + for _, m := range parsed.Data.MessageView { + if m.MsgID == "" { + continue + } + srcSlug := chainSlug(m.OriginChainID) + + if !m.IsDelivered || m.DeliveryOccurredAt == nil { + // Count undelivered once and keep out of dedupe so we + // observe the eventual delivered transition on later poll. + if !seen.contains(m.MsgID) { + hyperlaneUndeliveredTotal.WithLabelValues(srcSlug).Inc() + } + continue + } + if seen.contains(m.MsgID) { + continue + } + sendTs, err1 := time.Parse(hyperlaneTimeLayout, m.SendOccurredAt) + recvTs, err2 := time.Parse(hyperlaneTimeLayout, *m.DeliveryOccurredAt) + if err1 != nil || err2 != nil { + seen.add(m.MsgID) + continue + } + deltaMs := float64(recvTs.Sub(sendTs).Milliseconds()) + if deltaMs < 0 || deltaMs > maxLatencyMs { + seen.add(m.MsgID) + continue + } + dstSlug := chainSlug(m.DestinationChainID) + hyperlaneLatencyMs.WithLabelValues(srcSlug, dstSlug).Observe(deltaMs) + hyperlaneSeenTotal.WithLabelValues(srcSlug, dstSlug).Inc() + seen.add(m.MsgID) + fresh++ + } + hyperlaneDedupeCacheSize.Set(float64(seen.size())) + log.Printf("poll: %d rows, %d fresh, dedupe=%d", len(parsed.Data.MessageView), fresh, seen.size()) + return nil +} + +func main() { + log.SetFlags(log.LstdFlags | log.Lmicroseconds) + log.Println("hyperlane-message-latency: starting") + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + client := &http.Client{Timeout: requestTimeout} + seen := newLRUSet(dedupeCacheMaxLen) + var mu sync.Mutex + + go func() { + http.Handle("/metrics", promhttp.Handler()) + http.HandleFunc("/health", func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(200) + _, _ = w.Write([]byte("ok")) + }) + log.Printf("metrics: listening on %s", metricsListenAddr) + if err := http.ListenAndServe(metricsListenAddr, nil); err != nil { + log.Fatalf("metrics server: %v", err) + } + }() + + mu.Lock() + if err := poll(ctx, client, seen); err != nil { + log.Printf("poll error: %v", err) + hyperlanePollErrors.Inc() + } + mu.Unlock() + + ticker := time.NewTicker(pollInterval) + defer ticker.Stop() + + sig := make(chan os.Signal, 1) + signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM) + + for { + select { + case <-ticker.C: + mu.Lock() + if err := poll(ctx, client, seen); err != nil { + log.Printf("poll error: %v", err) + hyperlanePollErrors.Inc() + } + mu.Unlock() + case <-sig: + log.Println("shutdown") + return + } + } +} diff --git a/harnesses/hyperlane-message-latency/cmd/script/metrics.go b/harnesses/hyperlane-message-latency/cmd/script/metrics.go new file mode 100644 index 00000000..578fe5c8 --- /dev/null +++ b/harnesses/hyperlane-message-latency/cmd/script/metrics.go @@ -0,0 +1,68 @@ +package main + +import ( + "github.com/prometheus/client_golang/prometheus" +) + +// hyperlane_message_latency_milliseconds is a histogram of Hyperlane +// end-to-end message delivery latency, bucketed by source + destination +// chain. +// Latency = `delivery_occurred_at - send_occurred_at` from the +// Hyperlane Explorer GraphQL feed. This is protocol-observed +// delivery latency (whichever relayer actually delivered the +// message), so it reflects the practical UX rather than a +// theoretical protocol floor. +// +// Emitted in MILLISECONDS to match the site's `unit: ms` display +// convention. Buckets 2s-30min. +var ( + hyperlaneLatencyMs = prometheus.NewHistogramVec( + prometheus.HistogramOpts{ + Name: "hyperlane_message_latency_milliseconds", + Help: "End-to-end delivery latency (ms) for a Hyperlane message, from source send to destination delivery, labeled by source + destination chain.", + Buckets: []float64{2_000, 5_000, 10_000, 15_000, 20_000, 30_000, 45_000, 60_000, 90_000, 120_000, 180_000, 300_000, 600_000, 900_000, 1_200_000, 1_800_000}, + }, + []string{"source_chain", "dest_chain"}, + ) + + hyperlaneSeenTotal = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "hyperlane_message_seen_total", + Help: "Total unique DELIVERED Hyperlane messages observed, by source + destination chain.", + }, + []string{"source_chain", "dest_chain"}, + ) + + // Undelivered messages counted separately (permissionless relayers + // mean some routes go under-served; a spike here surfaces stuck + // relayer coverage rather than protocol latency). + hyperlaneUndeliveredTotal = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "hyperlane_message_undelivered_total", + Help: "Hyperlane messages observed with is_delivered=false, by source chain.", + }, + []string{"source_chain"}, + ) + + hyperlanePollErrors = prometheus.NewCounter( + prometheus.CounterOpts{ + Name: "hyperlane_message_poll_errors_total", + Help: "Total failed pollings of the Hyperlane Explorer GraphQL feed.", + }, + ) + + hyperlaneDedupeCacheSize = prometheus.NewGauge( + prometheus.GaugeOpts{ + Name: "hyperlane_message_dedupe_cache_size", + Help: "Number of recent Hyperlane message ids the monitor keeps for deduplication.", + }, + ) +) + +func init() { + prometheus.MustRegister(hyperlaneLatencyMs) + prometheus.MustRegister(hyperlaneSeenTotal) + prometheus.MustRegister(hyperlaneUndeliveredTotal) + prometheus.MustRegister(hyperlanePollErrors) + prometheus.MustRegister(hyperlaneDedupeCacheSize) +} diff --git a/harnesses/hyperlane-message-latency/go.mod b/harnesses/hyperlane-message-latency/go.mod new file mode 100644 index 00000000..82332bb0 --- /dev/null +++ b/harnesses/hyperlane-message-latency/go.mod @@ -0,0 +1,18 @@ +module hyperlane-message-latency + +go 1.24.0 + +require github.com/prometheus/client_golang v1.23.2 + +require ( + github.com/beorn7/perks v1.0.1 // indirect + github.com/cespare/xxhash/v2 v2.3.0 // indirect + github.com/kr/text v0.2.0 // indirect + github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect + github.com/prometheus/client_model v0.6.2 // indirect + github.com/prometheus/common v0.66.1 // indirect + github.com/prometheus/procfs v0.17.0 // indirect + go.yaml.in/yaml/v2 v2.4.2 // indirect + golang.org/x/sys v0.36.0 // indirect + google.golang.org/protobuf v1.36.10 // indirect +) diff --git a/harnesses/hyperlane-message-latency/go.sum b/harnesses/hyperlane-message-latency/go.sum new file mode 100644 index 00000000..f415d46b --- /dev/null +++ b/harnesses/hyperlane-message-latency/go.sum @@ -0,0 +1,46 @@ +github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= +github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo= +github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= +github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o= +github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg= +github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk= +github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE= +github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9ZoGs= +github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA= +github.com/prometheus/procfs v0.17.0 h1:FuLQ+05u4ZI+SS/w9+BWEM2TXiHKsUQ9TADiRH7DuK0= +github.com/prometheus/procfs v0.17.0/go.mod h1:oPQLaDAMRbA+u8H5Pbfq+dl3VDAvHxMUOVhe0wYB2zw= +github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= +github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog= +github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= +github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= +go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= +go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= +go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI= +go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU= +golang.org/x/sys v0.36.0 h1:KVRy2GtZBrk1cBYA7MKu5bEZFxQk4NIDV6RLVcC8o0k= +golang.org/x/sys v0.36.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks= +google.golang.org/protobuf v1.36.10 h1:AYd7cD/uASjIL6Q9LiTjz8JLcrh/88q5UObnmY3aOOE= +google.golang.org/protobuf v1.36.10/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=