ChainBench · Flotapponnier · Jul 9, 2026 · Jul 9, 2026 · Jul 9, 2026 · Jul 9, 2026
@@ -5,7 +5,7 @@ number: "001"
 title: Fastest crypto price API, live head lag across Mobula, Codex, GeckoTerminal
 seo_title: "Fastest crypto price API 2026"
 seo_description: "{{best_name}} leads fastest crypto price API at {{best_p50}} (p50, 24h). Mobula WebSocket, Codex GraphQL, GeckoTerminal REST live across Base, BNB, Solana."
-subtitle: Wall-clock head lag in seconds from on-chain swap event to API emission, measured live for Mobula, Codex and GeckoTerminal on Base, BNB Chain and Solana.
+subtitle: Wall-clock head lag in seconds from on-chain swap event to API emission, measured live for Mobula, Codex and GeckoTerminal on Base, BNB Chain, Solana and Robinhood Chain.
 
 per_chain_explainer:
   - slug: base
@@ -20,6 +20,10 @@ per_chain_explainer:
     h2: "Lowest Solana price feed head lag"
     body: |
       {{best_name:chain:solana}} currently leads Solana head lag at {{best_p50:chain:solana}} (p50, 24h). Solana runs a 400 ms slot under Proof of History with confirmed status at typically 1-2 slots, so any second-scale provider latency lives entirely in the indexer pipeline rather than in the chain. Providers that subscribe to Yellowstone gRPC streams of confirmed blocks beat REST-pollers by definition. Measured against a canonical archive node from us-east, eu-west and Singapore.
+  - slug: robinhood
+    h2: "Lowest Robinhood Chain price feed head lag"
+    body: |
+      {{best_name:chain:robinhood}} currently leads Robinhood Chain head lag at {{best_p50:chain:robinhood}} (p50, 24h). Robinhood Chain is an Arbitrum Orbit L2 with sub-second block cadence, so nearly all of the latency you see here is provider pipeline (indexer, fan-out, edge) rather than chain time. It is also the youngest chain on this bench: indexer coverage is uneven across providers, which makes the leader-to-trailer spread wider than on mature chains. Measured on the highest-activity pool (USDG/WETH, about 40k swaps per day) from us-east, eu-west and Singapore.
 
 category: Aggregators
 status: live
@@ -35,7 +39,7 @@ seo_intro: |
   number. Mobula's WebSocket, Codex's GraphQL feed and GeckoTerminal's
   REST endpoint are watched from three regions (us-east, eu-west, sgp)
   against canonical-tip archive
-  nodes on Base, BNB Chain and Solana. Because each chain has its own
+  nodes on Base, BNB Chain, Solana and Robinhood Chain. Because each chain has its own
   base block cadence (Solana's ~400 ms slots vs Base/BNB's 2-3 s
   blocks), the headline winner shifts by chain. On Solana the leader
   is {{best_name:chain:solana}} at {{best_p50:chain:solana}}; on Base
@@ -51,19 +55,19 @@ seo_intro: |
 abstract: |
   We measure the gap between a transaction settling on chain and the same
   transaction appearing on each aggregator's data feed. The harness watches
-  reference pools across Base, BNB Chain and Solana from three regions
+  reference pools across Base, BNB Chain, Solana and Robinhood Chain from three regions
   (us-east, eu-west, sgp), records every new on-chain event with millisecond
   timestamps, and observes when each aggregator's WebSocket reflects the
   same event.
 
 methodology:
   - "Aggregators measured: Mobula, Codex, GeckoTerminal."
-  - "Chains: Base, BNB Chain, Solana."
+  - "Chains: Base, BNB Chain, Solana, Robinhood Chain."
   - "Regions: us-east, eu-west, sgp. Cross-region median reported in the headline."
   - "Reference: archive nodes per chain, validated against block hashes."
   - "Metric: gauge `head_lag_seconds`, sampled every 15 seconds. Aggregated over the 24-hour window using `quantile_over_time`."
   - "Success rate: presence ratio. Share of expected sampling slots where a value was actually emitted (5,760 expected per provider per day at the 15s cadence). 100% means the aggregator's feed was reachable for the full window."
-  - "Cardinality: 3 aggregators × 3 chains × 3 regions = 27 active series."
+  - "Cardinality: 3 aggregators × 4 chains × 3 regions = 36 active series."
 
 findings:
   - "Across the unfiltered cross-chain aggregate the leader is {{best_name}} at {{best_p50}} (p50, 24 h) over {{count}} measured providers, but that headline number mixes chains with very different baseline cadences (Solana 400 ms slots vs Base/BNB 2-3 s blocks) so the per-chain breakdown below is what to read for an integration decision."
@@ -90,7 +94,7 @@ faq:
   - q: "How fast does GeckoTerminal update prices?"
     a: "{{name:geckoterminal}} currently sits at {{p50:geckoterminal}} (p50, 24 h) head lag on this benchmark. REST endpoints update at the publisher's chosen cadence rather than the moment a swap lands, which is the architectural reason REST polling reads slower than WebSocket feeds for the same underlying chain data."
   - q: "How is head lag measured on OpenChainBench?"
-    a: "The harness holds a live WebSocket subscription to canonical-tip archive nodes on each chain (Base, BNB Chain, Solana). Every new on-chain swap is timestamped on receipt with millisecond precision. The harness then watches each provider's feed for the same event and computes the delta. p50, p90 and p99 are aggregated over 24 hours via Prometheus `quantile_over_time`. Errors and dropouts are tracked separately as a presence ratio so a fast feed that disappears does not silently rank above a slower feed that stays up."
+    a: "The harness holds a live WebSocket subscription to canonical-tip archive nodes on each chain (Base, BNB Chain, Solana, Robinhood Chain). Every new on-chain swap is timestamped on receipt with millisecond precision. The harness then watches each provider's feed for the same event and computes the delta. p50, p90 and p99 are aggregated over 24 hours via Prometheus `quantile_over_time`. Errors and dropouts are tracked separately as a presence ratio so a fast feed that disappears does not silently rank above a slower feed that stays up."
   - q: "Which data provider should I pick for a real-time trading interface?"
     a: "Read both the p50 and the p99 columns. p50 tells the typical experience, p99 tells the worst minute a user will see when a chain spikes or a region's path degrades. A 500 ms p50 with a 30 s p99 means most of the time the UI feels snappy and once per minute it stutters. The leaderboard splits both, plus the success rate column flags providers whose feed dropped during the measurement window."
 
@@ -99,10 +103,11 @@ faq:
 # every PromQL selector via lib/spec.ts:injectChainFilter.
 dimensions:
   chain:
-    - { value: all,    label: All chains }
-    - { value: base,   label: Base }
-    - { value: bnb,    label: BNB Chain }
-    - { value: solana, label: Solana }
+    - { value: all,       label: All chains }
+    - { value: base,      label: Base }
+    - { value: bnb,       label: BNB Chain }
+    - { value: solana,    label: Solana }
+    - { value: robinhood, label: Robinhood }
   # Region values match the raw harness labels (REGION env on each
   # Railway probe: us-east / eu-west / sgp). Declared so the
   # /compare/[slug] page can render a per chain x per region matrix

@@ -52,6 +52,15 @@ var geckoTerminalPools = []struct {
 		PoolID:  "160787671",
 		Chain:   "bnb",
 	},
+	{
+		// Highest txn-frequency pool on Robinhood Chain (~40k swaps/24h).
+		// Internal GT pool_id source: app.geckoterminal.com/api/p1/robinhood/pools/<address>
+		// Pool address: 0x69bfaf19c9f377bb306a89aed9f6b07e2c1a8d9a
+		Name:    "USDG/WETH Robinhood",
+		Network: "robinhood",
+		PoolID:  "328167634",
+		Chain:   "robinhood",
+	},
 }
 
 // ActionCable message structures

@@ -16,6 +16,14 @@ import (
 // Measures indexation latency: time between on-chain event and WebSocket receipt
 // ============================================================================
 
+// Post-reconnect grace period: Mobula WS replays buffered/old trades right
+// after a (re)connect, which show up as multi-second "lag" that is NOT real
+// indexation latency. During this window we do NOT feed the alerting gauges
+// (mobula_head_lag_detailed_seconds & co) so #alerting-aggregator-latency
+// stays quiet on reconnect bursts. The head_lag_seconds bench gauge keeps
+// being fed (no spike alert on it; keeps AggregatorHeadLagStale happy).
+const reconnectGracePeriod = 10 * time.Second
+
 // Pool configurations for head lag monitoring
 type HeadLagPool struct {
 	Name       string // Human readable name
@@ -28,16 +36,10 @@ type HeadLagPool struct {
 // Pools to monitor - high activity pools for accurate lag measurement
 var headLagPools = []HeadLagPool{
 	{
-		// Switched 2026-05-29 to the Raydium SOL/USDC ($8.7M liq, $2.6M
-		// vol24h vs the previous Orca pool 7qbRF6Y… at $206K / $46K).
-		// This is the only Solana pool GMGN pushes per-pair events for,
-		// so all 4 providers measure the same reference pool → true
-		// apples-to-apples instead of GMGN-on-slot-heartbeat vs others-
-		// on-per-swap. Bonus: much higher event rate tightens stats.
 		Name:       "SOL/USDC Raydium",
 		Blockchain: "solana",
 		NetworkID:  1399811149,
-		Address:    "58oQChx4yWmvKdwLLZzBi4ChoCc2fqCUWBkwMihLYQo2",
+		Address:    "7qbRF6YsyGuLUVs6Y1q64bdVrfe4ZcUUz1JRdoVNUJnm",
 		ChainName:  "solana",
 	},
 	{
@@ -48,19 +50,20 @@ var headLagPools = []HeadLagPool{
 		ChainName:  "base",
 	},
 	{
-		// Switched 2026-05-29 off the WBNB/BUSD pool ($104K vol24h —
-		// BUSD was deprecated by Binance in 2023 so swap events on it
-		// are too sparse for percentile stats: the bench was reporting
-		// zero BNB samples across all 3 regions). New pool is
-		// PancakeSwap V3 WBNB/USDT 0.01% with $17M liq + $100M vol24h —
-		// ~1000x event rate, the canonical BNB pair every aggregator
-		// indexes.
-		Name:       "WBNB/USDT PancakeSwap V3",
+		Name:       "WBNB/BUSD PancakeSwap",
 		Blockchain: "evm:56",
 		NetworkID:  56,
-		Address:    "0x172fcd41e0913e95784454622d1c3724f546f849",
+		Address:    "0x58f876857a02d6762e0101bb5c46a8c1ed44dc16",
 		ChainName:  "bnb",
 	},
+	{
+		// Highest txn-frequency pool on Robinhood Chain (~40k swaps/24h).
+		Name:       "USDG/WETH Robinhood",
+		Blockchain: "evm:4663",
+		NetworkID:  4663,
+		Address:    "0x69bfaf19c9f377bb306a89aed9f6b07e2c1a8d9a",
+		ChainName:  "robinhood",
+	},
 }
 
 // ============================================================================
@@ -97,9 +100,11 @@ func runMobulaHeadLagMonitor(config *Config, stopChan <-chan struct{}, wg *sync.
 			return
 		default:
 			err := connectAndMonitorMobula(config, stopChan)
+			RecordWSConnected("mobula", config.MonitorRegion, false)
 			if err != nil {
-				log.Printf("[HEAD-LAG][MOBULA] Connection error: %v. Reconnecting in %v...", err, reconnectDelay)
-
+				RecordWSReconnect("mobula", config.MonitorRegion)
+				log.Printf("[HEAD-LAG][MOBULA] 🔌 DISCONNECTED: %v. Reconnecting in %v...", err, reconnectDelay)
+
 				select {
 				case <-stopChan:
 					return
@@ -149,7 +154,9 @@ func connectAndMonitorMobula(config *Config, stopChan <-chan struct{}) error {
 		return fmt.Errorf("subscribe failed: %w", err)
 	}
 
-	fmt.Printf("[HEAD-LAG][MOBULA] Subscribed to %d pools\n", len(items))
+	connectedAt := time.Now()
+	RecordWSConnected("mobula", config.MonitorRegion, true)
+	log.Printf("[HEAD-LAG][MOBULA] ✅ CONNECTED — subscribed to %d pools (alert-gauge grace period: %v)", len(items), reconnectGracePeriod)
 
 	// Start ping goroutine
 	pingDone := make(chan struct{})
@@ -223,19 +230,28 @@ func connectAndMonitorMobula(config *Config, stopChan <-chan struct{}) error {
 			// Track latest tx per pool so alert annotations can link to it
 			RecordMobulaLastTx(chainName, config.MonitorRegion, trade.Pair, trade.Hash)
 
-			// Record detailed metric with breakdown
-			RecordMobulaHeadLagDetailed(
-				chainName,
-				config.MonitorRegion,
-				trade.Pair,
-				trade.Hash,
-				totalLagMs,
-				mobulaLagMs,
-				networkLagMs,
-				onChainTime.Format("2006-01-02T15:04:05Z"),
-				mobulaProcessTime.Format("2006-01-02T15:04:05Z"),
-				receiveTime.Format("2006-01-02T15:04:05Z"),
-			)
+			// Post-reconnect grace: don't feed the alerting gauges with
+			// replayed/buffered trades from the fresh connection.
+			if time.Since(connectedAt) < reconnectGracePeriod {
+				if totalLagMs > 2000 {
+					log.Printf("[HEAD-LAG][MOBULA][%s] ⏭️  GRACE: late trade %.2fs within %v of reconnect — alert gauge NOT updated | Tx: %s",
+						chainName, lagSeconds, reconnectGracePeriod, trade.Hash)
+				}
+			} else {
+				// Record detailed metric with breakdown
+				RecordMobulaHeadLagDetailed(
+					chainName,
+					config.MonitorRegion,
+					trade.Pair,
+					trade.Hash,
+					totalLagMs,
+					mobulaLagMs,
+					networkLagMs,
+					onChainTime.Format("2006-01-02T15:04:05Z"),
+					mobulaProcessTime.Format("2006-01-02T15:04:05Z"),
+					receiveTime.Format("2006-01-02T15:04:05Z"),
+				)
+			}
 
 			// Enhanced logging for spikes
 			if totalLagMs > 3000 {
@@ -269,6 +285,8 @@ func getChainNameFromBlockchain(blockchain string) string {
 		return "base"
 	case "BNB Smart Chain (BEP20)", "BSC", "evm:56":
 		return "bnb"
+	case "Robinhood Chain", "evm:4663":
+		return "robinhood"
 	default:
 		return blockchain
 	}
@@ -338,7 +356,9 @@ func runCodexHeadLagMonitor(config *Config, stopChan <-chan struct{}, wg *sync.W
 			log.Printf("[HEAD-LAG][CODEX] 🔄 Connection attempt #%d (proxy rotation enabled)", attemptNum)
 
 			err := connectAndMonitorCodex(config, stopChan)
+			RecordWSConnected("codex", config.MonitorRegion, false)
 			if err != nil {
+				RecordWSReconnect("codex", config.MonitorRegion)
 				consecutiveFailures++
 				log.Printf("[HEAD-LAG][CODEX] ❌ Connection attempt #%d failed (%d consecutive): %v", attemptNum, consecutiveFailures, err)
 
@@ -446,15 +466,18 @@ func connectAndMonitorCodex(config *Config, stopChan <-chan struct{}) error {
 
 		var subMsg map[string]interface{}
 
-		// Solana: use onUnconfirmedEventsCreated (pre-finalized, lowest latency)
+		// Solana: commitmentLevel Processed (pre-finalized, lowest latency).
+		// This replaces onUnconfirmedEventsCreated, deprecated by Codex on
+		// 2026-04-20 and enforcement-killed for API keys ~2026-07-06 (hard
+		// WS close 1006). Verified delivering on the JWT path 2026-07-09.
 		if pool.ChainName == "solana" {
 			poolID := fmt.Sprintf("%s:%d", pool.Address, pool.NetworkID)
 			subMsg = map[string]interface{}{
 				"type": "subscribe",
 				"id":   subID,
 				"payload": map[string]interface{}{
-					"query": `subscription OnPoolEvents($id: String!) {
-						onUnconfirmedEventsCreated(id: $id, quoteToken: token0) {
+					"query": `subscription OnPoolEvents($id: String!, $cl: [EventCommitmentLevel!]) {
+						onEventsCreated(id: $id, commitmentLevel: $cl) {
 							address
 							networkId
 							events {
@@ -467,6 +490,7 @@ func connectAndMonitorCodex(config *Config, stopChan <-chan struct{}) error {
 					}`,
 					"variables": map[string]interface{}{
 						"id": poolID,
+						"cl": []string{"Processed"},
 					},
 				},
 			}
@@ -505,13 +529,57 @@ func connectAndMonitorCodex(config *Config, stopChan <-chan struct{}) error {
 
 	log.Printf("[HEAD-LAG][CODEX] ✅ Subscribed to %d pools", len(headLagPools))
 	log.Printf("[HEAD-LAG][CODEX] 🎉 Connection fully established! Waiting for events...")
+	RecordWSConnected("codex", config.MonitorRegion, true)
 
 	// graphql-transport-ws keepalive: send {"type":"ping"} every 20s.
 	// Without this, the rotating proxy (or Codex itself) silently kills idle
 	// TCP after ~3min, manifesting as "i/o timeout" or "1006 unexpected EOF".
 	// Writes to the websocket conn are serialized via writeMu since we now have
 	// two goroutines writing (this ping loop + the subscribe path on next reconnect).
 	var writeMu sync.Mutex
+
+	// Per-chain flow watchdog. A high-activity pool that goes silent for
+	// 10 minutes means the subscription is dead even if the connection
+	// looks healthy (pongs + other subs satisfy the read deadline). On
+	// trip: purge the frozen gauges for that chain so the bench page
+	// never shows a stale value, then force a reconnect by closing the
+	// connection (the read loop errors out and the outer loop redials).
+	var lastEventMu sync.Mutex
+	lastEventByChain := make(map[string]time.Time)
+	firstEventBySub := make(map[string]bool)
+	for _, pool := range headLagPools {
+		lastEventByChain[pool.ChainName] = time.Now()
+	}
+	const flowSilence = 10 * time.Minute
+	watchdogDone := make(chan struct{})
+	go func() {
+		t := time.NewTicker(time.Minute)
+		defer t.Stop()
+		for {
+			select {
+			case <-watchdogDone:
+				return
+			case <-t.C:
+				lastEventMu.Lock()
+				tripped := ""
+				for chain, last := range lastEventByChain {
+					if time.Since(last) > flowSilence {
+						tripped = chain
+						break
+					}
+				}
+				lastEventMu.Unlock()
+				if tripped != "" {
+					log.Printf("[HEAD-LAG][CODEX] 🪦 %s subscription silent for >%s — purging gauges and forcing reconnect", tripped, flowSilence)
+					DeleteHeadLagSeries("codex", tripped, config.MonitorRegion)
+					_ = conn.Close()
+					return
+				}
+			}
+		}
+	}()
+	defer close(watchdogDone)
+
 	pingDone := make(chan struct{})
 	go func() {
 		t := time.NewTicker(20 * time.Second)
@@ -564,11 +632,29 @@ func connectAndMonitorCodex(config *Config, stopChan <-chan struct{}) error {
 				continue
 			}
 
+			// graphql-transport-ws: the server terminates ONE subscription
+			// with {"type":"complete","id":...} or {"type":"error",...}.
+			// Swallowing these (the pre-2026-07-09 behaviour) leaves a dead
+			// subscription on a live connection: no read timeout (other
+			// subs keep the socket busy), frozen gauges, blind alerts.
+			// Codex killed the Solana sub exactly this way. Force a full
+			// reconnect + resubscribe instead.
+			if wsMsg.Type == "complete" || wsMsg.Type == "error" {
+				return fmt.Errorf("subscription %q terminated by server (type=%s)", wsMsg.ID, wsMsg.Type)
+			}
+
 			// Skip non-data messages
 			if wsMsg.Type != "next" || wsMsg.Payload == nil {
 				continue
 			}
 
+			// One-shot log per subscription id: tells us whether the legacy
+			// unconfirmed sub, the Processed sub, or both deliver on this tier.
+			if !firstEventBySub[wsMsg.ID] {
+				firstEventBySub[wsMsg.ID] = true
+				log.Printf("[HEAD-LAG][CODEX] 📬 First event on subscription %q", wsMsg.ID)
+			}
+
 			// Parse event data
 			payloadBytes, _ := json.Marshal(wsMsg.Payload)
 			var eventData CodexEventData
@@ -611,6 +697,10 @@ func connectAndMonitorCodex(config *Config, stopChan <-chan struct{}) error {
 				// Get chain name
 				chainName := getChainNameFromNetworkID(networkID)
 
+				lastEventMu.Lock()
+				lastEventByChain[chainName] = time.Now()
+				lastEventMu.Unlock()
+
 				// Record metrics with tx hash
 				RecordHeadLag("codex", chainName, lagMs, lagSeconds, config.MonitorRegion, event.TransactionHash)
 				RecordCodexBlockNumber(chainName, event.BlockNumber, config.MonitorRegion)
@@ -645,6 +735,8 @@ func getChainNameFromNetworkID(networkID int) string {
 		return "base"
 	case 56:
 		return "bnb"
+	case 4663:
+		return "robinhood"
 	default:
 		return fmt.Sprintf("network_%d", networkID)
 	}