DATA-RECOVERY-001: Add feed recovery, stale handling, fallback, and provider health

[jamalfrnk]

Problem

Once DATA-HL-001 lands a real Hyperliquid feed, it needs production-grade reliability: reconnect/backoff, staleness detection, fallback behavior, and an exposed health surface -- none of which exist for the current CoinGecko polling loop either (it has basic staleness via STALE_AFTER_MS but no reconnect/backoff/fallback concept since it's simple HTTP polling).

Repository evidence

server/src/market-data/ingestion.ts has a staleness threshold but no WS reconnect logic (there's nothing to reconnect -- it's poll-based today). This issue's scope only becomes concrete once DATA-HL-001's WS client exists.

Risk

P1 -- without this, a Hyperliquid WS disconnect silently stalls prices with no visible degraded state, which directly enables one of the mission's explicit hard requirements: never let stale data drive a new paper fill under a normal-looking success state.

Scope

Exponential reconnect backoff with jitter and a max cap; heartbeat/liveness + silent-stall detection; duplicate and out-of-order event handling; missed-candle recovery via REST backfill; resubscription after reconnect; resource cleanup; update coalescing/backpressure; last-known-good state with timestamp; stale-data thresholds; primary-failure, fallback-failure, and total-failure states; reconnection-storm protection. Expose: last successful provider message time, last successful REST snapshot, consecutive failures, reconnect attempts, current provider, current mode (live/degraded/fallback/unavailable), data age by symbol.

Non-goals

Not implementing a second live provider beyond CoinGecko-as-fallback-display (already exists conceptually); not building alerting/paging (that's operational, out of scope here).

Dependencies

DATA-HL-001 (needs the real WS client to attach recovery logic to).

Proposed implementation

A MarketHealthService tracking per-symbol freshness and connection state, wired into the existing getIngestionHealth()//api/market-data/health endpoint (extended, not replaced) and into PaperPricingService's reference-price gating (no new paper order when reference price is untrustworthy).

Acceptance criteria

Simulated disconnect triggers backoff+reconnect+resubscribe; simulated stale feed blocks new paper-order execution with a clear rejection reason while preserving last-known-good display data; health endpoint reflects all listed states accurately.

Definition of done

Server tests simulate disconnect/reconnect, duplicate/out-of-order events, and stale-price order rejection; reviewer PASS.

Test plan

Unit tests against a mocked WS transport: reconnect backoff sequence, resubscription after reconnect, duplicate-candle dedup, out-of-order rejection, stale-price paper-order rejection.

Security review

Reconnection-storm protection prevents this from becoming a self-inflicted DoS vector against Hyperliquid's public endpoints.

Accessibility review

N/A (server-side; client-facing freshness indicators are CHART-001's concern).

Observability requirements

This issue is the observability work for market data -- see acceptance criteria.

Performance considerations

Coalesce update fan-out so one price tick doesn't cause N redundant broadcasts; document the coalescing interval chosen.

Rollback plan

Revert PR; falls back to DATA-HL-001's baseline behavior without recovery hardening.

Documentation requirements

Extend docs/architecture/market-data.md.

Completion evidence

(added on merge)

Reviewer verdict

(pending)

[jamalfrnk]

Completion evidence

PR: #59 (merged, 4088caf).

Independent review verdict: PASS WITH WARNINGS. Reviewer worked in a fresh isolated worktree and independently reproduced every safety-critical claim against real infrastructure, not just the diff:

• npm run build clean, npm test 191/191 passing (matches PR claim exactly).
• Live Hyperliquid WS connect confirmed via log line; GET /api/market-data/health mode: "live" confirmed genuinely computed (not hardcoded) by observing lastMessageAt advance between two calls 5s apart.
• Real WS client measured 1.13 updates/sec/symbol (BTC/ETH/SOL) over an 8s window -- matches the claimed ~1/sec.
• Risk gate independently exercised end-to-end against a real Postgres DB: new order with market.source='coingecko' -> REJECTED with a reason mentioning coingecko; same order with source restored to hyperliquid -> FILLED (control, no false positive). Resting limit order becoming marketable with source='coingecko' -> stays ACKNOWLEDGED (correctly blocked); same order with source='hyperliquid' -> FILLED (control). All test DB state was restored and verified afterward.
• Guest-session paper order regression check: real order still fills correctly against a live Hyperliquid-derived price.
• CI: 3/3 checks green. No CRLF/line-ending issues. No secrets in the diff.

Two Medium (non-blocking) findings, tracked as fast-follow work:

1. checkTrustworthySource's actual call sites in paperEngine.ts (processNewOrder, sweepLimitOrders) have no automated regression test -- only the standalone risk-check function and its dispatch from evaluateTrade are unit-tested. The reviewer verified the wiring is correct by hand this run, but a future refactor could silently break it undetected by CI.
2. The 1s marketUpdate broadcast loop in server.ts always labels source: 'hyperliquid', even though the paired change24h/volume fields actually come from the last REST ingestion cycle's getLastKnownMarketMeta, which could itself be CoinGecko-sourced in a narrow window (WS up, REST degraded). This doesn't affect risk-gating (which reads the DB markets.source column directly, independently verified correct) -- it's a UI/labeling-accuracy issue.

Both findings are being addressed in a small follow-up PR. Closing this issue now; the follow-up will reference it.