diff --git a/README.md b/README.md index 4cd7e810..8288e9d6 100644 --- a/README.md +++ b/README.md @@ -1,351 +1,148 @@ # OpenChainBench -> Open, reproducible benchmarks for crypto infrastructure. aggregators, bridges, RPCs, price feeds. Same metric, same conditions, every provider. Live at [openchainbench.com](https://openchainbench.com). +> Open, reproducible benchmarks for crypto infrastructure. Aggregators, bridges, RPCs, oracles, price feeds, prediction markets, perp DEXes, cross-chain messaging. Same metric, same conditions, every provider. Live at **[openchainbench.com](https://openchainbench.com)**. -OpenChainBench publishes one benchmark at a time, each one shipping with the script that produces its data. The goal is to make performance an observable property of crypto infra. measured in the open, by anyone who wants to add a provider or a metric. +[![License: MIT](https://img.shields.io/badge/code-MIT-blue.svg)](./LICENSE) [![Data: CC-BY-4.0](https://img.shields.io/badge/data-CC--BY--4.0-lightgrey.svg)](https://creativecommons.org/licenses/by/4.0/) [![Benchmarks](https://img.shields.io/badge/benchmarks-94_live-green.svg)](https://openchainbench.com/benchmarks) [![MCP](https://img.shields.io/badge/MCP-ready-purple.svg)](https://openchainbench.com/mcp) [![llms.txt](https://img.shields.io/badge/llms.txt-yes-orange.svg)](https://openchainbench.com/llms.txt) -The project is community-run, MIT-licensed, and accepts PRs from any party including the providers it benchmarks. +OpenChainBench publishes one benchmark at a time, each shipping with the harness that produces its data. The goal is to make performance an observable property of crypto infra — measured in the open, by anyone who wants to add a provider or a metric. -## What's inside +Community-run, MIT-licensed harnesses + CC-BY-4.0 data, PRs from any party including the providers we benchmark. -``` -benchmarks/ Spec files. one YAML per published benchmark (30 and counting) -├── aggregator-head-lag.yml №001 · onchain data provider latency -├── bridge-quote-latency.yml №002 · cross-chain bridge quote latency -├── hyperliquid-frontends.yml №030 · HL builder-code revenue (100+ frontends) -├── perp-funding.yml №036 · perp funding rates across venues -├── pm-rate-limits.yml №037 · prediction-market API rate limits -├── … (run `ls benchmarks/` for the full list) -└── README.md Spec format reference + submission guide - -harnesses/ The runners that produce the metrics (18 dirs) -├── aggregator-head-lag/ Go service: WebSocket monitor (:2112/metrics) -├── bridge-monitor/ Go service: 4-bridge quote loop + execution (:9090/metrics) -├── hyperliquid-frontends/ Go service: tails a local hl-node fill stream; -│ feeds benches №030/№036 + HIP-3 deployers, -│ builder registry in builders.json (100+ entries) -├── l1-finality/ Go service: per-chain finality latency probe -├── … (oracle-deviation, solana-tx-landing, validator-yield, …) -└── README.md Contract for new harnesses - -worker/ Materialization worker (Railway): sweeps every bench - against Prometheus every 60s, publishes snapshots - to the Redis store the site reads +## At a glance -alternatives/ YAML-driven /alternatives/ SEO landing pages -└── README.md Format for "Pump Portal alternatives", "Relay alternatives", … +- **94** live benchmark specs across 8 categories (RPCs, Trading, Aggregators, Bridges, Blockchains, RWA, Explorers, NFT APIs) +- **50** self-contained Go harnesses, each shipping a `/metrics` Prometheus endpoint +- **32** curated question/answer pages · **23** "alternatives to X" landing pages +- **26** machine-readable API routes (citation, badges, MCP, feeds, per-format quotes) +- One shared Prometheus, one materialization worker, CDN-fastpath snapshot store — cold reads in < 100 ms edge, no origin fan-out on hot paths -infrastructure/ Shared services every harness depends on -└── prometheus/ Single shared Prometheus that scrapes all harnesses +## Quickstart -src/ Next.js 16 site (App Router, ISR, Tailwind v4) -├── app/ Pages: -│ / — hero ("Highest accuracy at every price point") -│ with animated radar dial, live Network Ecosystem, -│ Latest deployed benchmarks table -│ /benchmarks — card grid (3-col) with category pill filter + search -│ /providers — "Products" list with brand logos, benchmarks / -│ top-1 counts (Top-1 green when > 0) -│ /contribute — 6-step tutorial + federation / timeline cards -│ /methodology — design principles + statistical conventions -│ /benchmarks/[slug] — bench detail with card-wrapped chart + ledger -│ /alternatives, /press, /mcp, /about -├── components/ -│ ├── site-header.tsx, site-footer.tsx, site-banner.tsx, site-logo.tsx -│ ├── theme-toggle.tsx Sun/moon toggle, html.dark class strategy, localStorage-persisted -│ ├── hero-radar.tsx Animated SVG dial on the home hero -│ ├── benchmark-card.tsx, benchmark-grid.tsx, home-bench-table.tsx -│ ├── time-series-chart, ledger-table, region-grid, chain-tabs, … -│ └── live/ Live page: dashboard, ticker, chart, compact-feed -├── data/ -│ ├── benchmarks.ts Spec loader (YAML + materialized snapshots → Benchmark[]) -│ └── provider-registry.ts Per-provider description, URL, Twitter handle -└── lib/ - ├── prometheus.ts Prometheus HTTP client + spec/formatting helpers - ├── providers.ts Aggregates each provider's benchmark appearances - ├── brand.ts Vivid brand-color table per chain / provider (legible both modes) - └── live/ Live page domain logic (types, config, chains, format, buckets) +```bash +pnpm install +pnpm dev # site at http://localhost:3000 +pnpm validate # bench spec Zod schema check (run before opening a PR) +pnpm typecheck # tsc --noEmit ``` -The live dashboard at the top of `/` is fed by a separate **stream relay** living in the -[`mobula-monorepo`](https://github.com/MobulaFi/mobula-monorepo) at -`miniapps/ocb-stream-relay/`. It is hosted by Mobula because it holds an -upstream API key; the browser talks to it directly over WebSocket. Vercel -only serves the static page shell. See [`docs/architecture.md`](./docs/architecture.md#the-live-page) -for the data flow. +Adding a benchmark, writing a harness, or fixing a number: see [CONTRIBUTING.md](./CONTRIBUTING.md) + the visual walkthrough at [`/contribute`](https://openchainbench.com/contribute). -## Cite OpenChainBench +## Machine-readable surfaces -Every benchmark is licensed CC-BY-4.0 and exposed through a small set of -machine-readable endpoints so journalists, devs, and AI agents can quote -us without screenshotting: +Every benchmark is licensed CC-BY-4.0 and exposed through endpoints designed for citation by journalists, devs, and AI agents: | Endpoint | Audience | What it returns | |---|---|---| -| [`/llms.txt`](https://openchainbench.com/llms.txt) | LLM crawlers (ChatGPT, Claude, Perplexity, Gemini) | Plain-text index of every benchmark + links to JSON. Follows the [llmstxt.org](https://llmstxt.org) convention. | +| [`/llms.txt`](https://openchainbench.com/llms.txt) · [`/llms-full.txt`](https://openchainbench.com/llms-full.txt) | LLM crawlers | Plain-text index (short) + full Markdown context (long) per the [llmstxt.org](https://llmstxt.org) convention. | | [`/api/citable`](https://openchainbench.com/api/citable) | Devs, agents | Flat JSON: every benchmark with current value, leader, headline sentence, citation URL, OG image URL. | | [`/api/stat/`](https://openchainbench.com/api/stat/aggregator-head-lag) | Devs, agents | Single benchmark: full rankings, sparkline (24h), methodology, paste-ready quote, attribution URL. | -| [`/api/freshness`](https://openchainbench.com/api/freshness) | Live UI, dashboards | Tiny `{slug → asOf ms}` map. Edge-cached 5 s, polled by the on-page "Live" indicator every 8 s for ~15-20 s p99 staleness. | -| [`/api/llm-context`](https://openchainbench.com/api/llm-context) | LLMs | Every benchmark with rankings + methodology in one Markdown blob. Paste into a system prompt for "here's everything you need to answer questions about crypto-infra performance today". | -| [`/benchmarks//opengraph-image`](https://openchainbench.com/benchmarks/aggregator-head-lag/opengraph-image) | Twitter/Slack/Discord/iMessage | 1200×630 PNG with the current value, leader, sparkline, watermark. Returned automatically as the OG image when someone shares a benchmark link. | -| [`/benchmarks//share-card?template=...&theme=...`](https://openchainbench.com/benchmarks/aggregator-head-lag/share-card) | Manual export | 5 share templates (ranking, snapshot, headline, compare, leaderboard) — supports `?theme=dark` so the export matches the user's site theme. | -| [`/api/openapi.json`](https://openchainbench.com/api/openapi.json) | LangChain, Custom GPTs, generic clients | OpenAPI 3.1 schema describing every endpoint. | -| [`/api/mcp/mcp`](https://openchainbench.com/api/mcp/mcp) | MCP-capable agents (Claude Desktop, Cursor) | MCP server exposing `list_benchmarks`, `get_benchmark`, `query_prom` tools. Streamable HTTP transport. | -| [`/api/badge//`](https://openchainbench.com/api/badge/aggregator-head-lag/mobula) | Provider sites, READMEs, blogs | Embeddable SVG badge with the provider's current rank and headline figure. 360×36, cache-aware. | - -Each bench detail page also has a **Copy API URL** strip under the title. one click and a journalist has the JSON endpoint for live numbers. - -### How journalists cite us - -```text +| `/api/cite//` | Zotero, Mendeley, Perplexity | Citation in `bibtex`, `apa`, `ris`, or `txt` with the correct MIME type. | +| [`/api/openapi.json`](https://openchainbench.com/api/openapi.json) | LangChain, custom GPTs, generic clients | OpenAPI 3.1 schema describing every endpoint. | +| [`/api/mcp/mcp`](https://openchainbench.com/api/mcp/mcp) | MCP clients (Claude Desktop, Cursor, ChatGPT tools) | MCP server: `list_benchmarks`, `get_benchmark`, `query_prom` tools + `openchainbench://benchmark/{slug}` resource. | +| [`/api/freshness`](https://openchainbench.com/api/freshness) | Live UI, dashboards | `{slug → asOf ms}` map. Edge-cached 5 s. | +| [`/api/llm-context`](https://openchainbench.com/api/llm-context) | LLM system prompts | Every benchmark with rankings + methodology in one Markdown blob. | +| [`/benchmarks//opengraph-image`](https://openchainbench.com/benchmarks/aggregator-head-lag/opengraph-image) | X, LinkedIn, Slack, iMessage | 1200×630 PNG with current value + leader + sparkline. Auto-served on link unfurl. | +| [`/benchmarks//share-card?template=...`](https://openchainbench.com/benchmarks/aggregator-head-lag/share-card) | Manual export | 5 templates (ranking / snapshot / headline / compare / leaderboard), `?theme=dark` supported. | +| [`/api/badge//`](https://openchainbench.com/api/badge/aggregator-head-lag/mobula) | Provider sites, READMEs | 360×36 embeddable SVG with the provider's rank + headline figure. | +| [`/rss.xml`](https://openchainbench.com/rss.xml) · [`/feed.json`](https://openchainbench.com/feed.json) | Feed readers, agent tooling | RSS 2.0 + JSON Feed 1.1 mirrors, one entry per live bench, updated on release. | +| [`/sitemap.xml`](https://openchainbench.com/sitemap.xml) · [`/robots.ts`](https://openchainbench.com/robots.txt) | Crawlers | Every AI crawler (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, …) explicitly allowed. | + +Structured data on every bench page: `Dataset` + `StatisticalReport` + `Observation` + `TechArticle` + `FAQPage` + `BreadcrumbList` JSON-LD, plus a schema.org Dataset microdata graph layered on the visible "At a glance" infobox. + +**Journalist citation format** (what `/api/stat/.headlineSentence` emits): +``` "Mobula leads head lag at 0.8s (p50, 24h) on Fastest onchain data provider. Source: OpenChainBench (https://openchainbench.com/benchmarks/aggregator-head-lag)" ``` -This is the `headlineSentence` exposed on `/api/stat/`. The page URL is stable, the OG preview unfurls automatically with the current value, and the data is CC-BY-4.0. - -### Product pages - -Every product (provider) that appears in at least one benchmark spec gets a page at `/providers/` (the URL stays `providers/` for SEO continuity; the UI labels everything as "Products"). The list at [`/providers`](https://openchainbench.com/providers) is now a flat avatar list with each row showing: - -- the product's brand logo + name + optional type pill (intent / protocol / aggregator / relay) -- the categories it appears in, colored by category (Aggregators / Bridges / Blockchains / Trading) -- two right-aligned counts: **Benchmarks** (total appearances) and **Top 1** (green when > 0, gray otherwise) - -A category pill filter and `⌘K` search box sit above the list. Drafts (products from harnesses temporarily unreachable) still surface in the list rather than disappearing — keeps the directory stable when a single harness is down. - -The bench detail page also keeps the per-product registry (description, URL, Twitter handle) from [`src/data/provider-registry.ts`](./src/data/provider-registry.ts) and renders an embeddable badge for every bench where the product is currently #1. - -### How agents query us - -**Via MCP (recommended)**: OpenChainBench ships an MCP server at `https://openchainbench.com/api/mcp/mcp` (Streamable HTTP transport, no auth). Any MCP-capable client (Claude Desktop, Cursor, ChatGPT custom tools, generic MCP clients) can connect and discover: - -- **3 tools**: `list_benchmarks`, `get_benchmark(slug, chain?, region?)`, `query_prom(query, windowSec?, steps?)` — `query_prom` is scoped to the published benchmark metric namespaces so the public endpoint can't be used to walk the underlying Prometheus catalog. -- **1 resource template**: `openchainbench://benchmark/{slug}` — every live benchmark is also exposed as an MCP resource (Markdown + JSON), so an agent can pin a bench into its context as a long-lived document instead of re-fetching every turn. - -#### Install in Claude Desktop - -Add to `~/Library/Application Support/Claude/claude_desktop_config.json` (or the equivalent on your OS): - -```json -{ - "mcpServers": { - "openchainbench": { - "url": "https://openchainbench.com/api/mcp/mcp" - } - } -} -``` - -Restart Claude Desktop. The tools appear under the 🔌 icon; ask *"what's the fastest crypto data aggregator on Base today?"* and watch it call `get_benchmark` live. - -#### Install in Cursor - -Settings → MCP → Add server → URL `https://openchainbench.com/api/mcp/mcp`. - -**Via REST**: hit `/api/citable` once to discover everything, then `/api/stat/` for specifics. Both are zero-auth, edge-cached for 60 s. For just the live freshness, `/api/freshness` is a few hundred bytes and refreshes every 5 s at the edge. - -## How a benchmark gets data. federation - -OpenChainBench is a federation of independently-hosted harnesses. Each spec declares the Prometheus its harness writes to. nothing forces every contributor onto the same instance. - -``` -[Mobula's infra] [Contributor B's infra] [Provider C's infra] - harness exposes harness exposes harness exposes - /metrics on /metrics on /metrics on - - │ │ │ - ▼ ▼ ▼ - Prometheus A Prometheus B Prometheus C - (operator's, (operator's, (operator's, - public read API) public read API) public read API) - │ │ │ - └──────────────────────────┼──────────────────────────────┘ - ▼ - openchainbench.com - (Next.js site on Vercel) - queries spec.prometheus.url - declared in each YAML -``` - -Each harness is run by whoever wrote it. Mobula for the existing aggregator and bridge benchmarks, independent contributors for any future ones, providers for self-benchmarks of their own services. They never share API keys with the project. They expose `/metrics` over HTTPS to **their own** Prometheus, and that Prom's read API is what the site queries. - -Prometheus is queried by a **materialization worker** (`worker/index.ts`, a Railway service colocated with the Prom), not by the site at request time. The worker sweeps every bench every 60 s via the standard HTTP API (`/api/v1/query`, `/api/v1/query_range`), applies per-provider carry-forward (a provider whose queries fail this sweep keeps its last good values, marked with a `staleSince` timestamp), and publishes complete, atomic snapshots to a Redis store. The site reads snapshots only (`READ_FROM_STORE=1`). Resilience layers, in order: - -1. **Carry-forward in the worker.** Transient per-provider query failures never produce holes in a leaderboard — last good values are served with honest staleness metadata instead. -2. **Snapshot freshness gate with live fallback.** If a snapshot is missing or older than the acceptance window (worker down), the site falls back to the original live-Prometheus loader for that render. -3. **Per-bench `unstable_cache` revalidate 60 s + ISR** on top of the store read, so a Redis blip on bench A does not poison bench B and pages stay static-fast. -4. **Vercel edge cache** (`s-maxage` + `stale-while-revalidate`) on every public route. A worker heartbeat key surfaces sweep staleness for ops. - -A spec's Prometheus URL goes through a two-layer SSRF guard: a schema-time rejection of non-HTTPS / RFC1918 / link-local / metadata IPs at PR-validate time (`pnpm validate`), and a runtime DNS resolution that refuses if the hostname currently lands on a private address. Plus `redirect: "manual"` on every Prom fetch so a 3xx into a private host gets blocked too. - -## The live dashboard on / - -A second data path feeds the live dashboard at the top of [openchainbench.com](https://openchainbench.com): a real-time stream of swap events from Mobula's fast-trade WebSocket. The data **does not flow through Prometheus**. it goes directly from the relay's WebSocket to the browser, with the Vercel site acting as a thin static shell. - -``` -[Mobula fast-trade WS] [Mobula REST /lighthouse,/all] - │ │ - ▼ ▼ - ocb-stream-relay (Railway) - │ - Multi-resolution chart store: - │ 10m window @ 5s buckets - │ 1h window @ 30s buckets - │ 24h window @ 10min buckets - │ - 24h vol counter - │ - lighthouse poller (5min) - │ - mcap poller (10min) - │ - snapshot sent on each client connect - ▼ wss://ocb-stream-relay-...up.railway.app/ws - browser (openchainbench.com) - Next.js Client Component, no Vercel relay -``` - -Single Railway box, in-memory fan-out. Cost stays flat regardless of viewer count. The dashboard collapses to a one-line ticker by default and expands to a cumulative-volume chart with hover crosshair, per-chain tooltip, and live trade feed when toggled. The chart's range picker (10 min / 1 hour / 24 h) reads from the relay's three ring buffers and the x-axis auto-fits to the data extent when a buffer is still filling. See [`docs/architecture.md`](./docs/architecture.md#the-live-page) for the full diagram. +## Site surfaces + +| Route | What lives there | +|---|---| +| [`/`](https://openchainbench.com/) | Hero + Latest benchmarks table + Live dashboard | +| [`/benchmarks`](https://openchainbench.com/benchmarks) · `/benchmarks/[slug]` | Catalog grid + per-bench leaderboard with chart, ledger, chain/region filters, share cards, FAQ | +| [`/benchmarks/[slug]/[chain]`](https://openchainbench.com/benchmarks/aggregator-head-lag/solana) | Chain-scoped bench view with per-chain explainer | +| [`/products/[slug]`](https://openchainbench.com/products/mobula) | Per-provider aggregated profile: every bench they appear in, top-1 counts, related products | +| [`/chains/[slug]`](https://openchainbench.com/chains/ethereum) | Per-chain hub: live native price, TVL history, every bench that touches this chain grouped by category | +| [`/compare/[a]-vs-[b]`](https://openchainbench.com/compare/codex-vs-mobula) | Head-to-head comparison across shared benches, canonical alphabetical order enforced at the edge | +| [`/alternatives/[slug]`](https://openchainbench.com/alternatives/alchemy) | "Alternatives to X" landing pages with live leaderboard from the referenced bench | +| [`/answers/[slug]`](https://openchainbench.com/answers/which-crypto-data-api-covers-the-most-blockchains) | Q&A pages backed by a live bench, formatted for AI answer engines | +| [`/hyperliquid`](https://openchainbench.com/hyperliquid) · `/hyperliquid/[slug]` | HyperLiquid frontends cohort + per-builder dashboard (revenue, volume, first-active date) | +| [`/perps`](https://openchainbench.com/perps) · [`/prediction-markets`](https://openchainbench.com/prediction-markets) · [`/rpc`](https://openchainbench.com/rpc) | Vertical hubs aggregating multiple benches per topic | +| [`/mcp`](https://openchainbench.com/mcp) | MCP server docs + install instructions | +| [`/methodology`](https://openchainbench.com/methodology) · [`/contribute`](https://openchainbench.com/contribute) · [`/team`](https://openchainbench.com/team) · [`/about`](https://openchainbench.com/about) · [`/press`](https://openchainbench.com/press) · [`/badges`](https://openchainbench.com/badges) | Static | ## Architecture -| Layer | Where it runs | Notes | -|---|---|---| -| Site (Next.js 16, ISR) | Vercel | Bench detail pages on `revalidate: 60`, static hubs. Reads materialized snapshots from the Redis store (`READ_FROM_STORE=1`) with a live-Prom fallback. Zero secrets beyond the store URL. | -| Materialization worker | Railway (`ocb-materialize-worker`) | Sweeps every bench against Prometheus every 60 s, applies per-provider carry-forward, publishes atomic snapshots to Redis. Its image clones OCB `dev` at build — rebuild it after spec provider-list changes. | -| Snapshot store | Railway Redis | Atomic per-(bench, variant) snapshot blobs + heartbeat key. | -| Prometheus | A small Railway service | Shared OpenChainBench time-series store. Open access (read-only public API). | -| Harnesses | Wherever the contributor wants to host them | Railway, Fly, Cloud Run, a VPS. each contributor owns their own runtime, secrets, and budget. | -| Live-stream relay | Mobula's Railway | Holds the Mobula API key, fans out Mobula fast-trade events to browsers. Not in this repo. | - -The split is intentional: Vercel for the globally-cached read path, Prometheus for the time-series store, and any compute platform for the long-running data producers. Nobody other than the harness operator needs the harness's secrets. - -A background **health-check cron** on Vercel (`/api/cron/health-check`, every 5 min) detects providers that have gone silent. It uses PromQL `present_over_time()` to test whether Prometheus has actually scraped a sample for a metric inside the freshness window, which works correctly for slow-cadence gauges where the value can be stable for minutes at a time. Sustained transitions (10 minute hysteresis) post a Slack message; the UI separately renders `availability: "unavailable"` for the affected provider rows. - -## Running the site locally - -```bash -pnpm install -pnpm dev # http://localhost:3000 ``` - -The site reads every `benchmarks/*.yml` at request time. Locally (no Redis store configured) it falls back to live Prometheus queries; specs whose data source the runtime can't reach render as drafts (no numbers, methodology only). - -```bash -pnpm validate # schema-lint every spec in benchmarks/ -pnpm spec:dry-run # query Prometheus and print numbers, no rendering -pnpm test # run unit tests (bun test src/) -pnpm check # validate + typecheck + lint + test (pre-PR gate) -pnpm build # production build +┌─────────────────────────┐ ┌───────────────────┐ ┌──────────────────────┐ +│ 50 Go harnesses on │ │ Shared Prometheus │ │ Materialization worker│ +│ ocb-par-main (Paris) │─── ▶│ (same VPS) │───▶│ (same VPS, docker) │ +│ each exposes /metrics │ │ federates a few │ │ sweeps every 60s │ +└─────────────────────────┘ │ Railway regional │ └──────┬───────────────┘ + │ harnesses │ │ + └───────────────────┘ ▼ + ┌─────────────────────────┐ + │ Redis + CDN blob store │ + │ (SRH at kv.opencha…) │ + └──────────┬──────────────┘ + │ + ┌─────────────────────────┐ + │ Vercel Fluid Compute: │ + │ pages read CDN blobs │ + │ first, SRH as fallback, │ + │ last-known-good on cold │ + └─────────────────────────┘ ``` -The live dashboard connects to the production relay by default. To point it at a local relay (see the relay README for instructions), set: +- **Harnesses + Prometheus + worker + Redis + SRH** run on one VPS (`ocb-par-main` in Paris) via docker compose. A handful of multi-region harnesses (rpc-capabilities-us/eu/sgp, evm-quote-latency, aggregator-head-lag regional) stay on Railway and federate into the VPS Prom. +- **Worker** publishes atomic snapshots every 60 s to Redis + CDN blobs at `kv.openchainbench.com/aggregate/{latest,benches/,variants//}.json`. Site reads CDN first (~20 ms edge), falls back to SRH, then to the last-known-good snapshot key on cold cache. +- **Site** is Next.js 16 App Router on Vercel Fluid Compute. Auto-deploy from `main` on every push (with sitemap smoke test + auto-rollback + bench-page warm-up); `dev` branch runs on staging preview. +- **Middleware** (`src/middleware.ts`) handles lowercase 308 normalization, `/compare/-vs-` → canonical alphabetical, 410 Gone for removed benches, 301 for renamed ones, and cache-key normalization on read-only public APIs. +- **Live stream relay** (WebSocket at `wss://stream.openchainbench.com/ws`) is a separate service run by Mobula because it holds upstream API keys; the browser talks to it directly, Vercel only serves the static shell. -```bash -NEXT_PUBLIC_RELAY_WS_URL=ws://localhost:2112/ws pnpm dev -``` +Full walkthrough with diagrams + a "clone-and-run-it-yourself" section is in [`docs/architecture.md`](./docs/architecture.md). -For production deployments, attaching a Vercel KV (or Upstash Redis via the Marketplace) is recommended but optional. Once connected, the env vars `KV_REST_API_URL` and `KV_REST_API_TOKEN` are injected automatically and the snapshot fallback activates. Without them the site behaves identically to a deployment without KV: a cold-start hit during a Prom blackout will fall through to a draft placeholder until the next successful scrape lands. +## Repo layout -## Running a harness locally - -Each harness is a standalone Go binary that exposes `/metrics` on a documented port (`:2112` for aggregator, `:9090` for bridge). They have no Prometheus / Grafana dependencies. that lives in [`infrastructure/`](./infrastructure/) and is shared. - -```bash -cd harnesses/aggregator-head-lag -cp .env.example .env # fill in API keys -go run ./cmd/script/ # or: docker build -t hh . && docker run -p 2112:2112 hh ``` - -To render the site against your local harness, run a local Prometheus scraping `localhost:` (the [`infrastructure/prometheus/README.md`](./infrastructure/prometheus/README.md) has notes) and point the corresponding YAML's `prom_url` at it. - -## Adding a benchmark - -Full guide in [CONTRIBUTING.md](./CONTRIBUTING.md). For a concrete end-to-end example, read [`docs/walkthrough.md`](./docs/walkthrough.md). Short version: - -1. **Open an issue** with the [📊 Propose a benchmark template](https://github.com/ChainBench/OpenChainBench/issues/new?template=new-benchmark.yml). Sketch the metric, providers, methodology. get feedback before you build. Want to brainstorm first? Use [Discussions → Ideas](https://github.com/ChainBench/OpenChainBench/discussions/categories/ideas) instead. -2. **Write the spec** at `benchmarks/.yml`. Format documented in [`benchmarks/README.md`](./benchmarks/README.md), validated by `src/lib/spec-schema.ts`. -3. **Build the harness** in `harnesses//`. Any language works as long as it exposes `/metrics` over HTTPS with the metric names and labels your spec references. The harness is a data producer only. no Prometheus, Grafana, or Alertmanager packaging. -4. **Deploy the harness** on whatever infra fits. Railway, Fly, Cloud Run, a VPS, even a home server with a static IP. Expose `/metrics` over HTTPS at a stable public URL. You own the runtime, the secrets and the budget. -5. **Add a scrape entry** to `infrastructure/prometheus/prometheus.yml` pointing at your public URL so the shared Prometheus picks up your harness. -6. **Open a PR.** CI runs schema validation, typecheck, lint, and build. Once merged, a maintainer redeploys the central Prometheus and the site renders your benchmark on the next ISR cycle (≤ 60 s). - -You never share API keys or wallet keys with the project. Your harness runs with your credentials, on your infra, on your budget. the maintainers only see the metric values your harness chooses to publish. - -### Filtering by chain / dimension - -A spec can declare `dimensions.chain` (or other dimensions) to expose a tab strip above the chart that filters every PromQL query by that label. Example from `benchmarks/aggregator-head-lag.yml`: - -```yaml -dimensions: - chain: - - { value: all, label: All chains } - - { value: base, label: Base } - - { value: bnb, label: BNB Chain } - - { value: solana, label: Solana } +benchmarks/ 94 YAML specs, one per public benchmark +harnesses/ 50 Go harnesses, each self-contained + Dockerized +answers/ 32 Q&A YAMLs backing /answers/ +alternatives/ 23 "alternatives to X" YAMLs backing /alternatives/ +worker/ Materialization worker (sweeps Prom → publishes CDN + Redis blobs) +infrastructure/ Shared Prometheus config +src/ Next.js 16 site (App Router, ISR, Tailwind 4, TypeScript, Zod) +docs/ architecture.md, methodology, walkthrough.md +scripts/ Spec validators + prebuild manifest generators +.github/ CI: prod deploy, staging preview, sync dev from main ``` -Selecting `Base` injects `chain="base"` into every selector in the spec, including the per-provider `regions:` subqueries. URLs are shareable via `?chain=base`. - -## Editorial conventions - -- **No pre-determined winners.** Specs do not declare a "best" provider. The leader on every page is computed at render time from the lowest p50. -- **Tail before mean.** Headlines use p50 and p99. The arithmetic mean is reported in the table but never used as a takeaway. -- **State the timeout.** Failures are excluded from latency aggregates and counted toward success rate. Both numbers are reported. -- **Methodology first.** A spec without a written methodology is rejected. -- **Corrections in place.** If a number is wrong we publish a dated note on the affected report; future readers see it on the masthead. - -## Stack - -- Next.js 16 (App Router, ISR, Turbopack) on Vercel -- Tailwind v4 (CSS-only theme, `@theme` tokens) with `@custom-variant dark` class strategy -- Light + dark mode, localStorage-persisted with a pre-paint inline script to avoid flash -- Inter / Inter Tight / JetBrains Mono via `next/font` (Source Serif 4 still loaded for the server-rendered share cards) -- Zod for spec validation -- Prometheus HTTP API (instant + range queries) -- Go 1.24 for the existing harnesses (any language is acceptable) -- WebSockets + a small Go relay for the live dashboard - -### Design system - -White paper, slate ink ramp, vivid orange accent (`#EA580C`). All chrome -colors are CSS variables in `src/app/globals.css`, with a `.dark` block -that overrides each token for dark mode. Components reference tokens -(`text-ink`, `bg-surface`, `border-rule`, …) instead of literal hex so -the theme switch is purely declarative. Brand colors per chain/product -live in `src/lib/brand.ts` — picked to be saturated enough to read on -both light and dark backgrounds. +## Contributing -Exported share-cards (`/benchmarks/[slug]/share-card`) mirror the active -site theme by reading `html.dark` and appending `?theme=dark` to the -PNG URL — the downloaded image matches what the user is looking at. +Read [CONTRIBUTING.md](./CONTRIBUTING.md) for the long-form guide. TL;DR: -## Community +1. Float an idea in [Discussions](https://github.com/ChainBench/OpenChainBench/discussions) +2. Open a `📊 Propose a benchmark` issue with the metric + methodology +3. Write the harness (Go recommended, any language shipping `/metrics` works) +4. Host it (anywhere with HTTPS + public `/metrics`) +5. Open a PR against `dev`: spec YAML + harness ref + Prom scrape config +6. Review on the staging preview; merge to `main` when data lands -- 💡 [Discussions → Ideas](https://github.com/ChainBench/OpenChainBench/discussions/categories/ideas). brainstorm new benchmarks before writing them up -- 🙋 [Discussions → Q&A](https://github.com/ChainBench/OpenChainBench/discussions/categories/q-a). methodology / harness / spec questions -- 📊 [Discussions → Show & tell](https://github.com/ChainBench/OpenChainBench/discussions/categories/show-and-tell). share forks and dashboards -- 🗺️ [Roadmap](https://github.com/orgs/OpenChainBench/projects). what's planned and what's live -- 🐞 [New issue](https://github.com/ChainBench/OpenChainBench/issues/new/choose). formal benchmark proposal, data-quality flag, or provider correction -- See [SUPPORT.md](./.github/SUPPORT.md) for the full triage matrix. - -## SEO TODOs - -Tracking the work to make benchmark pages rank for the queries they target. - -- **Auto-crawl ping on data update.** Today Google decides when to re-crawl on its own (often 1-2 weeks for a young domain). Build a Vercel cron that: - - POSTs to [IndexNow](https://www.indexnow.org/) (Bing, Yandex, others) with the changed URLs each day → free, no auth - - Re-submits `sitemap.xml` to Search Console via the official URL Inspection or Indexing API - - Why: even though our HTML is regenerated every 60 s via ISR, search snippets keep showing the timestamp Google last crawled. Faster re-crawl = fresher SERP snippet = higher CTR on freshness-sensitive queries ("fastest crypto data API today"). Side benefit: new pages (`/alternatives/*`, new benches) get indexed in 24-48 h instead of weeks. -- **FAQ JSON-LD on /benchmarks/[slug].** Add a `FAQPage` block with 3-5 Q&As per bench ("Which provider is fastest on X?"). Rich snippet in Google + extra keyword density. -- **Keyword densification on high-competition benches** (aggregator-head-lag, bridge-fee). Currently the title contains the target keyword once. Etoffer subtitle + abstract to repeat naturally 2-3×, plus a `headlineSentence()` tweak so the snippet contains the keyword. -- **Internal linking with descriptive anchors.** Ledger-table now links provider names to `/products/[slug]` but with the provider name as anchor. Add cross-references between related benches with anchors like "fastest data provider API" pointing to the relevant /benchmarks/[slug]. -- **Headline sentence formatter bug.** Output currently concatenates samples + provider count without separator (e.g. `522,88211 providers`). Split with `·` for readability and Google snippet quality. +## Editorial conventions -## Links +- **Same metric, same conditions**. If two providers can't be measured identically, they don't share a bench. +- **Provider marketing does not shape published numbers**. Public methodology + reproducible harness are the only source of truth. +- **Retirement over removal**. Retired benches return `410 Gone` with a link to the current catalog so previously-indexed URLs decay cleanly. +- **Freshness signals everywhere**. Every citable surface carries `dateModified` + `asOf` so LLMs can tell fresh data from stale. +- **Draft benches never quote a number**. If quorum drops, the page renders "awaiting samples" rather than a fabricated leader. -- Site. [openchainbench.com](https://openchainbench.com) -- Live stream. folded into [openchainbench.com](https://openchainbench.com) -- Twitter. [@OpenChainBench](https://x.com/OpenChainBench) -- GitHub. [ChainBench/OpenChainBench](https://github.com/ChainBench/OpenChainBench) +## Community + license -## License +- **Discussions**: [github.com/ChainBench/OpenChainBench/discussions](https://github.com/ChainBench/OpenChainBench/discussions) +- **Twitter/X**: [@OpenChainBench](https://x.com/OpenChainBench) +- **Wikidata**: [Q140172649](https://www.wikidata.org/wiki/Q140172649) +- **Zenodo DOI**: [10.5281/zenodo.20800311](https://doi.org/10.5281/zenodo.20800311) +- **Hugging Face dataset mirror**: [OpenChainBench/benchmarks](https://huggingface.co/datasets/OpenChainBench/benchmarks) (daily parquet snapshots) -Code: [MIT](./LICENSE). -Reports & figures: [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/). +**License**: harness code MIT · dataset CC-BY-4.0 · attribution required. See [LICENSE](./LICENSE).