scrape-autorouter

Route each web scrape to the provider that returns the most information for the price.

Vaaya runs many scraping providers behind one endpoint. This repo is the routing core that picks between them. It ranks providers by measured information yield, not by vendor success codes. A 200 response that is a CAPTCHA wall returns no information, so it scores zero and steers routing away.

The code here is pure and has no database or network dependency. Vaaya wires it to Postgres in production. You can wire it to anything, or run it in memory.

The idea

Most routers pick the cheapest provider, or the one that returns HTTP 200. Neither is what you want. You want the page's content.

So the router scores every billed result on a single axis: information yield, a number from 0 to 1. Yield is the fraction of requested formats that came back with real content. Thin bodies and bot walls score near zero even when the vendor returned success.

Ranking is then lexicographic:

1. Most information first.
2. Cheapest provider within a small yield band.
3. Lowest measured latency as the tiebreak.
4. Static priority order last.

New providers ride an optimistic prior, so they earn real traffic instead of being starved by an incumbent's record. A daily benchmark seeds the priors before live traffic sees a new provider.

Install

npm install
npm test

Rank providers

import { rankByInformation, InMemoryStatsStore } from 'scrape-autorouter'

const store = new InMemoryStatsStore()

const order = rankByInformation(
  [
    { provider: 'crw', priceCents: 1, priorityIndex: 0 },
    { provider: 'firecrawl', priceCents: 1, priorityIndex: 1 },
  ],
  store.read('scrape', 'default'),
)
// order[0] is the provider to try first.

Score a result

import { assessScrapeYield } from 'scrape-autorouter'

const { yieldScore, flags } = assessScrapeYield(vendorResponse, ['markdown'])
// yieldScore: 0..1
// flags: 'blocked' | 'thin_content' | 'missing_format'

Feed the score back so the next call ranks on real data:

store.record({ provider: 'crw', action: 'scrape', ok: true, yieldScore, latencyMs: 820 })

Benchmark

The benchmark runs a sealed set of tasks against every provider. Every provider runs the identical tasks, so the results compare. The task set is versioned. Changing a URL means a new version, never an in-place edit, so rows stay comparable within a version.

The protocol is adapted from AgentSLABench (arXiv:2608.00805): identical sealed tasks, deterministic yield judging, benchmark-seeded priors.

npm run bench

bench/run.ts ships two mock providers so the harness runs with no API keys. Replace them with real fetchers to benchmark your own stack.

What is here

File	What it does
src/yield.ts	Scores a scrape result for information yield. No model, only string tests.
src/rank.ts	The pure lexicographic ranking. Most information, cheapest, fastest.
src/store.ts	A storage-agnostic stats port and an in-memory implementation with the EWMA math.
src/probe-tasks.ts	The sealed benchmark task set.
bench/run.ts	A runnable benchmark harness with mock providers.

License

MIT