One application, built twice. An emoji product-catalog app, implemented behind a single shared
DataSource contract — once on Harper (data + compute + messaging in one co-located system) and once on the
Vercel stack (Vercel functions + Neon Postgres + Upstash Redis + Ably realtime). Same UI,
same AI generator, same data. The only variable is the architecture.
We ran 474 load tests across 3 trials per coast — 843,240 requests, from two U.S. coasts, across eight scenarios ("acts") — under an open-model, single-target load methodology that measures each platform's true offered-load behavior. This is the story of what we found.
Outcome: Harper and Vercel each win the half of the web they were built for, and this methodology makes the line sharp. Harper's co-located, in-process architecture wins every live, personalized-data path — single reads, injected live values, server-side streaming, write-to-read freshness, and read fan-out at normal load — often several-fold, up to ~14×, and symmetrically across coasts. Vercel wins cacheable content (its CDN is superb) and broadcast-only realtime (Ably's purpose-built edge — though Harper additionally persists each event, so that one isn't apples-to-apples; see Act 8). And the honest catch: Vercel's serverless autoscaling outlasts Harper's single free node under high sustained fan-out load, where that one node hits a throughput ceiling. Pick by workload — live data → Harper; cache-heavy edge delivery and high-concurrency fan-out → Vercel.
Jump to the code: the two apps live at apps/harper/ and apps/vercel/ — each has
its own architecture + data-flow README. They share packages/shared/ (the DataSource contract, seed data, UI).
📊 Full per-act detail + methodology: RESULTS.md · 🔬 Exact conditions: MANIFEST.md · 📁 Raw data: bench/results/dataset.json + latency-quantiles.csv · 📈 Charts: bench/results/charts/
Vercel is genuinely excellent at what it was built for. The developer experience is best-in-class: git push
and you're global. Its CDN is superb — a cacheable page is served from an edge node milliseconds from your user,
anywhere on earth. Serverless functions scale to zero and back to thousands without you thinking about it. And the
ecosystem — Neon, Upstash, Ably — gives you a managed building block for every need. For a marketing site, a blog, a
docs portal, or any read-mostly, cacheable workload, it's hard to beat.
But modern apps aren't static. They're personalized, live, and interactive — a product page tailored to you, inventory that's correct right now, a feed that updates as things happen. The moment content can't be cached at the edge, Vercel's architecture has to do something fundamental: leave the function and cross the network to a separate service to get the data. Vercel doesn't store your data — Neon does. It doesn't cache — Upstash does. It doesn't do realtime — Ably does. Every personalized read, every cache check is a round-trip between services.
Harper is the structural answer to that hop. On Harper, the application runs inside the database. Reading data is a function call in the same process — not a network request to Neon. Caching is in-process — not a round-trip to Upstash. The "cross the network to another service" step simply isn't there. That's why Harper wins the live-data paths below, often by an order of magnitude, and does it identically on both coasts.
The honest trade-off. Co-location means the work happens on that node — and the free tier is a single 1-core / 1 GB node per region, the smallest compute in this comparison. Under high sustained concurrency on heavy pages that one node hits a throughput ceiling while Vercel's functions autoscale past it — a real result we measure directly (Act 5). The rest of this README shows all of it, with numbers.
A benchmark is only worth reading if it's honest, and the load model is where comparisons quietly go wrong: a shared load generator that hits both platforms in one loop lets the slower platform throttle the load offered to the faster one, hiding its true behavior under load. We avoid that entirely:
- Open-model, single-target. Each act runs once per platform (
constant-arrival-rate): the test fires a fixed number of requests per second regardless of how fast either platform answers. Offered load is set by us, never by one platform's latency. Neither platform can throttle or shield the other. - We sweep the offered rate (5 → 100 req/s) on the fan-out workload to find each platform's saturation knee — the point where it can no longer keep up. That's how the free-node ceiling above shows up as data, not a footnote.
- Same app, same code path. Both builds implement the identical
DataSourceinterface and render the identical UI. - AI is stubbed. Both call the same
generateDescription, switched to deterministic instant text (GENERATE_STUB=1) so we measure platform overhead, never model latency. Claude is in zero measured numbers. - Both platforms at genuine best case. Vercel: pooled Neon driver, Redis
SET NXlock for stampede,revalidatePathfor freshness, client-direct to Ably for realtime. Harper: write-through freshness, realtime node-pinning, an in-engine relational LEFT JOIN for fan-out. We steelmanned both sides, then measured them apart. - Principle: keep real advantages, remove only fake advantages and artificial handicaps.
Result: 843,240 requests, 1 failed — a single Vercel request under the deliberately-saturating knee sweep, i.e. 99.9999% success (no other 404s/500s); every run passed gating.
Full detail — load levels, key distribution, instance specs, metric provenance — is in RESULTS.md § Methodology. The essentials:
- 3 trials per coast (n=3), from Grafana k6 Cloud, two AWS zones (the open-model, single-target design is in How it's fair, above).
- Load levels: static/shell/stream 100 req/s, one-read probe 5 req/s, fan-out 10 req/s at the reference rate plus a 5→100 req/s sweep on three page shapes, stampede 50 in lockstep, realtime 25 WebSocket connections.
- Keys: uniform random over a small warm hot set (5 hot products / 25 hot pairs / 100 discover IDs) over 1,000 products / 50 personas / 50k interactions of seed data — so these are warm-path numbers (platform overhead).
- We report the in-process read where we have it (our key differentiator) at P50 and P95, plus TTFB and full response time, with full quantile distributions in RESULTS § Appendix.
- Network latency we can't control, stated plainly: West-Coast load comes from N. California (us-west-1, ~350 mi from Harper's LA node); East-Coast from Ashburn, VA (~12 ms from Harper's Columbus node). The in-process read (~0.4 ms) is geography-neutral — that's the architectural metric.
| Component | Where | Tier |
|---|---|---|
| Harper (app + data + realtime) | One auto-synced cluster — GCP us-west2 (Los Angeles) + us-east5 (Columbus, OH) | Free (Fabric START — 2× 1 vCPU / 1 GB colocated nodes) |
| Vercel functions | us-east-1 — Virginia (iad1) |
Pro (paid) |
| Neon Postgres | us-east-1 — Virginia | Free |
| Upstash Redis | us-east-1 — Virginia | Pay-as-you-go (paid) |
| Ably realtime | global edge | Free |
| Load generators (Grafana k6 Cloud) | N. California (us-west-1) and Ashburn, VA | Free tier + paid VUH (see cost) |
Harper was multi-region automatically — with zero effort on my part. My only choice was "deploy to the US." Harper Fabric provisioned a two-node cluster (Los Angeles + Columbus), kept in sync automatically, with Akamai global traffic routing in front so every user is served by the nearest node. I never configured replication, regions, or routing. (To match this on the Vercel side you'd bolt multi-region Neon read-replicas + routing onto the stack — see Complexity.)
So we tested from both coasts — N. California and Ashburn, VA. Vercel's data lives in one region (us-east-1),
so it's quick for East users and slower for West; Harper's cluster has a node near both. The in-process read
(server_lookup) is per node and geography-neutral; TTFB shows what a real user on each coast feels. (How to read the
West vs. East columns: see the note above the result tables.)
This benchmark isolates a single question: when content can't be cached at the edge and must be fetched live, what does each architecture pay to get it — an in-process call, or a network hop to a separate service? To answer that cleanly we hold the data path at its warm, in-memory best case: a 1,000-product catalog whose working set fits in each node's RAM, read over a small hot key set. That's deliberate — it strips out disk and cold-cache variance so the only thing left is the architectural difference. Read the numbers accordingly: they are an honest characterization of that gap at its core, and an upper bound on co-location's advantage. As a workload's working set grows past a node's RAM, Harper must go to storage and the in-process lead narrows toward a networked database's. Two regimes this run deliberately does not stress — working sets larger than node RAM, and sustained-saturation throughput beyond the free node — are the planned next chapter.
Format: West / East = load from N. California / Ashburn, VA. Each cell is the median of 3 trials per coast (n=3); where trial-to-trial spread matters (realtime, the knee, naive fan-out) the (min–max) range is shown inline, with full per-trial ranges in RESULTS.
How to read West vs. East on the dynamic paths (Acts 2–4): the East column is the in-region, like-for-like number — Vercel's data lives in the East (
us-east-1), so East is its home turf and the fairest head-to-head. The West column reflects Harper's automatically multi-region cluster vs. Vercel's single-region data (a West user crossing the country to East-coast Neon). That West gap is a real Harper advantage — tagged here, at the tables, so you can weigh it deliberately rather than read it as raw per-request speed.
| TTFB p50 (W / E) | full p50 (W / E) | full p95 (W / E) | |
|---|---|---|---|
| Harper (origin) | 11.5 / 12.5 ms | 11.8 / 12.8 ms | 13.3 / 14.9 ms |
| Vercel (CDN edge) | 7.5 / 8.8 ms | 7.7 / 9.0 ms | 21.9 / 26.5 ms |
Why: this is a CDN's home turf — the page is identical for everyone, so Vercel serves it from an edge node next to the user, no compute, no data. A CDN should win cacheable content, and it does. One nuance worth noting: Harper's p95 is actually tighter (13–15 ms vs 22–27 ms) — the origin cache is more consistent than the CDN's tail here — but Vercel's median is the number that matters for this workload, and it wins. We put this first to be honest.
| in-proc read p50 (W / E) | TTFB p50 (W / E) | TTFB p95 (W / E) | |
|---|---|---|---|
| Harper | 0.35 / 0.42 ms | 11.1 / 12.7 ms | 12.4 / 13.6 ms |
| Vercel + Neon | 2.98 / 3.30 ms | 93.7 / 34.3 ms | 119.8 / 61.7 ms |
Why: now the content is per-user, so it can't be cached — someone must fetch it. On Harper that's an in-process call (~0.35–0.42 ms, geo-neutral); on Vercel it's a ~3 ms round-trip to Neon — a ~8× in-process gap that is the seed of everything below (a warm-path result — see What this benchmark measures, above). End-to-end, Harper wins TTFB on both coasts: a West user's Vercel read round-trips cross-country to Neon (94 ms) vs. Harper's LA node (11 ms); on the East, Harper's Columbus node (13 ms) still beats Vercel's in-region read (34 ms). The network component (TTFB − in-process) is ~11–12 ms for Harper vs. 91 / 31 ms for Vercel — that's the hop to a separate database, made visible.
Live read (in-process), W / E: Harper p50 0.32 / 0.38 ms · p95 0.5 / 0.6 ms vs Vercel p50 3.2 / 3.2 ms · p95 4.3 / 4.4 ms.
TTFB — cached shell vs. dynamic api (W / E):
| shell p50 | shell p95 | api p50 | api p95 | |
|---|---|---|---|---|
| Harper | 11.5 / 12.4 ms | 13.2 / 13.7 ms | 11.1 / 12.0 ms | 12.4 / 13.4 ms |
| Vercel | 7.3 / 8.1 ms | 20.0 / 24.8 ms | 90.6 / 29.3 ms | 112.6 / 53.5 ms |
Why: the realistic pattern — cache the page frame, inject one live per-user value. The shell is cacheable, so
Vercel's CDN serves it fast on both coasts (~7–8 ms). The injected live read is ~8–10× faster in-process on Harper,
and the dynamic api that delivers it is 11–12 ms on Harper, both coasts, vs 91 ms West / 29 ms East on Vercel.
| TTFB p50 (W / E) | TTFB p95 (W / E) | full p50 (W / E) | full p95 (W / E) | |
|---|---|---|---|---|
| Harper | 11.8 / 12.4 ms | 13.1 / 13.9 ms | 12.0 / 12.5 ms | 13.5 / 14.3 ms |
| Vercel | 97.3 / 37.1 ms | 135.2 / 79.2 ms | 99.8 / 39.9 ms | 143.4 / 84.0 ms |
Why: flush the shell, then stream the dynamic hole. Harper streams in ~12 ms on both coasts; Vercel is 37–40 ms in its home region and degrades to ~97–100 ms for West users crossing the country to Neon. Closing Vercel's West gap means the multi-region Neon engineering described under Complexity; Harper's free cluster already spans both coasts.
Act 5 — Read fan-out (the marquee) → Harper far faster per request; Vercel's autoscaling wins under sustained load
A real page isn't one read — it's many connected reads (reads = 1 + n·(4 + 2·depth)). This act has two findings, and
keeping them separate is the whole point.
(a) The per-request compute tax. server_assemble = the server-side time to assemble the page, at the
reference rate. One asymmetry to read carefully: on Harper this timer is pure in-process compute (geography-neutral); on
Vercel the same timer also wraps the Neon/Upstash network round-trips — so it's CPU vs. CPU-plus-network, which if
anything understates Harper's edge. Optimized = each platform's best hand-tuned path; naive = how most teams write
it first. p50 ms, W / E:
| page (cards × depth, reads) | Harper opt | Vercel opt | Harper naive | Vercel naive |
|---|---|---|---|---|
| 10 × d2 (81) | 2.5 / 2.7 | 34 / 37 | 2.8 / 3.0 | 299 / 276 |
| 50 × d2 (401) | 7.2 / 8.0 | 33 / 33 | 9.8 / 11.1 | 1,565 / 1,483 |
| 50 × d6 (801) | 7.3 / 8.5 | 54 / 49 | 16.5 / 18.2 | 3,809 / 3,785 |
| 100 × d12 (2,801) | 13.7 / 14.7 | 85 / 80 | 54.3 / 65.5 | (not run — proj. > 90 s) |
The same shapes at p95 (server_assemble, ms, W / E):
| page (cards × depth, reads) | Harper opt | Vercel opt | Harper naive | Vercel naive |
|---|---|---|---|---|
| 10 × d2 (81) | 4.1 / 5.3 | 46 / 46 | 4.2 / 4.9 | 373 / 334 |
| 50 × d2 (401) | 13.5 / 14.9 | 48 / 39 | 14.5 / 20.0 | 3,211 / 1,622 |
| 50 × d6 (801) | 14.0 / 16.9 | 68 / 58 | 28.2 / 34.1 | 20,578 / 16,961 |
| 100 × d12 (2,801) | 24.2 / 29.1 | 116 / 109 | 161.8 / 237.6 | (not run — proj. > 90 s) |
At normal load Harper's in-process assembly is ~4–14× faster than Vercel's batched-JOIN path, because even a batched read is a network round-trip on Vercel and an in-memory call on Harper. And the naive trap is brutal on Vercel: the N+1-over-the-network pattern — how almost everyone writes it first — balloons to ~3.8 s on both coasts at 50×depth-6 (p50; p95 tails to ~17–21 s). We measured Vercel naive through 50×depth-6; the 100×d12 cell is not run — at 2,801 reads (≈3.5× more sequential round-trips) it would exceed k6's 90 s per-request timeout, so that cell is a projection, not a measurement. Harper's naive path stays close to its optimized one (no per-read network tax), so Harper naive still beats Vercel optimized at every size shown.
(b) The throughput ceiling — the knee. Read this scope first: that per-request speed is on Harper's free tier — a
single 1 vCPU / 1 GB node per region, the smallest compute in this entire comparison. Sweep the offered rate and that
one tiny node's ceiling appears, while Vercel's serverless autoscaling holds flat. server_assemble p50 vs offered rate
(✕ = saturated: the node can't sustain the offered rate and p50 collapses):
| page | 5 req/s | 10 | 25 | 50 | 100 |
|---|---|---|---|---|---|
| 50×d6 — Harper · West | 8 ms | 7 ms | 7 ms | 8 ms | 815 ms ✕ |
| 50×d6 — Vercel · West | 52 ms | 49 ms | 51 ms | 55 ms | 58 ms |
| 50×d6 — Harper · East | 8 ms | 9 ms | 9 ms | 8 ms | 1,196 ms ✕ |
| 50×d6 — Vercel · East | 49 ms | 47 ms | 52 ms | 53 ms | 56 ms |
| 100×d12 — Harper · West | 14 ms | 14 ms | 13 ms | 442 ms ✕ | 1,925 ms ✕ |
| 100×d12 — Vercel · West | 76 ms | 86 ms | 86 ms | 101 ms | 103 ms |
| 100×d12 — Harper · East | 15 ms | 15 ms | 15 ms | 2,032 ms ✕ | 2,310 ms ✕ |
| 100×d12 — Vercel · East | 71 ms | 75 ms | 84 ms | 97 ms | 118 ms |
The same sweep at p95 (server_assemble, ms):
| page | 5 req/s | 10 | 25 | 50 | 100 |
|---|---|---|---|---|---|
| 50×d6 — Harper · West | 14 ms | 13 ms | 18 ms | 120 ms | 1,245 ms ✕ |
| 50×d6 — Vercel · West | 60 ms | 61 ms | 79 ms | 78 ms | 87 ms |
| 50×d6 — Harper · East | 17 ms | 20 ms | 22 ms | 155 ms | 1,442 ms ✕ |
| 50×d6 — Vercel · East | 58 ms | 56 ms | 67 ms | 76 ms | 82 ms |
| 100×d12 — Harper · West | 23 ms | 24 ms | 110 ms | 963 ms ✕ | 2,310 ms ✕ |
| 100×d12 — Vercel · West | 88 ms | 104 ms | 129 ms | 180 ms | 340 ms |
| 100×d12 — Harper · East | 25 ms | 27 ms | 134 ms | 2,441 ms ✕ | 2,638 ms ✕ |
| 100×d12 — Vercel · East | 79 ms | 90 ms | 137 ms | 168 ms | 449 ms |
The ✕ marks genuine saturation — p50 collapsing to seconds because the node can't sustain the offered rate. Both platforms also dropped a handful of iterations (≤45, <1%) at the very top rates, but with latency unchanged — a brief ramp/cold-start transient, not saturation, so it isn't flagged.
Within the swept window (free tier, up to 100 req/s, both coasts), only Harper's free node reaches a true knee — and only on the heavy pages (light pages stayed flat to 100 req/s); we did not locate one for Vercel. That isn't evidence it has none — a stateless platform answers load by adding instances, holding latency flat (~50–120 ms across the sweep) as long as it can keep spawning compute. The difference is where the ceiling lands: Harper's fixed free node turns sustained overload into a latency wall, while Vercel turns it into cost — every extra concurrent request is another billed invocation. So the workload decides: moderate fan-out → Harper is far faster and cheaper; heavy fan-out at high sustained concurrency → Vercel's autoscaling wins. (Finding Vercel's own knee would mean driving load past this tier's range — out of scope here.)
Read this ceiling for exactly what it is: the limit of one free 1 vCPU / 1 GB node, not of Harper's architecture. It's bound by that single node's CPU and RAM; a paid tier with more cores, memory, and nodes raises it substantially (a planned follow-up). Sweeping the offered rate — rather than measuring one load point — is what makes the knee visible at all, and it reproduces on both coasts across 3 trials (same shapes, same rates, dropped-iteration surges tracking each collapse), so it's a real property of the free node, not a single-cell artifact.
| Harper (write-through) | Vercel (revalidatePath) |
|---|---|
| ~128 ms to serve fresh content | ~387 ms to serve fresh content |
What this measures. Data changes — how long until a reader sees the new version while the page is still served from cache? That's the cache-coherence problem. The test is identical for both: warm the cached page, fire one invalidation, poll until fresh content appears, report wall-clock from signal → fresh.
Why Harper wins, structurally. On Harper the write path and the cache live in the same process as the data, so "data
changed → re-render → replace the cache entry" is one local, synchronous step (write-through), and the next read is a fresh
local hit. On Vercel the data (Neon), cache (CDN/ISR), and compute (function) are three separate services:
revalidatePath marks the path stale, then a regeneration re-fetches from Neon across the network, re-renders, and
repopulates the edge — each step a hop. That cross-service orchestration is the ~3.0× difference. Both are event-driven
and sub-second, and we poll both sides at the same 25 ms interval so the comparison is apples-to-apples. It matters for
anyone who caches for speed but whose data changes and must stay current: live inventory/pricing, a CMS "publish,"
dashboards, feature flags, collaborative edits.
50 requests hit one cold key simultaneously. Both coalesce to 1 regeneration — Harper via in-process single-flight
(automatic, inherent), Vercel via a Redis SET NX lock (effective when coded correctly, though the lock can fail open if
Redis is unavailable). Same result; different guarantees.
| p50 (W / E) | p95 (W / E) | delivery (W / E) | |
|---|---|---|---|
| Vercel + Ably (client-direct, broadcast-only) | 17 / 24 ms | 53 / 52 ms | 100% / 100% |
| Harper (native WS, node-pinned, persist + broadcast) | 34 / 55 ms | 252 / 171 ms | 100% / 100% |
Read the work asymmetry first — this is not apples-to-apples, and the asymmetry favors the winner. The Vercel path publishes client-direct to Ably's edge (
DIRECT=1): the Vercel function and Neon are out of the path, and the event is never persisted — it is broadcast-only. Harper's path persists every event to the database and broadcasts it in one in-process step. So this measures "Ably broadcast-only" against "Harper persist + broadcast" — Harper is doing strictly more durable work. A like-for-like durable path on the Vercel side (function → Neon write → fan-out) would add the very cross-service hop that costs ~90 ms elsewhere in this suite. We report Ably's best case anyway, because it is the realistic way to build realtime on that stack — but the latency win is for broadcast-only delivery, not a persist-and-deliver guarantee.
Why: Ably operates a purpose-built global edge — clients connect to the nearest PoP and messages route within Ably's backbone, delivering at 17–24 ms p50, a tight ~52 ms p95 tail, and clean 100% delivery on both coasts. Harper's native WebSocket also delivers cleanly — 100% on both coasts — but it is slower on the median (34 / 55 ms) and heavier in the tail (171–252 ms p95) than Ably's purpose-built edge. The delivery-ratio guard (received / sent) confirms neither side silently drops messages — a latency number only counts at clean delivery. One nuance the guard surfaced: in one of the three West trials the LA node briefly delivered ~19% of events twice (an active-active replication echo, 119% delivery, ~1.25 s tail) — it did not recur in the other West trials, so the median delivery is a clean 100% and it is not the steady-state behavior. For realtime, the specialized global edge wins on speed and tail — and Harper, which does the extra work of persisting every event, still delivers every message.
A note on Vercel WebSockets (new — beta): As this benchmark was being finalized, Vercel shipped native WebSocket support in public beta (changelog). We benchmarked the Vercel stack's established realtime path, Ably — serverless functions are stateless and historically can't hold the persistent connections realtime needs, which is the very reason the production stack adds a dedicated realtime service. We did not test the brand-new beta path. This is a fast-moving space: if you benchmark Vercel's native WebSockets against Harper's, we'd love to see your numbers — share them with us.
Look at what each platform is:
VERCEL'S STACK (4 services, 3 networks to cross) HARPER (1 system)
┌────────────┐ net ┌──────────┐ ┌──────────────────────┐
│ Vercel fn │ ────▶ │ Neon DB │ (every read) │ app + data + cache │
│ │ ────▶ │ Upstash │ (every cache op) │ + realtime, in one │
│ │ ────▶ │ Ably │ (every realtime) │ process, in memory │
└────────────┘ └──────────┘ └──────────────────────┘
4 dashboards, 4 bills, 4 SLAs, 1 dashboard, 1 bill,
4 sets of keys, 3 network hops 0 inter-service hops
Vercel functions deliberately don't do persistence, caching, or realtime — they're stateless compute — so a real app is assembled from four vendors wired together over the network. That network hop is the direct cause of the live-data gaps above: the ~0.4 ms vs 3 ms read (Act 2) is the hop to Neon; the multi-second naive fan-out (Act 5) is that hop paid per read; the freshness gap (Act 6) is the cross-service revalidation path. Harper removes the hop by co-locating everything in one process — and that's why it wins those paths, with one dashboard and one bill.
But co-location cuts both ways, and we measured both edges. The same single process that makes Harper's reads sub-millisecond is also a single free node with a finite ceiling (Act 5's knee), and Vercel's four-service sprawl is also four independently-autoscaling tiers. So the honest summary is architectural fit, not a blanket winner: co-location wins the live-data, moderate-load majority of real apps; service-sprawl-with-autoscaling wins cacheable delivery, realtime, and heavy fan-out at high sustained concurrency. And the multi-region version of the Vercel stack — needed to match Harper's both-coasts symmetry — means adding Neon read-replicas, replica routing, and cross-region cache coherence on top of the four services. Harper's free deploy was already a multi-region, auto-synced cluster (Los Angeles + Columbus) with zero config.
This entire benchmark — 1,000 products, 50k interactions, 843,240 requests, eight scenarios — ran on Harper's free tier (a START cluster that, with zero configuration, was already multi-region: Los Angeles + Columbus, auto-synced).
The Vercel stack could not be tested on free tiers. To run the same workload we had to pay across the stack:
| Service | Free tier allowed… | Why we had to pay |
|---|---|---|
| Vercel → Pro (paid) | Hobby: 1 M function calls, 4 CPU-hrs, DDoS shield on | The DDoS/attack mitigation tripped on our load-test traffic and blocked the run; the benchmark also exceeds Hobby's 1 M-invocation / 4-CPU-hour ceilings. |
| Upstash → Pay-as-you-go (paid) | Free: 500 K commands/month | Every Vercel read is a Redis cache check; the run issued millions of commands. |
| Neon | Free tier sufficed | — |
| Ably | Free tier sufficed (200 connections / 6 M msgs·mo; we used 25) | — |
| Harper | Free START cluster ran the entire benchmark — multi-region out of the box | Nothing to upgrade. |
The same test that Harper ran for $0 required paid plans on two of Vercel's four services — more services means more independent limits to blow past. (For completeness: the k6 Cloud load generator is our testing tool, not part of either platform's bill, and doesn't enter the cost comparison.)
(Full list in RESULTS § Caveats.)
- The free-node ceiling is scoped to the free tier (Act 5 knee). The saturation is the limit of a single 1 vCPU / 1 GB free node, not of Harper's architecture; Vercel's autoscaling doesn't hit it. More cores/RAM (a paid tier) raise the ceiling substantially — a larger-tier sweep is a planned follow-up.
- Harper realtime. Harper delivers cleanly — 100% on both coasts — but is slower on the median and tail than Ably's edge, so Vercel wins realtime. In one of three West trials the LA node briefly delivered ~19% of events twice (an active-active replication echo with a heavy tail); it did not recur, so median delivery is 100%. Disclosed via the delivery-ratio guard.
- 3 trials per coast (n=3); stampede and realtime are n=6 (3 per coast). Table cells are the median across trials; full per-trial ranges are in RESULTS § Appendix.
- Working set fits in RAM (the biggest scope boundary). These are warm-path numbers — the corpus fits in each node's RAM — so they capture Harper's in-process advantage at its maximum and don't extrapolate to working sets larger than RAM. Full framing and the planned follow-up are in What this benchmark measures, above.
- CDN and realtime are real Vercel wins. Where content is cacheable (Acts 1, 3-shell) or pub/sub-realtime (Act 8), Vercel's edge genuinely wins — we show it.
- Harper is multi-region (fully disclosed). Its free deploy is automatically a 2-node cluster (LA + Columbus) behind Akamai routing; the geography-neutral metric is the per-node in-process read (~0.4 ms).
- RESULTS.md — every act, full numbers + quantiles, methodology, the knee sweep, and per-run quantile distributions (min→max).
- MANIFEST.md — exact tiers, regions, timestamps, commit, env, cache state.
- Raw data: bench/results/dataset.json + latency-quantiles.csv (every metric, every run, min→max).
Regenerate:
node bench/build-dataset.mjs→node bench/analyze-v2.mjs→node bench/make-charts.mjs; verify withnode bench/verify-runs.mjs. - bench/run-regression-v5.sh — the open-model, single-target orchestrator (resumable); run IDs in bench/results/regression-runmap-v5.tsv.
- The two apps: apps/harper/ and apps/vercel/ (each with an architecture README), sharing packages/shared/.
platform-comparison/
├─ README.md ← you are here (the narrative + headline results)
├─ RESULTS.md ← detailed per-act results, methodology, full quantiles
├─ MANIFEST.md ← exact run conditions (tiers, regions, commit, env)
├─ PLAN.md ← the original design/fairness plan
├─ apps/
│ ├─ harper/ ← the Harper build (app inside Harper) + its architecture README
│ ├─ vercel/ ← the Vercel build (functions + Neon + Upstash + Ably) + README
│ └─ reference/ ← the shared-UI reference app both builds derive from
├─ packages/
│ ├─ shared/ ← DataSource contract, seed data, generator, UI components
│ └─ e2e/ ← Playwright end-to-end tests (functional-parity checks)
└─ bench/ ← k6 load scripts + analysis
├─ lib/scenario.js ← shared open-model (constant-arrival-rate) helpers
├─ mode1/mode2-load/mode3/fanout/act6-freshness/act7-stampede/rt-* ← the 8 acts' scripts
├─ run-regression-v5.sh ← open-model, single-target orchestrator
├─ build-dataset.mjs / analyze-v2.mjs / make-charts.mjs / verify-runs.mjs ← the data pipeline
└─ results/ ← dataset.json, latency-quantiles.csv, run map, charts/, raw/