Skip to content

benchmark.md

Latest commit

 

History

History
274 lines (227 loc) · 16.3 KB

benchmark.md

File metadata and controls

274 lines (227 loc) · 16.3 KB
Raw

Speedometer 3.1 across browsers, including BOSS's fluck browser

Benchmark: Speedometer 3.1 (browserbench.org, run against the live site - not a local mirror) Date: 2026-07-30 Machine: MacBook Pro Mac15,8, Apple M3 Max (12 performance + 4 efficiency cores), 128 GB, macOS 26.5.2 (25F84), on AC power at 100%

Speedometer 3's score is "iterations per minute" - higher is better. Each run used iterationCount=10 and startAutomatically=1, so every browser ran the identical workload via the identical URL.

Every number below is the median of 3 independent runs. Read the Caveats before quoting any of it: this machine was not quiet, and one browser's figure is not directly comparable to the rest.

Results

Browser Version Engine Runs Median Spread Method
BOSS fluck browser BOSS 9.2.63 - 47.9 / 47.5 / 48.4 47.9 ⚠️ 1.9% MCP browser_run_js
Comet 150.0.7871.228 Chromium 150 45.9 / 46.2 / 46.5 46.2 ⚠️ 1.3% CDP
Google Chrome 151.0.7922.71 Chromium 151 36.2 / 35.3 / 35.5 35.5 2.5% CDP
ChatGPT Atlas 1.2026.189.1 Chromium 150.0.7871.115 34.8 / 34.6 / 34.4 34.6 1.2% screenshot
Safari 26.5.2 WebKit 626 28.8 / 29.9 / 29.9 29.9 ⚠️ 3.7% screenshot
Firefox 153.0.1 Gecko / SpiderMonkey 22.5 / 30.9 / 22.0 22.5 40% WebDriver

⚠️ fluck and Comet are tied within measurement precision - the 1.7-point gap between them is not a result. See Is fluck actually faster than Comet?. ⚠️ Safari's number is not comparable to the others - see Caveats.

The one ordering this data supports firmly is that fluck and Comet both sit ~30% above Chrome and Atlas - a gap an order of magnitude larger than the run-to-run noise.

Is fluck actually faster than Comet?

Not established. Treat the top two rows as tied.

The runs above have a structural flaw at the top of the table. run-all.sh interleaves browsers round-robin precisely because load drift on this machine is worth 20-40% (caveat 4) - but the fluck browser is a tab inside the BOSS host, driven over MCP, so it could not join that sweep. Its three runs were collected separately, afterwards. The headline margin was therefore the one number not produced by the controlled protocol, and at 3.7% it was far smaller than the drift the protocol defends against.

So it was re-measured properly: Comet and fluck alternated back-to-back, three times, each pair within ~2 minutes (paired/PAIRED.md).

Pair Comet fluck fluck margin
1 44.7 46.5 +4.0%
2 46.1 46.95 +1.8%
3 45.4 48.3 +6.4%
median 45.4 46.95 +3.4%

fluck was ahead in all three pairs. But +3.4% is the same size as each browser's own spread within this experiment (Comet 3.1%, fluck 3.9%), and 3/3 is p = 0.125 under a sign test - directionally consistent, not significant. Separating these two would need many more runs on a quiet machine.

Supporting this reading: fluck's per-suite means sit uniformly 1-3% under Comet's with no signature, and Comet actually wins Charts-chartjs. A flat small offset is what an ambient-load difference looks like - contrast the Chrome/Comet gap below, which has a clear DOM-mutation signature.

Not benchmarked

Target Why not
Dia 1.42.1 Installed (WebKit-based, links /System/Library/Frameworks/WebKit.framework). Blocked on a one-time first-run onboarding - it plays an intro and never loads the requested URL until a human completes sign-in. Needs an account, so it was not automated.
Antigravity 1.11.5 Not a browser. It is a VS Code fork (an IDE). Its "browser" feature drives a separate real Chrome install via an extension, so benchmarking it would just re-measure Chrome.
Codex Not a browser. /opt/homebrew/bin/codex is OpenAI's CLI - no browser engine, nothing to run a web benchmark in.

The headline finding: the engine version barely matters here

Comet and ChatGPT Atlas ship the same Chromium 150 build family - yet they differ by 34%:

Embedder Chromium Median
Comet 150.0.7871.228 46.2
ChatGPT Atlas 150.0.7871.115 34.6

Two browsers, essentially one engine build (.228 vs .115), a third of the score apart - with Chrome 151 landing between them at 35.5. Whatever separates these browsers on Speedometer 3.1, it is not the engine version; it is how each product configures and wraps the engine it ships. That is the main reason a score here should be read as a property of the browser, not of its underlying engine.

Where Chrome loses to Comet: DOM mutation, not rendering

Comparing per-suite means from chrome-run2.json and comet-run2.json (ratio > 1 means Chrome is slower):

Suite Chrome ms Comet ms Ratio
TodoMVC-WebComponents 19.59 10.36 1.89
TodoMVC-jQuery 114.14 62.33 1.83
TodoMVC-Preact-Complex-DOM 14.24 7.78 1.83
TodoMVC-JavaScript-ES6-Webpack 37.20 24.23 1.54
TodoMVC-Vue 20.50 13.46 1.52
TodoMVC-Lit-Complex-DOM 14.46 9.96 1.45
Editor-TipTap 66.89 46.95 1.42
…
React-Stockcharts-SVG 50.18 47.52 1.06
Perf-Dashboard 27.62 26.03 1.06
Charts-chartjs (canvas) 28.79 27.63 1.04

The gap is concentrated in DOM-mutation / style-recalc workloads and nearly absent in canvas rendering (Charts-chartjs, 1.04×), SVG (1.06×), and page navigation (NewsSite-Next 1.07×, NewsSite-Nuxt 1.08×). This is a specific signature, not a uniform slowdown - which is what you would expect from a feature/configuration difference in the DOM path rather than a slower CPU or a slower JIT.

What was ruled out

  • Chrome's field-trial / variations config - REFUTED. Re-running Chrome with --disable-field-trial-config --disable-features=OptimizationHints scored 35.1, statistically identical to 35.5 with them enabled. Experiment groups do not explain Chrome's position.
  • Chromium version - REFUTED. Atlas (150.0.7871.115) and Comet (150.0.7871.228) differ by 34% on the same build family.
  • Background CPU load - ruled out for the Chrome/Comet gap specifically. Median ambient CPU recorded during the runs was comparable: Chrome 843/858/1138%, Comet 912/861/878%. Chrome was not measured under a heavier machine.

The cause is not established. The evidence narrows it to embedder configuration affecting the DOM path (candidates: site-isolation and sandboxing policy, injected agent/automation instrumentation, or differing Blink feature flags), but this benchmark did not isolate which. Do not quote a cause from this document.

Caveats - read before quoting these numbers

  1. The machine was under heavy competing load. Ambient CPU during the runs was 800–1300% (8–13 of 16 cores) - Docker Desktop's Linux VM alone held ~400%, plus endpoint-security agents (~218%) and Spotlight/Siri indexing (~105%). All absolute scores here are depressed and should not be compared to published browserbench numbers from a quiet machine. Relative comparison is the usable signal.
  2. Safari ran with a real profile; the Chromium browsers did not. Chrome, Comet, and Atlas each got a fresh throwaway --user-data-dir (no extensions, no tabs, no prior state). Safari was measured with the operator's live profile - roughly 14 background tabs and installed extensions. That handicap is unquantified and Safari's 29.9 is very likely an underestimate of WebKit on this hardware. Safari's figure is not comparable to the fresh-profile numbers, and it is the one row in the table that should not be used to rank engines.
  3. Firefox has 40% run-to-run spread (22.0–30.9) where every Chromium browser held within 2.5% at the same ambient load. Its median is the least trustworthy number in the table; three runs are not enough for Gecko here.
  4. Run order was interleaved, deliberately. Runs go round-robin (run 1 of every browser, then run 2, …) rather than three-in-a-row per browser. An earlier grouped-by-browser sweep produced badly skewed results - Firefox read 23.1 → 18.0 → 15.8 and Comet 45.2 → 36.0 purely from load drift and a leaked process landing on whichever browser ran last. Those numbers were discarded, not reported.
  5. Screenshot-derived scores are transcribed by eye. Safari and Atlas expose no automation endpoint, so their scores were read off Speedometer's own result screen (cropped images under results-final/cropped/) with no programmatic value to cross-check against. Chrome, Comet, Firefox, and fluck numbers came from reading #result-number directly out of the live DOM.
  6. Speedometer reported valid: true for every CDP/WebDriver run (read from summary.classList), and every viewport exceeded Speedometer's 850×650 minimum (1280×783 Chrome, 1270×780 Comet, 1280×786 Firefox, 1432×729 fluck). For the screenshot-derived rows (Safari, Atlas) validity is recorded as null, not true - the committed crops show the score but not Speedometer's validity banner, so asserting valid: true there would claim more than the evidence supports.
  7. fluck's viewport is 1432×729 against 1280×783 for the others. Both clear the minimum, but Speedometer's DOM work is viewport-dependent, so this is an uncontrolled variable in the fluck comparison - another reason to read the top two rows as tied.
  8. Ambient CPU is sampled from ps -Ao pcpu, which is a decaying average over up to a minute rather than instantaneous load. It is a co-tenancy indicator, not a precise measure of load during the run.

How to reproduce

All scripts live in benchmarks/speedometer/. They are dependency-free - Node ≥21 (for the built-in global WebSocket) plus geckodriver for Firefox.

macOS only. Every runner shells to osascript, open, screencapture, sips and pgrep -f, and hardcodes /Applications paths. run-all.sh exits 2 on any other platform rather than failing halfway through.

cd benchmarks/speedometer
brew install geckodriver                     # Firefox only

# Full interleaved sweep, 3 runs per browser
RESULTS=results-final REPEATS=3 ITERATIONS=10 \
  BROWSERS="chrome comet firefox safari atlas" ./run-all.sh

The sweep exits non-zero and lists which runs failed, so a missing result file cannot be mistaken for full coverage.

Keep the benchmark window uncovered for the whole run. A window the OS considers occluded gets its requestAnimationFrame throttled, which produces a low score rather than an error. run-chromium.mjs now refuses to start in that state and flags a run that becomes hidden partway, but the screenshot runner cannot check this - so don't stack windows over a running benchmark, and don't leave a full-screen app on another Space.

Script Role
run-all.sh Orchestrator. Round-robin interleaving, REPEATS/ITERATIONS/BROWSERS knobs.
run-chromium.mjs Chrome, Comet. Drives CDP over Node's built-in WebSocket; fresh profile per run; records ambient CPU; verifies the browser actually died before returning.
run-webdriver.mjs Firefox via geckodriver (W3C WebDriver is plain HTTP + JSON, so fetch suffices). Also supports Safari if you run sudo safaridriver --enable.
run-screenshot.mjs Safari, Atlas, Dia - browsers with no automation endpoint. Detects run completion from CPU falling back toward a pre-launch baseline, then captures the result screen.
crop-score.sh Crops a full-screen capture to the score card.

Raw per-run JSON (including 197–206 per-suite metrics for the CDP/WebDriver browsers) is in results-final/. fluck.json holds the fluck browser's full run.

Why the fluck browser is driven differently

The fluck browser is a tab inside BOSS, not a separate process, so it is driven through the host's own MCP tools (tab_focus, browser_navigate, browser_run_js) rather than by run-all.sh.

One non-obvious requirement: the fluck tab must be the active tab, with BOSS frontmost, for the whole run. Speedometer 3.1 measures with requestAnimationFrame by default, and a background BOSS tab reports document.hidden === true, which throttles rAF to a crawl - an early attempt sat at 11/580 steps after 40 seconds. Verify before trusting a run:

// must report hidden:false
JSON.stringify({ hidden: document.hidden, vw: innerWidth, vh: innerHeight })

Harness bugs found and fixed while building this

Recorded because each one silently produced plausible-looking wrong numbers:

  • An occluded window silently throttles the benchmark - and still reports valid: true. This is the worst of the set. macOS marks a fully-covered window hidden, and Chromium then throttles requestAnimationFrame in it; Speedometer 3.1 measures with rAF. A run in a covered window does not fail - it crawls (284/580 after five minutes, observed) and, if it finishes, yields a much lower score that is indistinguishable from a genuinely slow browser. The harness had checked viewport size and Speedometer's validity flag but never visibility. run-chromium.mjs now passes --disable-backgrounding-occluded-windows, refuses to start a run whose window reports document.hidden (retrying the raise first), samples it on every poll, and records occludedDuringRun in each result. Every run reported here recorded false; the fluck arms were checked before each run with completion time as the cross-check.
  • Leaked browser processes poisoned later runs. A failed Comet run left its process alive; the next run then contended with it and read 36.0 instead of ~46 - which looked exactly like "Comet is as slow as Chrome". run-chromium.mjs now polls until no process matching the run's profile directory remains.
  • Absolute CPU-idle threshold hung Safari. Matching com.apple.WebKit catches every WKWebView app on the machine, so "idle means < 15%" never fired while an unrelated chat app was busy, and the run would have spun to its 45-minute timeout. Now measured relative to a pre-launch baseline, with a hard 180s cap.
  • A command-line URL is only a hint. Comet intermittently showed its own startup page and dropped the benchmark URL. The runner now falls back to Target.createTarget over CDP.
  • Ambient CPU counted the browser's own helpers. Excluding by main-binary path missed Chromium's renderer/GPU processes under Contents/Frameworks/…, inflating "ambient". Now excluded by .app bundle.
  • sips exits 0 on a missing input. crop-score.sh reported "wrote …" having produced nothing. It now checks the input exists and the output is non-empty.

Notes for the operator

  • ChatGPT Atlas disappeared mid-session. Atlas 1.2025.288.14 was present at the start (its Info.plist read com.openai.atlas), then vanished from /Applications - bundle and Application Support both gone, nothing in Trash, no unified-log evidence. It went missing after being launched with --remote-debugging-port, a pattern EDR products flag as browser credential theft, and this machine runs managed endpoint-security/EDR agents. Cause unproven. It was reinstalled from the chatgpt-atlas Homebrew cask as 1.2026.189.1 (newer than what was there), and that is the version benchmarked.
  • Atlas has no usable --remote-debugging-port - the outer bundle is a Swift/SwiftUI shell and the engine is a nested app at Contents/Support/ChatGPT Atlas.app, so the flag never reaches Chromium.
  • Four orphaned WebKit.WebContent processes (parented to launchd, Safari not running, PIDs long predating this session) were burning ~180% CPU during the early runs. They exited on their own before the final sweep.
  • Newly installed for this benchmark: firefox, comet, geckodriver, chatgpt-atlas (Homebrew), and Dia 1.42.1 from releases.diabrowser.com.