Skip to content

Commit 6d3fc9f

Browse files
committed
benchmark: add live view CDP latency comparison tool and results
Benchmark tool and results comparing CDP operation latency across four image variants: headless baseline, Approach 1 (Xvfb/noVNC, PR #174), Approach 2 (CDP screencast, PR #176), and headful. Covers 40+ CDP operations across 9 categories (screenshot, JS eval, DOM, input, network, page, emulation, target, composite) plus concurrent load testing. Includes results from Docker (4 vCPU / 1 GB headless, 8 vCPU / 8 GB headful), Docker with constrained headful (4 vCPU / 1 GB), and KraftCloud (Unikraft) environments. Key findings: - Approach 2 (CDP screencast) adds near-zero overhead vs baseline - Approach 1 (Xvfb/noVNC) adds ~30% overhead on input/screenshot ops - Headful under headless constraints is not viable (39% idle memory) - On Unikraft, Approach 2 remains the better choice for live view Made-with: Cursor
1 parent 4a321ab commit 6d3fc9f

60 files changed

Lines changed: 9779 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

benchmarks/liveview/.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
bench

benchmarks/liveview/README.md

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
# Live View Benchmark
2+
3+
Compares CDP operation latency and resource usage across four image variants:
4+
5+
| Variant | Branch | Image | Description |
6+
|---|---|---|---|
7+
| `baseline` | `main` | headless | Current production headless (no live view) |
8+
| `approach1` | `feat/headless-live-view` | headless | Xvfb + noVNC live view (PR #174) |
9+
| `approach2` | `headless-cdp-live-view` | headless | CDP screencast live view (PR #176) |
10+
| `headful` | `main` | headful | Full headed browser (reference) |
11+
12+
## Test sites (15 sites across 5 categories)
13+
14+
| Category | Sites |
15+
|---|---|
16+
| Static | example.com, httpbin.org, lite.duckduckgo.com |
17+
| Content | Hacker News, Wikipedia, Reuters, Lobsters |
18+
| SPA | Google Maps, GitHub, Reddit |
19+
| Media | Unsplash, BBC News, CNN |
20+
| Complex | Amazon, eBay |
21+
22+
## CDP operations tested (40+ operations across 9 categories)
23+
24+
**Navigation** — full page load to each site category
25+
26+
**Screenshots** — JPEG (q80), PNG, full-page capture, clipped region
27+
28+
**JS Evaluation** — trivial expression, querySelectorAll count, innerText length, getComputedStyle, scroll to bottom, DOM manipulation (create/remove 100 elements), bounding rect collection
29+
30+
**DOM** — getDocument at depth 1/5/full, querySelector, getOuterHTML
31+
32+
**Input** — mouse move, click (press+release), type 21 chars, scroll (5x wheel events)
33+
34+
**Network** — getCookies, fetch page body via Runtime.evaluate
35+
36+
**Page** — reload (to load event), getNavigationHistory, getLayoutMetrics, printToPDF
37+
38+
**Emulation** — set desktop viewport, toggle mobile viewport, set geolocation
39+
40+
**Target** — getTargets, create+close tab round-trip
41+
42+
**Composite scenarios** (realistic multi-step automations):
43+
- Navigate + screenshot
44+
- Scrape all links on page
45+
- Inject form + fill + read back
46+
- Click first link + wait for navigation
47+
- 10 rapid screenshots in sequence
48+
- Scroll 5 steps with screenshot at each position
49+
50+
**Concurrent load test** — N parallel CDP connections all firing screenshots, evaluates, and DOM queries simultaneously for a configurable duration
51+
52+
## Resource allocation
53+
54+
Containers are constrained to match production allocations:
55+
56+
| Type | Memory | CPUs | Variants |
57+
|---|---|---|---|
58+
| Headless | 1024 MB | 4 | baseline, approach1, approach2 |
59+
| Headful | 8192 MB | 8 | headful |
60+
61+
Override with env vars:
62+
63+
```bash
64+
HEADLESS_MEMORY=2048m HEADLESS_CPUS=4 HEADFUL_MEMORY=8192m HEADFUL_CPUS=8 ./run.sh
65+
```
66+
67+
## Quick start
68+
69+
```bash
70+
./run.sh
71+
72+
# Customize
73+
ITERATIONS=5 WARMUP=2 CONCURRENT_WORKERS=5 CONCURRENT_DURATION=60s ./run.sh
74+
75+
# Skip concurrent test for faster runs
76+
SKIP_CONCURRENT=true ITERATIONS=3 ./run.sh
77+
78+
# Run specific variants only
79+
VARIANTS="baseline:main:headless: approach2:headless-cdp-live-view:headless:-e=ENABLE_LIVE_VIEW=true=-p=8080:8080" ./run.sh
80+
```
81+
82+
## Requirements
83+
84+
- Docker with BuildKit
85+
- Go 1.25+
86+
- python3 (for summary table generation; optional)
87+
88+
## Output
89+
90+
Results go to `results/<timestamp>/`:
91+
92+
```
93+
results/20260310-143000/
94+
├── SUMMARY.md # Combined comparison table
95+
├── baseline/
96+
│ ├── results.md # Markdown latency table
97+
│ ├── results.json # Machine-readable results
98+
│ ├── docker-stats.csv # CPU/memory time series
99+
│ ├── memory.txt # Idle + post-workload memory snapshots
100+
│ ├── container.log # Full container logs
101+
│ └── bench.log # Benchmark stderr
102+
├── approach1/
103+
├── approach2/
104+
└── headful/
105+
```
106+
107+
## Running the Go benchmark standalone
108+
109+
```bash
110+
cd server
111+
go build -o ../benchmarks/liveview/bench ../benchmarks/liveview/main.go
112+
113+
# Against an already-running container
114+
../benchmarks/liveview/bench \
115+
-cdp-url ws://127.0.0.1:9222 \
116+
-iterations 3 \
117+
-label mytest \
118+
-concurrent-workers 3 \
119+
-concurrent-duration 30s
120+
121+
# JSON output
122+
../benchmarks/liveview/bench -cdp-url ws://127.0.0.1:9222 -iterations 3 -label mytest -json
123+
124+
# Quick run without concurrent test
125+
../benchmarks/liveview/bench -cdp-url ws://127.0.0.1:9222 -iterations 2 -skip-concurrent -label quick
126+
```

0 commit comments

Comments
 (0)