Skip to content

rerun the benchmark suite and put the cross-language state up front #619

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kacy merged 1 commit into from
Jul 29, 2026
Merged

rerun the benchmark suite and put the cross-language state up front #619

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 26 additions & 9 deletions bench/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,14 +39,15 @@ cargo build --release --manifest-path bench/event_ledger_rust/Cargo.toml
zig build-exe -O ReleaseFast -femit-bin=bench/event_ledger_zig bench/event_ledger.zig
```

Latest measured results on this machine, 200000 events, median of 5:
Latest measured results on this machine, 200000 events, median of 5
(2026-07-29):

| lang | gen | parse | analyze | sign | total |
|---|---:|---:|---:|---:|---:|
| pith | 322 | 199 | 56 | 0 | 575 |
| go | 127 | 354 | 16 | 0 | 490 |
| rust | 31 | 62 | 24 | 0 | 122 |
| zig | 19 | 104 | 9 | 0 | 136 |
| pith | 325 | 202 | 61 | 0 | 618 |
| go | 100 | 372 | 16 | 0 | 481 |
| rust | 29 | 61 | 26 | 0 | 117 |
| zig | 22 | 114 | 11 | 0 | 137 |

(ms; `gen` builds the stream, `parse` decodes it into structs, `analyze`
runs the map/set rollup, `sign` is the HMAC.)
Expand Down Expand Up @@ -97,7 +98,10 @@ bench/http_bench.sh ./bench/http_server_go 8081 120
```

It prints a per-10s RSS table and a summary line (requests, throughput, and
RSS start / end / peak). The Pith server holds flat RSS across the run.
RSS start / end / peak). The Pith server holds flat RSS across the run: on
the 2026-07-29 30-second run it served 8079 req/s with 4 kb of RSS growth,
against the Go server's 28453 req/s with 6.1 mb of growth. Go is ~3.5x
faster here; the flat line is the part this benchmark exists to watch.

## catalog service benchmark

Expand Down Expand Up @@ -189,7 +193,14 @@ candidate index lists for common region/active filters, which is closer to how
an actual in-memory service would avoid rescanning the full catalog on every
request.

latest measured results on this machine, using the median of 5 trials:
latest measured results on this machine, using the median of 5 trials
(2026-07-29, direct 200k-iteration runs interleaved):

| iterations | go total | pith total | ratio | rust total |
|---|---:|---:|---:|---:|
| `200000` | `~436 ms` | `~114 ms` | `0.26x` | `~80 ms` |

the earlier 1m-iteration medians, for the trend:

| iterations | go total | pith total | ratio | go batch | pith batch |
|---|---:|---:|---:|---:|---:|
Expand Down Expand Up @@ -318,6 +329,10 @@ green wake-path work described below):
| rust | 135 | 7.4 m | 2.4 mb |
| zig | 135 | 7.4 m | 2.7 mb |

the 2026-07-29 rerun (medians of 7, interleaved) held the same shape with
the usual comparator drift: green ~133 (bimodal, best runs at ~69), green
pinned to one worker ~46, go ~75, rust ~75, zig ~240.

read the rows that oversubscribe os threads (pith os-thread, rust, zig)
with the box in mind: eight threads on two cores, so they swing run to run.
across two suite runs a week apart rust moved 135 -> 94 ms and zig 135 -> 204
Expand Down Expand Up @@ -400,11 +415,13 @@ and takes it away from the thing being measured — enough of it, and the
oom killer starts taking builds out mid-run. go's default cache is on
disk, which is what you want.

latest measured results on this machine, using the median of 5 trials:
latest measured results on this machine, using the median of 5 trials
(totals rerun 2026-07-29; the phase breakdown below is from the 2026-07-21
run and its shape still holds):

| records | go total | rust total | pith total | pith/go | pith/rust |
|---|---:|---:|---:|---:|---:|
| `50000` | `324 ms` | `136 ms` | `482 ms` | `1.49x` | `3.54x` |
| `50000` | `~352 ms` | `~177 ms` | `~576 ms` | `1.63x` | `3.25x` |

phase breakdown from the same run:

Expand Down
121 changes: 82 additions & 39 deletions docs/performance.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,35 +9,70 @@ the helpers in `bench/` before trusting them on different hardware.

## where pith stands

the short version, all on the same 2-core machine. the concurrency rows are the
green backend, which is the default on linux; the rows marked `PITH_GREEN=0`
are the os-thread opt-out, kept for contrast:
the short version, all on the same 2-core machine, fully rerun 2026-07-29
after the july hardening pass (details in the summary below). the concurrency
rows are the green backend, which is the default on linux; rows marked
`PITH_GREEN=0` are the os-thread opt-out, kept for contrast:

| coordination | pith | go | rust | zig |
|---|---:|---:|---:|---:|
| chan_fanout, 1m msgs | **171 ms** | 69 ms | 94-135 ms | 135-204 ms |
| chan_fanout, pinned to 1 worker | **~46 ms** | 69 ms | — | — |
| chan_fanout, `PITH_GREEN=0` | ~438 ms | 69 ms | — | — |
| context switches over the run | 2.5k | 258 | 4.8k | 60k |
| 20k spawn + join | **~50 ms** | ~27 ms | — | — |
| 20k spawn, peak rss | 10 mb | 11 mb | — | — |
| 20k spawn, `PITH_GREEN=0` | ~1450 ms / 174 mb | — | — | — |

| compute | pith | go | rust | zig |
|---|---:|---:|---:|---:|
| event_ledger, 200k events | 955 ms (1.16x go) | 820 ms | 178 ms | 199 ms |
| std_pipeline, 50k records | 476 ms (1.54x go) | 310 ms | 135 ms | — |
| grpc unary echo, conc=8 | 8075 calls/s (1 worker) | 13417 | 11012 | — |

on coordination the green backend beats rust and zig outright, sits ~2.3x
behind go at the default worker count, and beats go pinned to one worker. on
spawn it is ~2x go at go's memory, down from ~29x before the coroutine stack
pool. compute is the older story: competitive with go, well behind rust and
zig, and what is left of that gap is string building rather than the runtime.
| chan_fanout, 1m msgs | **~133 ms** (bimodal, best 69) | ~75 ms | ~75 ms | ~240 ms |
| chan_fanout, pinned to 1 worker | **~46 ms** | ~75 ms | — | — |
| 20k spawn + join (batches of 64) | **~56 ms / 3.2 mb** | ~8 ms / 3.8 mb | — | — |
| 20k spawn, `PITH_GREEN=0` | ~1017 ms / 3.5 mb | — | — | — |

the rows that oversubscribe os threads swing run to run on a 2-core box (see
the caveat on the fan-out table below); the green and go rows are the stable
ones.
| services and compute | pith | go | rust | zig |
|---|---:|---:|---:|---:|
| catalog workload, 200k requests | **~114 ms** | ~436 ms | ~80 ms | — |
| grpc unary echo, conc=8, 16 B | ~8000 calls/s | 14409 | 11604 | — |
| http server under wrk, 30 s | 8079 req/s, rss flat (+4 kb) | 28453 req/s (+6.1 mb) | — | — |
| event_ledger, 200k events | 618 ms (1.28x go) | 481 ms | 117 ms | 137 ms |
| std_pipeline, 50k records | 576 ms (1.63x go) | 352 ms | 177 ms | — |

reading it honestly: on channel coordination green matches rust and beats zig,
runs ~1.8x behind go at the default worker count (placement is still decided
by first-resume luck, hence the bimodal spread), and beats go pinned to one
worker. raw spawn/await is where go's scheduler still clearly wins; what the
green backend buys over pith's own os-thread backend there is ~18x the speed
at flat memory. the service-shaped rows are the strongest: the catalog
workload — lookups, filtered scans, json batch scoring — runs ~4x faster than
go's version, because a flat struct decodes in one pass with no reflection,
and the http server holds byte-flat rss across a sustained load where the go
server grows. the compute rows keep their long-standing shape: modestly behind
go, well behind rust and zig, with string building the remaining gap.

## july 2026 hardening, in numbers

between 2026-07-26 and 2026-07-29 the green backend became the linux default,
every blocking call got a yield point, and an ownership sweep fixed seven
leak/use-after-free defects in the emitter and runtime. what that changed,
each measured before and after on this machine:

| what | before | after |
|---|---:|---:|
| a co-tenant cpu task while another task waits on dns | blocked until the lookup ended | decoupled (~78 ms) |
| the same, while another task appends to a log file | 257-273 ms | 122-124 ms |
| the same, while another task waits on a child process | 1013-1018 ms | 8-9 ms |
| one bare tcp connect to a tls server | server dead, health checks green | survives; >512 junk handshakes hold no slots |
| json parse per request, 20k docs on one task | 89 mb, degrading | 10 mb flat, ~30% faster |
| map/list eviction churn, 800k rounds | 38 mb | 10 mb flat |
| `container[expr()]` index keys, 800k | 26-75 mb | flat |
| `xs.map(f)` result lists, 800k rounds | 269 mb | flat |
| fn values named in a loop, 800k | 309 mb | flat |
| struct values stored in containers, 800k | 318 mb | flat |

three of those "leaks" were masking live use-after-free bugs (a lambda stored
in a `List[fn]` segfaulted the moment its leak was fixed; sitegen's tag pages
came out named after the output path when struct stores stopped over-counting),
which is the argument for fixing leaks even when the memory alone would be
tolerable. the sweep is pinned by `make leak-check`, a growth gate that runs
six churn shapes at two round counts in ci and fails on ~2 mb of drift — it
was proven by reverting each fix and watching it go red.

the cost side, stated plainly: single-task tight-loop file reads pay ~2x for
the blocking-pool handoff that keeps file i/o from stalling a green worker,
and event_ledger/std_pipeline drifted a few percent slower as hot paths gained
the releases they had been skipping. flat memory was bought at list price.

all numbers from one 2-core machine, medians of 5 where quick enough to
repeat; the tables below were fully rerun 2026-07-15 (after the arc
Expand All @@ -56,9 +91,9 @@ in a single pass, filled straight into the struct.
`bench/chan_fanout`, the coordination benchmark, was the one pith lost
outright — ~580ms against go's ~69 for a million messages between eight
tasks. the green wake-path work on 2026-07-26 (details below) brought
the green backend to 171ms, ahead of rust and zig and 2.3x go, with
~46ms — faster than go on this box — when the pipeline is pinned to one
worker. the batch benchmarks measure compute and pith is competitive
the green backend to ~133ms on the 2026-07-29 rerun — level with rust,
ahead of zig, ~1.8x go — with ~46ms, faster than go on this box, when the
pipeline is pinned to one worker. the batch benchmarks measure compute and pith is competitive
there; coordination is now a genuine strength of the green backend
rather than the standing embarrassment it was.

Expand Down Expand Up @@ -179,9 +214,11 @@ coroutine stack pool, the channel condvar fix, and the slab-free wake), so
they understate green as it stands. re-running the same shape on 2026-07-27
at a smaller batch put pith at 5067 calls/sec os-thread, 8075 at one green
worker (+59%), and 7083 at two (+40%) — the same relationships the table
below records (+53% and +41%), so its conclusions hold. the table is left at
its original batch size rather than replaced with a smaller, non-comparable
run; re-measuring it at the documented batch is a tracked follow-up.
below records (+53% and +41%), so its conclusions hold. the 2026-07-29
rerun of the cross-language shape (16-byte payload, conc=8, prebuilt
client) put pith at ~7500-8900 calls/sec against go's 14409 and rust's
11604 on the same server. the table is left at its original batch size
rather than replaced with a smaller, non-comparable run.

| 16 B, conc=8 | calls/sec | ctx-switches/call | cpu |
|---|---|---|---|
Expand Down Expand Up @@ -297,27 +334,33 @@ interleaved with a go canary on the 2-core box (2026-07-26):

| 20k spawn + join | elapsed | peak rss |
|---|---:|---:|
| os threads | ~1450 ms | 174 mb |
| os threads (2026-07-26) | ~1450 ms | 174 mb |
| os threads (2026-07-29) | ~1017 ms | 3.5 mb |
| green, before the pool | ~580 ms | 10 mb |
| green, after the pool | ~50 ms | 10 mb |
| go | ~27 ms | 11 mb |
| green (2026-07-29) | ~56 ms | 3.2 mb |
| go (batch twin, 2026-07-29) | ~8 ms | 3.8 mb |

page faults over the run fell 21832 -> ~2500 and context switches 25748 ->
~3000. shrinking the stack from 1 MiB to 64 KiB changed nothing before the
pool, which is the tell: the cost was the *number* of mappings, not their
size. green is now within ~2x of go on this shape at the same memory, from
~29x, and roughly 30x faster than the os-thread backend, which pays a real
thread per task.
size. the 2026-07-29 rerun, against a minimal go twin of the same batch shape,
puts go clearly ahead on raw spawn/await (~7x) — go's scheduler reuses
goroutine stacks with no fd or slab work at all — while green holds an
~18x lead over pith's own os-thread backend at flat memory. the os-thread
rss collapse from 174 mb to 3.5 mb came free with the july ownership
sweep: the task table's records now release their payloads on eviction
like every other container.

`bench/chan_fanout` — the same fan-out shape with cross-language
comparators: four producer tasks push one million messages through a
bounded channel (capacity 256) and four consumer tasks drain them,
folding each into an order-independent sum. all four languages print the
same checksum. medians of 9 on this 2-core box, 2026-07-26:

| | pith | pith green | go | rust | zig |
| | pith `PITH_GREEN=0` | pith green | go | rust | zig |
|---|---:|---:|---:|---:|---:|
| total | 438ms | 171ms | **75ms** | 135ms | 135ms |
| total (2026-07-26) | 438ms | 171ms | **75ms** | 135ms | 135ms |
| total (2026-07-29) | — | ~133ms | ~75ms | ~75ms | ~240ms |
| peak rss | 3.0 mb | 3.0 mb | 2.0 mb | 2.4 mb | 2.7 mb |

read the rows that oversubscribe os threads (pith os-thread, rust, zig)
Expand Down
Loading