Transactor throughput after the write channel: loop cost, grouping, and scale-out

[tvanhens]

Context

#641 raised transactor throughput and made the operation path cost one transactor request per Worker batch instead of two per operation. #642 makes a WebSocket write channel the write path, which removes the per-object request ceiling. This issue collects what is left once frames are the write path: the commit loop itself becomes the limit, at about 1.1 ms of wall time per raw write and about 1.9 ms per operation on Cloudflare, and per-item cost is linear so batch size does not amortize it.

The numbers below come from bun run bench:cf (Cloudflare, 256 in flight) and bun run bench:op (local, profiled with bun --cpu-prof). Stage-to-stage variance on Cloudflare is up to 2x, so treat them as ratios within a run.

Directions, in order of expected value

1. Cut the loop's per-operation cost

Locally an operation costs 3.7x a raw write inside the loop; on Cloudflare about 1.7x. The remaining cost after #641, from the local profile:

• The per-operation receipt claim is a durable write before the body runs. It cannot be batched without reordering admission relative to earlier bodies in the same batch, because admission for a later operation sees the staged effects of earlier ones. Options that keep the ordering: a compact claim row instead of full receipt JSON, and completing receipts in one statement per batch keyed by batch id.
• Admission compiles a policy predicate and resolves the principal per operation. The principal lookup depends on db state and changes per transaction, but the compiled predicate for a unit and principal could be reused within a batch when the batch's datoms touch nothing the predicate observed, the same dirty check the replay fence now uses.
• Receipt preparation canonicalizes the invocation input and hashes it. The scope digest is cached; the invocation digest is per operation by definition.
• Transaction expansion and unique probes are shared with the raw path and are the floor.

Halving the operation cost is the only way past roughly 1,000 ops/s per database once frames are the write path.

2. Multi-operation invocations

One invocation carrying several operations under one receipt and one admission pass amortizes everything in item 1 at the API level. It introduces group semantics (all-or-nothing across the group, one replay fence), so it is a product decision, but it is the cheapest lever for clients that naturally write in groups.

3. Fewer storage crossings per batch

Log rows already commit in multi-row inserts (32 rows, under the Durable Object's 100 bound parameters). Receipt completions still run one UPDATE per operation inside the commit transaction. Packing completions per batch, or storing terminal receipts in a per-batch row, reduces the per-commit statement count to a constant.

4. Scale out by database

The transactor is one Durable Object per database and the overload queue is per object. Splitting tenants or workloads across databases scales linearly today and is the only option that raises the ceiling rather than approaching it more slowly. Worth documenting as the deployment model for write-heavy tenants.

5. Replication write path

Outbox replication reaches the transactor through the replica object one request per transaction. Once #642 lands it should ride the same socket, and until then it would benefit from the batch route the /op path now uses.

Measured and not worth pursuing

• Moving indexing off the transactor. Suppressing index runs entirely changed raw throughput by a few percent, and an external indexer would leave the writer's node cache cold after every run, putting R2 reads on the unique-probe path.
• Larger batches or a bigger RAMOSE_BATCH_BUDGET_MS. Per-item cost is linear. The budget is also measured with performance.now(), which Workers freeze during synchronous work, so on Cloudflare it only advances across awaits.
• Concurrency inside the loop. Admission and bodies are CPU work in one isolate with synchronous SQLite reads; there is no I/O to overlap.
• Moving authorization in front of the transactor on its own. The commit-time check must remain authoritative, so this only pays together with the dirty-check fence, which Raise transactor throughput and add a Cloudflare write benchmark #641 already added inside the loop.

Tooling

• bench:cf phases: transact, operation, info (request ceiling), socket-ping and socket-write (frame ceiling and frame-driven loop ceiling).
• bench:op for local ranking of loop costs by CPU profile, since Workers do not expose CPU timing.
• Compare phases within one Cloudflare run, and run paired comparisons in both deployment orders before drawing conclusions from absolute numbers.