[task] nothing measures zstd against lz4 as the ClickHouse sink's compression

[MarcusKainth]

What kind

None of the form's kinds apply cleanly. This is a benchmark scenario, filed on the feature form because blank issues are disabled.

Which components

ClickHouse sink (spate-clickhouse)

What you are trying to do, and what stops you

Measure what compression: zstd costs the ClickHouse sink compared with the default compression: lz4, so the choice of default rests on a number. Nothing measures this today.

ClickHouse 26.9 changed its own default codec from LZ4 to ZSTD(3) for table data and network traffic (ClickHouse/ClickHouse#108786). That pull request says ZSTD(3) "gives substantially better compression ratios at a still-modest CPU cost". Its CI performance comparison does not test the claim: the harness pins both servers to LZ4. #458 keeps lz4 as the sink's default. This task decides whether it should follow the server to zstd.

The cost shows up in two places. The sink compresses every insert body on the pipeline's I/O runtime, which has pipeline.io_threads workers (default 2). ClickHouse decompresses the body on arrival. Both are measured.

Two questions:

1. Does throughput under the benchmark's 32-CPU envelope change when the sink compresses with zstd?
2. How much sink CPU per row and ClickHouse CPU per row does zstd add or save?

Rig. The kafka_avro_clickhouse bench in spate-etl/benchmark on c8gd-metal-24xl-ec2-docker, using the spate arm's published native variant. Only compression changes. results/c8gd-metal-24xl-ec2-docker/spate/2026-08.jsonl has the captured lz4 figures for that variant. Its rowbinary runs are infra_bound, so they cannot show a sink-side cost, and native goes first.

Arms.

Arm	compression
A	lz4
B	zstd (level 3)
C	zstd:1, optional

Baseline. The August capture ran spate 0.2.0 against ClickHouse 26.3.22.7. A zstd run on today's spate commit or a newer server needs a fresh arm A in the same session. The capture then shows whether the lz4 figures have drifted since August. Interleave the arm order, and run at least three replicates per arm, matching the capture.

To build. The spate arm cannot set compression today. entrants/spate/README.md lists it among the framework values that no descriptor reaches. The arm needs a compression knob, recorded in the result's variant so every JSONL row names its codec.

Recorded per arm. rows_per_s, cpu_us_per_row, data_plane_cores_used, throttled_us, ch_cpu_us_per_row and ch_inserted_bytes.

Confounder. The rig runs everything on one host, so the sink reaches ClickHouse over a local bridge with bandwidth to spare. A smaller body saves nothing there. A deployment where the sink and ClickHouse sit on separate hosts would gain from it, and this rig cannot show that gain.

Semver impact, as best you can tell

Additive — nothing existing changes

Does this touch any of these?

• INV-1 — delivery semantics, or anything that could commit a watermark past unacknowledged data
• INV-2 — the source poll loop, which must never block on a channel send
• INV-3 — the checkpoint tracker, which is synchronous and free of async runtimes
• INV-5 — the sink intake path, which must not await outside its select!
• INV-6 — adding a connector or 0.x dependency type to a public signature in spate-core
• INV-8, INV-9 or INV-10 — metric naming, cardinality, or gauge ownership

None. The bench observes and changes no crate's behavior.

Would you want to implement it?

Yes — maintainer task, on the benchmark rig.