Measure what a served request costs us, with no model behind it

[v-positronic]

A probe that stands the real PolicyServer up on loopback behind a model that answers a fixed chunk instantly, replays a recorded episode through the rig-side stack and the real client, and divides every round trip into the phases the server reports.

What

positronic/offboard/serving_cost.py, run as uv run --locked python -m positronic.offboard.serving_cost --dataset.path=<episode root>. It builds the rig-side half a DROID endpoint declares — StopOnFault | TemporalStack(25 frames of two cameras, the pose and the grip) | ChunkedSchedule | RestrictImageSize(1024x288) — runs a recorded episode through it, keeps the observations that stack would have put on the wire, and sends each one to a server serving the same pipeline over remote(compress_images=True) and an InstantChunk model. Per request it reports the payload size, the client's JPEG encode, its pack, the round trip, the server's own served/decode/queued/infer, and what is left outside the server's span.

The stack depth, sampling rate, image bound, chunk rows and compression are all flags, so the same probe answers what a different payload would cost.

Why

Nothing measured what our serving path costs, so a slow round trip against a hosted model could not be divided between the model and us. Reading a real episode rather than synthesising frames matters: JPEG size and decode time both follow image content, and a stack of noise is several times the size of a real one.

The one contract change is positronic/policy/remote.py: _prepare_obs becomes prepare_obs, because the probe times that encode separately from the round trip that follows it. Its test already imported the private name.

Verification

On one DROID episode of the 27 August round, 20 requests of 846 KiB each, medians:

	ms
client JPEG-encodes the 50 frames	21.0
client packs the message	0.2
round trip	57.5
— server decode (msgpack plus 50 JPEG decodes)	27.4
— server inference (the instant chunk)	0.3
— outside the server's span (socket both ways, and the receive before the span opens)	29.3

Every figure is proportional to the payload: at 12 frames (410 KiB) the round trip is 42 ms, at 5 frames (171 KiB) it is 18 ms.

That measurement also found uvicorn's default websocket implementation reassembling a received message twice as slowly as its sans-io one — 58 ms against 29 ms for 846 KiB, against a 28.6 ms floor for a bare websockets echo of the same bytes. So PolicyServer names websockets-sansio, which took the round trip above from 86 ms to 57 ms. The constant lives in server.py and the probe and the test fixture read it, so all three serve on the same implementation.

pytest positronic pimm is green (1493 passed), and the two new tests pin the phase split and the shape of a captured payload.

Base

Branched off client-latency-spans -> server-timing (#725), whose protocol.TIMING_* fields the probe reads. It needs retargeting once #722 and #725 merge; until then it draws no Codex review and no CI, as they only run against the default branch.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 607d5366c9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Include the advertised encode phase in the report

With the current PolicyServer, session.served_timing never contains TIMING_ENCODE: server.py calls timing.report() while the encode phase is still open, before its context manager records that phase. Consequently **served silently omits the advertised encode_ms column from every benchmark result; arrange for the completed encode duration to reach the response or stop claiming that this probe divides out encoding.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Match the benchmark defaults to the declared DROID stack

The repository's DROID pipeline does not declare these defaults: positronic/vendors/dreamzero/codecs.py:248-253 configures a 23-frame lookback at stride 8, yielding four samples, and positronic/vendors/dreamzero/server.py:373-387 bounds the DROID frames to 320x180. Running this command unchanged instead sends 25 contiguous 1024x288 frames per camera, so its payload and latency figures do not represent the DROID serving path it says it measures; derive the defaults from that pipeline or describe this as a synthetic load.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Convert the output path at the CLI boundary

Rule primitive-type violated:
main carries the CLI's filesystem destination as str | None until the final write. Keep the configuronic-facing input as a string if the framework requires it, but convert it once at entry to a Path | None local and use that typed value for writing and display.

AGENTS.md reference: AGENTS.md:L7-L8

Useful? React with 👍 / 👎.