fix(n2): send the whole screenshot history, pruning only to fit the wire cap

[juanpin]

Why

The n2 loop windowed every request down to the two newest image-bearing messages and left [older image omitted] where each older frame had been.

That buys the model nothing. The API's n2 handler applies exactly that window itself before serving the model — keep_images_in_last_n_image_messages(messages, n=N2_IMAGE_MESSAGE_WINDOW), with N2_IMAGE_MESSAGE_WINDOW = 2 — and its comment calls it "the agreed n2 serving context policy (current + previous observation), not a provider constraint". api.md has told callers to "send the full conversation" for as long as the loop has been trimming it.

What the window did cost is the replay. The request log a run's replay is built from records exactly the messages the client sent, so a client that trims first is the only reason a macOS run's story stops after the last two frames while a playground run of the same length shows every step. This is the bug that started the investigation: [older image omitted] showing up in place of a run's history.

What changed

prune_n2_screenshots_to_budget replaces the window: send everything that fits under the 10 MB cap, drop oldest-first when it does not, never the newest. This is the reference harness's pruneScreenshotsToBudget (playground-core/src/server/computer-use-loop.ts), including:

• its headroom — 64 KB, not 500 KB. The larger allowance only threw away frames that would have fit.
• its silence — a dropped frame leaves no marker. That is what the server's own window does to the frames it strips; injecting a marker per dropped frame handed the model text the reference harness never produces. [older image omitted] stays the server's marker for its per-message image cap (cap_images_within_message), which is where it was trained.

retain_n2_image_window stays exported for a harness with its own reason to send less than it has, and as the executable statement of the server's window. fit_n2_request_images_to_budget is superseded and removed — it was neither exported from yutori.navigator nor documented in api.md.

prepare_n2_image_data_url now reads the media type off the data-URL header instead of base64-decoding the payload to compare a string. A request walks every frame each step now, and the common case is a pass-through compare.

Behaviour

Measured against synthetic 1920x1080 WebP q80 frames at ~121 KB each (a noisy worst case; real UI frames compress to roughly half that):

steps	before	dropped	kept	on the wire
20	2.50 MB	0	20	2.50 MB
50	6.26 MB	0	50	6.26 MB
100	12.51 MB	21	79	9.89 MB
200	25.03 MB	121	79	9.91 MB

Most runs never prune. A long one settles at the cap instead of losing its history.

The cost is upload. The playground pays this too, but it runs in-datacenter; a macOS run uploads from the user's own connection, so a run past ~50 steps sends several MB per step. That is what parity means here, and it is a deliberate trade — the alternative is the empty replay this PR is fixing.

Not a gap after all

I checked screenshotPolicy: 'on_demand' (playground #12806/#12810) and the loop already matches it: shell, browser-navigation and file calls return their text with no capture (the early returns in execute_n2_computer_call), and translate_n2_batch rejects shell/file/browser members outright, so a computer_batch is never all-non-visual. Existing tests already lock both halves in (test_current_file_tools_return_one_text_result_without_a_screenshot, and the bash-only turn in the harness test).

Test plan

• pytest -m "not slow" — 865 passed, 9 skipped
• ruff check . — clean
• New coverage: a history that fits keeps every frame; oldest-first drops leave no marker; the newest frame is protected; extra images sharing the newest message are still droppable; the raise when even one frame cannot fit.
• Rewritten: the loop-level test now asserts three frames for a three-click run, and the harness test asserts no marker anywhere in the request.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes n2 request payload size and replay fidelity (larger uploads on long runs) and replaces core message-prep logic; behavior is well-tested but affects every multi-step computer-use run.

Overview
Navigator n2 no longer windows outgoing requests to two image-bearing messages. N2ComputerAgent sends every screenshot from the run on each request (matching what the API replay log records); the server still applies its own two-message image window before the model sees the conversation.

Budget handling is replaced and tightened. fit_n2_request_images_to_budget is removed in favor of exported prune_n2_screenshots_to_budget, which mutates messages in place: drop oldest image parts first when serialized messages exceed the cap, never drop the newest observation, leave no [older image omitted] placeholder (dropped frames look like server-stripped history). Request headroom for non-message JSON shrinks from 500 KB to 64 KB so fewer frames are discarded unnecessarily.

Small performance/doc tweaks: prepare_n2_image_data_url checks the data-URL header for format before decoding; retain_n2_image_window stays public for harnesses; api.md and tests assert full-frame requests and the new prune semantics.

^{Reviewed by Cursor Bugbot for commit 305dbf8. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 305dbf8. Configure here.}

[cursor]

Newest screenshot can be dropped

Medium Severity

prune_n2_screenshots_to_budget puts the newest message's content list into image_contents[:-1] whenever that message holds more than one image. After the first pass drains the extras, the second pass walks those same lists again and _drop_first_image can remove the remaining current observation instead of raising ValueError.

Additional Locations (1)

• yutori/navigator/n2_payload.py#L208-L230

 

^{Reviewed by Cursor Bugbot for commit 305dbf8. Configure here.}