Skip to content

feat(macos): show the shell rail and the model conversation for computer-use runs - #370

Merged
juanpin merged 3 commits into
mainfrom
juan/macos-background-transcript
Sep 3, 2026
Merged

feat(macos): show the shell rail and the model conversation for computer-use runs#370
juanpin merged 3 commits into
mainfrom
juan/macos-background-transcript

Conversation

@juanpin

@juanpin juanpin commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Why

The conversation with the model was only ever readable where it happened to be painted. A foreground run showed reasoning in the cursor capsule for as long as a step lasted and nothing after; a background (window-scope) run drives one app window and paints nothing on the desktop, so everything it did was compressed into a single menu bar caption line — no shell commands, no reasoning, no way to read a step that had already scrolled past.

What runs show now

  • The activity window, in both modes (menu: Show activity). A floating, non-activating, resizable panel carrying the conversation with the model — the task, its thinking, every action, and every shell command — in the playground chat's vocabulary, transcribed from Tailwind to plain CSS on a new page (navigator-activity.{html,css,js}). Rows are addressed by id, so a shell command is revised in place as it moves from running to its exit code. The page is built up front and hidden, so the transcript starts at step one no matter when it is opened; rows arriving while it loads are buffered. A window-scope run puts the driven window's live frame above the conversation; a foreground run shows no frame pane (the operator is already looking at what the model sees) and the window is hidden with the overlay for every capture, so the model never sees it.
  • The shell rail for background runs. The same click-through phosphor "run command" panels a foreground run paints under the menu bar, in their own borderless non-activating panel. It ignores mouse events, and window-scope capture sees only the target window, so neither the operator's work nor the model's view is disturbed. A command running on this Mac no longer requires opening a window to notice.

The status menu is unchanged apart from the new Show activity item: one caption line about the latest step, above the frame thumbnail and Stop. The window keeps the history now, so the menu does not have to.

The transcript opens with the task itself: run() and resume() present a task event, so a presentation shows the conversation from its first message. Background runs also record shell telemetry, which only the overlay path did.

Both status-mode pages are extras next to a menu bar item that still carries Stop, so one that fails to load is dropped rather than treated as fatal (the overlay page stays fatal — it is the run's only surface).

Verification

  • pytest -m "not slow": 862 passed. New coverage for the transcript stream in both modes, in-place shell rows, the status-mode rail and its counts, entry clamping, and the activity page's integrity check (it is hashed and verified like every other packaged asset).
  • Drove the compiled host with realistic payloads in both modes and screenshotted the result: the activity window, the menu, and — with captureHide in flight — the desktop with the window correctly absent from what the model would capture.
  • Two live background runs against the model on this build: Calculator (12x12 plus a shell command), and a browsing task on yutori.com that read the team page and wrote a text file. Both kept the presentation available with no degradation; the second streamed 240 preview frames into the activity window while it ran.

Downstream

yutori-mcp needs a pin bump once this releases, plus wording updates in computer_use/result.py ("Menu bar status", "streamed to the menu bar item"). Users pick the new assets up by re-running computer-use setup, which recompiles the host.

🤖 Generated with Claude Code


Note

Medium Risk
Touches macOS overlay host, capture hide/show, and presentation paths used during live agent runs; failures are mostly fail-soft for auxiliary UI, but incorrect capture hiding could leak UI into model screenshots.

Overview
macOS computer-use presentation now surfaces the full model conversation (task, reasoning, actions, shell, errors, final answer) in a new activity window (Show activity), shared by foreground overlay runs and window-scope status mode—with a live driven-window frame above the transcript in background runs only.

Window-scope (status) runs gain the same click-through shell rail under the menu bar as foreground runs (dedicated borderless panel), so shell work is visible without opening the menu. The old floating “live view” is replaced by this activity UI; preview streaming keys off activityOpen instead of liveView.

Python/Swift wiring: MacOSPresentationController maps events to transcript host commands (_transcript_entry, shell rows updated in place by id); N2ComputerAgent.run/resume emit an initial task event. New packaged assets navigator-activity.{html,css,js} are built, integrity-checked, and passed via activityHtml config. Non-fatal load failures for rail/activity pages in status mode degrade gracefully; overlay page failure remains fatal.

Docs and tests update api.md and add coverage for transcript streaming, status-mode rail counts, activity page tampering, and preview demand.

Reviewed by Cursor Bugbot for commit cc9a54f. Bugbot is set up for automated code reviews on this repo. Configure here.

…round runs

A window-scope run drives one app window and paints nothing on the desktop, so
everything it did was compressed into a single menu bar caption line: no shell
commands, no reasoning, no way to read a step that had already scrolled past.
Foreground runs show all of it (capsule, cursor, shell rail); background runs
are exactly the ones the operator is not watching, and had the least.

Status mode now carries three surfaces:

- The same click-through shell rail under the menu bar, in its own borderless
  non-activating panel. It ignores mouse events, and window-scope capture sees
  only the target window, so neither the operator's work nor the model's view
  is disturbed -- a command running on this Mac no longer requires opening a
  window to notice.
- The activity window (menu: Show activity): the live frame of the driven
  window above the conversation with the model -- the task, its thinking, every
  action, and every shell command -- in the playground chat's vocabulary,
  transcribed from Tailwind to plain CSS on a new page the host loads in a
  floating, non-activating, resizable panel. Rows are addressed by id, so a
  shell command is revised in place as it moves from running to its exit code.
  The page is built up front and hidden, so the transcript starts at step one
  no matter when it is opened; rows that arrive while it loads are buffered.
- The menu keeps the last four steps rather than only the newest.

The transcript opens with the task itself: `run()` and `resume()` now present a
`task` event, so a presentation shows the conversation from its first message.
Background runs also record shell telemetry, which only the overlay path did.

Both status-mode pages are extras next to a menu bar item that still carries
Stop, so one that fails to load is dropped rather than taken as fatal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comment thread yutori/navigator/macos/assets/macos-overlay-host.swift
Which surfaces exist should not depend on how a run drives the Mac. The desktop
overlay says what is happening right now; the activity window is where the
operator reads back what happened -- the task, the model's thinking, every
action and shell command -- and a foreground run has exactly as much of that to
show as a background one.

The host now builds the window in both modes and puts Show activity in the
overlay run's menu bar menu, and the controller feeds the transcript from the
one `present()` path both modes share. Two consequences follow from foreground
capture being the whole desktop: the window is hidden with the overlay for
every capture, so the model never sees it or clicks into it, and it shows no
frame pane, because the operator is already looking at what the model sees --
the pane appears only when a run streams frames of its own.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@juanpin juanpin changed the title feat(macos): show the shell rail and the model conversation for background runs feat(macos): show the shell rail and the model conversation for computer-use runs Sep 3, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 61f3cfb. Configure here.

Comment thread yutori/navigator/macos/assets/macos-overlay-host.swift
The menu grew to the last four steps back when it was the only place a
background run's history existed. The activity window keeps that history now,
and better, so the menu goes back to what it is good at: one glanceable line
about the latest step, above the frame and Stop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@juanpin
juanpin merged commit 106f16a into main Sep 3, 2026
16 checks passed
@juanpin
juanpin deleted the juan/macos-background-transcript branch September 3, 2026 23:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant