Skip to content

feat(chat): chat pane as the default bot surface - #20

Open
mrinaldi2 wants to merge 30 commits into
ahilles107:mainfrom
mrinaldi2:codex/chat-pane
Open

mrinaldi2 wants to merge 30 commits into
ahilles107:mainfrom
mrinaldi2:codex/chat-pane

Conversation

@mrinaldi2

Copy link
Copy Markdown
Contributor

Stacked on #18 and #19. This branch builds on codex/peer-bots, so the diff against main also shows those 14 commits. This PR's own changes are the last 10 commits, from c402609 to 358afee. Merge #18 and #19 first.

Refs #13. This does not close #13 yet: the issue asks for a live Claude Code check of Allow/Deny from the card, and that has not been run (see Limitations).

What

Makes a chat the default surface when a bot is opened. The Claude Code terminal stays a tab away. Design spec: docs/superpowers/specs/2026-09-16-chat-pane-design.md.

Phase 1, daemon chat model. The daemon parses each bot's runtime transcript into turns:

  • Turn contents: what woke the bot, its text, tool steps with diff counts and images, and Gravity's own bus calls shown as sent messages, results and decisions.
  • Triggers: the owner from the composer or the terminal, another bot, a routine, a ruling, or a background job.
  • Live updates: a per-bot reader extends a chat incrementally, and a watcher pushes chat_turns updates.
  • Requests: new read-grant list_chat, get_chat_step, get_chat_image, list_artifacts and read_file, under a chat capability. read_file only reaches the bot's own directory and its project's artifacts.

Phase 2, desktop chat.

  • Chat tab: the default tab, showing turns, folded step groups that open to their command, diff and output, image thumbnails with a viewer, and markdown with highlighted code. Bot output keeps the Control Center's link and image policy.
  • Composer: sends through send_user_message and shows each message's delivery state.
  • Files tab: a new side tab with artifact previews.
  • Shortcuts: ⌘1–3 switch tabs and ⌘L focuses the composer.

Phase 3, permission cards.

  • Hook: a blocking Claude Code PermissionRequest hook. This is not PreToolUse, which fires for every call the bot is already allowed to make.
  • Cards: Allow once, Allow for this session (session-scoped rules only, never written to settings), or Deny with a reason.
  • Fail closed: an unanswered card is denied when its window closes.
  • Terminal fallback: the daemon only holds a prompt while an app announcing features: ["permission_cards"] is connected with the control grant. Otherwise the terminal prompt appears as before.
  • Requests: list_permissions and answer_permission, plus the permission_request and permission_resolved pushes.

Phase 4, Codex parity.

  • Observations: codex-observations.jsonl records commands, file changes with diffs, MCP calls and turn ends in Claude Code's transcript shape.
  • Approvals: App Server approvals become the same cards. Whichever answers first, the card or the native CLI, decides, and the other is withdrawn.

Phase 5, linked bots. A linked bot's real chat, steps, images and files are read across the peer link, and its turn updates are streamed back (peer_chat capability).

Phase 6, beyond v1.

  • Decisions: answered from the chat in one step, as answer-and-publish.
  • Attachments: uploaded in chunks through write_artifact into artifacts/uploads/.
  • Slash commands: typed into the terminal.
  • Search: find-in-chat with highlighting and next/previous.

Also:

  • Tasks tab: shows Now, Waiting on others, Upcoming and Done (list_tasks, get_task).

Fixes found along the way:

  • Transcript folders: transcript_dir kept _, but Claude Code files paths as -, so bots under such paths had no activity preview.
  • Routine completion (two bugs): the Stop hook did not forward its payload on Unix, and the run-id scan missed Claude Code's peer-message wrapper. Together they meant routine runs never completed from the transcript.
  • Routine listener: the scheduler's routine-completion listener ended for good on the first non-BotDone internal event, or on a lag.
  • Paging field: the desktop's list_messages sent before_id to a daemon that reads before_num.
  • Double start: included from the feat(runtime): choose Codex CLI or Claude Code per bot #18 follow-up.

Why

Issue #13. The terminal is the truth, but it is a poor way to read what a bot did: images are invisible, diffs scroll past, and files are paths to go and open. A permission prompt is a keystroke typed into a terminal.

How tested

  • Daemon: new tests in tests/chat.rs, tests/permissions.rs, tests/codex_runtime.rs, tests/peer_chat.rs and tests/tasks_view.rs, plus unit tests for the transcript builder, envelopes, step titles, Codex diff hunks and permission rules.
    • The permission tests run a real /hook/permission call through allow-once, allow-session, deny, expiry, abandonment, and "no app to answer".
    • A runtime-raised permission is answered end to end through the session.
  • Transcript parser: smoke-tested on 21 real bot transcripts (about 3,300 steps, all matched to their results).
  • Desktop: pnpm check passes with 650 tests, and the build passes.
  • Daemon checks: cargo fmt, cargo clippy -D warnings, the full workspace tests and the file-length check pass.
  • Live: installed and used on macOS against real bots: chat, Tasks and Files.

Rollout

  • Daemon and app together: ship them together. An older app hides the chat; an older daemon makes the app fall back to the terminal-first view.
  • Hook settings: bots' .claude/settings.json gains a PermissionRequest hook on their next start, on both macOS and Windows.
  • New config: permission_timeout_seconds, default 600.
  • Peers: a linked bot's full chat needs both daemons on this branch.

Limitations

  • Live permission check: not yet run against a real Claude Code session. Chat pane as the default bot surface #13 requires it before closing.
  • Visual baselines: the new Ladle stories need baselines generated by CI and adopted.
  • Linked bots: their permission prompts are answered on the machine they run on.
  • Search: covers loaded turns only.
  • gravity_os: moving the iPhone app off Gravity Lens is a follow-up in that repository.

🤖 Generated with Claude Code

mrinaldi2 and others added 29 commits October 1, 2026 09:34
Bot creation and the supervision tick could both enter start_bot for a
new bot before either registered its session. With Codex, the second
runtime resumed the thread the first had just created and failed with
"thread already has an active writer"; the failure path then cleared the
healthy first session, crashing the bot until the next retry.

start_bot now claims the bot under the supervisor lock before launching
anything, and releases the claim when it returns, so a concurrent caller
backs off instead of starting a second runtime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pair two of the owner's daemons over Tailscale and stand a bot from one
in as a linked bot in a project on the other. A linked bot is an ordinary
bot row with peer_id set, so addressing, task budgets and history work
unchanged; the delivery worker forwards deliveries for it over the /peer
link instead of writing to a local session.

The receiving daemon stores each forwarded message as if the sender were
local: a task is mirrored with the sender's deadline and hop count, a
result or cancel closes the mirrored task, and the sender gets a linked
bot of its own on first contact so replies route back. Files listed in a
result travel with it (from the bot's own directory or the project
artifacts only, size-capped) and land in the recipient project's
artifacts directory.

Pairing is a one-time invite holding a link token bound to the first
daemon that presents it. Peers can only list bots, link one, and deliver
to bots linked to them; revoking drops the link. `gravityd peer ...`
drives pairing and linking until the desktop UI lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs ahilles107/gravity#13.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 1 of the chat pane (ahilles107/gravity#13). The daemon now reads a
bot's runtime transcript and serves it as turns: what woke the bot (the
owner from the composer or the terminal, another bot, a routine, a ruling,
a background job), its text, its tool steps with diff counts and images,
and Gravity's own bus calls as sent messages, results and decisions.

A per-bot reader folds the transcript in incrementally and a watcher pushes
chat_turns for loaded chats as they grow. Step detail, images and files are
fetched on demand; read_file only reaches the bot's own directory and its
project's artifacts. New read-grant requests list_chat, get_chat_step,
get_chat_image, list_artifacts and read_file, behind a `chat` capability.

Also fixes two bugs the research turned up:
- transcript_dir kept `_`, while Claude Code files every non-alphanumeric
  character as `-`, so bots under such paths had no activity preview.
- Routine runs never completed from the transcript on Unix: the Stop hook
  did not forward Claude Code's payload (so no transcript_path), and the
  run-id scan did not expect Claude Code's peer-message wrapper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 2 of the chat pane (ahilles107/gravity#13). Opening a bot now shows
its conversation as turns read from the daemon's chat model: who woke it,
its answer as markdown with highlighted code, tool steps folded into
groups that expand to their command, diff and output, step images with a
viewer, sent messages, results with their files, and decisions raised.
The composer sends through send_user_message and shows a message's
delivery state until the bot reads it. The chat follows chat_turns
pushes and pages back through older turns.

The Terminal and Routines tabs stay (⌘1-3, ⌘L focuses the composer),
the side panel gains a Files tab listing the project's artifacts with
previews, and a linked bot from a peer shows its bus conversation. A
daemon without the chat capability keeps the terminal-first view.

Bot markdown keeps the Control Center's policy: links and remote images
are inert; images render only from bytes the daemon served. Also fixes
list_messages paging, which sent before_id to a daemon reading before_num.

Visual baselines for the new stories still need to come from CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 3 of the chat pane (ahilles107/gravity#13). Bots get a blocking
PermissionRequest hook (not PreToolUse, which fires for every call the
bot is already allowed to make). The hook posts the prompt to
/hook/permission and waits; the app shows a card above the bot's tabs
with Allow once, Allow for this session and Deny (with a reason the bot
sees), and the answer goes back as Claude Code's decision.

Nothing defaults to yes:
- An unanswered prompt is denied when its window closes
  (permission_timeout_seconds, default 600, below the hook's timeout).
- "Allow for this session" sends Claude Code's suggested rules with the
  destination forced to session; mode changes and persistent rules are
  dropped, so no settings file is ever written.
- The daemon only holds a prompt while an app that shows cards (hello
  features: permission_cards) is connected with the control grant.
  Otherwise, or if the hook fails, Claude Code's own terminal prompt
  appears as before.
- Answering is `control`, the same grant as typing into the terminal.

New list_permissions / answer_permission requests and permission_request
/ permission_resolved pushes. Also splits chat/builder.rs (505 lines,
over the limit since phase 1) and moves the Unix hook settings out of
paths.rs into paths/unix_hooks.rs.

The live Claude Code check the issue requires is still to be run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Next to Info and Files, a bot's Tasks tab shows its work in four
sections: Now (open tasks it was given), Waiting on others (open tasks
it delegated, with the peer machine for linked bots), Upcoming (its
enabled routines' next runs) and Done (finished, cancelled or expired
tasks, opening to the result that closed them). It refetches after bus
traffic and routine runs.

A new read-grant `list_tasks` request serves a bot's tasks from both
ends, newest first, with the request, the result and when it closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 4 of the chat pane (ahilles107/gravity#13).

The Codex worker now records every command, file change (with its diff
as Claude Code-style structured patch hunks) and MCP call in
codex-observations.jsonl as tool_use / tool_result pairs, and marks each
turn's end with a turn_duration record, so the chat reads a Codex bot's
turns, steps, diffs and bus calls the same way as a Claude bot's.

Command, file-change and permission approvals become permission cards:
the worker raises a structured SessionEvent::Permission, the supervisor
relays it on the internal bus, the approval broker shows the card, and
the answer goes back through RuntimeSession::answer_permission as the
App Server's accept / acceptForSession / decline (or a turn- or
session-scoped permission grant). Whichever answers first, the card or
the native CLI, decides; serverRequest/resolved withdraws the card.
Questions and elicitations stay in the native CLI.

The broker is split into a shared decide step with Claude hook and
runtime front ends. The supervisor's session event loop moves to its own
file. Also fixes the scheduler's routine-completion listener, which
ended for good on the first internal event that was not BotDone, or on
a lagged burst.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 5 of the chat pane (ahilles107/gravity#13). A linked bot's chat
used to show only its bus conversation. Now list_chat, get_chat_step,
get_chat_image and read_file for a linked bot are forwarded to the
daemon it runs on, which serves them for bots linked to that peer only
(read_file stays confined to the bot's own directory and its project's
artifacts there). Turns come back with the local linked bot's id.

While a peer has a chat open, the daemon running the bot streams its
chat_turns as event frames (no req_id, no answer) and the other daemon
pushes them to its clients under the linked bot's id. Daemons advertise
`peer_chat`; the app shows the full chat for linked bots on such a
daemon, with a note saying which machine the bot runs on, and keeps
the bus-conversation view otherwise. Permission prompts of a linked bot
are still answered on its own machine.

The two-daemon test setup moves to tests/common/peers.rs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 6 of the chat pane (ahilles107/gravity#13), the parts the issue
left out of v1:

- Decisions in the thread: a decision a bot raised shows its options in
  the chat. Picking one, or writing an answer, publishes it in one step
  (publish_decisions with an inline ruling), as "Answer and publish" does
  in the Control Center; it needs the approve grant. The card links to
  the Control Center for drafts, holds and the discussion.
- Attachments: attach or paste files in the composer. They upload in
  chunks through a new control-grant write_artifact request into the
  project's artifacts/uploads/ (16 MB cap, name confined to its last
  component), and the message lists their paths for the bot.
- Slash commands: a message starting with / is typed into the bot's
  terminal, as if typed there, with a hint under the composer.
- Search: ⌘F filters the loaded turns by their text, steps and trigger.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Search now works like find in page: every match in the loaded turns is
  highlighted in place (CSS custom highlights), the current one apart, and
  ▲/▼ or Enter / Shift+Enter step between them, scrolling each into view.
  Folded step groups open while searching so their text is found. Turns
  are no longer filtered out.
- Tasks: requests, results and routine prompts clamped to two lines get a
  "Show more" / "Show less" when the clamp hides something or a result is
  waiting, instead of only an ellipsis.
- Image thumbnails keep their image's aspect ratio (fixed height, width
  from the image) instead of a square crop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Tasks list trims each request to 400 characters and each result to
600, so "Show more" could only reveal a cut-off preview. The list now says
when it trimmed (request_truncated, result_truncated), a new read-grant
get_task request returns one task with its full request and result, and
opening a row loads it when the preview was cut. Opened requests and
results render as the markdown bots write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A mic button in the composer turns speech into text with the operating
system's own recognizer, so audio never goes to Gravity or a third
party. On macOS, AVAudioEngine feeds the microphone to Apple's Speech
framework (objc2-speech / objc2-avf-audio), set to recognize on the
device whenever the Mac supports it. Partial transcripts stream into the
draft after what was already typed, and the final one stays; pressing
the button again stops.

The first use asks for Speech Recognition and Microphone access: the app
now carries NSSpeechRecognitionUsageDescription and
NSMicrophoneUsageDescription (src-tauri/Info.plist), and signed releases
get the hardened-runtime audio-input entitlement (Entitlements.plist).
The mic is hidden where dictation is unavailable: a plain browser, and
Windows until its recognizer (Windows.Media.SpeechRecognition) lands.

The composer's attachments move into useAttachments and its buttons into
ComposerParts. Notices gain the seven objc2 framework crates, which ship
no license file and use the reviewed objc2 LICENSE override like their
siblings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The full-size image viewer reused the command palette's overlay panel,
which is 580px wide, at most 60vh tall and clips its content, so wide
screenshots were cropped into a box. Images now open in a full-window
lightbox: fitted to the window at their own aspect ratio, or at actual
size (click the image) with the stage scrolling. ←/→ and the side
buttons page through a step group's images; Esc or the backdrop closes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The composer's mic was macOS-only. On Windows, SAPI's in-process
recognizer with its dictation grammar now feeds the same partial and
final events, so audio stays on the PC. WinRT dictation was not used
because it requires Windows' online speech service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ation

fec6522 changed apps/desktop/src-tauri/Cargo.toml and Cargo.lock without
regenerating the notice inventory, so `scripts/notices.py` refused them. It
added no crates: the regenerated records and bundles are identical, so only
the two input hashes change. The records are kept as committed rather than
regenerated here, since a macOS run swaps the marketing build tools'
Windows platform packages for macOS ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eleven stories added with the chat pane had no baseline, so the visual
suite failed on each: chat composer, code blocks, lightbox, permission
cards, tasks panel and turn view. Generated in the pinned Playwright
container with --update-snapshots=missing, so the existing 72 baselines
are untouched and still match byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	.github/workflows/windows.yml
#	CONTRIBUTING.md
#	apps/desktop/src/components/bot/BotRuntimePicker.tsx
#	crates/gravityd/src/activity.rs
#	crates/gravityd/src/app.rs
#	crates/gravityd/src/db/peers.rs
#	crates/gravityd/src/paths.rs
#	crates/gravityd/src/paths/trust.rs
#	crates/gravityd/src/paths/windows_hooks.rs
#	crates/gravityd/src/peer.rs
#	crates/gravityd/src/peer/hub.rs
#	crates/gravityd/src/peer/inbound.rs
#	crates/gravityd/src/runtime/codex.rs
#	crates/gravityd/src/runtime/codex/approvals.rs
#	crates/gravityd/src/runtime/codex/native.rs
#	crates/gravityd/src/runtime/codex/observations.rs
#	crates/gravityd/src/runtime/codex/worker.rs
#	crates/gravityd/src/runtime/mod.rs
#	crates/gravityd/src/runtime/pty/tests.rs
#	crates/gravityd/src/server.rs
#	crates/gravityd/src/supervisor.rs
#	crates/gravityd/src/supervisor/lifecycle.rs
#	crates/gravityd/src/ws.rs
#	crates/gravityd/src/ws/dispatch.rs
#	crates/gravityd/tests/codex_runtime.rs
#	crates/gravityd/tests/fixtures/codex-server.mjs
#	crates/gravityd/tests/peer_bots.rs
#	docs/bot-runtimes.md
#	docs/peer-bots.md
#	docs/protocol.md
#	docs/windows.md
#	third-party/inventory.json
@ahilles107

Copy link
Copy Markdown
Owner

I'm not sure if chat interface in the app is a great idea. One of the project original rules was to keep things and features list simple, and decision to go with the terminal view helped that a lot. With it we get most of the provider (claude, codex) features for free without a heavy maintenance work. Chat view add the whole new surface which requires constant development and maintenance.

@mrinaldi2

Copy link
Copy Markdown
Contributor Author

I understand I found having 12 agents in the project very hard to use, tracking files, looking at screenshot the agent tooks, understand when a task is completed or not, the chat is based on the log not on the terminal and with AI maintain eventually will mean ask Opus to do un update

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Chat pane as the default bot surface

2 participants