Skip to content

fix(engine): bound idle git work by checkout, not by subscriber - #27

Merged
hoangvu12 merged 1 commit into
mainfrom
fix/idle-git-storm
Oct 17, 2026
Merged

hoangvu12 merged 1 commit into
mainfrom
fix/idle-git-storm

Conversation

@hoangvu12

Copy link
Copy Markdown
Owner

Problem

While idle, the engine spawned ~20–60 git.exe processes per second (measured on v0.10.1: 190–590 new children per 10 s, 1,118 in one 30 s window). This starved the engine: lock-free RPCs took up to 918 ms, list_chats 6.7 s, and a rename took up to 9 s to reach the chat list.

Root causes

  1. PR-status watches inspect Git once per subscription, not once per checkout. Each WatchCheckoutChangeRequest stream re-ran inspect_checkout (~8 git processes: rev-parse --show-toplevel, branch --show-current, @{upstream}, branch.<x>.remote, remote.pushDefault, remote get-url --push, symbolic-ref --quiet …/HEAD, …) every 15 s, even though it already shared the gh lookup. Cost therefore grew with the number of subscribers (chats × clients) rather than the number of checkouts, and orchestrators keep adding side chats.
  2. Checkout authorization ran the full refs() on every request. Repos::workspace_checkout ran branch -a + symbolic-ref + branch --show-current + worktree list (~5 processes) only to check that a folder is the repo or one of its worktrees. It runs on every workspace-file request (list/read/search/write/git-status, mention search, command catalogs), on every change_request_root space scan, and on project actions.

The open puzzle: why did watches run with sidebarShowPullRequest: false?

The desktop wasn't the subscriber; the paired Android phone was.

  • The desktop UI does honour the setting. Shell::new applies it synchronously in the same app-init closure that starts bootstrap, so reconcile_change_request_watches never runs with the true default once the registry attaches. Dropping a watch task drops its RpcSubscription, which sends a cancel frame, and the server aborts the stream task (rpc/src/server.rs). The new UI test runs that path (setting off before the registry attaches, chats arriving, selection changes) and holds zero watch tasks. It passes on both the old and new code, which rules the desktop out.
  • On the live engine, GetRemoteAccess listed a paired session "S23 Ultra", last seen 06:32 local, inside the measurement window. Get-NetTCPConnection showed 5 established connections from roboco-tailcat.exe on the remote-access listener (the Android path), next to 4 roboco mcp clients.
  • The mobile client (crates/mobile-client/src/workspace/change_requests.rs) subscribes WatchCheckoutChangeRequest for up to 40 (repo root, branch) targets per engine: every non-archived top-level chat, idle ones included, sorted by recency. It never sees the desktop's sidebar setting, which is per-viewport UI state; the engine doesn't know about it. That matches the signature: inspections for idle chats' branches such as implement-core-remaining and feat/orchestration-v2.

So the fix has to live in the engine. Engine cost must be bounded by checkouts, not by subscriber count or client type. That is what (1) below does.

After the app restarted at 07:12 (not by me), a fresh 30 s sample showed 840 git children, dominated by the refs() signature (~130 sets) with only ~7 PR inspections. That points at (2), workspace-file authorization, which the cache below removes.

Changes

Engine: change_requests.rs

  • One CheckoutInspection per checkout id, shared by every subscription on that checkout whatever branch it asks about (weak map; subscriptions hold the strong refs). It runs single-flight, at most once per poll interval.
  • Re-inspection is skipped while a stamp (size + mtime) of the worktree's HEAD and the common config / packed-refs is unchanged, up to a 5-minute max age. The stamp is read before inspecting, so a change racing the inspection triggers another. A missing or unreadable stamp (2 s timeout for dead mounts) falls back to the 15 s interval. Failed inspections are never extended by the stamp.
  • The locked section runs as a spawned task. Without that, a subscriber that stops polling mid-inspection keeps its pending unfold future, and with it the lock every other subscriber on the checkout waits for. The existing failure_keeps_the_last_success test hung on exactly this before I made the change.

Engine: repos.rs

  • workspace_checkout reads linked worktrees from a single git worktree list --porcelain. The listing is cached per repository and revalidated against the mtime of .git/worktrees (worktree add/remove/prune show up immediately), with a 10 s TTL for worktree move and repos without a stampable .git dir. Misses are cached too, which matters for change_request_root, since it scans every space.
  • Test-only git_runs counter on Repos::git (the seam for counting).

UI: change_requests.rs, state.rs

  • Watch targets: top-level non-archived chats, plus a side chat only while it or its parent is selected. Side chats do render PR badges, in the parent's child-session rows (files/sections.rs) and in their own composer footer. So they're kept for those surfaces instead of being dropped outright. select_chat now reconciles watches.

Git-invocation counts (tests written first; they fail on main)

Test main this branch
watchers_on_one_checkout_share_one_inspection_per_interval: 8 watchers, 10 × 15 s intervals, unstampable git dir 96 inspections (~770 git) 11 (one per interval)
unchanged_git_metadata_skips_reinspection_until_head_moves: 4 watchers on an idle checkout, 10 intervals 34 inspections 1, then 2 after HEAD moves (seen within one interval)
workspace_checkout_lists_worktrees_once_until_they_change: 20 authorizations (10 hits + 10 misses) 100 git processes 1; a new worktree adds 1, a removed one adds 1
change_request_watches_follow_visibility_and_badge_surfaces (UI) watches [lead, lead-side, other, other-side] [lead, other]; selection-scoped side chats; zero tasks with visibility off (passes on main too, so the desktop isn't the leaker)

Verification (Windows):

  • cargo test -p roboco-engine --lib: 344 passed, 1 ignored.
  • cargo test -p roboco-ui --lib: 1483 passed, 5 ignored.
  • cargo test -p roboco-engine --test workspace_files: 6 passed, 1 failed. write_rejects_changed_checkout_even_when_contents_match fails identically on main: the worktree checkout's README hash differs under core.autocrlf=true. It's unrelated to this change.
  • cargo clippy -p roboco-engine -p roboco-ui --lib --tests: no warnings in the touched code. The engine test target needs -A clippy::unused_io_amount to get past a pre-existing deny-level lint at rpc.rs:3232. cargo fmt isn't clean on main (it rewrites ~80 files), so the diff only formats the lines it touches.

Deliberately left out

  • Mobile target selection is unchanged: it still watches up to 40 targets regardless of what's on screen. With shared inspections plus the stamp skip, an idle watch costs no git, so the cap no longer turns into process spawns. Narrowing it to visible rows is a product call for the Android app.
  • gh pr list cadence. The provider lookup is still per (checkout, branch): 45 s without a PR, 2 min with one. gh spawns its own git children (grandchildren of Roboco, not in the counts above). With ~27 idle branches that is about 0.6 gh/s. Worth a longer no-PR TTL, or skipping the lookup for branches without an upstream, in a follow-up.
  • checkout_identity (2 git per successful workspace-file resolution) is not cached; the live sample showed it at ~1/s.
  • Detached linked worktrees are now accepted by workspace_checkout. The old code only counted worktrees with a branch line, so a worktree mid-rebase was rejected. They're the repo's own checkouts, so I treated the old behaviour as a bug rather than a rule.
  • I didn't verify on the live app, because the installed build won't have this until a release. The tests are the proof.

PR-status watches re-ran the ~8-process checkout inspection once per
subscription every 15 s, so cost grew with chats x clients (a paired
phone watches up to 40 targets regardless of the desktop's hidden-badge
setting). Workspace-file authorization ran the full refs() (~5
processes) on every request.

- Share one single-flight inspection per checkout and skip it while a
  stamp of HEAD/config/packed-refs is unchanged (5 min max age). The
  locked section runs detached so an unpolled subscriber cannot strand
  the lock.
- Authorize checkouts from one cached `git worktree list`, revalidated
  against the mtime of .git/worktrees (10 s TTL otherwise).
- Desktop: watch side chats only while they or their parent are
  selected; reconcile on selection.
@hoangvu12
hoangvu12 merged commit 810ff4e into main Oct 17, 2026
13 of 16 checks passed
@hoangvu12
hoangvu12 deleted the fix/idle-git-storm branch October 17, 2026 01:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant