Skip to content

fix: eliminate Athena resource spikes and freeze regressions - #190

Merged
luckeyfaraday merged 1 commit into
mainfrom
fix/resource-stability
Jul 20, 2026
Merged

fix: eliminate Athena resource spikes and freeze regressions#190
luckeyfaraday merged 1 commit into
mainfrom
fix/resource-stability

Conversation

@luckeyfaraday

Copy link
Copy Markdown
Owner

Summary

Athena's freeze/heat problem is a combination of software-composited xterm painting, terminal delivery contracts that acknowledge before xterm drains, periodic whole-corpus session parsing, and agent fleets that can bypass memory admission.

The live Surface Laptop 3 profile that motivated this PR measured:

  • Electron renderer: 128.6% mean CPU (137% peak)
  • Chromium Viz compositor: 61.3% mean CPU (71% peak)
  • Electron main: 9.8% CPU / about 493 MiB RSS
  • Athena process tree: about 2.5 GiB RSS with four live agent panes

This PR fixes those causes as one resource/lifecycle contract and adds permanent tests for the regressions seen across #26, #57, #95-#100, #146, #154, #169-#172, #181, and #184.

What changed

Terminal delivery, visibility, and layout

  • Replace preload-dispatch ACKs with epoch/sequence delivery and ACK only after the xterm write callback drains.
  • Retain the exact in-flight payload for retry; stale, duplicate, and wrong-epoch ACKs are harmless.
  • Make attach atomic: a versioned bounded snapshot precedes contiguous live output for renderer and control/SSE consumers.
  • Keep queues bounded per consumer. Overflow becomes an explicit reset snapshot rather than a silent wedge or unbounded heap.
  • Store terminal history in bounded chunks and choose Unicode/ANSI-safe replay boundaries without repeatedly joining the discarded prefix.
  • Cap renderer remount replay at 64 KiB while preserving the independent 200k control buffer.
  • Stop broadcasting raw bytes to hidden panes; attention classification now emits small main-process state events.
  • Separate visible from focused. Only the selected terminal owns focus, and one coalesced lifecycle/layout scheduler handles fit/refresh.
  • Replace browser-owned pane heights with React-owned sizing and remove the press theme's full-window turbulence filter.

Session discovery and allocation bounds

  • Move Hermes file parsing out of Electron main into a below-normal-priority metadata worker.
  • Persist only compact metadata/path hints, not up to 300k characters of transcript search text per session.
  • Make persisted-index load single-flight and retire/restart a wedged worker after timeout/send failure instead of reusing it forever.
  • Bound Electron's workspace TTL/LRU cache and backend provider scans/caches; preserve last-known-good Hermes metadata on transient corruption.
  • Apply workspace predicates before provider limits and align POSIX-case/root, Windows, WSL, and UNC identity rules across renderer, Electron, and backend.

Launch admission, restore, and process lifecycle

  • Put UI, control, and restore terminal launches behind one atomic Electron admission service using physical MemAvailable only.
  • Reserve 640 MiB per heavyweight agent tree, stagger every non-shell grid launch by 750 ms, and make critical-memory UI approval one-shot.
  • Add equivalent bounded admission to the legacy FastAPI spawn route; return structured rejection/defer results instead of false success.
  • Recheck pressure between restore entries, preserve missing provider identities instead of silently starting fresh, and keep restore crash evidence unless PTY shutdown is confirmed before the quit deadline.
  • Bound backend stdout/stderr capture, use process groups, gate late spawns during shutdown, await owned service cleanup, and force-reap stubborn descendants.

Adaptive Linux graphics

  • Healthy Linux launches use hardware acceleration, addressing the measured software-compositor heat.
  • Persist an accelerated-pending marker before Chromium starts. An unclean accelerated exit or GPU-process crash quarantines the next launch into safe mode.
  • Keep explicit safe/accelerated settings and environment overrides.
  • Ensure a rejected packaged second instance cannot mutate the primary instance's graphics marker or log.
  • Keep the Linux native beep disabled; later Harden terminal rendering and disable Linux native beep crash path #100 evidence implicated that path in the original crash sequence.

Regression protection and diagnostics

  • Add permanent PR-lineage suites for terminal stream/ACK, GPU policy, pane layout/focus, session indexing, admission, and backend output bounds.
  • Add CI for the full Python, renderer, Electron, production-build, and permanent-regression matrix.
  • Surface terminal produced/delivered/acked/reset/replay counters, hidden raw IPC, session-index scan stats, event-loop lag, and process-tree diagnostics in Settings.

Why this avoids the previous regressions

Validation

  • Production client build: pass
  • Renderer/chat/layout/path tests: 31/31 pass
  • Electron/helper tests: 222/222 pass
  • Permanent historical-regression tests: 72/72 pass
  • Python/backend tests: 207/207 pass
  • git diff --check: pass

Short hardware canary on the affected Surface Laptop 3 (Intel Iris Plus G7, 8086:8a52, i915):

  • accelerated launch and clean window shutdown completed without a runtime GPU-crash signal;
  • one mounted xterm processed a bounded 5,000-line shell burst;
  • after the burst settled, the renderer and GPU processes averaged about 1% combined CPU over a five-second sample;
  • graphics and launch state both recorded a clean exit.

This canary validates startup, mounted-xterm rendering, and clean fallback state on the known-risk device. It is not a substitute for the long soak below.

Known limits / follow-up gates

  • No two-hour CPU/RSS soak or sustained four-pane 64 KiB/s hardware load was run in this PR session.
  • No real Windows ConPTY hardware run was available.
  • Bounded reset/replay is explicit and safe, but it is not yet a full authoritative serialized VT checkpoint; sufficiently old scrollback may be intentionally discarded.
  • Electron and legacy FastAPI admission reservations are process-local. A narrowly concurrent legacy HTTP spawn and desktop spawn can still race before OS RSS becomes visible; the legacy path is separately capped and was not implicated in the measured incident.
  • External CLIs that deliberately self-daemonize outside their inherited process group still need release-soak verification.

Checklist

  • Reviewed the prior performance/freeze PR lineage and reconstructed regression chains
  • Added regression tests before changing the riskiest contracts
  • Preserved bounded-memory and restore/idempotency invariants
  • Exercised accelerated Chromium and a mounted xterm on the affected Surface hardware
  • Added rollback controls for Linux graphics mode
  • Complete the two-hour multi-pane soak and Windows ConPTY hardware gate before release promotion

@luckeyfaraday

Copy link
Copy Markdown
Owner Author

Manual validation update: the resource-stability AppImage built from a497aa4 was tested by the user on the affected machine; they report that it looks good and works. This is additional smoke-test evidence. The two-hour multi-pane soak and Windows ConPTY hardware gate listed in the PR remain open.

@luckeyfaraday
luckeyfaraday merged commit ead7a46 into main Jul 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant