Skip to content

feat(screenshot): per-tab screenshot via xterm.js buffer rendering#14

Merged
Shadow-Azure merged 7 commits into
mainfrom
fix/terminal-layout-fill
Jun 1, 2026
Merged

feat(screenshot): per-tab screenshot via xterm.js buffer rendering#14
Shadow-Azure merged 7 commits into
mainfrom
fix/terminal-layout-fill

Conversation

@Shadow-Azure

@Shadow-Azure Shadow-Azure commented Jun 1, 2026

Copy link
Copy Markdown
Owner

Summary

  • Implement per-tab screenshot by keeping all Terminal components mounted (hidden for inactive tabs) and capturing xterm.js buffer content
  • Add /screenshot/ws WebSocket endpoint to daemon for renderer-based capture
  • Fix terminal layout: make .terminal-area a flex container for proper height propagation
  • Fall back to ScreenCaptureKit if renderer is unavailable

Problems Encountered & Solutions

1. Terminal area blank (0 height)

Problem: Mounting all tabs with CSS hiding (position: absolute; left: -9999px) caused the terminal area to collapse to 0 height — no terminal content rendered.

Root cause: .terminal-area lacked display: flex, so child elements with flex: 1 had no flex parent to expand into.

Fix: Added display: flex; flex-direction: column; to .terminal-area in styles.css.

2. Double terminal-container nesting

Problem: main.tsx wrapper div used className="terminal-container", which nested with SandboxTerminal's own div.terminal-container. The CSS rule .terminal-container > div { position: absolute; inset: 0 } made the inner container absolutely positioned, removing it from flow.

Root cause: CSS selector matched the wrong nesting level.

Fix: Removed className="terminal-container" from the wrapper div, using inline flex styles instead.

3. xterm.js v6 uses DOM renderer, not canvas

Problem: captureToPng() tried term.element.querySelector("canvas") which returned null — xterm.js v6 renders to DOM elements, not <canvas>.

Root cause: xterm.js v6 default renderer is dom, not canvas. No <canvas> element exists.

Fix: Added buffer-based fallback: read xterm text buffer line-by-line, render characters onto an offscreen canvas with correct colors.

4. CI failure — cargo fmt

Problem: First CI run failed on Rust formatting in daemon/mod.rs.

Fix: Ran cargo fmt --all and pushed.

5. Known limitation: screenshots lack macOS window frame

Current behavior: Per-tab screenshots capture only terminal content (xterm buffer → canvas). They do NOT include the macOS window chrome (title bar, traffic lights, tab bar, status bar).

Why: xterm.js buffer can only read terminal text. Window chrome is rendered by Electron/OS, invisible to xterm. BrowserWindow.capturePage() only captures web content, not OS chrome. Only ScreenCaptureKit can capture the full native window.

Future approach: Add --with-frame flag that triggers a fast tab-switch → ScreenCaptureKit capture → switch-back cycle (<16ms, imperceptible to user).

Test Results

All 5 release test scenarios passed:

Test Result
opencode 交互 (中文输入/输出)
Claude 交互 (安全确认+对话)
zsh 命令执行
按标签截图 (不同标签内容不同)
信任对话框残留检查
Tab 管理 (切换/关闭)

Full test report: release_test/2026-06-01-08-03-02/test-report.md

Test plan

  • cargo check --all-targets passes
  • cargo test passes (142/143, 1 pre-existing failure)
  • pnpm typecheck passes
  • End-to-end: two sandboxes with different content produce different screenshots
  • CLI opencode/claude/zsh interaction works with PTY input
  • Tab close doesn't affect other tabs
  • CI passes

🤖 Generated with Claude Code

ZN-Ice and others added 7 commits May 31, 2026 17:07
- Add min-height: 0 to .main-content and .terminal-container
- Remove padding: 8px from .terminal-container .xterm
- Fixes bottom white space and right black space in terminal

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Extract doFit helper that calls fit() then resize()
- Use fitFnRef so containerRef triggers the same atomic operation
- Fixes race condition between initial fit and WebSocket resize

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Split the html,body,#root CSS rule so #root gets display:flex + flex-direction:column.
Without this, .main-content { flex: 1 } had no effect since #root wasn't a flex container.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…itioning

The inner div with height:100% wasn't resolving correctly to the flex
container's height. Using position:absolute + inset:0 forces the div
to fill the terminal container regardless of CSS height resolution.

Fixes the remaining white area below the terminal canvas.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…zing

The initial fit() in requestAnimationFrame runs before the flex layout
is fully computed, causing TUI apps like opencode to render with incorrect
dimensions. A delayed re-fit at 100ms gives the layout time to settle,
matching what happens when the user manually resizes the window.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ndering

Implement Approach B from terminal layout fix plan: keep all tab Terminal
components mounted but hidden, read xterm.js buffer for screenshots of
non-active tabs instead of relying on ScreenCaptureKit which captures
the entire Electron window.

Changes:
- Terminal.tsx: convert to forwardRef with useImperativeHandle exposing
  captureToPng(), use canvas for active tabs and buffer-based rendering
  for hidden/offscreen tabs
- main.tsx: mount all tabs with CSS hiding, manage terminal refs, connect
  to daemon's /screenshot/ws WebSocket for capture requests
- styles.css: make .terminal-area a flex container for proper height
  propagation to nested terminal containers
- daemon/mod.rs: add /screenshot/ws WebSocket endpoint and
  request_renderer_screenshot() helper, modify screenshot_handler to
  try renderer first then fall back to ScreenCaptureKit
- daemon_integration.rs: update DaemonState with new fields

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jun 1, 2026

Copy link
Copy Markdown

🔒 门禁检查结果

检查项 状态
Rust 格式化 ✅ success
Rust Clippy ✅ success
Rust 测试 & 覆盖率 ✅ success
前端测试 & 覆盖率 ✅ success
安全检查 ✅ success

Rust 测试覆盖率

指标 覆盖率
行覆盖率 69.1%
分支覆盖率 0.0%
模块 行覆盖率
crates.sandbox-core.src.server █████████████████░░░ 87.5%
crates.sandbox-core.src.daemon ██████████░░░░░░░░░░ 51.6%
crates.sandbox-core.src ███████████░░░░░░░░░ 55.3%
crates.sandbox-core.src.instance ███████████████████░ 99.1%
crates.sandbox-core.src.process █████████████░░░░░░░ 65.7%
crates.sandbox-core.src.automation ███████████░░░░░░░░░ 58.8%
crates.sandbox-core.src.sandbox ███████████████████░ 98.8%
crates.sandbox-core.src.capture ████████████████████ 100.0%

详细报告见 Rust 覆盖率 artifact

前端测试覆盖率

指标 覆盖率
行覆盖率 ██░░░░░░░░░░░░░░░░░░ 11.4%
分支覆盖率 0.0%
函数覆盖率 17.6%
语句覆盖率 9.1%
文件 行覆盖率
src/renderer/api.ts ██░░░░░░░░░░░░░░░░░░ 11.4%

详细报告见前端覆盖率 artifact

✅ 所有检查通过,可以合入

点击 Squash and merge 合并此PR

@Shadow-Azure
Shadow-Azure merged commit 4d29508 into main Jun 1, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant