Skip to content

feat(web): full-page screenshots, sensitive-page guard and receipts - #13

Merged
Verlintas merged 1 commit into
mainfrom
feat/web-vision-phase2
Nov 11, 2026
Merged

Verlintas merged 1 commit into
mainfrom
feat/web-vision-phase2

Conversation

@Verlintas

Copy link
Copy Markdown
Owner

What this adds

Phase 2 of the web visual channel:

  • Full-page screenshots — web_research screenshot(full_page=true) uses WebView.capturePicture() to grab the whole document and crops it into up to six readable viewport segments. Longer pages are explicitly marked truncated. Scrolling + View.draw() was abandoned because the virtual-display WebView reports the new scroll offset but returns a blank draw after scrolling.
  • Sensitive-page guard — screenshots are refused with screenshot_blocked when the page has a visible password field, a card/payment form, a Stripe/PayPal-like payment iframe, a checkout/payment URL plus payment text, or a CAPTCHA challenge. The model is pointed to snapshot / read / show_to_user instead.
  • User-visible receipts — AgentEvent.ToolCallFinished carries ephemeral ToolImages. When the existing keepExecutionScreenshots setting is on, ChatViewModel imports them as assistant-message attachments, so the user can see exactly what the model saw. Assistant attachments are stripped from future model input, as before.
  • Main-screen tools — take_screenshot and screen_ocr now return OCR plus the image when the model is a known vision family (or vision=true); vision=false keeps them text-only.
  • Startup WebView mode — enableSlowWholeDocumentDraw() is enabled in Application.onCreate, before any WebView loads, so capturePicture() includes content below the first viewport.
  • Debug tooling — generic DebugToolActivity for any registered tool, and DebugOcrActivity can OCR a file path.

Verification

API 35 emulator, devtest package; daily app untouched.

  • 5-viewport local page: 5 images, 799×1600 each; OCR read ALPHA-111, BRAVO-222, CHARLIE-333, DELTA-444, ECH0-555.
  • 9-viewport page: 6 images with a clear truncation note.
  • full_page=true&vision=false: 5 OCR segments, images=0.
  • password.html: screenshot_blocked: ... password field.
  • payment.html (autocomplete=cc-number/cc-csc): screenshot_blocked: ... card/payment form.
  • Single viewport chart page: still 720×1600 / 135 KB with OCR.
  • take_screenshot with vision=true: 92 KB PNG + OCR + images=1.
  • screen_ocr with vision=true: OCR + images=1 (entire-screen MediaProjection grant).
  • Unit tests: WebScreenshotPolicy (allowed article, password/card, payment frame/page, CAPTCHA) and AgentEvent image propagation.
  • test, lintDebug, assembleDebug, assembleRelease green.

Known limits

  • Full-page is capped at six segments; very long pages need follow-up scrolling or read.
  • capturePicture() may miss video/WebGL/SurfaceView layers on some WebView builds.
  • enableSlowWholeDocumentDraw() raises memory/time for very long pages.
  • Main-screen screenshots still rely on the user's MediaProjection grant and may contain notifications; the tool description warns against password/payment screens.

Add full_page segmented capture via capturePicture with slow whole-document draw, default blocking of password/payment/CAPTCHA pages, user-visible screenshot receipts on assistant messages, and the same visual channel for take_screenshot and screen_ocr.

Verified on API 35 emulator: 5-segment and 9-segment pages, password/card-page blocks, vision on/off, and main-screen tools. test/lintDebug/assembleDebug/assembleRelease green.
@Verlintas
Verlintas merged commit 5299290 into main Nov 11, 2026
1 check passed
@Verlintas
Verlintas deleted the feat/web-vision-phase2 branch November 11, 2026 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant