Repository navigation
feat(web): full-page screenshots, sensitive-page guard and receipts - #13
Merged
Merged
Conversation
Add full_page segmented capture via capturePicture with slow whole-document draw, default blocking of password/payment/CAPTCHA pages, user-visible screenshot receipts on assistant messages, and the same visual channel for take_screenshot and screen_ocr. Verified on API 35 emulator: 5-segment and 9-segment pages, password/card-page blocks, vision on/off, and main-screen tools. test/lintDebug/assembleDebug/assembleRelease green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Phase 2 of the web visual channel:
web_research screenshot(full_page=true)usesWebView.capturePicture()to grab the whole document and crops it into up to six readable viewport segments. Longer pages are explicitly marked truncated. Scrolling +View.draw()was abandoned because the virtual-display WebView reports the new scroll offset but returns a blank draw after scrolling.screenshot_blockedwhen the page has a visible password field, a card/payment form, a Stripe/PayPal-like payment iframe, a checkout/payment URL plus payment text, or a CAPTCHA challenge. The model is pointed tosnapshot/read/show_to_userinstead.AgentEvent.ToolCallFinishedcarries ephemeralToolImages. When the existingkeepExecutionScreenshotssetting is on, ChatViewModel imports them as assistant-message attachments, so the user can see exactly what the model saw. Assistant attachments are stripped from future model input, as before.take_screenshotandscreen_ocrnow return OCR plus the image when the model is a known vision family (orvision=true);vision=falsekeeps them text-only.enableSlowWholeDocumentDraw()is enabled inApplication.onCreate, before any WebView loads, socapturePicture()includes content below the first viewport.DebugToolActivityfor any registered tool, andDebugOcrActivitycan OCR a file path.Verification
API 35 emulator, devtest package; daily app untouched.
ALPHA-111,BRAVO-222,CHARLIE-333,DELTA-444,ECH0-555.full_page=true&vision=false: 5 OCR segments,images=0.password.html:screenshot_blocked: ... password field.payment.html(autocomplete=cc-number/cc-csc):screenshot_blocked: ... card/payment form.take_screenshotwithvision=true: 92 KB PNG + OCR +images=1.screen_ocrwithvision=true: OCR +images=1(entire-screen MediaProjection grant).WebScreenshotPolicy(allowed article, password/card, payment frame/page, CAPTCHA) andAgentEventimage propagation.test,lintDebug,assembleDebug,assembleReleasegreen.Known limits
read.capturePicture()may miss video/WebGL/SurfaceView layers on some WebView builds.enableSlowWholeDocumentDraw()raises memory/time for very long pages.