Screen Solver

[jpka]

Destination

An implementation-ready spec for a Windows 11 local server that captures a chosen browser window when the user clicks a "solve now" button in the web client, calls a vision LLM on that capture, logs every answer locally, and streams answers to any browser — desktop or phone — that opens its web client.

Done means: nothing left to decide before someone starts coding.

Notes

Domain. Greenfield Windows 11 app. Node 24.13 / npm 11.6 available; no Python, .NET, or Rust toolchain installed.

Tracker. GitHub Issues (this map + its child issues). See docs/agents/issue-tracker.md. Original planning history lives at .scratch/solver/.

Posture. Plan, don't do. Every ticket resolves a decision; the map is done when the way is clear. Do not build the app from inside this map.

Skills to consult. /grilling and /domain-modeling by default; /research for research tickets; /prototype for prototype tickets; /codebase-design when designing the provider seam.

Settled constraints. Pinned down while naming the destination — these fix the scope and are not up for re-litigation inside a ticket:

Constraint	Answer
Destination	Implementation-ready spec (plan, don't build)
First target	Coding exercises / katas (LeetCode, Exercism, Advent of Code, HackerRank)
Trigger	Manual — the user clicks a "solve now" button in the web client; no interval, no automatic change detection. The client therefore sends at least one command upstream — a settled fact The stream contract and The control surface both inherit
Capture target	A named window the user picks from a list, captured by handle so it follows moves and resizes
Scope boundary	Strictly read-only — the app displays answers and never acts on the page
LLM provider	Anthropic API only for v1 (decided, Vision provider comparison), behind a seam designed to admit a second provider later
Output surface	A local HTTP server plus a browser web client — no desktop output window. Any device that can reach the host, phone included
Answer log	Every answer persists locally whether or not a client is connected; a client joining mid-session loads previously logged answers. The loop never gates on a viewer
Packaging	Out of scope. The app runs from a local checkout

Decisions so far

• Window capture on Windows 11 — Target Windows.Graphics.Capture; Electron's desktopCapturer is already WGC-backed and sidesteps PrintWindow's black-frame problem on GPU-composited Chrome/Edge. Two findings from the now-out-of-scope packaging ticket are really capture facts and live here: graphicsCaptureWithoutBorder turns out to be a general-use capability rather than a restricted one (correcting this ticket) — but that's moot, because WebRTC's WGC session never sets IsBorderRequired and Electron exposes no way to, so no packaging format can suppress the yellow border from that capture path. It ships as permanent, expected behaviour and doubles as a free OS-drawn "watching" indicator that vanishes on pause — provided the app holds one long-lived capture session rather than one per tick, or it strobes. The host runtime found those two clauses in tension and resolved it: the session is held per run, torn down on pause and rebuilt on resume, so the border is honest rather than permanently lit. The plain-Node alternative's backend, which this ticket left unverified, has since been answered by Capture without Electron — it is PrintWindow, not WGC. Stale-vs-black behaviour on minimized/other-desktop windows stays open.

• Vision provider comparison — Decided directly by the user, skipping the planned cross-provider research: Anthropic API only for v1. The seam (Provider seam design) is still designed to admit a second provider later.

• App shell and stack — Electron + TypeScript. Capture loop, change detection, and the streaming Anthropic call all live in the main process; the output window is a thin renderer driven by IPC events, never touching the network or capture API directly. Reopened in part by the redraw, and now re-confirmed: The host runtime keeps Electron + TypeScript — and found that the renderer half never actually left. It changes job from displaying answers to holding the capture session, because Electron's live capture path is renderer-only. The rule that the display surface never touches capture or the network survives, with the browser client in the renderer's place.

• Provider seam design — A deep module: createProvider(config) → Provider with solve(image, {signal}) → AsyncIterable. System prompt is configured once at construction, not per-call. Streaming normalizes to delta/done{usage}/error{kind}; transient errors (rate limit, overloaded, network) retry internally, auth/refusal surface immediately. Budget enforcement, change detection, and image capture stay outside the seam. How those events reach a browser is The stream contract's job; the seam itself is unaffected.

• The output window — "Focus pane": one dominant streaming answer pane, history/settings behind titlebar icons, new problem interrupts-and-replaces (previous pushed to history, tagged interrupted). Three variants prototyped; code wraps rather than horizontal-scrolls, window is always-on-top by default and remembers size/position across restarts. Superseded as a desktop window by the redraw — but the focus-pane content design survives intact and is carried into The web client. What doesn't survive is everything that assumed an owned window: always-on-top, remembered size and position, and a titlebar to hang icons on.

• Configuration and API key storage — API key stored via Electron's safeStorage (DPAPI-backed, no extra native dep), entered once through an in-app masked input, decrypted into a main-process-only variable and never sent over IPC or logged. Settings (interval, target window, provider/model, change-detection params) live in one config.json in userData, alongside the encrypted key blob; target window is identified by process name + title (not the unstable OS handle), re-resolved on startup with a fallback to reopening the picker. All settings apply live, no restart needed. The interval and change-detection params this originally specified are now moot — triggering is manual (see Out of scope) — but the rest of the config shape, the window-identity scheme, and the live-apply rule survive for the settings that remain. The safeStorage half is now reversed outright by The host runtime — not re-mechanized: the key is an ANTHROPIC_API_KEY env var, no encrypted blob, no masked input anywhere, and a missing key means refuse to start. userData survives as the state root and resolves to %APPDATA%\screen-solver\. The config shape, the window-identity scheme, the live-apply rule, and the principle that the key never leaves the host process all survive.

• Cost control — Four nested limits — near-duplicate suppression, a sticky runaway breaker (8 solves/5min), a 20s cooldown, a 40/hour ceiling, and a $5/day spend cap — sized against a computed cost model: a solve costs $0.02–$0.09 ($0.21 worst case), and an unguarded misfiring detector on a 5s interval costs $18–$153/hour. Defaults ship conservative (claude-sonnet-5, effort: medium, max_tokens: 8000, capture downscaled to 1568px, no crop) because effort moves cost more than image size does — thinking bills at the output rate. Limits escalate silent → auto-recovering → sticky; hitting one stops and warns, never silently degrades the model. Keeping the last 3 solved baselines for near-duplicate suppression also partially closed the scroll-vs-navigate risk from the automatic change-detection gate — moot now that the gate itself is out of scope (triggering moved to a manual client button; see Out of scope). Whether the rest of this ticket's limits still fit a manual trigger is open (see Not yet specified). Prompt caching was left conditional on Solve-call prompt design, which has since measured the prompt over the minimum cacheable prefix and turned it on at ttl: 1h. All five enforcement mechanisms are now reversed to out of scope by The answer log — decided directly by the user: the threat model here was an automatic detector misfiring, which is gone now that triggering is a manual click. What survives is pure cost visibility — every attempted call is still recorded with its usage and cost in usage.jsonl, uncapped and unenforced. See Out of scope.

• Solve-call prompt design — Answer-first (# title → fenced code → two or three short paragraphs), matching The output window's code-streams-first contract; the heading doubles as the history drawer's per-entry label. Prompt committed, run against the real kata captures. The biggest correctness lever wasn't on the ticket's list: the answer must reproduce the editor pane's exact signature, and the sample tests pin the return shape more precisely than the prose does — which is why Cost control was right to refuse cropping. Partial screens are judged by determinacy, not by whether anything is cut off (a scrolled frame with title, examples, signature and tests is solved silently; only a missing load-bearing fact earns a > **Missing:** line, placed before the code because the code streams first). Prompt caching turns on — the prompt measures 1196–1496 t against claude-sonnet-5's 1024 minimum, and above that threshold a longer prompt is a cheaper one. The measured visible answer is 255–305 t, under 4% of max_tokens: 8000, so that ceiling is in practice a thinking budget and the right answer to Cost control #9's truncation question is to make truncation detectable (stop_reason) rather than assert it won't happen. A literal # No exercise on screen bail token closes the map's "detection as a separate pass" fog: no separate detector ships in v1 — the in-prompt bail is the v1 detector, and counting how often it fires produces exactly the suppression rate Cost control #9's 4% break-even arithmetic was missing, turning a quality guess into a measurement. No API key was reachable, so every figure carries its measured-vs-estimated label and its direction of bias.

• Capture without Electron — node-screenshots is not a drop-in for desktopCapturer, and the reason is the backend, not the build. It pins xcap 0.4.1, which has no WGC code at all: capture_window is PrintWindow with the undocumented PW_RENDERFULLCONTENT flag, falling back to an undocumented flag 4 and then BitBlt — xcap ported WebRTC's legacy GDI capturer, not the WgcCapturerWin that makes Electron work. Confirmed against the shipped artifact, not just the source: the published .node imports PrintWindow/BitBlt/GetDIBits and imports nothing from combase.dll, d3d11.dll or dxgi.dll. The worst part is that failure is silent — PrintWindow is documented to return nonzero on success with no relationship to pixels drawn, so a black frame skips every fallback as dead code and reaches JS as a valid all-black PNG with no error, which for a per-frame-billed app is a silent bill. Conversely the toolchain fear was unfounded: prebuilt win32-x64/arm64 binaries ship with no install script, no node-gyp and no download step — so "no Rust" rules out compiling Rust on the target, not consuming a Rust-built binary. Recommendation into The host runtime: keep Electron for capture; the one lever that flips it is forking the binding onto xcap >= 0.8.3 with features = ["wgc"], which needs Rust once on a build machine and never on the target. Four findings transfer whatever the runtime turns out to be — gate captures on isMinimized(), treat "vanished from enumeration" as ambiguous between closed and moved-to-another-desktop, expect pixel dimensions to change when the window crosses monitors with different scale factors, and check a non-black-pixel ratio before spending an LLM call. Findings: capture-without-electron.md. Run on Linux, so every runtime-pixel claim is labelled inferred or unresolvable with its settling test named.

• The host runtime — Electron stays, and the framing question had a false premise. Electron's live capture path is renderer-only — desktopCapturer returns ids and thumbnails, but turning one into a stream is getUserMedia, and navigator doesn't exist in main — so "headless Electron" is really Electron plus a hidden BrowserWindow holding the MediaStream. The renderer half of App shell and stack never left; it changed job. One process tree: main owns the HTTP server, detection, guards and the provider call (the "detection" piece has since gone moot with the manual-trigger scope change — read this as main owns the HTTP server, guards, and the provider call); the hidden renderer does mechanism only (grab, downscale to 1568px on canvas, encode, send), so the whole decision layer is testable in plain Node with no Electron and no real window. The main-process-only variant (poll getSources({thumbnailSize}) per tick) is rejected — it enumerates every window each tick and opens a fresh WGC session, which is the strobing border. Configuration and API key storage is reversed, not re-mechanized: ANTHROPIC_API_KEY env var only, deleted from process.env before any renderer exists, missing key means refuse to start. That deletes The control surface's hardest bullet — key entry is host-only by construction — at two honest costs: the at-rest story is plaintext rather than DPAPI, and ERR_CONNECTION_REFUSED now covers crashed, never-started and misconfigured alike. State root is %APPDATA%\screen-solver\ via app.getPath('userData') — chosen over a checkout-local data/ because git clean -xdf would wipe the 30-day spend history, and a budget guard that forgets is a money bug. requestSingleInstanceLock() is authoritative rather than letting the port bind serve as the lock, which makes EADDRINUSE mean exactly one thing regardless of whatever port-collision behaviour ends up shipping (port-collision handling itself is no longer a decided question — see Reach and trust in Out of scope); being userData-keyed, it also makes two checkouts correctly one instance. The capture session is per run, resolving a contradiction the map was carrying — a process-lifetime session leaves the yellow border lit while the loop is paused, so the map's one free indicator would have lied in the reassuring direction. Money counters derive from the durable usage record; Cost control's sticky breaker is in-memory and clears on restart, since with no auto-start a restart is the human acknowledgment and the loss is bounded to ~8 solves. (The breaker itself is since reversed to out of scope — see The answer log.)

• The answer log — Two unbounded, append-only JSONL files under %APPDATA%\screen-solver\: answers.jsonl (title, final text only — no delta stream, no image — timestamp, model, usage, source window; one entry per done/interrupted outcome, never for a bail or error) and usage.jsonl (one entry per attempted call, every outcome including bail/error, tracking cost regardless of content). In-flight text is memory-only, written once at the terminal event — a mid-stream crash loses that answer, accepted as a rare failure the user recovers from by re-clicking. This also decided, directly by the user, that Cost control's five enforcement mechanisms move to out of scope — usage.jsonl becomes pure observability with no cap and nothing acting on the running total, closing the map's "whether Cost control #9's limits still fit a manual trigger" fog. Hands The stream contract its answer: a joining client is handed the full contents of answers.jsonl, in order.

• The stream contract — SSE for the live answer stream plus a separate POST /solve for the trigger, not one WebSocket carrying everything (the traffic is one-directional broadcast plus occasional discrete commands, which doesn't need bidirectional machinery). Three endpoints: POST /solve (always accepts and interrupts whatever's in flight — no busy rejection, matching The output window — returns 202 after only synchronous validation, 400/503); GET /answers (the full answers.jsonl backlog as JSON, fetched once on page load, kept separate from the live connection); GET /events (SSE, one shared broadcast, no per-client filtering, no auth per Reach and trust). Named SSE events — start, delta{text}, done{usage}, error{kind}, and a sync{text} sent only to a client connecting mid-flight, carrying the accumulated text so far in place of start. The server holds the in-flight accumulated text in memory to make sync possible. EventSource's built-in auto-reconnect falls out of this for free — a reconnect is indistinguishable from a fresh mid-flight join, so no Last-Event-ID replay logic is needed. Closes half of the map's "watching away from the host" fog: a connection sitting between start/sync and done is the phone's equivalent of the host's yellow WGC border. Endpoint-serving details (same-origin vs. separate) are The web client's; the error{kind} taxonomy is Failure and edge-case behavior's.

• The control surface — Decided directly by the user: the web client is the control surface, wide open, no auth, no host-only restriction — v1 is personal-use-only, so the map's earlier "lean host-only absent a specific reason to trust a networked client" default doesn't extend here (it still applies to the API key, unchanged). Startup with no target window configured is "start and wait," not refuse-to-start like the key — the server starts normally and the client shows the window picker in place of the focus pane until one's chosen; POST /solve without a target returns 400. The picker is persistent and live-editable, not one-time-only, matching Configuration and API key storage's no-restart rule. Multiple clients: last-write-wins, broadcast to everyone via a new config{target} event on The stream contract's existing SSE channel. Only the API key stays an unavoidable host-side step (env var, refuse-to-start, per The host runtime); everything else is fully browser-configurable from the first visit. Graduates half the map's "watching away from the host" fog into Capture session lifetime across manual triggers.

• Failure and edge-case behavior — Two channels: SSE error{kind} for per-solve outcomes (always shown immediately in the pane), a standing status pill for app-wide conditions (silent → auto-recovering → sticky ladder). Corrects Cost control's pill vocabulary: throttled/paused (budget)/paused (runaway guard) no longer exist since The answer log reversed budget enforcement to pure visibility — the pill survives only as a concept, repopulated here. Pre-flight checks (window gone, minimized/off-desktop, black/zero-size frame) fail silently before any spend, with only a button flash as acknowledgment; a bail (# No exercise on screen) did spend a call and gets a low-emphasis pane message instead. Mid-run target loss is a three-way split on an app-tracked intent flag — deliberate pause is ignored, unexpected loss triggers one silent re-resolution before falling back to the picker, and a renderer crash gets its own auto-restart with escalation on repeat. Auth rejection (key valid at startup, later revoked) is the one per-solve error that also flips the pill sticky, since it will recur on every future click. A dead stream keeps its partial pane text visible with an appended error marker. Port-bind failure refuses to start like a missing key; client disconnect/reconnect was already answered by The stream contract; an unwatched "loop running for hours" is moot with no automatic loop left; a blocking firewall is undetectable from the app's side and explicitly out of scope. Loudness policy: the pill shows only in whatever browser client is open, no push/toast — except sticky states also get one console line on the host, for whoever's watching the terminal.

• The web client — Two layouts, picked by orientation automatically rather than one responsive layout or a user setting: portrait is a continuous log (one feed, live answer expanded and outlined at top, past answers collapsed to a line and toggling open in place), landscape is a split rail (132px list rail — cut from 220px, which spent a quarter of an 844px viewport on a mostly-empty list — plus the answer pane). The log layout drops The output window's focus-pane framing, and that is the gain: history needs no drawer, sheet, or overlay of its own. Six variants built and driven on a real phone; the two rejected landscape variants are instructive — a code|prose two-column split was the only one that eliminated scrolling entirely at 844×390 but gives up a single readable answer column, and a horizontal filmstrip pays two-axis scrolling in the orientation with the least vertical room. Orientation is detected off innerWidth > innerHeight directly (matchMedia and orientationchange demoted to change triggers, since a compound query can silently fail to match and a dimension comparison cannot), with a 480px floor so the rule keys off shape, not device type, and swaps live with no reload and no interruption to the stream. That swap has a real cost the spec must carry: the two layouts model selection differently — the rail has an explicit live-vs-history pane, the log does not — so the client must normalise state across rotation (drop the pane mode, keep which entry was open, land it as an expanded card) or the live entry renders a history-coloured dot labelled "live". Confirms The output window #7's wrap-over-horizontal-scroll choice at 390px (with a per-block toggle as an escape hatch for a pathological long line), collapses connection state to two signals on one indicator (socket state while live, replaced wholesale by "viewing history" when reading a past answer), and finds The stream contract's sync{text} needs no rendering beyond a transient syncing… tag. Nothing replaces always-on-top — a second tab or the phone is the always-on-top. Fullscreen was added to both layouts on request and is confirmed working on the target device (Android Chrome); iPhone Safari exposes no Fullscreen API for arbitrary elements, so the button is feature-detected and renders visibly disabled rather than failing silently.

• Capture session lifetime across manual triggers — One long-lived WGC session, not per-click: opens the moment a target window is selected (The control surface's config{target} event), not at server startup or deferred to first solve; a target change closes the old session and opens a new one immediately under the same rule. The yellow border is the only host-side affordance — no console line for routine open/close, which stays reserved for Failure and edge-case behavior's sticky failure ladder. Failure and edge-case behavior #13's mid-run target-loss policy (one silent re-resolution, then fallback to the picker) carries over unchanged to this idle long-lived session — it was never really about runs, just about what to do when the held window identity turns out to be gone. Closes the second half of the map's "watching away from the host" fog; the remote half was already answered by The stream contract.

Not yet specified

Out of scope

• Auto-typing answers into the browser — read-only was chosen deliberately; input injection is a different category of tool.
• Clipboard copy of the answer — strictly-read-only was chosen over the read-only-plus-clipboard variant.
• Quizzes, puzzles, crosswords, general Q&A — katas are the first and only target; generalizing the detection problem is a separate effort.
• macOS and Linux — Windows only.
• A local on-device model as the v1 provider — the seam should not preclude one later, but v1 does not ship it.
• Comparing or supporting non-Anthropic vision providers for v1 — decided directly (Vision provider comparison): Anthropic only, no comparison research. The seam stays generic enough to add one later, but nothing beyond Anthropic ships now.
• Silently degrading to a cheaper model when the budget is hit — considered and rejected in Cost control. The app's whole value is a correct answer; changing answer quality without the user asking, and invisibly, is worse than pausing. Switching models stays a deliberate one-click setting.
• Packaging, installers, auto-start, and updates — dropped wholesale by the redraw; the app runs from a local checkout. This closes out Packaging and distribution, whose electron-builder NSIS answer (per-user, unsigned, opt-in auto-start, manual rebuilds) no longer applies, and it subsumes the earlier narrower boundary against Store submission, code-signing certificates, and a hosted update feed. Two of that ticket's findings weren't really about packaging and were kept: the unsuppressable yellow border and the one-long-lived-capture-session requirement moved to Window capture on Windows 11, and its single-instance-lock mandate — which exists because two instances is two budget guards and two bills — moved to The host runtime.
• Remote or internet access — tunnels, port forwarding, a hosted relay. "Local server" means the host machine and the network it is already on.
• Client URL discovery — mDNS, QR codes, broadcast beacons. Whoever opens the web client is assumed to know the address.
• Native mobile or desktop app clients — a browser is the only client.
• Keeping an Electron output window alongside the web client — the client replaces it rather than joining it; two display surfaces is two things to keep in sync.
• A dedicated bind/port/auth/trust decision for the local server — decided directly by the user via a map update; closes out Reach and trust. The destination still requires a phone to be able to reach the server, but v1 does not spend a ticket deciding bind address, port-collision handling, authentication, TLS, or firewall-prompt messaging — it ships with whatever the simplest default turns out to be when built.
• Automatic change detection — decided directly by the user via a map update replacing the trigger with a manual "solve now" button; closes out What counts as "changed". The loop no longer runs on an interval or decides for itself when to solve, so there is no unattended tick for a dHash-and-debounce gate to protect. The prototype and its measured findings remain on the branch if a future automatic mode is ever considered, but nothing from it ships in v1.
• Budget enforcement — near-duplicate suppression, the runaway circuit breaker, the 20s cooldown, the 40/hour ceiling, and the $5/day spend cap — decided directly by the user in The answer log, reversing Cost control's mechanisms wholesale rather than retuning them. The threat they guarded against — an automatic detector misfiring unattended — no longer exists now that triggering is a manual click; a human is the rate limiter. Only cost visibility ships: usage.jsonl records every attempted call's usage and cost, uncapped, with nothing pausing or throttling on the total.

Tickets

• Window capture on Windows 11 #2 — Window capture on Windows 11
• App shell and stack #3 — App shell and stack
• Vision provider comparison #4 — Vision provider comparison
• Provider seam design #5 — Provider seam design
• The output window #7 — The output window
• Configuration and API key storage #8 — Configuration and API key storage
• Cost control #9 — Cost control
• Solve-call prompt design #11 — Solve-call prompt design
• Failure and edge-case behavior #13 — Failure and edge-case behavior
• Capture without Electron #15 — Capture without Electron
• The host runtime #16 — The host runtime
• The answer log #17 — The answer log
• The stream contract #18 — The stream contract
• The control surface #20 — The control surface
• The web client #21 — The web client
• Capture session lifetime across manual triggers #23 — Capture session lifetime across manual triggers

---

Generated by Claude Code