Screen Solver v1: implementation spec

[jpka]

Part of #1

Problem Statement

When working through coding exercises (LeetCode, Exercism, Advent of Code, HackerRank, and similar kata sites), the user has to break their flow to reason through an approach and write it out by hand — there's no fast way to get a second opinion on what's on screen without alt-tabbing to a separate tool, describing the problem in prose, and losing the visual context of the editor pane and sample tests. The user also wants to glance at that answer from a device other than the one running the exercise — a phone on the same desk — without setting up screen sharing or giving the tool any ability to act on the page itself.

Solution

A Windows 11 local server (Electron main process) watches one browser window that the user names explicitly. Clicking "Solve now" in a browser-based web client — open on the same desktop or on a phone — captures that window via Windows.Graphics.Capture, sends the image to Claude with a coding-exercise-tuned system prompt, and streams the answer live to every client that has the web page open. Every attempted call is logged locally regardless of outcome, so cost is fully visible even though nothing is capped. The app never types into or otherwise acts on the watched page — it only reads and answers.

User Stories

Setup & configuration

1. As a user, I want the app to refuse to start if ANTHROPIC_API_KEY isn't set, so I never end up running the app in a state where a solve fails on auth after I've already started working.
2. As a user, I want my API key read once from an environment variable and never written to disk, sent over IPC, or logged, so my key isn't exposed at rest or in transit within the app.
3. As a user, I want my settings (target window, provider/model) stored in one config.json under my user data folder, so they survive an app restart.
4. As a user, I want every setting to apply live with no restart required, so I can change the target window mid-session without losing anything.

Picking a window

5. As a user, I want to pick which browser window is being watched from a list of open windows, so the app only ever sees the tab I'm actually solving in.
6. As a user opening the web client before any target window is configured, I want to see a window picker in place of the answer pane, so it's obvious what to do first.
7. As a user, I want the target window identified by process name plus title rather than a raw OS handle, so the app can re-find it across restarts even though handles aren't stable.
8. As a user whose target window unexpectedly disappears mid-run, I want one silent re-resolution attempt before the app falls back to showing the picker, so a transient loss doesn't interrupt a working session for nothing.
9. As a user who changes the target window from the client, I want that change broadcast to every connected client immediately, so no client is left watching a stale target.

Triggering a solve

10. As a user looking at a coding exercise, I want to click a single "Solve now" button in the web client, so I never have to leave the page or take my own screenshot.
11. As a user, I want "Solve now" to always be accepted and to interrupt whatever answer is currently streaming, so I'm never blocked by a stale in-flight request.
12. As a user who clicks "Solve now" with no target window configured, I want a clear rejection rather than a silent no-op, so I know to pick a window first.

Capture correctness

13. As a user, I want the capture to be one held session for the current target rather than a fresh grab per click, so the yellow capture-indicator border doesn't flicker on and off with every solve.
14. As a user, I want the captured frame downscaled but never cropped, so a partial answer is never caused by the app itself cutting off part of the problem.
15. As a user whose target window is minimized or sitting on another virtual desktop, I want the app to refuse to spend a call on a stale or black frame, so I'm never billed for a screenshot of nothing.
16. As a user, I want a black or zero-size captured frame detected before the model is ever called, so a transient rendering glitch doesn't cost me money for no answer.

Answer streaming & display

17. As a user, I want the model's answer to stream in as it's generated, so I get useful signal well before the full answer finishes.
18. As a user, I want the answer to lead with a heading naming the exercise, then the complete solution in a fenced code block, then a short explanation, so I can grab the code first and read the reasoning after.
19. As a user on a screen with no exercise open, I want the app to say so plainly instead of spending a call inventing an answer, so I'm never misled by a hallucinated solution.
20. As a user whose screenshot is missing something load-bearing (a cut-off constraint, a scrolled-off signature), I want a single line telling me what was assumed, so I know exactly what to double-check.
21. As a user who triggers a new solve while a previous answer is still streaming, I want that previous partial answer pushed into history (marked interrupted) rather than discarded, so I can still see how far it got.

History & persistence

22. As a user, I want every completed or interrupted answer to persist locally, so a client that connects later still sees everything that happened while it was away.
23. As a user, I want a client that connects mid-answer to receive the text streamed so far, so I don't miss the start of an answer just because I opened the client late.
24. As a user, I want a bail or a failed solve to not clutter my visible answer history, while still being recorded for cost tracking, so my history stays a list of actual answers.
25. As a user, I want to see the cost of every attempted call, including failed and bailed ones, so I have full cost visibility even with nothing capped or enforced.

Multiple clients / multiple devices

26. As a user, I want to open the web client from my phone on the same network and see the same live stream as my desktop, so I can check an answer without a second monitor.
27. As a user with two clients open at once, I want both to show identical live state, so there's no "primary" client that's more current than the other.
28. As a user whose phone locks and later reconnects, I want that reconnect to happen automatically and pick up wherever the stream currently is, so I never have to manually refresh.
29. As a user on my phone in portrait, I want a single continuous feed — the live answer expanded at top, past answers collapsed to one line each — so history never needs a separate drawer or overlay.
30. As a user on my phone in landscape, I want a narrow list rail next to the answer pane, so I can jump between past answers without losing the live one.
31. As a user who rotates my phone mid-answer, I want the layout to swap live with no reload and no interruption to the stream, so rotating never costs me my place.
32. As an Android Chrome user, I want a fullscreen button that actually works, so I can view answers without browser chrome around them.
33. As an iPhone Safari user, I want the fullscreen button to render visibly disabled rather than fail silently, so I'm not left wondering why nothing happened.

Failure & edge cases

34. As a user whose API key is revoked after the app already started, I want that failure surfaced clearly and to stay visible, since it will keep failing on every future click until I fix it.
35. As a user, if the server can't bind its port, I want the app to refuse to start with a clear message, so I find out immediately rather than by a client silently failing to connect.
36. As a user whose answer stream dies mid-answer, I want the partial text to stay visible with an error marker appended, so I don't lose what had already streamed.
37. As a user, I want transient provider errors (rate limit, overload, brief network blip) retried automatically without my involvement, so a flaky moment doesn't force me to re-click.
38. As a user, I want a standing status indicator for whether anything is currently wrong with the app as a whole, so I don't have to infer app health from the content of one answer.

Extensibility & cost

39. As a developer extending this app later, I want the model call behind one provider interface, so adding a second vision provider touches a single module rather than a rewrite.
40. As a user, I want prompt caching enabled on the fixed system prompt, so repeated solves don't re-bill the same prompt tokens every time.

Scope discipline

41. As a user, I want the app strictly read-only — it never types into or otherwise acts on the page it watches — so I keep full control over what actually gets submitted on the exercise site.

Implementation Decisions

Process architecture. Electron main process plus exactly one hidden BrowserWindow (never shown to the user). Main owns everything decision-related: the HTTP server, request routing, window-resolution and black-frame guards, the provider call, answer-log writes, and SSE broadcast. The hidden renderer does mechanism only — turn a desktopCapturer source into a live MediaStream via getUserMedia, draw it to a canvas, downscale to 1568px on the long edge, encode, and hand the bytes to main over IPC. No decision logic runs in the renderer; this is what makes the decision layer testable in plain Node.

Process lifecycle. requestSingleInstanceLock() is the authoritative single-instance guard (not port-bind-as-lock), keyed on userData, so two checkouts pointed at the same state root are correctly treated as one instance. A missing ANTHROPIC_API_KEY or a port-bind failure both mean refuse to start.

State root. app.getPath('userData'), resolving to %APPDATA%\screen-solver\. Holds config.json, answers.jsonl, and usage.jsonl.

API key handling. ANTHROPIC_API_KEY env var only — no encrypted blob, no masked input UI anywhere in the app. Deleted from process.env before the hidden renderer is created, so the key is host-process-only by construction. A missing key means refuse to start.

Config shape. One config.json: target window identity (process name + title, not OS handle) and provider/model selection. Fully live-reloadable — every field can change with no restart, and a change broadcasts to all connected clients.

Capture session lifecycle. One long-lived WGC session per target window, not one per click. It opens the moment a target is selected (the same event that broadcasts config{target} to clients) — not at server startup, not deferred to first solve. A target change tears down the old session and opens the new one immediately under the same rule. The session stays open while idle; it is not opened and closed per solve. The OS-drawn yellow capture border is the only host-side "watching" indicator and must stay honest — lit exactly when a session is open, silent about individual solves.

Provider seam. createProvider(config) → Provider, where Provider.solve(image, {signal}) → AsyncIterable<Event>. Event is one of delta{text}, done{usage}, or error{kind} with kind ∈ {auth, refusal, transient}. The system prompt is configured once at construction, not per call. Transient errors (rate limit, overload, network) retry internally inside the seam; auth and refusal surface immediately without retry. Budget accounting, capture, and window resolution all stay outside the seam — it wraps the model call and nothing else. This is the module a second provider would replace.

Model defaults. claude-sonnet-5, effort: medium, max_tokens: 8000, capture downscaled to 1568px with no crop. Chosen conservative because effort moves cost more than image size does — thinking bills at the output rate. The 8000-token ceiling is in practice a thinking budget (measured visible answers run 255–305 tokens); truncation is made detectable via stop_reason rather than assumed not to happen.

HTTP surface. Three endpoints:

• POST /solve — synchronous validation only (is a target window configured); always interrupts any in-flight solve, never rejects as busy. Returns 202 on accept, 400 with no target configured, 503 if the server itself isn't ready to accept a solve.
• GET /answers — the full answers.jsonl backlog as a JSON array, fetched once on page load, independent of the live connection.
• GET /events — Server-Sent Events, one shared broadcast stream, no per-client filtering, no auth.

SSE event vocabulary. start (a new solve begins), delta{text} (streamed answer text), done{usage} (terminal, success), error{kind} (terminal, failure), sync{text} (sent only to a client connecting mid-flight, carrying the accumulated text so far in place of start), config{target} (target window changed, broadcast to everyone). The server keeps the in-flight accumulated text in memory to construct sync. EventSource's built-in reconnect is relied on directly — a reconnect is treated identically to a fresh mid-flight join, so no Last-Event-ID replay logic exists.

Answer log (answers.jsonl). One JSON object per line, written only for done and interrupted outcomes — never for a bail or an error. Fields: title (the answer's # heading), final answer text only (no delta history, no image), timestamp, model, usage, source window identity, and an interrupted: true tag where applicable. In-flight text is memory-only and written once at the terminal event; a mid-stream crash loses that answer's text, accepted as a rare, user-recoverable failure.

Usage log (usage.jsonl). One line per attempted call, every outcome including bail and error, always recording usage and cost regardless of content. This is pure observability — nothing reads the running total to cap or throttle anything. All previously-considered enforcement (near-duplicate suppression, a runaway breaker, a cooldown, an hourly ceiling, a daily spend cap) is explicitly not implemented.

System prompt. Answer-first contract: a single # heading naming the exercise, one fenced code block (paste-ready over the site's starter scaffolding, matching its exact visible function/class signature, no test harness or demo calls), then two to three short prose paragraphs on approach and complexity. A literal # No exercise on screen heading is the entire v1 "is there a problem here" detector — no separate pre-call detection pass exists; that bail still spends a call and is recorded in usage.jsonl but not answers.jsonl. A single > **Missing:** … line, placed before the code block, fires at most once when something load-bearing was off-screen; a determinate-but-cropped screen is solved silently with no crop caveat. Prompt caching is on, ttl: 1h, since the measured prompt (1196–1496 tokens) sits above the model's 1024-token minimum cacheable prefix.

Window capture. Windows.Graphics.Capture via Electron's desktopCapturer (already WGC-backed) — not PrintWindow, which is documented to return nonzero-success black frames on GPU-composited Chrome/Edge with no error signal, making failure silent and, for a per-call-billed app, a silent bill. The renderer turns a desktopCapturer source into a live stream via getUserMedia, not a one-shot thumbnail grab.

Not-capturable / bad-frame detection. Gate captures on isMinimized() (IsIconic). Treat "vanished from window enumeration" as ambiguous between closed and moved-to-another-desktop — this triggers the re-resolution flow, not an immediate hard failure. Check a non-black-pixel ratio on the captured frame before spending a model call. A failing pre-flight check is a silent no-spend: a button flash only, no SSE error event, since no money was spent.

Failure taxonomy and surfaces:

Condition	Spends a call?	Surface
Window gone/unresolvable, minimized/off-desktop, black/zero-size frame	No	Silent — button flash only, no SSE event
Bail (# No exercise on screen)	Yes	Normal done stream, rendered low-emphasis in the client; usage.jsonl entry, no answers.jsonl entry
Auth rejection (key revoked after startup)	Yes (attempted)	error{kind: 'auth'}; also flips the standing status pill sticky, since it will recur on every future click
Transient error, retries exhausted	Yes (attempted)	error{kind: 'transient'}
Stream dies mid-answer	Partially	Partial pane text stays visible with an appended error marker
Mid-run target loss	N/A	Three-way split on an app-tracked intent flag: deliberate pause → ignored; unexpected loss → one silent re-resolution, then fallback to the picker; renderer crash → auto-restart, escalating on repeat
Port-bind failure	N/A	Refuse to start, same as a missing key

Status pill ladder: silent → auto-recovering → sticky. Sticky states also print one line to the host's console, for whoever is watching the terminal; there is no push/toast/notification for anyone not currently looking at an open client.

Web client layout. Orientation drives layout directly off innerWidth > innerHeight (not matchMedia/orientationchange, which are demoted to mere change triggers, since a compound media query can silently fail to match), with a 480px floor so very narrow phones still get portrait treatment. Portrait = continuous log: one feed, live answer expanded and outlined at top, past answers collapsed to a line each, tap to expand in place. Landscape = split rail: a 132px list rail plus the answer pane. A rotation mid-session normalizes state across the swap — drop the pane-mode concept the rail has and the log doesn't, keep whichever entry was open, land it as an expanded card — so the live entry is never shown with a history-colored indicator.

Connection-state indicator. Two signals collapsed onto one indicator: live socket state while watching the live stream, replaced wholesale by a "viewing history" label when reading a past entry. The sync{text} catch-up window shows as a transient "syncing…" tag and nothing more.

Fullscreen. Feature-detected per platform: works on Android Chrome (the confirmed target device); renders visibly disabled, not silently broken, where the Fullscreen API doesn't support arbitrary elements (iPhone Safari).

Reach and trust. No auth, no host-only restriction on the web client — v1 is personal-use-only. Bind address, port-collision handling, and firewall messaging are left as whatever the simplest default turns out to be; this is a deliberate non-decision, not an oversight. The API key remains the one host-only exception, unaffected by this.

Testing Decisions

A good test here exercises externally observable behavior — HTTP responses, the SSE event sequence, and the resulting answers.jsonl/usage.jsonl contents — never internal call graphs, IPC message shapes, or Electron plumbing.

Primary seam: the local HTTP server. Run the server module with a fake capture function (returns canned image buffers, or a "black frame"/"minimized" signal on demand) and a fake Provider (canned delta/done/error sequences with controllable timing) injected in place of the real ones, against a temp userData directory per test. Drive it exactly like a real client — POST /solve, read GET /events, GET /answers — and assert on the wire events and the resulting JSONL files. Cases drawn directly from the user stories above:

• Happy path: start → delta* → done, one answers.jsonl entry written, matching usage.jsonl entry.
• No target configured: POST /solve returns 400, no SSE traffic emitted.
• Interrupt-and-replace: second POST /solve while one is in flight marks the first interrupted in answers.jsonl, no data lost.
• Mid-flight join: a client connecting to /events after start has fired receives sync{text}, not start.
• Bail: usage.jsonl entry written, no answers.jsonl entry, low-emphasis marker present on the wire.
• Auth error: error{kind:'auth'} on the wire, no answers.jsonl entry, usage.jsonl entry recorded, standing status goes sticky.
• Transient-then-recovered: no error event surfaces to the client; exactly one usage.jsonl entry reflects the eventual successful outcome.
• Pre-flight failure (minimized/black frame): no SSE traffic at all, no usage.jsonl entry (no call was ever attempted).
• Target-window loss mid-run: re-resolution-then-fallback behavior per the intent-flag rule.
• Multi-client fan-out: two simultaneous /events connections observe an identical event sequence.
• Reconnect mid-answer: a fresh connection receives sync, not a full history replay.

Secondary seam: the provider module. Unit tests for createProvider/solve in isolation, against a fake Anthropic transport (canned stream chunks, canned error responses). Covers the delta/done/error normalization and specifically the retry-vs-surface-immediately split — transient errors retry silently inside the seam; auth/refusal surface on first occurrence with no retry.

Prior art. None yet — per package.json, this repo currently ships no product code ("Wayfinder map and spec... No product code yet"), so there's no existing suite or framework convention to match. Test framework choice (e.g. node:test vs. an added dependency) is left to whoever picks up implementation, consistent with the project's Node 24.13 / npm-only toolchain (no Python, .NET, or Rust installed).

Explicitly not unit-tested. The hidden-renderer capture mechanism (WGC session open/close, getUserMedia → canvas → downscale → encode) needs a real window and a real composited desktop, so it stays manual/E2E-verified against the real target sites during implementation — carrying forward four findings already banked from prior research: gate on isMinimized(), treat "vanished from enumeration" as ambiguous rather than definitive, expect pixel-dimension shifts across per-monitor DPI boundaries, and check a non-black-pixel ratio before spending a call. The web client's orientation-swap behavior and fullscreen feature-detection are likewise manual-only — both were already validated against a real phone during prototyping and are viewport/DOM-driven rather than server-logic-driven.

Out of Scope

• Auto-typing answers into the browser, or clipboard copy of the answer — strictly read-only was chosen deliberately.
• Quizzes, puzzles, crosswords, or general Q&A — katas are the only v1 target.
• macOS and Linux — Windows 11 only.
• A local/on-device model as the v1 provider — the seam shouldn't preclude one later, but v1 doesn't ship it.
• Any non-Anthropic vision provider for v1.
• Silently degrading to a cheaper model when a budget is hit — there is no budget enforcement to degrade from; switching models stays a deliberate, explicit setting.
• Packaging, installers, auto-start, and updates — the app runs from a local checkout.
• Remote or internet access — tunnels, port forwarding, a hosted relay. "Local server" means the host machine and the network it's already on.
• Client URL discovery — mDNS, QR codes, broadcast beacons. Whoever opens the web client is assumed to already know the address.
• Native mobile or desktop app clients — a browser is the only client.
• Keeping an Electron output window alongside the web client — the web client replaces it.
• A dedicated bind address / port-collision / auth / TLS / firewall-messaging decision — ships with whatever the simplest default turns out to be.
• Automatic change detection or interval-based triggering — the loop never runs unattended; a human clicking "Solve now" is the only trigger.
• Budget enforcement of any kind (near-duplicate suppression, a runaway circuit breaker, a cooldown, an hourly ceiling, a daily spend cap) — usage.jsonl is visibility only, uncapped.

Further Notes

This spec synthesizes all 16 closed decision tickets under the wayfinder map (#1) into one implementation-ready document, per the map's own definition of done ("nothing left to decide before someone starts coding"). The full decision trail, with citations and dissenting/superseded findings, lives in the map issue and its closed children (#2–#23); prototypes and measured findings referenced above live under .scratch/solver/ on main, with the web client's rejected layout variants on the prototype/21-web-client branch.

Two threads the map explicitly left open, worth watching during implementation but not blocking it: stale-vs-black frame behavior specifically for minimized/other-desktop windows (flagged open under #2), and the exact bind/port default (deliberately deferred under #19, see Out of Scope).

No CONTEXT.md exists yet in this repo, so the domain vocabulary used throughout this spec (focus pane, bail, status pill, capture session, intent flag) is drawn directly from the map and its tickets rather than a settled glossary — /domain-modeling is the natural next step to formalize it once implementation starts.