feat: Playwright skills (screenshot, browser) + scrape backend

README.md

@@ -69,8 +69,13 @@ These handle concerns devkit doesn't — methodology, specialized reviews, and c
 ```bash
 brew install rtk       # Token optimization (60-90% savings on Bash output)
 brew install ast-grep  # AST-based repo mapping (used by onboard skill)
+
+# Browser automation — enables scrape (JS-rendered), screenshot, and browser skills
+npx playwright install chromium
 ```
 
+**Playwright** (optional) enables three skills: enhanced `scrape` for JS-heavy sites, `screenshot` for page captures, and `browser` for full automation (clicking, form filling, multi-step flows, codegen). Free and local — no API keys. Install only the browsers you need (`chromium` is ~170MB).
+
 ### Verify
 
 ```bash
@@ -173,6 +178,8 @@ Skills activate automatically based on context. No slash command needed.
 | "research X" | `research` |
 | "deep research", "validate this" | `deep-research` |
 | "scrape this URL" | `scrape` |
+| "screenshot this page" | `screenshot` (requires Playwright) |
+| "automate this browser flow" | `browser` (requires Playwright) |
 | "create an ADR" | `adr` |
 
 Coding principles (`clean-code`, `dry`, `yagni`, `dont-reinvent`, `executing`, `stuck`, `scratchpad`) are injected as condensed rules (~120 tokens) per workflow step — not loaded as full skill files.
@@ -264,7 +271,7 @@ Terminal fallback (devkit workflow run <name>):
 ```
 devkit/
 ├── commands/          # 6 slash commands (tab-completable entry points)
-├── skills/            # 20 context-activated skills + _principles.yml
+├── skills/            # 22 context-activated skills + _principles.yml
 ├── agents/            # 6 agents (reviewer, researcher, improver, ...)
 ├── hooks/             # 12 hooks (safety, security, quality gates, workflow enforcement)
 ├── workflows/         # 18 YAML workflow definitions

ROADMAP.md

@@ -5,7 +5,7 @@
 - **MCP engine** — Go server exposes `devkit_start`, `devkit_advance`, `devkit_status`, `devkit_list` tools inside Claude Code. Step ordering enforced via MCP tool scoping + PreToolUse hook exit 2. Session state in session.json (hot path, <50ms hook reads) + SQLite (cold history). ~65% token reduction vs old monolithic prompts.
 - **6 slash commands** — Tab-completable entry points for things that need explicit invocation (tri-review, tri-debug, tri-security, status, setup-rules, workflow); 18 former commands now context-activated via skills or invoked via MCP tools (pr-monitor folded into pr-ready workflow; pr-ready is now a natural-language skill)
 - **Deterministic workflow conversion** — All command logic moved from LLM-interpreted markdown to Go-engine-driven YAML workflows; ~3,600 lines of inline logic removed
-- **20 context-activated skills** — 10 auto-trigger workflows (test-gen, doc-gen, changelog, onboard, research, deep-research, scrape, autoloop, adr, pr-ready) + 6 coding principles (executing, clean-code, DRY, YAGNI, dont-reinvent, stuck) + 2 tools (gcli, creating-workflows) + 1 iteration memory (scratchpad) + 1 orchestration (mega-pr)
+- **22 context-activated skills** — 10 auto-trigger workflows (test-gen, doc-gen, changelog, onboard, research, deep-research, scrape, autoloop, adr, pr-ready) + 6 coding principles (executing, clean-code, DRY, YAGNI, dont-reinvent, stuck) + 2 tools (gcli, creating-workflows) + 1 iteration memory (scratchpad) + 1 orchestration (mega-pr) + 2 browser automation (screenshot, browser)
 - **6 agents** — Scoped tool access, worktree isolation, model assignment
 - **12 hooks** — Safety (destructive command blocking, edit-time security patterns, PR gate), observability (audit trail, slop detection, post-validation, subagent verification, language-aware code review), optimization (RTK token compression), workflow enforcement (devkit-guard, devkit-stop-guard)
 - **Graceful degradation** — tri:* commands work with 1-3 agents depending on installed CLIs

skills/browser/SKILL.md

@@ -0,0 +1,161 @@
+---
+name: browser
+description: Automate a web browser — use when asked to click buttons, fill forms, navigate multi-step flows, extract data from JS-rendered pages, log into a site, record user interactions, or test a web app in a real browser. Use for anything beyond simple article scraping.
+---
+
+# Browser Automation
+
+Drive a real browser via Playwright to interact with pages — click, fill forms, extract data from SPAs, run multi-step flows, or record user interactions.
+
+Use this skill for anything beyond simple article scraping: JS-heavy data extraction, form filling, multi-step auth flows, codegen recording, or E2E testing. For static article→markdown use `scrape`; for pure image capture use `screenshot`.
+
+## Step 1: Verify Playwright
+
+Run `npx playwright --version`. If it fails, stop and tell the user: `Playwright required — install with: npx playwright install chromium`. Do not attempt workarounds.
+
+## Step 2: Parse the Request
+
+- **Target URL(s)** — starting page (validate per Rules before launching)
+- **Actions** — navigate, click, fill, extract, wait
+- **Data** — what fields, what format (JSON usually)
+- **Auth** — are credentials needed? Read from env, never hardcode
+- **Repeatability** — one-off throwaway, or save script for reuse?
+
+If the request is vague ("scrape this site"), clarify which fields, whether login is needed, and whether it'll be re-run.
+
+## Step 3: Choose the Right Mode
+
+### Mode A — Codegen (recording)
+
+When the user wants to figure out selectors or hand off a repeatable flow:
+
+```bash
+npx playwright codegen {url}
+```
+
+Opens a browser. User interacts. Playwright prints the equivalent script. Best for:
+- Complex pages where selectors aren't obvious
+- Building reusable flows
+- Teaching users how Playwright works
+
+### Mode B — Inline script (one-off)
+
+Write the script to a file that reads args from `process.argv`, then pass values via shell variables:
+
+```bash
+cat > /tmp/devkit-browser-flow.mjs <<'EOF'
+import { chromium } from 'playwright';
+
+const url = process.argv[2];
+if (!url) { console.error('usage: node devkit-browser-flow.mjs <url>'); process.exit(2); }
+
+const browser = await chromium.launch();
+let exitCode = 0;
+try {
+  const page = await browser.newPage();
+  await page.goto(url, { waitUntil: 'networkidle', timeout: 30000 });
+
+  // Example: extract items from a JS-rendered list
+  const data = await page.evaluate(() =>
+    Array.from(document.querySelectorAll('.item')).map(el => ({
+      title: el.querySelector('h3')?.textContent?.trim(),
+      link: el.querySelector('a')?.href,
+    }))
+  );
+
+  console.log(JSON.stringify(data, null, 2));
+} catch (err) {
+  console.error('flow failed:', err.message);
+  exitCode = 1;
+} finally {
+  try { await browser.close(); } catch (e) { /* suppress close errors so the real error propagates */ }
+}
+process.exit(exitCode);
+EOF
+
+URL=https://example.com node /tmp/devkit-browser-flow.mjs "$URL"
+```
+
+### Mode C — Persistent test (for web apps)
+
+If the user is building a web app and wants E2E tests:
+
+```bash
+# Scaffold Playwright test framework
+npm init playwright@latest
+
+# Generates playwright.config.ts and tests/ directory
+# Run with:
+npx playwright test
+```
+
+Then write test specs in `tests/*.spec.ts`.
+
+### Mode D — Form fill + auth flow
+
+Same pattern as Mode B, plus `page.fill` / `page.click` for the login form. Credentials come from env, never from hardcoded strings or the command line:
+
+```bash
+cat > /tmp/devkit-browser-auth.mjs <<'EOF'
+import { chromium } from 'playwright';
+
+const loginUrl = process.argv[2];
+const targetUrl = process.argv[3];
+if (!loginUrl || !targetUrl) {
+  console.error('usage: node devkit-browser-auth.mjs <login_url> <target_url>');
+  process.exit(2);
+}
+
+const browser = await chromium.launch();
+let exitCode = 0;
+try {
+  const page = await browser.newPage();
+  await page.goto(loginUrl);
+  await page.fill('input[name="email"]', process.env.EMAIL);
+  await page.fill('input[name="password"]', process.env.PASSWORD);
+  await page.click('button[type="submit"]');
+  await page.waitForURL('**/dashboard');
+
+  await page.goto(targetUrl);
+  const content = await page.content();
+  console.log(content);
+} catch (err) {
+  console.error('auth flow failed:', err.message);
+  exitCode = 1;
+} finally {
+  try { await browser.close(); } catch (e) { /* suppress close errors */ }
+}
+process.exit(exitCode);
+EOF
+
+EMAIL="$EMAIL" PASSWORD="$PASSWORD" node /tmp/devkit-browser-auth.mjs "$LOGIN_URL" "$TARGET_URL"
+```
+
+## Step 4: Report
+
+After running, report:
+
+- What was done (pages visited, actions performed)
+- What was extracted (or where it was saved)
+- Any failures (timeouts, missing elements, auth errors)
+- The script file path (if saved for reuse)
+
+## Rules
+
+- **URL validation** — only accept `http://` and `https://`. Reject `file://`, `ftp://`, `data:`, and all other schemes. Reject private/reserved IPs: `localhost`, `127.0.0.1`, `0.0.0.0`, `169.254.x.x` (cloud metadata — AWS/GCP/Azure), `10.x.x.x`, `172.16-31.x.x`, `192.168.x.x`, `[::1]` — unless the user is explicitly testing localhost. Reject URLs containing `@` (credential-in-URL attacks). On rejection: **report the exact reason and stop** — never silently skip.
+- **Credentials** — never hardcode. Read from env or ask the user. Never log them.
+- **No raw interpolation** — write scripts to files that read args from `process.argv`, then pass values via shell variables. Never build via `-e "..."` with interpolated user input.
+- **Always close the browser safely** — use `try/catch/finally` with the `close()` call in its own try/catch, so a close-time error never masks the real error.
+- **Headless by default** — only use `headless: false` when user explicitly wants it (e.g., codegen, debugging).
+- **Respect the site** — no CAPTCHA bypass, no hammering, no scraping the user doesn't own.
+- **Default timeout 30s** — if slow, use `page.waitForSelector()` rather than longer fixed waits, and explain why.
+
+## Troubleshooting
+
+| Symptom | Fix |
+|---|---|
+| `"Executable doesn't exist"` | `npx playwright install chromium` |
+| Timeout exceeded | **Report the timeout to the user first with the URL and selector** — do not retry silently. Then investigate: wrong selector, page slow, content gated by JS that never fires. Use `waitForSelector()` with a sensible timeout rather than longer fixed waits. |
+| Element not visible | `await locator.scrollIntoViewIfNeeded()` before action |
+| Site blocks headless | Try user agent override; last resort: `headless: false` |
+| Flaky data extraction | Use `networkidle` wait condition instead of `domcontentloaded` |

skills/scrape/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: scrape
-description: Scrape a URL to clean Markdown — use when asked to scrape, fetch, extract content from, or read a webpage and convert it to Markdown. Uses Jina Reader, Firecrawl, or WebFetch.
+description: Scrape a URL to clean Markdown — use when asked to scrape, fetch, extract content from, or read a webpage and convert it to Markdown. Uses Jina Reader, Playwright, Firecrawl, or WebFetch.
 ---
 
 # Web Scrape to Markdown
@@ -13,7 +13,7 @@ Fetch a URL and convert it to clean, LLM-ready Markdown. Supports multiple backe
 /devkit:scrape https://example.com
 /devkit:scrape https://example.com --json
 /devkit:scrape https://example.com https://other.com
-/devkit:scrape https://example.com --backend firecrawl
+/devkit:scrape https://example.com --backend playwright
 ```
 
 ## Arguments
@@ -24,9 +24,10 @@ Fetch a URL and convert it to clean, LLM-ready Markdown. Supports multiple backe
 
 ## Backends (in priority order)
 
-1. **Jina Reader** — prepend `https://r.jina.ai/` to the URL. Returns Markdown by default. Use if `JINA_API_KEY` is set (higher rate limits) or anonymously (~20 RPM).
-2. **Firecrawl** — use if `FIRECRAWL_API_KEY` is set. Best for JS-heavy sites and anti-bot bypass.
-3. **WebFetch fallback** — use Claude's built-in `WebFetch` tool. No API key needed, but returns raw content (less clean).
+1. **Jina Reader** — prepend `https://r.jina.ai/` to the URL. Returns Markdown by default. Use if `JINA_API_KEY` is set (higher rate limits) or anonymously (~20 RPM). Best for articles and docs.
+2. **Playwright** — use if `npx playwright --version` succeeds (optional dep, install with `npx playwright install chromium`). Best for JS-heavy SPAs and sites that block headless scrapers. Free and local — no API keys.
+3. **Firecrawl** — use if `FIRECRAWL_API_KEY` is set. Paid API. Good for anti-bot bypass when Playwright isn't enough.
+4. **WebFetch fallback** — use Claude's built-in `WebFetch` tool. No API key needed, but returns raw content (less clean).
 
 ## Execution
 
@@ -45,6 +46,43 @@ This returns: { "url": "...", "title": "...", "content": "..." }
 where "content" is the Markdown.
 ```
 
+**Playwright:**
+
+Validate the URL first (see Rules: http/https only, no private IPs, no `@`).
+
+Check availability: `npx playwright --version`. If not installed, tell the user once per session: `Playwright not installed. Install with: npx playwright install chromium`, then fall through to the next backend. The winning backend is always reported in Step 3 (Error Handling), so the user sees which one served the result.
+
+Never build scripts via inline `-e "..."` strings. Write the script to a file that reads the URL from `process.argv[2]`, then invoke it:
+
+```bash
+cat > /tmp/devkit-scrape.mjs <<'EOF'
+import { chromium } from 'playwright';
+
+const url = process.argv[2];
+if (!url) { console.error('usage: node devkit-scrape.mjs <url>'); process.exit(2); }
+
+const browser = await chromium.launch();
+try {
+  const page = await browser.newPage();
+  await page.goto(url, { waitUntil: 'networkidle', timeout: 30000 });
+  const title = await page.title();
+  const html = await page.content();
+  process.stdout.write(JSON.stringify({ title, html }));
+} catch (err) {
+  console.error('playwright failed:', err.message);
+  process.exit(1);
+} finally {
+  try { await browser.close(); } catch (e) { /* suppress close errors */ }
+}
+EOF
+
+node /tmp/devkit-scrape.mjs "$URL"
+```
+
+Pass the URL as a shell variable (`URL=https://example.com`), never inline into the command string. The `.mjs` reads it from `process.argv[2]` so quotes, backticks, and `$()` in the URL can't escape.
+
+Then convert HTML → Markdown. Prefer a local converter if available (pandoc, turndown). If none available, strip script/style/nav/footer tags and extract text from article/main/body.
+
 **Firecrawl (if FIRECRAWL_API_KEY is set and --backend firecrawl):**
 ```
 Use Bash to call (use jq to safely construct JSON — never interpolate URLs directly):
@@ -72,9 +110,11 @@ When given multiple URLs, scrape them in parallel:
 
 ### Error Handling
 
-- If Jina Reader returns an error or empty content, fall back to WebFetch
-- If a URL is unreachable, report the error and continue with remaining URLs
-- Never silently drop a URL — always report what happened
+**Fallback chain (single linear order):** Jina Reader → Playwright → Firecrawl → WebFetch. Each backend is tried in order. Move to the next only on error, empty content, or missing dependency.
+
+- **Report which backend actually ran** for each URL, and (if fallbacks happened) why the earlier ones failed. Never silently substitute content — the user must know a paywall/cookie-wall/SPA-skeleton from one backend wasn't the real page fetched by another.
+- If a URL is unreachable by every backend, report the error and continue with remaining URLs.
+- Never silently drop or substitute a URL — always report what happened, with the winning backend named.
 
 ## Output
 
@@ -106,8 +146,8 @@ For multiple URLs, output a JSON array.
 
 ## Rules
 
-- **URL validation** — only accept `http://` and `https://` URLs. Reject `file://`, `ftp://`, `data:`, and all other schemes. Reject URLs targeting private/reserved IPs: `localhost`, `127.0.0.1`, `0.0.0.0`, `169.254.x.x`, `10.x.x.x`, `172.16-31.x.x`, `192.168.x.x`, `[::1]`. Reject URLs containing `@` (credential-in-URL attacks).
-- **No shell injection** — never interpolate user URLs directly into shell command strings. Use `jq` to construct JSON payloads for curl.
+- **URL validation** — only accept `http://` and `https://` URLs. Reject `file://`, `ftp://`, `data:`, and all other schemes. Reject URLs targeting private/reserved IPs: `localhost`, `127.0.0.1`, `0.0.0.0`, `169.254.x.x` (cloud metadata — AWS/GCP/Azure), `10.x.x.x`, `172.16-31.x.x`, `192.168.x.x`, `[::1]`. Reject URLs containing `@` (credential-in-URL attacks). When rejecting, report the exact reason and stop — never silently skip a URL in a batch.
+- **No shell injection** — never interpolate user URLs directly into shell command strings. Use `jq` to construct JSON payloads for curl. For Playwright, write scripts to files that read URLs from `process.argv`, and pass the URL as a shell variable — never inline into `-e` strings.
 - **API keys from env only** — never hardcode API keys. Always reference `$JINA_API_KEY`, `$FIRECRAWL_API_KEY` from environment variables.
 - Always try Jina Reader first — it's free and produces the cleanest output
 - Respect rate limits — if scraping many URLs, add a brief pause between requests