diff --git a/README.md b/README.md index c3aa995..966c0fe 100644 --- a/README.md +++ b/README.md @@ -69,8 +69,13 @@ These handle concerns devkit doesn't — methodology, specialized reviews, and c ```bash brew install rtk # Token optimization (60-90% savings on Bash output) brew install ast-grep # AST-based repo mapping (used by onboard skill) + +# Browser automation — enables scrape (JS-rendered), screenshot, and browser skills +npx playwright install chromium ``` +**Playwright** (optional) enables three skills: enhanced `scrape` for JS-heavy sites, `screenshot` for page captures, and `browser` for full automation (clicking, form filling, multi-step flows, codegen). Free and local — no API keys. Install only the browsers you need (`chromium` is ~170MB). + ### Verify ```bash @@ -173,6 +178,8 @@ Skills activate automatically based on context. No slash command needed. | "research X" | `research` | | "deep research", "validate this" | `deep-research` | | "scrape this URL" | `scrape` | +| "screenshot this page" | `screenshot` (requires Playwright) | +| "automate this browser flow" | `browser` (requires Playwright) | | "create an ADR" | `adr` | Coding principles (`clean-code`, `dry`, `yagni`, `dont-reinvent`, `executing`, `stuck`, `scratchpad`) are injected as condensed rules (~120 tokens) per workflow step — not loaded as full skill files. @@ -264,7 +271,7 @@ Terminal fallback (devkit workflow run ): ``` devkit/ ├── commands/ # 6 slash commands (tab-completable entry points) -├── skills/ # 20 context-activated skills + _principles.yml +├── skills/ # 22 context-activated skills + _principles.yml ├── agents/ # 6 agents (reviewer, researcher, improver, ...) ├── hooks/ # 12 hooks (safety, security, quality gates, workflow enforcement) ├── workflows/ # 18 YAML workflow definitions diff --git a/ROADMAP.md b/ROADMAP.md index 0871b88..ef58b0e 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -5,7 +5,7 @@ - **MCP engine** — Go server exposes `devkit_start`, `devkit_advance`, `devkit_status`, `devkit_list` tools inside Claude Code. Step ordering enforced via MCP tool scoping + PreToolUse hook exit 2. Session state in session.json (hot path, <50ms hook reads) + SQLite (cold history). ~65% token reduction vs old monolithic prompts. - **6 slash commands** — Tab-completable entry points for things that need explicit invocation (tri-review, tri-debug, tri-security, status, setup-rules, workflow); 18 former commands now context-activated via skills or invoked via MCP tools (pr-monitor folded into pr-ready workflow; pr-ready is now a natural-language skill) - **Deterministic workflow conversion** — All command logic moved from LLM-interpreted markdown to Go-engine-driven YAML workflows; ~3,600 lines of inline logic removed -- **20 context-activated skills** — 10 auto-trigger workflows (test-gen, doc-gen, changelog, onboard, research, deep-research, scrape, autoloop, adr, pr-ready) + 6 coding principles (executing, clean-code, DRY, YAGNI, dont-reinvent, stuck) + 2 tools (gcli, creating-workflows) + 1 iteration memory (scratchpad) + 1 orchestration (mega-pr) +- **22 context-activated skills** — 10 auto-trigger workflows (test-gen, doc-gen, changelog, onboard, research, deep-research, scrape, autoloop, adr, pr-ready) + 6 coding principles (executing, clean-code, DRY, YAGNI, dont-reinvent, stuck) + 2 tools (gcli, creating-workflows) + 1 iteration memory (scratchpad) + 1 orchestration (mega-pr) + 2 browser automation (screenshot, browser) - **6 agents** — Scoped tool access, worktree isolation, model assignment - **12 hooks** — Safety (destructive command blocking, edit-time security patterns, PR gate), observability (audit trail, slop detection, post-validation, subagent verification, language-aware code review), optimization (RTK token compression), workflow enforcement (devkit-guard, devkit-stop-guard) - **Graceful degradation** — tri:* commands work with 1-3 agents depending on installed CLIs diff --git a/skills/browser/SKILL.md b/skills/browser/SKILL.md new file mode 100644 index 0000000..9e9e438 --- /dev/null +++ b/skills/browser/SKILL.md @@ -0,0 +1,161 @@ +--- +name: browser +description: Automate a web browser — use when asked to click buttons, fill forms, navigate multi-step flows, extract data from JS-rendered pages, log into a site, record user interactions, or test a web app in a real browser. Use for anything beyond simple article scraping. +--- + +# Browser Automation + +Drive a real browser via Playwright to interact with pages — click, fill forms, extract data from SPAs, run multi-step flows, or record user interactions. + +Use this skill for anything beyond simple article scraping: JS-heavy data extraction, form filling, multi-step auth flows, codegen recording, or E2E testing. For static article→markdown use `scrape`; for pure image capture use `screenshot`. + +## Step 1: Verify Playwright + +Run `npx playwright --version`. If it fails, stop and tell the user: `Playwright required — install with: npx playwright install chromium`. Do not attempt workarounds. + +## Step 2: Parse the Request + +- **Target URL(s)** — starting page (validate per Rules before launching) +- **Actions** — navigate, click, fill, extract, wait +- **Data** — what fields, what format (JSON usually) +- **Auth** — are credentials needed? Read from env, never hardcode +- **Repeatability** — one-off throwaway, or save script for reuse? + +If the request is vague ("scrape this site"), clarify which fields, whether login is needed, and whether it'll be re-run. + +## Step 3: Choose the Right Mode + +### Mode A — Codegen (recording) + +When the user wants to figure out selectors or hand off a repeatable flow: + +```bash +npx playwright codegen {url} +``` + +Opens a browser. User interacts. Playwright prints the equivalent script. Best for: +- Complex pages where selectors aren't obvious +- Building reusable flows +- Teaching users how Playwright works + +### Mode B — Inline script (one-off) + +Write the script to a file that reads args from `process.argv`, then pass values via shell variables: + +```bash +cat > /tmp/devkit-browser-flow.mjs <<'EOF' +import { chromium } from 'playwright'; + +const url = process.argv[2]; +if (!url) { console.error('usage: node devkit-browser-flow.mjs '); process.exit(2); } + +const browser = await chromium.launch(); +let exitCode = 0; +try { + const page = await browser.newPage(); + await page.goto(url, { waitUntil: 'networkidle', timeout: 30000 }); + + // Example: extract items from a JS-rendered list + const data = await page.evaluate(() => + Array.from(document.querySelectorAll('.item')).map(el => ({ + title: el.querySelector('h3')?.textContent?.trim(), + link: el.querySelector('a')?.href, + })) + ); + + console.log(JSON.stringify(data, null, 2)); +} catch (err) { + console.error('flow failed:', err.message); + exitCode = 1; +} finally { + try { await browser.close(); } catch (e) { /* suppress close errors so the real error propagates */ } +} +process.exit(exitCode); +EOF + +URL=https://example.com node /tmp/devkit-browser-flow.mjs "$URL" +``` + +### Mode C — Persistent test (for web apps) + +If the user is building a web app and wants E2E tests: + +```bash +# Scaffold Playwright test framework +npm init playwright@latest + +# Generates playwright.config.ts and tests/ directory +# Run with: +npx playwright test +``` + +Then write test specs in `tests/*.spec.ts`. + +### Mode D — Form fill + auth flow + +Same pattern as Mode B, plus `page.fill` / `page.click` for the login form. Credentials come from env, never from hardcoded strings or the command line: + +```bash +cat > /tmp/devkit-browser-auth.mjs <<'EOF' +import { chromium } from 'playwright'; + +const loginUrl = process.argv[2]; +const targetUrl = process.argv[3]; +if (!loginUrl || !targetUrl) { + console.error('usage: node devkit-browser-auth.mjs '); + process.exit(2); +} + +const browser = await chromium.launch(); +let exitCode = 0; +try { + const page = await browser.newPage(); + await page.goto(loginUrl); + await page.fill('input[name="email"]', process.env.EMAIL); + await page.fill('input[name="password"]', process.env.PASSWORD); + await page.click('button[type="submit"]'); + await page.waitForURL('**/dashboard'); + + await page.goto(targetUrl); + const content = await page.content(); + console.log(content); +} catch (err) { + console.error('auth flow failed:', err.message); + exitCode = 1; +} finally { + try { await browser.close(); } catch (e) { /* suppress close errors */ } +} +process.exit(exitCode); +EOF + +EMAIL="$EMAIL" PASSWORD="$PASSWORD" node /tmp/devkit-browser-auth.mjs "$LOGIN_URL" "$TARGET_URL" +``` + +## Step 4: Report + +After running, report: + +- What was done (pages visited, actions performed) +- What was extracted (or where it was saved) +- Any failures (timeouts, missing elements, auth errors) +- The script file path (if saved for reuse) + +## Rules + +- **URL validation** — only accept `http://` and `https://`. Reject `file://`, `ftp://`, `data:`, and all other schemes. Reject private/reserved IPs: `localhost`, `127.0.0.1`, `0.0.0.0`, `169.254.x.x` (cloud metadata — AWS/GCP/Azure), `10.x.x.x`, `172.16-31.x.x`, `192.168.x.x`, `[::1]` — unless the user is explicitly testing localhost. Reject URLs containing `@` (credential-in-URL attacks). On rejection: **report the exact reason and stop** — never silently skip. +- **Credentials** — never hardcode. Read from env or ask the user. Never log them. +- **No raw interpolation** — write scripts to files that read args from `process.argv`, then pass values via shell variables. Never build via `-e "..."` with interpolated user input. +- **Always close the browser safely** — use `try/catch/finally` with the `close()` call in its own try/catch, so a close-time error never masks the real error. +- **Headless by default** — only use `headless: false` when user explicitly wants it (e.g., codegen, debugging). +- **Respect the site** — no CAPTCHA bypass, no hammering, no scraping the user doesn't own. +- **Default timeout 30s** — if slow, use `page.waitForSelector()` rather than longer fixed waits, and explain why. + +## Troubleshooting + +| Symptom | Fix | +|---|---| +| `"Executable doesn't exist"` | `npx playwright install chromium` | +| Timeout exceeded | **Report the timeout to the user first with the URL and selector** — do not retry silently. Then investigate: wrong selector, page slow, content gated by JS that never fires. Use `waitForSelector()` with a sensible timeout rather than longer fixed waits. | +| Element not visible | `await locator.scrollIntoViewIfNeeded()` before action | +| Site blocks headless | Try user agent override; last resort: `headless: false` | +| Flaky data extraction | Use `networkidle` wait condition instead of `domcontentloaded` | diff --git a/skills/scrape/SKILL.md b/skills/scrape/SKILL.md index 4275fac..69f11f2 100644 --- a/skills/scrape/SKILL.md +++ b/skills/scrape/SKILL.md @@ -1,6 +1,6 @@ --- name: scrape -description: Scrape a URL to clean Markdown — use when asked to scrape, fetch, extract content from, or read a webpage and convert it to Markdown. Uses Jina Reader, Firecrawl, or WebFetch. +description: Scrape a URL to clean Markdown — use when asked to scrape, fetch, extract content from, or read a webpage and convert it to Markdown. Uses Jina Reader, Playwright, Firecrawl, or WebFetch. --- # Web Scrape to Markdown @@ -13,7 +13,7 @@ Fetch a URL and convert it to clean, LLM-ready Markdown. Supports multiple backe /devkit:scrape https://example.com /devkit:scrape https://example.com --json /devkit:scrape https://example.com https://other.com -/devkit:scrape https://example.com --backend firecrawl +/devkit:scrape https://example.com --backend playwright ``` ## Arguments @@ -24,9 +24,10 @@ Fetch a URL and convert it to clean, LLM-ready Markdown. Supports multiple backe ## Backends (in priority order) -1. **Jina Reader** — prepend `https://r.jina.ai/` to the URL. Returns Markdown by default. Use if `JINA_API_KEY` is set (higher rate limits) or anonymously (~20 RPM). -2. **Firecrawl** — use if `FIRECRAWL_API_KEY` is set. Best for JS-heavy sites and anti-bot bypass. -3. **WebFetch fallback** — use Claude's built-in `WebFetch` tool. No API key needed, but returns raw content (less clean). +1. **Jina Reader** — prepend `https://r.jina.ai/` to the URL. Returns Markdown by default. Use if `JINA_API_KEY` is set (higher rate limits) or anonymously (~20 RPM). Best for articles and docs. +2. **Playwright** — use if `npx playwright --version` succeeds (optional dep, install with `npx playwright install chromium`). Best for JS-heavy SPAs and sites that block headless scrapers. Free and local — no API keys. +3. **Firecrawl** — use if `FIRECRAWL_API_KEY` is set. Paid API. Good for anti-bot bypass when Playwright isn't enough. +4. **WebFetch fallback** — use Claude's built-in `WebFetch` tool. No API key needed, but returns raw content (less clean). ## Execution @@ -45,6 +46,43 @@ This returns: { "url": "...", "title": "...", "content": "..." } where "content" is the Markdown. ``` +**Playwright:** + +Validate the URL first (see Rules: http/https only, no private IPs, no `@`). + +Check availability: `npx playwright --version`. If not installed, tell the user once per session: `Playwright not installed. Install with: npx playwright install chromium`, then fall through to the next backend. The winning backend is always reported in Step 3 (Error Handling), so the user sees which one served the result. + +Never build scripts via inline `-e "..."` strings. Write the script to a file that reads the URL from `process.argv[2]`, then invoke it: + +```bash +cat > /tmp/devkit-scrape.mjs <<'EOF' +import { chromium } from 'playwright'; + +const url = process.argv[2]; +if (!url) { console.error('usage: node devkit-scrape.mjs '); process.exit(2); } + +const browser = await chromium.launch(); +try { + const page = await browser.newPage(); + await page.goto(url, { waitUntil: 'networkidle', timeout: 30000 }); + const title = await page.title(); + const html = await page.content(); + process.stdout.write(JSON.stringify({ title, html })); +} catch (err) { + console.error('playwright failed:', err.message); + process.exit(1); +} finally { + try { await browser.close(); } catch (e) { /* suppress close errors */ } +} +EOF + +node /tmp/devkit-scrape.mjs "$URL" +``` + +Pass the URL as a shell variable (`URL=https://example.com`), never inline into the command string. The `.mjs` reads it from `process.argv[2]` so quotes, backticks, and `$()` in the URL can't escape. + +Then convert HTML → Markdown. Prefer a local converter if available (pandoc, turndown). If none available, strip script/style/nav/footer tags and extract text from article/main/body. + **Firecrawl (if FIRECRAWL_API_KEY is set and --backend firecrawl):** ``` Use Bash to call (use jq to safely construct JSON — never interpolate URLs directly): @@ -72,9 +110,11 @@ When given multiple URLs, scrape them in parallel: ### Error Handling -- If Jina Reader returns an error or empty content, fall back to WebFetch -- If a URL is unreachable, report the error and continue with remaining URLs -- Never silently drop a URL — always report what happened +**Fallback chain (single linear order):** Jina Reader → Playwright → Firecrawl → WebFetch. Each backend is tried in order. Move to the next only on error, empty content, or missing dependency. + +- **Report which backend actually ran** for each URL, and (if fallbacks happened) why the earlier ones failed. Never silently substitute content — the user must know a paywall/cookie-wall/SPA-skeleton from one backend wasn't the real page fetched by another. +- If a URL is unreachable by every backend, report the error and continue with remaining URLs. +- Never silently drop or substitute a URL — always report what happened, with the winning backend named. ## Output @@ -106,8 +146,8 @@ For multiple URLs, output a JSON array. ## Rules -- **URL validation** — only accept `http://` and `https://` URLs. Reject `file://`, `ftp://`, `data:`, and all other schemes. Reject URLs targeting private/reserved IPs: `localhost`, `127.0.0.1`, `0.0.0.0`, `169.254.x.x`, `10.x.x.x`, `172.16-31.x.x`, `192.168.x.x`, `[::1]`. Reject URLs containing `@` (credential-in-URL attacks). -- **No shell injection** — never interpolate user URLs directly into shell command strings. Use `jq` to construct JSON payloads for curl. +- **URL validation** — only accept `http://` and `https://` URLs. Reject `file://`, `ftp://`, `data:`, and all other schemes. Reject URLs targeting private/reserved IPs: `localhost`, `127.0.0.1`, `0.0.0.0`, `169.254.x.x` (cloud metadata — AWS/GCP/Azure), `10.x.x.x`, `172.16-31.x.x`, `192.168.x.x`, `[::1]`. Reject URLs containing `@` (credential-in-URL attacks). When rejecting, report the exact reason and stop — never silently skip a URL in a batch. +- **No shell injection** — never interpolate user URLs directly into shell command strings. Use `jq` to construct JSON payloads for curl. For Playwright, write scripts to files that read URLs from `process.argv`, and pass the URL as a shell variable — never inline into `-e` strings. - **API keys from env only** — never hardcode API keys. Always reference `$JINA_API_KEY`, `$FIRECRAWL_API_KEY` from environment variables. - Always try Jina Reader first — it's free and produces the cleanest output - Respect rate limits — if scraping many URLs, add a brief pause between requests diff --git a/skills/screenshot/SKILL.md b/skills/screenshot/SKILL.md new file mode 100644 index 0000000..e0506b0 --- /dev/null +++ b/skills/screenshot/SKILL.md @@ -0,0 +1,105 @@ +--- +name: screenshot +description: Capture a screenshot or PDF of a web page — use when asked to screenshot a URL, capture a page, take a picture of a website, generate a visual snapshot, or save a page as PDF. Supports full page, element selector, device emulation, and custom viewport. +--- + +# Web Page Screenshot + +Capture screenshots and PDFs of web pages using Playwright. + +## Step 1: Verify Playwright + +Run `npx playwright --version`. If it fails, stop and tell the user: `Playwright required — install with: npx playwright install chromium`. Do not attempt workarounds. + +## Step 2: Parse and Validate + +Extract from the user's request: + +- **URL** — required; validated below +- **Output path** — default `./screenshot-{timestamp}.png` in cwd +- **Capture mode** — viewport (default), full-page, element selector +- **Device / viewport** — optional emulation or custom size +- **Format** — PNG (default) or PDF + +**Validate the URL** (see Rules for the full list): only `http(s)`, no private/reserved IPs or cloud metadata, no `@`. On rejection, report the exact reason and stop — never silently skip in a batch. + +## Step 3: Execute + +Choose the command that matches the capture mode. Always quote the URL and output path. + +**Viewport screenshot:** +```bash +npx playwright screenshot "{url}" "{output}" +``` + +**Full-page screenshot:** +```bash +npx playwright screenshot --full-page "{url}" "{output}" +``` + +**Custom viewport** (format is `W,H` with no spaces, e.g. `1280,800`): +```bash +npx playwright screenshot --viewport-size=1280,800 "{url}" "{output}" +``` + +**Device emulation:** +```bash +npx playwright screenshot --device="{device}" "{url}" "{output}" +``` + +**PDF output:** +```bash +npx playwright pdf "{url}" "{output}.pdf" +``` + +**Element selector** (Playwright CLI doesn't expose this directly, so write a script file and pass args via `process.argv` — never interpolate user strings into `node -e`): + +```bash +cat > /tmp/devkit-shot-element.mjs <<'EOF' +import { chromium } from 'playwright'; + +const [, , url, selector, output] = process.argv; +if (!url || !selector || !output) { + console.error('usage: node devkit-shot-element.mjs '); + process.exit(2); +} + +const browser = await chromium.launch(); +try { + const page = await browser.newPage(); + await page.goto(url, { waitUntil: 'networkidle', timeout: 30000 }); + await page.locator(selector).screenshot({ path: output }); + console.log(output); +} catch (err) { + console.error('screenshot failed:', err.message); + process.exit(1); +} finally { + try { await browser.close(); } catch (e) { /* suppress close errors */ } +} +EOF + +node /tmp/devkit-shot-element.mjs "$URL" "$SELECTOR" "$OUTPUT" +``` + +Pass all user values via shell variables. The `.mjs` reads them from `process.argv` so quoting, `$()`, and backticks in the input can't escape. + +## Step 4: Report + +Output the absolute path and any metadata: + +``` +Screenshot saved: /abs/path/to/file.png +Size: 342 KB +Mode: full-page +``` + +If the page failed to load, report the error clearly with the HTTP status or timeout reason. + +## Rules + +- **URL validation** — only `http://` and `https://`. Reject `file://`, `ftp://`, `data:`, and all other schemes. Reject private/reserved IPs: `localhost`, `127.0.0.1`, `0.0.0.0`, `10.x.x.x`, `172.16-31.x.x`, `192.168.x.x`, `169.254.x.x` (cloud metadata — AWS/GCP/Azure), `[::1]` — unless user explicitly tests localhost. Reject URLs with `@`. On rejection: report exact reason and stop, never silently skip. +- **No raw interpolation** — for the element-selector script, pass values via shell variables and `process.argv`, never via `node -e "..."` with user input. +- **Headless only** — never open a visible browser window. +- **Respect gated content** — don't screenshot login-walled or paywalled content the user doesn't own. +- **Default to cwd** — put output files in the current working directory unless the user specifies otherwise. +- **Close safely** — when using an inline script, wrap `browser.close()` in its own try/catch inside `finally` so a close-time error can't mask the real error.