From 371ee95164a29a2a40b56222a84826459b9f782c Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Fri, 10 Apr 2026 00:14:59 -0400 Subject: [PATCH 1/4] feat: add Playwright-powered browser skills (screenshot, browser) + scrape backend MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add two new auto-invoked skills that leverage Playwright as an optional dependency: - screenshot: capture web pages as PNG/PDF (full-page, viewport, element, device emulation) - browser: general browser automation (click, fill, navigate, extract, codegen, E2E testing) Also add Playwright as a new backend in the existing scrape skill, slotted between Jina Reader and Firecrawl for JS-heavy pages without paid API dependencies. Playwright stays optional — users install it themselves via `npx playwright install chromium`. Skills check for availability and provide install instructions when missing. Research in homebase/docs/research/browser-automation-for-devkit.md covers the full landscape comparison (Playwright, shot-scraper, Puppeteer, Selenium, Rod, Crawlee, Skyvern, Browser Use, Stagehand, Steel, Browserbase, AgentQL). --- README.md | 9 +- skills/browser/SKILL.md | 175 +++++++++++++++++++++++++++++++++++++ skills/scrape/SKILL.md | 37 ++++++-- skills/screenshot/SKILL.md | 114 ++++++++++++++++++++++++ 4 files changed, 328 insertions(+), 7 deletions(-) create mode 100644 skills/browser/SKILL.md create mode 100644 skills/screenshot/SKILL.md diff --git a/README.md b/README.md index d18ceeb..638f5b9 100644 --- a/README.md +++ b/README.md @@ -69,8 +69,13 @@ These handle concerns devkit doesn't — methodology, specialized reviews, and c ```bash brew install rtk # Token optimization (60-90% savings on Bash output) brew install ast-grep # AST-based repo mapping (devkit workflow run repo-map) + +# Browser automation — enables scrape (JS-rendered), screenshot, and browser skills +npx playwright install chromium ``` +**Playwright** (optional) enables three skills: enhanced `scrape` for JS-heavy sites, `screenshot` for page captures, and `browser` for full automation (clicking, form filling, multi-step flows, codegen). Free and local — no API keys. Install only the browsers you need (`chromium` is ~170MB). + ### Verify ```bash @@ -153,6 +158,8 @@ Skills activate automatically based on context. No slash command needed. | "research X" | `research` | | "deep research", "validate this" | `deep-research` | | "scrape this URL" | `scrape` | +| "screenshot this page" | `screenshot` (requires Playwright) | +| "automate this browser flow" | `browser` (requires Playwright) | | "create an ADR" | `adr` | Coding principles (`clean-code`, `dry`, `yagni`, `dont-reinvent`, `executing`, `stuck`, `scratchpad`) load as reference when relevant. @@ -292,7 +299,7 @@ Self-Improvement (self-* workflows) ``` devkit/ ├── commands/ # 8 slash commands (tab-completable entry points) -├── skills/ # 19 context-activated skills +├── skills/ # 21 context-activated skills ├── agents/ # 6 agents (reviewer, researcher, improver, ...) ├── hooks/ # 10 hooks (safety, security, quality gates) ├── workflows/ # 18 YAML workflow definitions diff --git a/skills/browser/SKILL.md b/skills/browser/SKILL.md new file mode 100644 index 0000000..b074f08 --- /dev/null +++ b/skills/browser/SKILL.md @@ -0,0 +1,175 @@ +--- +name: browser +description: Automate a web browser — use when asked to click buttons, fill forms, navigate multi-step flows, extract data from JS-rendered pages, log into a site, record user interactions, or test a web app in a real browser. Use for anything beyond simple article scraping. +--- + +# Browser Automation + +Drive a real browser via Playwright to interact with pages — click, fill forms, extract data from SPAs, run multi-step flows, or record user interactions. + +## When to Use This vs Related Skills + +| User asks for... | Use | +|---|---| +| "fetch this article as markdown" | `scrape` (Jina is faster for static content) | +| "screenshot this page" | `screenshot` | +| "extract data from this JS-heavy page" | **`browser`** | +| "log into X and download Y" | **`browser`** | +| "fill this form and submit" | **`browser`** | +| "record me clicking through this flow" | **`browser`** (codegen) | +| "test my web app end-to-end" | **`browser`** | + +## Step 1: Verify Playwright + +```bash +npx playwright --version +``` + +If not installed, stop and tell the user: + +``` +This requires Playwright (optional devkit dependency). + +Install with: + npx playwright install chromium +``` + +Do not attempt workarounds. + +## Step 2: Parse the Request + +Understand what the user wants: + +- **Target URL(s)** — starting page +- **Actions** — navigate, click, fill, extract, screenshot, wait +- **Data to extract** — what fields, what format (JSON usually) +- **Auth** — are credentials needed? (prompt if user hasn't provided) +- **Repeatability** — one-off or should this become a reusable script? + +If the user's request is vague ("scrape this site"), clarify: +- Which data fields do you want? +- Does it require login? +- Is it a one-shot or will you re-run this? + +## Step 3: Choose the Right Mode + +### Mode A — Codegen (recording) + +When the user wants to figure out selectors or hand off a repeatable flow: + +```bash +npx playwright codegen {url} +``` + +Opens a browser. User interacts. Playwright prints the equivalent script. Best for: +- Complex pages where selectors aren't obvious +- Building reusable flows +- Teaching users how Playwright works + +### Mode B — Inline script (one-off) + +For quick, throwaway extractions, write a small script to `/tmp/` and run it: + +```bash +cat > /tmp/flow.mjs <<'EOF' +import { chromium } from 'playwright'; + +const browser = await chromium.launch(); +try { + const page = await browser.newPage(); + await page.goto('{url}', { waitUntil: 'networkidle' }); + + // Example: extract items from a JS-rendered list + const data = await page.evaluate(() => + Array.from(document.querySelectorAll('.item')).map(el => ({ + title: el.querySelector('h3')?.textContent?.trim(), + link: el.querySelector('a')?.href, + })) + ); + + console.log(JSON.stringify(data, null, 2)); +} finally { + await browser.close(); +} +EOF + +node /tmp/flow.mjs +``` + +Never build scripts via long `-e "..."` strings with interpolated user input — write the script to a file, then run it. + +### Mode C — Persistent test (for web apps) + +If the user is building a web app and wants E2E tests: + +```bash +# Scaffold Playwright test framework +npm init playwright@latest + +# Generates playwright.config.ts and tests/ directory +# Run with: +npx playwright test +``` + +Then write test specs in `tests/*.spec.ts`. + +### Mode D — Form fill + auth flow + +For "log in and grab something": + +```bash +cat > /tmp/flow.mjs <<'EOF' +import { chromium } from 'playwright'; + +const browser = await chromium.launch(); +try { + const page = await browser.newPage(); + await page.goto('{login_url}'); + await page.fill('input[name="email"]', process.env.EMAIL); + await page.fill('input[name="password"]', process.env.PASSWORD); + await page.click('button[type="submit"]'); + await page.waitForURL('**/dashboard'); + + // Now do whatever the user wanted + await page.goto('{target_url}'); + const content = await page.content(); + console.log(content); +} finally { + await browser.close(); +} +EOF + +EMAIL='...' PASSWORD='...' node /tmp/flow.mjs +``` + +Never hardcode credentials. Always read from env vars or prompt the user. + +## Step 4: Report + +After running, report: + +- What was done (pages visited, actions performed) +- What was extracted (or where it was saved) +- Any failures (timeouts, missing elements, auth errors) +- The script file path (if saved for reuse) + +## Rules + +- **URL validation** — only `http(s)`. Reject private IPs unless testing localhost is explicit. Reject URLs with `@`. +- **Credentials** — never hardcode. Read from env or ask the user. Never log them. +- **No raw interpolation** — write scripts to files, don't build via `-e "..."` with user input. +- **Always close the browser** — use `try/finally` to avoid orphaned processes. +- **Headless by default** — only use `headless: false` when user explicitly wants it (e.g., codegen, debugging). +- **Respect the site** — no CAPTCHA bypass, no hammering, no scraping the user doesn't own. +- **Auth state reuse** — for repeated runs against the same site, save `storageState` to avoid re-login. +- **Default timeout 30s** — if slow, use `page.waitForSelector()` rather than longer fixed waits, and explain why. + +## Troubleshooting + +| Symptom | Fix | +|---|---| +| `"Executable doesn't exist"` | `npx playwright install chromium` | +| Timeout exceeded | Selector wrong or page slow — use `waitForSelector()` | +| Element not visible | `await locator.scrollIntoViewIfNeeded()` before action | +| Site blocks headless | Try user agent override; last resort: `headless: false` | +| Flaky data extraction | Use `networkidle` wait condition instead of `domcontentloaded` | diff --git a/skills/scrape/SKILL.md b/skills/scrape/SKILL.md index 4275fac..6f9db78 100644 --- a/skills/scrape/SKILL.md +++ b/skills/scrape/SKILL.md @@ -1,6 +1,6 @@ --- name: scrape -description: Scrape a URL to clean Markdown — use when asked to scrape, fetch, extract content from, or read a webpage and convert it to Markdown. Uses Jina Reader, Firecrawl, or WebFetch. +description: Scrape a URL to clean Markdown — use when asked to scrape, fetch, extract content from, or read a webpage and convert it to Markdown. Uses Jina Reader, Playwright, Firecrawl, or WebFetch. --- # Web Scrape to Markdown @@ -13,7 +13,7 @@ Fetch a URL and convert it to clean, LLM-ready Markdown. Supports multiple backe /devkit:scrape https://example.com /devkit:scrape https://example.com --json /devkit:scrape https://example.com https://other.com -/devkit:scrape https://example.com --backend firecrawl +/devkit:scrape https://example.com --backend playwright ``` ## Arguments @@ -24,9 +24,10 @@ Fetch a URL and convert it to clean, LLM-ready Markdown. Supports multiple backe ## Backends (in priority order) -1. **Jina Reader** — prepend `https://r.jina.ai/` to the URL. Returns Markdown by default. Use if `JINA_API_KEY` is set (higher rate limits) or anonymously (~20 RPM). -2. **Firecrawl** — use if `FIRECRAWL_API_KEY` is set. Best for JS-heavy sites and anti-bot bypass. -3. **WebFetch fallback** — use Claude's built-in `WebFetch` tool. No API key needed, but returns raw content (less clean). +1. **Jina Reader** — prepend `https://r.jina.ai/` to the URL. Returns Markdown by default. Use if `JINA_API_KEY` is set (higher rate limits) or anonymously (~20 RPM). Best for articles and docs. +2. **Playwright** — use if `npx playwright --version` succeeds (optional dep, install with `npx playwright install chromium`). Best for JS-heavy SPAs, paywalled content, and sites that block headless scrapers. Free and local — no API keys. +3. **Firecrawl** — use if `FIRECRAWL_API_KEY` is set. Paid API. Good for anti-bot bypass when Playwright isn't enough. +4. **WebFetch fallback** — use Claude's built-in `WebFetch` tool. No API key needed, but returns raw content (less clean). ## Execution @@ -45,6 +46,29 @@ This returns: { "url": "...", "title": "...", "content": "..." } where "content" is the Markdown. ``` +**Playwright (if installed and --backend playwright, or as auto-fallback for JS-heavy sites):** +``` +Check availability first: npx playwright --version +If not installed, tell the user: "Playwright not installed. Run: npx playwright install chromium" + and fall through to the next backend. + +Extract HTML + convert to markdown using Playwright's CLI + a small inline script: + npx playwright cr -e " + const page = await context.newPage(); + await page.goto({url}, { waitUntil: 'networkidle' }); + const title = await page.title(); + const html = await page.content(); + console.log(JSON.stringify({ title, html })); + await browser.close(); + " + +Then convert HTML → Markdown. Prefer using a local converter if available (pandoc, turndown). +If none available, strip script/style/nav/footer tags and extract text from article/main/body. + +For simpler cases, use: npx playwright screenshot --full-page {url} /tmp/page.png + (screenshots only, not markdown) +``` + **Firecrawl (if FIRECRAWL_API_KEY is set and --backend firecrawl):** ``` Use Bash to call (use jq to safely construct JSON — never interpolate URLs directly): @@ -72,7 +96,8 @@ When given multiple URLs, scrape them in parallel: ### Error Handling -- If Jina Reader returns an error or empty content, fall back to WebFetch +- If Jina Reader returns an error or empty content, fall back to Playwright (if installed), then WebFetch +- If Playwright fails or isn't installed, fall back to Firecrawl (if API key set), then WebFetch - If a URL is unreachable, report the error and continue with remaining URLs - Never silently drop a URL — always report what happened diff --git a/skills/screenshot/SKILL.md b/skills/screenshot/SKILL.md new file mode 100644 index 0000000..acd38af --- /dev/null +++ b/skills/screenshot/SKILL.md @@ -0,0 +1,114 @@ +--- +name: screenshot +description: Capture a screenshot or PDF of a web page — use when asked to screenshot a URL, capture a page, take a picture of a website, generate a visual snapshot, or save a page as PDF. Supports full page, element selector, device emulation, and custom viewport. +--- + +# Web Page Screenshot + +Capture screenshots and PDFs of web pages using Playwright. + +## Step 1: Verify Playwright + +Before doing anything else, check that Playwright is installed: + +```bash +npx playwright --version +``` + +If the command fails or Playwright is not installed, stop and tell the user: + +``` +This requires Playwright (optional devkit dependency). + +Install with: + npx playwright install chromium + +Chromium is ~170MB. You can also install firefox or webkit. +``` + +Do NOT attempt to work around a missing install. The skill cannot function without it. + +## Step 2: Parse the Request + +From the user's message, extract: + +- **URL** — the page to capture (required; validate http/https only) +- **Output path** — where to save (default: `./screenshot-{timestamp}.png` in cwd) +- **Capture mode** — viewport (default), full-page, or specific element +- **Device emulation** — iPhone, Pixel, iPad, etc. (optional) +- **Viewport size** — custom width x height (optional) +- **Format** — PNG (default) or PDF +- **Selector** — CSS selector for element-only capture (optional) + +## Step 3: Validate the URL + +Reject: +- Non-`http(s)` schemes (`file://`, `ftp://`, `data:`) +- Private/reserved IPs: `localhost`, `127.0.0.1`, `0.0.0.0`, `10.x.x.x`, `172.16-31.x.x`, `192.168.x.x`, `169.254.x.x`, `[::1]` — unless the user explicitly wants to test localhost +- URLs containing `@` (credential-in-URL attack vector) + +## Step 4: Execute + +Choose the command that matches the capture mode. Always quote the URL and output path. + +**Viewport screenshot:** +```bash +npx playwright screenshot "{url}" "{output}" +``` + +**Full-page screenshot:** +```bash +npx playwright screenshot --full-page "{url}" "{output}" +``` + +**Custom viewport:** +```bash +npx playwright screenshot --viewport-size={width},{height} "{url}" "{output}" +``` + +**Device emulation:** +```bash +npx playwright screenshot --device="{device}" "{url}" "{output}" +``` + +**PDF output:** +```bash +npx playwright pdf "{url}" "{output}.pdf" +``` + +**Element selector** (Playwright CLI doesn't expose this directly, so use a small script): +```bash +node -e " +const { chromium } = require('playwright'); +(async () => { + const browser = await chromium.launch(); + const page = await browser.newPage(); + await page.goto({url_as_json}); + await page.locator({selector_as_json}).screenshot({ path: {output_as_json} }); + await browser.close(); +})(); +" +``` + +Use `JSON.stringify`-safe values (never interpolate raw user strings into `-e`). + +## Step 5: Report + +Output the absolute path and any metadata: + +``` +Screenshot saved: /abs/path/to/file.png +Size: 342 KB +Mode: full-page +``` + +If the page failed to load, report the error clearly with the HTTP status or timeout reason. + +## Rules + +- **URL validation first** — never launch the browser on an invalid or private URL +- **No shell injection** — never concatenate user URLs into shell strings without proper quoting +- **Headless only** — never open a visible browser window +- **Respect gated content** — don't screenshot login-walled or paywalled content the user doesn't own +- **Default to cwd** — put output files in the current working directory unless the user specifies otherwise +- **One browser, one close** — always `await browser.close()` to avoid orphaned processes From b2b04f45ce35588e763fc735cf50b64d5e35016b Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Fri, 10 Apr 2026 00:28:46 -0400 Subject: [PATCH 2/4] fix: address mega-pr review findings on Playwright skills MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Critical: - scrape: remove fabricated `npx playwright cr -e` subcommand (not a real Playwright CLI command) — replace with file-based .mjs script that reads the URL from process.argv, invoked via shell variable. Kills shell injection and script injection via URL contents. - scrape: expand URL validation to match screenshot's full blocklist (include 169.254.169.254 cloud metadata IP). Require reject-and-stop reporting in batch mode. - browser: expand one-line URL validation to full blocklist matching scrape and screenshot. Add cloud metadata IP. Require reject-and-stop. Important: - scrape: fallback chain contradicted itself (Jina→Playwright→WebFetch vs Jina→Playwright→Firecrawl→WebFetch). Now single linear chain: Jina → Playwright → Firecrawl → WebFetch. Require reporting which backend actually served the result + why prior backends failed, so silent content substitution (paywall/cookie-wall) can't sneak through. - scrape: remove paywall contradiction — listed Playwright as good for paywalled content while Rules forbade scraping paywalled content. - screenshot: element-selector block self-contradicted (told agent not to interpolate into -e while showing a node -e example with interpolation placeholders). Rewrite as file-based script reading from process.argv with shell variable pass-through. - browser: wrap browser.close() in its own try/catch inside finally so a close-time crash can't mask the real error via JS exception replacement semantics. Apply to all three script examples. - browser: troubleshooting table for "Timeout exceeded" was training the agent to retry with longer waits instead of surfacing the failure. Now requires reporting first, investigating second. Review sources: pr-review-toolkit code-reviewer, silent-failure-hunter, and Codex rescue agent (gemini rescue failed to return output). --- skills/browser/SKILL.md | 53 ++++++++++++++++++++--------- skills/scrape/SKILL.md | 70 +++++++++++++++++++++++--------------- skills/screenshot/SKILL.md | 43 ++++++++++++++++------- 3 files changed, 110 insertions(+), 56 deletions(-) diff --git a/skills/browser/SKILL.md b/skills/browser/SKILL.md index b074f08..936917e 100644 --- a/skills/browser/SKILL.md +++ b/skills/browser/SKILL.md @@ -70,14 +70,20 @@ Opens a browser. User interacts. Playwright prints the equivalent script. Best f For quick, throwaway extractions, write a small script to `/tmp/` and run it: +Write the script to a file that reads the URL from `process.argv`, then pass it via a shell variable. Never interpolate user input into the script source itself. + ```bash -cat > /tmp/flow.mjs <<'EOF' +cat > /tmp/devkit-browser-flow.mjs <<'EOF' import { chromium } from 'playwright'; +const url = process.argv[2]; +if (!url) { console.error('usage: node devkit-browser-flow.mjs '); process.exit(2); } + const browser = await chromium.launch(); +let exitCode = 0; try { const page = await browser.newPage(); - await page.goto('{url}', { waitUntil: 'networkidle' }); + await page.goto(url, { waitUntil: 'networkidle', timeout: 30000 }); // Example: extract items from a JS-rendered list const data = await page.evaluate(() => @@ -88,15 +94,19 @@ try { ); console.log(JSON.stringify(data, null, 2)); +} catch (err) { + console.error('flow failed:', err.message); + exitCode = 1; } finally { - await browser.close(); + try { await browser.close(); } catch (e) { /* suppress close errors so the real error propagates */ } } +process.exit(exitCode); EOF -node /tmp/flow.mjs +URL=https://example.com node /tmp/devkit-browser-flow.mjs "$URL" ``` -Never build scripts via long `-e "..."` strings with interpolated user input — write the script to a file, then run it. +Never build scripts via long `-e "..."` strings with interpolated user input — write the script to a file that reads args from `process.argv`, then pass values as shell variables. ### Mode C — Persistent test (for web apps) @@ -118,31 +128,42 @@ Then write test specs in `tests/*.spec.ts`. For "log in and grab something": ```bash -cat > /tmp/flow.mjs <<'EOF' +cat > /tmp/devkit-browser-auth.mjs <<'EOF' import { chromium } from 'playwright'; +const loginUrl = process.argv[2]; +const targetUrl = process.argv[3]; +if (!loginUrl || !targetUrl) { + console.error('usage: node devkit-browser-auth.mjs '); + process.exit(2); +} + const browser = await chromium.launch(); +let exitCode = 0; try { const page = await browser.newPage(); - await page.goto('{login_url}'); + await page.goto(loginUrl); await page.fill('input[name="email"]', process.env.EMAIL); await page.fill('input[name="password"]', process.env.PASSWORD); await page.click('button[type="submit"]'); await page.waitForURL('**/dashboard'); - // Now do whatever the user wanted - await page.goto('{target_url}'); + await page.goto(targetUrl); const content = await page.content(); console.log(content); +} catch (err) { + console.error('auth flow failed:', err.message); + exitCode = 1; } finally { - await browser.close(); + try { await browser.close(); } catch (e) { /* suppress close errors */ } } +process.exit(exitCode); EOF -EMAIL='...' PASSWORD='...' node /tmp/flow.mjs +EMAIL="$EMAIL" PASSWORD="$PASSWORD" node /tmp/devkit-browser-auth.mjs "$LOGIN_URL" "$TARGET_URL" ``` -Never hardcode credentials. Always read from env vars or prompt the user. +Never hardcode credentials. Always read from env vars or prompt the user. Pass URLs as shell variables, never inline into the script source. ## Step 4: Report @@ -155,10 +176,10 @@ After running, report: ## Rules -- **URL validation** — only `http(s)`. Reject private IPs unless testing localhost is explicit. Reject URLs with `@`. +- **URL validation** — only accept `http://` and `https://`. Reject `file://`, `ftp://`, `data:`, and all other schemes. Reject private/reserved IPs: `localhost`, `127.0.0.1`, `0.0.0.0`, `169.254.x.x` (cloud metadata — AWS/GCP/Azure), `10.x.x.x`, `172.16-31.x.x`, `192.168.x.x`, `[::1]` — unless the user is explicitly testing localhost. Reject URLs containing `@` (credential-in-URL attacks). On rejection: **report the exact reason and stop** — never silently skip. - **Credentials** — never hardcode. Read from env or ask the user. Never log them. -- **No raw interpolation** — write scripts to files, don't build via `-e "..."` with user input. -- **Always close the browser** — use `try/finally` to avoid orphaned processes. +- **No raw interpolation** — write scripts to files that read args from `process.argv`, then pass values via shell variables. Never build via `-e "..."` with interpolated user input. +- **Always close the browser safely** — use `try/catch/finally` with the `close()` call in its own try/catch, so a close-time error never masks the real error. - **Headless by default** — only use `headless: false` when user explicitly wants it (e.g., codegen, debugging). - **Respect the site** — no CAPTCHA bypass, no hammering, no scraping the user doesn't own. - **Auth state reuse** — for repeated runs against the same site, save `storageState` to avoid re-login. @@ -169,7 +190,7 @@ After running, report: | Symptom | Fix | |---|---| | `"Executable doesn't exist"` | `npx playwright install chromium` | -| Timeout exceeded | Selector wrong or page slow — use `waitForSelector()` | +| Timeout exceeded | **Report the timeout to the user first with the URL and selector** — do not retry silently. Then investigate: wrong selector, page slow, content gated by JS that never fires. Use `waitForSelector()` with a sensible timeout rather than longer fixed waits. | | Element not visible | `await locator.scrollIntoViewIfNeeded()` before action | | Site blocks headless | Try user agent override; last resort: `headless: false` | | Flaky data extraction | Use `networkidle` wait condition instead of `domcontentloaded` | diff --git a/skills/scrape/SKILL.md b/skills/scrape/SKILL.md index 6f9db78..3a5ecdd 100644 --- a/skills/scrape/SKILL.md +++ b/skills/scrape/SKILL.md @@ -25,7 +25,7 @@ Fetch a URL and convert it to clean, LLM-ready Markdown. Supports multiple backe ## Backends (in priority order) 1. **Jina Reader** — prepend `https://r.jina.ai/` to the URL. Returns Markdown by default. Use if `JINA_API_KEY` is set (higher rate limits) or anonymously (~20 RPM). Best for articles and docs. -2. **Playwright** — use if `npx playwright --version` succeeds (optional dep, install with `npx playwright install chromium`). Best for JS-heavy SPAs, paywalled content, and sites that block headless scrapers. Free and local — no API keys. +2. **Playwright** — use if `npx playwright --version` succeeds (optional dep, install with `npx playwright install chromium`). Best for JS-heavy SPAs and sites that block headless scrapers. Free and local — no API keys. 3. **Firecrawl** — use if `FIRECRAWL_API_KEY` is set. Paid API. Good for anti-bot bypass when Playwright isn't enough. 4. **WebFetch fallback** — use Claude's built-in `WebFetch` tool. No API key needed, but returns raw content (less clean). @@ -47,28 +47,43 @@ where "content" is the Markdown. ``` **Playwright (if installed and --backend playwright, or as auto-fallback for JS-heavy sites):** -``` -Check availability first: npx playwright --version -If not installed, tell the user: "Playwright not installed. Run: npx playwright install chromium" - and fall through to the next backend. - -Extract HTML + convert to markdown using Playwright's CLI + a small inline script: - npx playwright cr -e " - const page = await context.newPage(); - await page.goto({url}, { waitUntil: 'networkidle' }); - const title = await page.title(); - const html = await page.content(); - console.log(JSON.stringify({ title, html })); - await browser.close(); - " - -Then convert HTML → Markdown. Prefer using a local converter if available (pandoc, turndown). -If none available, strip script/style/nav/footer tags and extract text from article/main/body. - -For simpler cases, use: npx playwright screenshot --full-page {url} /tmp/page.png - (screenshots only, not markdown) + +First validate the URL against the rules below (http/https only, no private IPs, no `@`). + +Check availability: `npx playwright --version`. If not installed, tell the user once: +`Playwright not installed. Install with: npx playwright install chromium` — then fall through to the next backend AND record this in the backend-ran report so the user knows which backend actually served the result. + +Never build scripts via inline `-e "..."` strings. Write the script to a file that reads the URL from `process.argv[2]`, then invoke it: + +```bash +cat > /tmp/devkit-scrape.mjs <<'EOF' +import { chromium } from 'playwright'; + +const url = process.argv[2]; +if (!url) { console.error('usage: node devkit-scrape.mjs '); process.exit(2); } + +const browser = await chromium.launch(); +try { + const page = await browser.newPage(); + await page.goto(url, { waitUntil: 'networkidle', timeout: 30000 }); + const title = await page.title(); + const html = await page.content(); + process.stdout.write(JSON.stringify({ title, html })); +} catch (err) { + console.error('playwright failed:', err.message); + process.exit(1); +} finally { + try { await browser.close(); } catch (e) { /* suppress close errors */ } +} +EOF + +node /tmp/devkit-scrape.mjs "$URL" ``` +Pass the URL as a shell variable (`URL=https://example.com`), never inline into the command string. The `.mjs` reads it from `process.argv[2]` so quotes, backticks, and `$()` in the URL can't escape. + +Then convert HTML → Markdown. Prefer a local converter if available (pandoc, turndown). If none available, strip script/style/nav/footer tags and extract text from article/main/body. + **Firecrawl (if FIRECRAWL_API_KEY is set and --backend firecrawl):** ``` Use Bash to call (use jq to safely construct JSON — never interpolate URLs directly): @@ -96,10 +111,11 @@ When given multiple URLs, scrape them in parallel: ### Error Handling -- If Jina Reader returns an error or empty content, fall back to Playwright (if installed), then WebFetch -- If Playwright fails or isn't installed, fall back to Firecrawl (if API key set), then WebFetch -- If a URL is unreachable, report the error and continue with remaining URLs -- Never silently drop a URL — always report what happened +**Fallback chain (single linear order):** Jina Reader → Playwright → Firecrawl → WebFetch. Each backend is tried in order. Move to the next only on error, empty content, or missing dependency. + +- **Report which backend actually ran** for each URL, and (if fallbacks happened) why the earlier ones failed. Never silently substitute content — the user must know a paywall/cookie-wall/SPA-skeleton from one backend wasn't the real page fetched by another. +- If a URL is unreachable by every backend, report the error and continue with remaining URLs. +- Never silently drop or substitute a URL — always report what happened, with the winning backend named. ## Output @@ -131,8 +147,8 @@ For multiple URLs, output a JSON array. ## Rules -- **URL validation** — only accept `http://` and `https://` URLs. Reject `file://`, `ftp://`, `data:`, and all other schemes. Reject URLs targeting private/reserved IPs: `localhost`, `127.0.0.1`, `0.0.0.0`, `169.254.x.x`, `10.x.x.x`, `172.16-31.x.x`, `192.168.x.x`, `[::1]`. Reject URLs containing `@` (credential-in-URL attacks). -- **No shell injection** — never interpolate user URLs directly into shell command strings. Use `jq` to construct JSON payloads for curl. +- **URL validation** — only accept `http://` and `https://` URLs. Reject `file://`, `ftp://`, `data:`, and all other schemes. Reject URLs targeting private/reserved IPs: `localhost`, `127.0.0.1`, `0.0.0.0`, `169.254.x.x` (cloud metadata — AWS/GCP/Azure), `10.x.x.x`, `172.16-31.x.x`, `192.168.x.x`, `[::1]`. Reject URLs containing `@` (credential-in-URL attacks). When rejecting, report the exact reason and stop — never silently skip a URL in a batch. +- **No shell injection** — never interpolate user URLs directly into shell command strings. Use `jq` to construct JSON payloads for curl. For Playwright, write scripts to files that read URLs from `process.argv`, and pass the URL as a shell variable — never inline into `-e` strings. - **API keys from env only** — never hardcode API keys. Always reference `$JINA_API_KEY`, `$FIRECRAWL_API_KEY` from environment variables. - Always try Jina Reader first — it's free and produces the cleanest output - Respect rate limits — if scraping many URLs, add a brief pause between requests diff --git a/skills/screenshot/SKILL.md b/skills/screenshot/SKILL.md index acd38af..2626102 100644 --- a/skills/screenshot/SKILL.md +++ b/skills/screenshot/SKILL.md @@ -43,10 +43,12 @@ From the user's message, extract: ## Step 3: Validate the URL Reject: -- Non-`http(s)` schemes (`file://`, `ftp://`, `data:`) -- Private/reserved IPs: `localhost`, `127.0.0.1`, `0.0.0.0`, `10.x.x.x`, `172.16-31.x.x`, `192.168.x.x`, `169.254.x.x`, `[::1]` — unless the user explicitly wants to test localhost +- Non-`http(s)` schemes (`file://`, `ftp://`, `data:`, all others) +- Private/reserved IPs: `localhost`, `127.0.0.1`, `0.0.0.0`, `10.x.x.x`, `172.16-31.x.x`, `192.168.x.x`, `169.254.x.x` (cloud metadata — AWS/GCP/Azure), `[::1]` — unless the user explicitly wants to test localhost - URLs containing `@` (credential-in-URL attack vector) +On rejection: **report the exact reason and stop**. Never silently skip an invalid URL — if the user passed a batch, name which URL failed validation and why. + ## Step 4: Execute Choose the command that matches the capture mode. Always quote the URL and output path. @@ -76,21 +78,36 @@ npx playwright screenshot --device="{device}" "{url}" "{output}" npx playwright pdf "{url}" "{output}.pdf" ``` -**Element selector** (Playwright CLI doesn't expose this directly, so use a small script): +**Element selector** (Playwright CLI doesn't expose this directly, so write a script file and pass args via `process.argv` — never interpolate user strings into `node -e`): + ```bash -node -e " -const { chromium } = require('playwright'); -(async () => { - const browser = await chromium.launch(); +cat > /tmp/devkit-shot-element.mjs <<'EOF' +import { chromium } from 'playwright'; + +const [, , url, selector, output] = process.argv; +if (!url || !selector || !output) { + console.error('usage: node devkit-shot-element.mjs '); + process.exit(2); +} + +const browser = await chromium.launch(); +try { const page = await browser.newPage(); - await page.goto({url_as_json}); - await page.locator({selector_as_json}).screenshot({ path: {output_as_json} }); - await browser.close(); -})(); -" + await page.goto(url, { waitUntil: 'networkidle', timeout: 30000 }); + await page.locator(selector).screenshot({ path: output }); + console.log(output); +} catch (err) { + console.error('screenshot failed:', err.message); + process.exit(1); +} finally { + try { await browser.close(); } catch (e) { /* suppress close errors */ } +} +EOF + +node /tmp/devkit-shot-element.mjs "$URL" "$SELECTOR" "$OUTPUT" ``` -Use `JSON.stringify`-safe values (never interpolate raw user strings into `-e`). +Pass all user values via shell variables. The `.mjs` reads them from `process.argv` so quoting, `$()`, and backticks in the input can't escape. ## Step 5: Report From e6ec5fe6edbe2423d57b23f6e0908b5213c30480 Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Fri, 10 Apr 2026 00:35:27 -0400 Subject: [PATCH 3/4] refactor: simplify Playwright skills per mega-pr review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From code-simplifier and comment-analyzer findings: browser: - Trim "When to Use" table (redundant with description frontmatter) - Collapse Step 1 preflight (3 lines vs 14) - Merge Parse + Clarify into one section - Remove mid-step restatements of the "no raw interpolation" rule (kept once in Rules section only) - Remove credential-restatement after Mode D (covered in Rules) - Remove dead "Auth state reuse / storageState" rule (no example, not actionable) screenshot: - Collapse Step 1 preflight (same as browser) - Merge Step 2 (Parse) + Step 3 (Validate) into one "Parse and Validate" step — validation is part of parsing the URL arg - Fix --viewport-size example: concrete "1280,800" instead of "{width},{height}" placeholder (could be left unsubstituted) - Renumber subsequent steps (old Step 4 → 3, Step 5 → 4) - Rewrite Rules: full URL validation block (matching other skills) + close-safely rule to match browser pattern. Drop vague "URL validation first" bullet; rule is now concrete. scrape: - Remove "or as auto-fallback for JS-heavy sites" parenthetical — contradicted the clean linear fallback chain in Error Handling - Remove duplicate "record this in the backend-ran report" from Step 2 (error handling section already mandates it) - Clarify "tell the user once per session" for Playwright install hint Net: ~65 lines removed across three files, zero functionality loss. Final: browser 161L, screenshot 105L, scrape 156L = 422L total. --- skills/browser/SKILL.md | 55 ++++++--------------------------- skills/scrape/SKILL.md | 7 ++--- skills/screenshot/SKILL.md | 62 +++++++++++--------------------------- 3 files changed, 31 insertions(+), 93 deletions(-) diff --git a/skills/browser/SKILL.md b/skills/browser/SKILL.md index 936917e..9e9e438 100644 --- a/skills/browser/SKILL.md +++ b/skills/browser/SKILL.md @@ -7,49 +7,21 @@ description: Automate a web browser — use when asked to click buttons, fill fo Drive a real browser via Playwright to interact with pages — click, fill forms, extract data from SPAs, run multi-step flows, or record user interactions. -## When to Use This vs Related Skills - -| User asks for... | Use | -|---|---| -| "fetch this article as markdown" | `scrape` (Jina is faster for static content) | -| "screenshot this page" | `screenshot` | -| "extract data from this JS-heavy page" | **`browser`** | -| "log into X and download Y" | **`browser`** | -| "fill this form and submit" | **`browser`** | -| "record me clicking through this flow" | **`browser`** (codegen) | -| "test my web app end-to-end" | **`browser`** | +Use this skill for anything beyond simple article scraping: JS-heavy data extraction, form filling, multi-step auth flows, codegen recording, or E2E testing. For static article→markdown use `scrape`; for pure image capture use `screenshot`. ## Step 1: Verify Playwright -```bash -npx playwright --version -``` - -If not installed, stop and tell the user: - -``` -This requires Playwright (optional devkit dependency). - -Install with: - npx playwright install chromium -``` - -Do not attempt workarounds. +Run `npx playwright --version`. If it fails, stop and tell the user: `Playwright required — install with: npx playwright install chromium`. Do not attempt workarounds. ## Step 2: Parse the Request -Understand what the user wants: - -- **Target URL(s)** — starting page -- **Actions** — navigate, click, fill, extract, screenshot, wait -- **Data to extract** — what fields, what format (JSON usually) -- **Auth** — are credentials needed? (prompt if user hasn't provided) -- **Repeatability** — one-off or should this become a reusable script? +- **Target URL(s)** — starting page (validate per Rules before launching) +- **Actions** — navigate, click, fill, extract, wait +- **Data** — what fields, what format (JSON usually) +- **Auth** — are credentials needed? Read from env, never hardcode +- **Repeatability** — one-off throwaway, or save script for reuse? -If the user's request is vague ("scrape this site"), clarify: -- Which data fields do you want? -- Does it require login? -- Is it a one-shot or will you re-run this? +If the request is vague ("scrape this site"), clarify which fields, whether login is needed, and whether it'll be re-run. ## Step 3: Choose the Right Mode @@ -68,9 +40,7 @@ Opens a browser. User interacts. Playwright prints the equivalent script. Best f ### Mode B — Inline script (one-off) -For quick, throwaway extractions, write a small script to `/tmp/` and run it: - -Write the script to a file that reads the URL from `process.argv`, then pass it via a shell variable. Never interpolate user input into the script source itself. +Write the script to a file that reads args from `process.argv`, then pass values via shell variables: ```bash cat > /tmp/devkit-browser-flow.mjs <<'EOF' @@ -106,8 +76,6 @@ EOF URL=https://example.com node /tmp/devkit-browser-flow.mjs "$URL" ``` -Never build scripts via long `-e "..."` strings with interpolated user input — write the script to a file that reads args from `process.argv`, then pass values as shell variables. - ### Mode C — Persistent test (for web apps) If the user is building a web app and wants E2E tests: @@ -125,7 +93,7 @@ Then write test specs in `tests/*.spec.ts`. ### Mode D — Form fill + auth flow -For "log in and grab something": +Same pattern as Mode B, plus `page.fill` / `page.click` for the login form. Credentials come from env, never from hardcoded strings or the command line: ```bash cat > /tmp/devkit-browser-auth.mjs <<'EOF' @@ -163,8 +131,6 @@ EOF EMAIL="$EMAIL" PASSWORD="$PASSWORD" node /tmp/devkit-browser-auth.mjs "$LOGIN_URL" "$TARGET_URL" ``` -Never hardcode credentials. Always read from env vars or prompt the user. Pass URLs as shell variables, never inline into the script source. - ## Step 4: Report After running, report: @@ -182,7 +148,6 @@ After running, report: - **Always close the browser safely** — use `try/catch/finally` with the `close()` call in its own try/catch, so a close-time error never masks the real error. - **Headless by default** — only use `headless: false` when user explicitly wants it (e.g., codegen, debugging). - **Respect the site** — no CAPTCHA bypass, no hammering, no scraping the user doesn't own. -- **Auth state reuse** — for repeated runs against the same site, save `storageState` to avoid re-login. - **Default timeout 30s** — if slow, use `page.waitForSelector()` rather than longer fixed waits, and explain why. ## Troubleshooting diff --git a/skills/scrape/SKILL.md b/skills/scrape/SKILL.md index 3a5ecdd..69f11f2 100644 --- a/skills/scrape/SKILL.md +++ b/skills/scrape/SKILL.md @@ -46,12 +46,11 @@ This returns: { "url": "...", "title": "...", "content": "..." } where "content" is the Markdown. ``` -**Playwright (if installed and --backend playwright, or as auto-fallback for JS-heavy sites):** +**Playwright:** -First validate the URL against the rules below (http/https only, no private IPs, no `@`). +Validate the URL first (see Rules: http/https only, no private IPs, no `@`). -Check availability: `npx playwright --version`. If not installed, tell the user once: -`Playwright not installed. Install with: npx playwright install chromium` — then fall through to the next backend AND record this in the backend-ran report so the user knows which backend actually served the result. +Check availability: `npx playwright --version`. If not installed, tell the user once per session: `Playwright not installed. Install with: npx playwright install chromium`, then fall through to the next backend. The winning backend is always reported in Step 3 (Error Handling), so the user sees which one served the result. Never build scripts via inline `-e "..."` strings. Write the script to a file that reads the URL from `process.argv[2]`, then invoke it: diff --git a/skills/screenshot/SKILL.md b/skills/screenshot/SKILL.md index 2626102..e0506b0 100644 --- a/skills/screenshot/SKILL.md +++ b/skills/screenshot/SKILL.md @@ -9,47 +9,21 @@ Capture screenshots and PDFs of web pages using Playwright. ## Step 1: Verify Playwright -Before doing anything else, check that Playwright is installed: +Run `npx playwright --version`. If it fails, stop and tell the user: `Playwright required — install with: npx playwright install chromium`. Do not attempt workarounds. -```bash -npx playwright --version -``` - -If the command fails or Playwright is not installed, stop and tell the user: - -``` -This requires Playwright (optional devkit dependency). - -Install with: - npx playwright install chromium +## Step 2: Parse and Validate -Chromium is ~170MB. You can also install firefox or webkit. -``` - -Do NOT attempt to work around a missing install. The skill cannot function without it. - -## Step 2: Parse the Request +Extract from the user's request: -From the user's message, extract: - -- **URL** — the page to capture (required; validate http/https only) -- **Output path** — where to save (default: `./screenshot-{timestamp}.png` in cwd) -- **Capture mode** — viewport (default), full-page, or specific element -- **Device emulation** — iPhone, Pixel, iPad, etc. (optional) -- **Viewport size** — custom width x height (optional) +- **URL** — required; validated below +- **Output path** — default `./screenshot-{timestamp}.png` in cwd +- **Capture mode** — viewport (default), full-page, element selector +- **Device / viewport** — optional emulation or custom size - **Format** — PNG (default) or PDF -- **Selector** — CSS selector for element-only capture (optional) - -## Step 3: Validate the URL - -Reject: -- Non-`http(s)` schemes (`file://`, `ftp://`, `data:`, all others) -- Private/reserved IPs: `localhost`, `127.0.0.1`, `0.0.0.0`, `10.x.x.x`, `172.16-31.x.x`, `192.168.x.x`, `169.254.x.x` (cloud metadata — AWS/GCP/Azure), `[::1]` — unless the user explicitly wants to test localhost -- URLs containing `@` (credential-in-URL attack vector) -On rejection: **report the exact reason and stop**. Never silently skip an invalid URL — if the user passed a batch, name which URL failed validation and why. +**Validate the URL** (see Rules for the full list): only `http(s)`, no private/reserved IPs or cloud metadata, no `@`. On rejection, report the exact reason and stop — never silently skip in a batch. -## Step 4: Execute +## Step 3: Execute Choose the command that matches the capture mode. Always quote the URL and output path. @@ -63,9 +37,9 @@ npx playwright screenshot "{url}" "{output}" npx playwright screenshot --full-page "{url}" "{output}" ``` -**Custom viewport:** +**Custom viewport** (format is `W,H` with no spaces, e.g. `1280,800`): ```bash -npx playwright screenshot --viewport-size={width},{height} "{url}" "{output}" +npx playwright screenshot --viewport-size=1280,800 "{url}" "{output}" ``` **Device emulation:** @@ -109,7 +83,7 @@ node /tmp/devkit-shot-element.mjs "$URL" "$SELECTOR" "$OUTPUT" Pass all user values via shell variables. The `.mjs` reads them from `process.argv` so quoting, `$()`, and backticks in the input can't escape. -## Step 5: Report +## Step 4: Report Output the absolute path and any metadata: @@ -123,9 +97,9 @@ If the page failed to load, report the error clearly with the HTTP status or tim ## Rules -- **URL validation first** — never launch the browser on an invalid or private URL -- **No shell injection** — never concatenate user URLs into shell strings without proper quoting -- **Headless only** — never open a visible browser window -- **Respect gated content** — don't screenshot login-walled or paywalled content the user doesn't own -- **Default to cwd** — put output files in the current working directory unless the user specifies otherwise -- **One browser, one close** — always `await browser.close()` to avoid orphaned processes +- **URL validation** — only `http://` and `https://`. Reject `file://`, `ftp://`, `data:`, and all other schemes. Reject private/reserved IPs: `localhost`, `127.0.0.1`, `0.0.0.0`, `10.x.x.x`, `172.16-31.x.x`, `192.168.x.x`, `169.254.x.x` (cloud metadata — AWS/GCP/Azure), `[::1]` — unless user explicitly tests localhost. Reject URLs with `@`. On rejection: report exact reason and stop, never silently skip. +- **No raw interpolation** — for the element-selector script, pass values via shell variables and `process.argv`, never via `node -e "..."` with user input. +- **Headless only** — never open a visible browser window. +- **Respect gated content** — don't screenshot login-walled or paywalled content the user doesn't own. +- **Default to cwd** — put output files in the current working directory unless the user specifies otherwise. +- **Close safely** — when using an inline script, wrap `browser.close()` in its own try/catch inside `finally` so a close-time error can't mask the real error. From d63f3717eead4af9705e42f0f8a6b74bc637e9bb Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Fri, 10 Apr 2026 00:39:45 -0400 Subject: [PATCH 4/4] =?UTF-8?q?fix(ci):=20update=20ROADMAP=20skill=20count?= =?UTF-8?q?=2020=20=E2=86=92=2022?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add screenshot + browser to ROADMAP's skill enumeration so validate-counts CI job passes. --- ROADMAP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ROADMAP.md b/ROADMAP.md index 0871b88..ef58b0e 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -5,7 +5,7 @@ - **MCP engine** — Go server exposes `devkit_start`, `devkit_advance`, `devkit_status`, `devkit_list` tools inside Claude Code. Step ordering enforced via MCP tool scoping + PreToolUse hook exit 2. Session state in session.json (hot path, <50ms hook reads) + SQLite (cold history). ~65% token reduction vs old monolithic prompts. - **6 slash commands** — Tab-completable entry points for things that need explicit invocation (tri-review, tri-debug, tri-security, status, setup-rules, workflow); 18 former commands now context-activated via skills or invoked via MCP tools (pr-monitor folded into pr-ready workflow; pr-ready is now a natural-language skill) - **Deterministic workflow conversion** — All command logic moved from LLM-interpreted markdown to Go-engine-driven YAML workflows; ~3,600 lines of inline logic removed -- **20 context-activated skills** — 10 auto-trigger workflows (test-gen, doc-gen, changelog, onboard, research, deep-research, scrape, autoloop, adr, pr-ready) + 6 coding principles (executing, clean-code, DRY, YAGNI, dont-reinvent, stuck) + 2 tools (gcli, creating-workflows) + 1 iteration memory (scratchpad) + 1 orchestration (mega-pr) +- **22 context-activated skills** — 10 auto-trigger workflows (test-gen, doc-gen, changelog, onboard, research, deep-research, scrape, autoloop, adr, pr-ready) + 6 coding principles (executing, clean-code, DRY, YAGNI, dont-reinvent, stuck) + 2 tools (gcli, creating-workflows) + 1 iteration memory (scratchpad) + 1 orchestration (mega-pr) + 2 browser automation (screenshot, browser) - **6 agents** — Scoped tool access, worktree isolation, model assignment - **12 hooks** — Safety (destructive command blocking, edit-time security patterns, PR gate), observability (audit trail, slop detection, post-validation, subagent verification, language-aware code review), optimization (RTK token compression), workflow enforcement (devkit-guard, devkit-stop-guard) - **Graceful degradation** — tri:* commands work with 1-3 agents depending on installed CLIs