web-use: add stdlib Browserless helpers

skills/web-use/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: web-use
-description: "Use for any task that touches a web page or web data. Routes between web_search/web_fetch, Browserless/TinyFish/protected extraction, OpenClaw browser, and live browser context for login, 2FA, CAPTCHA, current tabs, carts, or checkout. Domain skills keep site-specific policy."
+description: "Use for any task that touches a web page or web data. Routes between web_search/web_fetch, Browserless/TinyFish/protected extraction, OpenClaw browser, and live browser context for login, 2FA, CAPTCHA, current tabs, or account state. Domain skills keep site-specific policy."
 ---
 
 # Web Use
@@ -23,7 +23,7 @@ Keep those questions separate even when one workflow needs both.
 | Read a simple public URL | `web_fetch` |
 | Inspect a JS-rendered page visually | OpenClaw `browser` |
 | Extract protected or bot-gated data | `references/extraction-backends.md` |
-| Use a current tab, login, 2FA, CAPTCHA, extension, cart, or checkout | `references/context-device.md` |
+| Use a current tab, login, 2FA, CAPTCHA, extension, or account state | `references/context-device.md` |
 | Site-specific paid API or domain policy | Relevant domain skill |
 
 Do not use a browser when search or fetch is enough. Do not use Browserless,
@@ -87,7 +87,7 @@ Use browser context routing when the task needs one of these:
 - preserving or using an existing logged-in tab/session
 - a visible browser on the user's device
 - a browser extension lane
-- CAPTCHA, 2FA, manual review, cart, checkout, or account state
+- CAPTCHA, 2FA, manual review, or account state
 - user-facing UX/design for browser intents
 
 Plain-English labels:
@@ -130,9 +130,10 @@ and phrasing guide.
 
 Use these from this skill directory when repeatable execution helps:
 
-- `scripts/browserless_extract.py` - Browserless `content`, `unblock`, or `stealth-bql`
+- `scripts/browserless_extract.py` - Browserless `content`, `unblock`, or `stealth-bql` using stdlib HTTP; reads `BROWSERLESS_TOKEN` or legacy `BROWSERLESS_API_KEY`
+- `scripts/browserless_media_requests.py` - Browserless `/function` network media discovery for rendered pages where media URLs appear only after playback/rendering
 - `scripts/browserless_session.py` - opt-in persistent Browserless session helper with redacted output and 0600 session files
-- `scripts/tinyfish_browser_extract.py` - TinyFish Browser API / CDP extraction helper
+- `scripts/tinyfish_browser_extract.py` - optional TinyFish Browser API / CDP extraction helper
 
 See `references/backends.md.example` for credential and command templates.
 
@@ -141,3 +142,7 @@ See `references/backends.md.example` for credential and command templates.
 - `references/context-device.md` - browser context/device/session matrix
 - `references/extraction-backends.md` - backend ladder and safety notes
 - `references/backends.md.example` - public-safe credential and command template
+
+## Baseline Checks
+
+Run `bash scripts/test.sh` after editing this skill.

skills/web-use/references/backends.md.example

@@ -13,7 +13,7 @@ Copy this file to `backends.md` and fill in the env var names or 1Password refer
 | Browserless (`/session`) | opt-in persistent protected extraction session | No | use sparingly only when cookies/localStorage/sessionStorage/cache must persist across repeated same-site BQL/CDP calls |
 | TinyFish Browser API (CDP session) | remote stealth browser primitive for brittle or multi-step flows | Secondary / situational | works on protected pages when used as a browser session |
 | Site-specific structured data API | clean structured fields for one site | No | domain skill owns credential, cost policy, and when it is worth a call |
-| OpenClaw browser | interactive human-visible browsing | No | use for login, 2FA, manual review, checkout |
+| OpenClaw browser | interactive human-visible browsing | No | use for login, 2FA, manual review, or account state |
 
 ---
 
@@ -33,6 +33,9 @@ export BROWSERLESS_TOKEN="..."
 export TINYFISH_API_KEY="..."
 ```
 
+`BROWSERLESS_API_KEY` is accepted as a legacy fallback by the local helper
+scripts, but new config should use `BROWSERLESS_TOKEN`.
+
 ---
 
 ## Policy
@@ -96,6 +99,18 @@ python3 skills/web-use/scripts/browserless_session.py stop \
 
 Do not paste the session file contents, `browserQL`, `connect`, or `stop` URLs into chat or logs. They include the Browserless token and should be treated as bearer credentials.
 
+### Browserless media-request helper
+
+```bash
+BROWSERLESS_TOKEN="<your token>" \
+  python3 skills/web-use/scripts/browserless_media_requests.py \
+  <URL>
+```
+
+Use this when a page exposes media only after rendering or playback, such as a
+public social post/Reel where `yt-dlp` failed. Returned CDN URLs are temporary;
+store source URL, method, fetch time, and summarized media metadata by default.
+
 ### TinyFish helper
 
 ```bash

skills/web-use/references/extraction-backends.md

@@ -72,7 +72,7 @@ session:
 - 2FA
 - CAPTCHA/manual solve
 - visual confirmation
-- final shopping steps
+- user-confirmed account actions
 
 ## Protected Page Escalation
 
@@ -87,6 +87,7 @@ When doing repeatable operational work, prefer the bundled helpers before
 rewriting one-off curl/CDP glue:
 
 - `../scripts/browserless_extract.py`
+- `../scripts/browserless_media_requests.py`
 - `../scripts/browserless_session.py`
 - `../scripts/tinyfish_browser_extract.py`
 

skills/web-use/scripts/browserless_extract.py

@@ -2,12 +2,12 @@
 """Fetch a page through Browserless and return normalized extraction JSON.
 
 Examples:
-  BROWSERLESS_TOKEN="<your token>" \
-    python3 skills/web-use/scripts/browserless_extract.py \
+  BROWSERLESS_TOKEN="<set-at-runtime>" \
+    python3 scripts/browserless_extract.py \
     https://example.com/protected-page
 
-  BROWSERLESS_TOKEN="<your token>" \
-    python3 skills/web-use/scripts/browserless_extract.py \
+  BROWSERLESS_TOKEN="<set-at-runtime>" \
+    python3 scripts/browserless_extract.py \
     https://example.com/protected-page \
     --mode unblock
 """
@@ -21,14 +21,41 @@
 import sys
 from html import unescape
 from typing import Any
-
-import requests
+from urllib.error import HTTPError, URLError
+from urllib.parse import urlencode
+from urllib.request import Request, urlopen
 
 DEFAULT_HOST = "https://production-sfo.browserless.io"
 DEFAULT_TIMEOUT = 120
 DEFAULT_SNIPPET_CHARS = 3000
 
 
+class BrowserlessRequestError(RuntimeError):
+    """Raised when Browserless returns a non-2xx response."""
+
+
+def post_browserless(url: str, payload: dict[str, Any], timeout: int) -> tuple[int, str]:
+    body = json.dumps(payload).encode("utf-8")
+    request = Request(
+        url,
+        data=body,
+        headers={"Content-Type": "application/json"},
+        method="POST",
+    )
+    try:
+        with urlopen(request, timeout=timeout) as response:
+            return response.status, response.read().decode("utf-8", errors="replace")
+    except HTTPError as exc:
+        error_body = exc.read().decode("utf-8", errors="replace")
+        raise BrowserlessRequestError(f"HTTP {exc.code}: {error_body[:1000]}") from exc
+    except URLError as exc:
+        raise BrowserlessRequestError(str(exc.reason)) from exc
+
+
+def browserless_url(host: str, path: str, token: str) -> str:
+    return f"{host}{path}?{urlencode({'token': token})}"
+
+
 def clean_text(text: str) -> str:
     return re.sub(r"\s+", " ", unescape(text or "")).strip()
 
@@ -77,44 +104,39 @@ def sanitize_error(error: Exception, token: str | None) -> str:
 
 
 def request_content(token: str, url: str, host: str, timeout: int) -> dict[str, Any]:
-    response = requests.post(
-        f"{host}/content",
-        params={"token": token},
-        json={"url": url},
-        timeout=timeout,
+    status, html = post_browserless(
+        browserless_url(host, "/content", token),
+        {"url": url},
+        timeout,
     )
-    response.raise_for_status()
-    html = response.text
     title = extract_title_from_html(html)
     body = strip_html(html)
     return {
         "provider": "browserless",
         "mode": "content",
         "url": url,
-        "status_code": response.status_code,
+        "status_code": status,
         "title": title,
         "body": body,
         **signals(title, body),
     }
 
 
 def request_unblock(token: str, url: str, host: str, timeout: int) -> dict[str, Any]:
-    response = requests.post(
-        f"{host}/unblock",
-        params={"token": token},
-        json={"url": url, "browserWSEndpoint": False},
-        timeout=timeout,
+    status, response_text = post_browserless(
+        browserless_url(host, "/unblock", token),
+        {"url": url, "browserWSEndpoint": False},
+        timeout,
     )
-    response.raise_for_status()
-    payload = response.json()
+    payload = json.loads(response_text)
     html = payload.get("content") or ""
     title = extract_title_from_html(html)
     body = strip_html(html)
     return {
         "provider": "browserless",
         "mode": "unblock",
         "url": url,
-        "status_code": response.status_code,
+        "status_code": status,
         "title": title,
         "body": body,
         "browser_ws_endpoint": payload.get("browserWSEndpoint"),
@@ -132,14 +154,12 @@ def request_stealth_bql(token: str, url: str, host: str, timeout: int, solve: bo
         'body: text(selector: "body") { text } '
         "}"
     )
-    response = requests.post(
-        f"{host}/stealth/bql",
-        params={"token": token},
-        json={"query": query, "variables": {"url": url}},
-        timeout=timeout,
+    _, response_text = post_browserless(
+        browserless_url(host, "/stealth/bql", token),
+        {"query": query, "variables": {"url": url}},
+        timeout,
     )
-    response.raise_for_status()
-    payload = response.json()
+    payload = json.loads(response_text)
     if payload.get("errors"):
         raise RuntimeError(json.dumps(payload["errors"], indent=2))
     data = payload.get("data") or {}
@@ -158,10 +178,7 @@ def request_stealth_bql(token: str, url: str, host: str, timeout: int, solve: bo
 
 
 def parse_args() -> argparse.Namespace:
-    parser = argparse.ArgumentParser(
-        description=__doc__,
-        formatter_class=argparse.RawDescriptionHelpFormatter,
-    )
+    parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument("url", help="Target URL")
     parser.add_argument(
         "--mode",
@@ -170,7 +187,11 @@ def parse_args() -> argparse.Namespace:
         help="Browserless surface to use",
     )
     parser.add_argument("--host", default=DEFAULT_HOST, help="Browserless host")
-    parser.add_argument("--token", default=os.environ.get("BROWSERLESS_TOKEN"), help="Browserless API token")
+    parser.add_argument(
+        "--token",
+        default=os.environ.get("BROWSERLESS_TOKEN") or os.environ.get("BROWSERLESS_API_KEY"),
+        help="Browserless API token",
+    )
     parser.add_argument("--timeout", type=int, default=DEFAULT_TIMEOUT, help="Request timeout in seconds")
     parser.add_argument(
         "--snippet-chars",