Skip to content

usage.md

Latest commit

 

History

History
174 lines (114 loc) · 6.03 KB

usage.md

File metadata and controls

174 lines (114 loc) · 6.03 KB
Raw

Usage Guide

Common ways to use pi-webaio from inside pi.

Examples

Search the web

Use aio-websearch to find the latest React documentation

Google search is on by default (via headless Chrome CDP). Reddit is opt-in: pass reddit: true to include the Reddit CDP lane (also requires Chrome). To skip Google:

Use aio-websearch to search for "Rust serde" (google: false)

Scout URLs with minimal tokens — one line per result, no snippet:

Use aio-websearch to find the Vite docs (compact: true)

TinyFish, FireCrawl, and Parallel run automatically as parallel bonus providers (FireCrawl needs no key; TinyFish and Parallel need TINYFISH_API_KEY / PARALLEL_API_KEY in ~/.piwebaio/config or ~/.piwebaio/.env). (FireCrawl needs no key; TinyFish needs TINYFISH_API_KEY in ~/.piwebaio/config or ~/.piwebaio/.env). For fetching a specific URL through one of them instead of the normal pipeline, pass a flag:

Use aio-webfetch with firecrawl: true to download https://example.com/page
Use aio-webfetch with tinyfish: true to download https://example.com/page

Fetch a single URL

Use aio-webfetch to download https://example.com/article

After fetching, use the built-in read tool to inspect the full saved file.

See a page's shape before reading it (outline mode)

Use aio-webfetch to download https://expressjs.com/en/guide/routing.html (outline: true)

Returns only a ~50-token heading outline (total + per-section word counts). The full content is still saved + cached — then fetch just the section you want with query, or the whole page with aio-webcontent.

Get a focused answer from one page

Use aio-webfetch to download https://example.com/long-article (query: "how does auth work")

Returns only the top-k BM25-ranked chunks that answer the query, with heading breadcrumbs. Full content stays cached.

Get a cited answer from several pages at once

Use aio-webfetch to download these URLs (query: "migration safety"):
  - https://example.com/docs/migrations
  - https://example.com/blog/schema-changes
  - https://example.com/guide/rollback

Fetches all URLs, ranks chunks across them, and returns the top-k most relevant chunks each cited with its source URL + heading + score — verbatim supporting text, not a generated answer.

Fetch multiple URLs in batch

Use aio-webfetch to download these URLs:
  - https://example.com/page1
  - https://example.com/page2
  - https://example.com/page3

Fetch as JSON for structured downstream processing

Use aio-webfetch to download https://api.github.com/repos/apmantza/pi-webaio (format: "json")

Returns a structured JSON object with url, title, author, published, site, language, wordCount, content, rawHtml. Useful for piping into other tools.

Fetch with RAG chunking

Use aio-webfetch to download https://en.wikipedia.org/wiki/Node.js (chunks: true, maxTokens: 512)

Splits the markdown into paragraph-bounded chunks with 50-token overlap. Result includes both the markdown and a chunks array.

Fetch a GitHub Actions run log

Use aio-webfetch to download https://api.github.com/repos/apmantza/pi-drykiss/actions/runs/27479618304/logs

Routes through gh run view --log (uses your existing gh auth login session) to get plain-text logs with auth + 302-redirect handling. No more HTTP 403.

Fetch with a specific browser fingerprint

Use aio-webfetch to download https://example.com (browser: "firefox_147", os: "linux")

Retrieve stored content (no re-download)

Use aio-webcontent to get the full content from https://example.com/article

See what changed since the last fetch (section-level diff):

Use aio-webcontent to diff https://example.com/article (diff: true)

Pull an entire site

Use aio-webpull to download https://docs.example.com (max: 50 pages)

Pull with URL pattern routing

Use aio-webpull to download https://example.com with routes:
  - { pattern: "*/api/*", mode: "fast" }
  - { pattern: "*/docs/*", mode: "browser" }

Routes different URL patterns to different fetcher modes. First match wins.

Pull with resume from checkpoint

Use aio-webpull to download https://docs.example.com (resume: true)

Skips pages that were already pulled (checks for existing .md files in the output directory).

Bypass a paywall (single URL)

Use aio-webfetch to download https://www.nytimes.com/2024/01/01/some-article (bypass: true)

If the normal fetch hits a paywall, pi-webaio tries archive → ua:googlebot → ua:bingbot → ua:facebookbot → referer:google → block_js → cookies in order, returning the first response that doesn't contain paywall markers.

Bypass with a custom strategy chain

Use aio-webfetch to download https://example.com/paywalled (bypass: true, bypassStrategies: ["archive", "ua:googlebot"])

Only tries Wayback Machine and Googlebot impersonation. Useful when you know a site only responds to specific strategies.

Bypass on a whole pull (every page)

Use aio-webpull to download https://www.ft.com (max: 50, bypass: true)

Applies the per-domain strategy chain to every page in the pull. NYT pages use block_js → archive; FT pages use block_js → archive; unknown sites fall through to the generic chain.

Headless Chrome

Chrome (Google/Reddit CDP search, AI summaries) is always headless (--headless=new on a dedicated port-9222 profile) — even on Linux desktops with $DISPLAY set, so no window pops up or steals focus on any operation.

  • Visible window is explicit opt-in only: GREEDY_SEARCH_VISIBLE=1.
  • --no-sandbox is passed only when needed: automatically when running as root, or forced with PI_WEBAIO_NO_SANDBOX=1 (force off with =0). As a normal user no flag is passed, so Chrome's "unsupported command-line flag" infobar never appears.
  • If an old visible Chrome is still running from before this change, stop it once with node bin/launch.mjs --kill — the next launch comes up headless.