From bc9314104c518bb5fcac5753e3ee2e57120c7497 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 3 Oct 2026 19:22:42 +0000 Subject: [PATCH] firecrawl: the tested CLI is 1.25.3, its real output captured and read before the move What changed: `TESTED_CLI_VERSION` is 1.25.3, with the comment block in `lib/firecrawl.mjs` extended by what the release is and what the capture showed. Four fixtures beside the 1.25.2 ones (`test/fixtures/firecrawl-{status,search,scrape}-1.25.3.*`, the scrape's stderr kept apart) and three tests feeding them through `parseStatus`, `normalizeSearch`, `creditsUsed` and `normalizeScrape`; the 1.25.2 status test now asserts its own release rather than the tested one, as the 1.24.6 one did before it. README's install line and table row, the architecture map's firecrawl row, the decision index row and the CHANGELOG say the pin moved. Why: the operator asked for the update after upgrading the CLI on the collector machine, where doctor reported 1.25.3 against a 1.25.2 pin (same major, so it passed). The release is one commit, "fix: make --objective optional for alexandria feedback" (firecrawl/cli#301, read from a shallow clone; the releases page answers 403 to this session), with nothing on scrape, search or --status, and the bundle is unchanged (SDK 4.40.0, axios 1.18.0, node >= 22). The decision project's rule (docs/decisions/2026-10-02-firecrawl-cli-1-25) is to move only on a capture of the real output, so it was captured anyway: a scratch npm prefix, the key given per command, three credits (2026-10-03). A vendor update is allowed under the freeze (ADR-0117). What it touched: lib/firecrawl.mjs, test/transport.test.mjs, four fixtures; README.md, docs/ARCHITECTURE.md (firecrawl row), docs/decisions/README.md, CHANGELOG Unreleased, research-kit/README.md count 1515. No ADR: the decision project is the record, and its brief is left as written. The fixtures were scanned for the session's keys (0 hits); the search and scrape fixtures carry the docs page's `fc-YOUR_KEY` placeholders, as the 1.25.2 ones do. The status capture shows the account's cycle is now 1,500 credits. What you verified: transport 99/1 before the pin move - the one red is the new status test's version assertion, the fixture's 1.25.3 against the tested 1.25.2 - and transport + doctor + cli 274/0 after; the fixtures parse (search: 3 web rows, 3 tools, creditsUsed 2; scrape: one JSON value on stdout, "Scrape ID" on stderr, 29,978 chars of markdown, status 200). The commit gate runs the full suite on this commit; the CI-equivalent gate runs on a fresh clone before the push. What you got wrong and fixed: my doc script asserted the front README names the pin twice; it names it three times (two on one line), so the script stopped before the map, changelog and index edits and they were applied in a second pass - caught by `git status` showing those files unchanged. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01S1fT6wSqD2LwRjsUuRvDau --- CHANGELOG.md | 5 ++ README.md | 4 +- docs/ARCHITECTURE.md | 2 +- docs/decisions/README.md | 2 +- research-kit/README.md | 2 +- research-kit/lib/firecrawl.mjs | 9 +++- .../fixtures/firecrawl-scrape-1.25.3.json | 1 + .../firecrawl-scrape-1.25.3.stderr.txt | 1 + .../fixtures/firecrawl-search-1.25.3.json | 1 + .../test/fixtures/firecrawl-status-1.25.3.txt | 9 ++++ research-kit/test/transport.test.mjs | 50 +++++++++++++++++-- 11 files changed, 77 insertions(+), 9 deletions(-) create mode 100644 research-kit/test/fixtures/firecrawl-scrape-1.25.3.json create mode 100644 research-kit/test/fixtures/firecrawl-scrape-1.25.3.stderr.txt create mode 100644 research-kit/test/fixtures/firecrawl-search-1.25.3.json create mode 100644 research-kit/test/fixtures/firecrawl-status-1.25.3.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index 128b8096..d0fe683d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,11 @@ Each release of Research-Kit, newest first. The version is `KIT_VERSION` in ## Unreleased +- The tested Firecrawl CLI is 1.25.3 (from 1.25.2): the release is one fix, `--objective` + optional for alexandria feedback (firecrawl/cli#301), with nothing on scrape, search or + `--status`; its output was captured from the real CLI anyway (three credits) and is read + unchanged by the adapter's parsers (`research-kit/test/fixtures/*-1.25.3.*`). `doctor`, the + install line and the workflows' install spec name 1.25.3. - Internal: one search session (`lib/search-session.mjs`, ADR-0135) owns how a query is put to the search providers - the merge, the one degrade to the fetch provider, the rate-limit patience, the meters, the failure rows and the refuse-before-spend readiness gate - and diff --git a/README.md b/README.md index e1e8ca6c..ebf3f883 100644 --- a/README.md +++ b/README.md @@ -85,7 +85,7 @@ exists for Claude Code only; every other agent gets the commit gate. | Node 22+ | everything | Node 22, 24 and 26 are each tested on every commit | | Git | everything | projects are git repositories; the commit gate is a git hook | | Python 3.11+ | the test suite only | the cross-language conformance runners; without it the suite blocks rather than passing | -| Firecrawl CLI 1.25.2 and a Firecrawl account | metered collection | `npm install -g firecrawl-cli@1.25.2`, then `firecrawl login`. Free tier: 1,000 credits a month, no card | +| Firecrawl CLI 1.25.3 and a Firecrawl account | metered collection | `npm install -g firecrawl-cli@1.25.3`, then `firecrawl login`. Free tier: 1,000 credits a month, no card | | Chromium or Chrome | the `browser` transport | free; reads pages built by JavaScript | | SerpAPI key or a SearXNG instance | searching, optional | a second meter, so search stops competing with fetching | @@ -445,7 +445,7 @@ evaluated and deliberately not wired in, because its terms allow retention for t **Firecrawl setup** ```bash -npm install -g firecrawl-cli@1.25.2 +npm install -g firecrawl-cli@1.25.3 firecrawl login ``` diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 45ce3c95..fae5c064 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -140,7 +140,7 @@ is an open decision (2026-09-30): | `credits.mjs` | **What a run does when the paying provider runs dry** — ADR-0086's one switch to the fallback, ADR-0129's stop-and-report. The one owner of a run's exhaustion state: `fellBack` / `stopped`, the `uncollected` and `skippedSearches` accounting, and the single `credits-exhausted` row the failure log keeps per run. The coordinator sequences and asks it which provider serves each call (`exhausted`, `live`, `onExhaustion`, `askLive`) and reads the state back for the summary; before the extraction the state lived as closures inside `runResearch` and `stopped.uncollected` was incremented from three loop sites. | ADR-0086, ADR-0129 | | `search-session.mjs` | **How a query is put to the search providers** (ADR-0135), for both coordinators - `runResearch` and `decompose` each carried a copy, and read side by side on 2026-10-03 the copies had drifted in six places (a failed first ask's usage counted in one, a plain failure in `searchFailures` in the other, `searchTransport` spelled `serpapi` by one and `serpapi+firecrawl-cli` by the other under a merge, the `notReady` refusal in one only, three log lines in one only, `covered` in the other only). `searchSession({ adapter, searchAdapter, searchAdapters, limit, maxRateLimitRetries, sleep, log, record, credits, askOne })` returns `name` (the paying provider, what the usage row's `searchTransport` says), `meter` (the dry run's label, every provider under a merge), `providers`, `assertReady()` (a `searchAdapter` reporting `notReady` throws `SEARCH_PROVIDER_NOT_READY` before any ask), `ask(text)` and `meters()`. `ask` is the loop: every provider under a merge, interleaved by `mergeByRank` and attributed, else the one provider with one bounded degrade to the fetch provider when it `canSearch` and is not the provider that failed; it answers `{ ok, query, results, provider, providers, merged, searchId, degraded }` or `{ ok: false, ..., error }`. **One accounting rule**: what an adapter reports is counted on its meter for every ask, succeeded or failed, and every failed ask is one failure on its provider - a query that failed on both providers is two. **The failure row is the session's, the sink the coordinator's**: `record` gets `{ query, error, provider }`, with `degraded: true, fellBackTo` on a degrade and `covered: true` when another provider of a merge answered; it is handed over at once (so a failure log keeps its chronology beside a `credits-exhausted` row) and gains `covered` afterwards. Provider-level lines (`waiting`, `search failed on`, `degrading to`, `searched`, `search found N`, `search failed`, `search failed on every provider`) are the session's; what a coordinator makes of an answer stays its own. `searchPatiently` and `mergeByRank` live here, re-exported by `research-run.mjs`. With a `credits` policy the policy's `askLive`/`live` decide which provider serves a call (ADR-0133); `askOne` is the one patient call a coordinator shares with that policy. | ADR-0027, ADR-0133, ADR-0135 | | `research-run.mjs` | Many URLs, one run: plan/query/url fan-out, discovery, candidate selection, budget tiers (`DEPTH_SCRAPES`), cache policy. The coordinator; collectOne does the writing, and what a run does when its credits run out is `credits.mjs`'s, asked through one policy object whose state is read back here. A query whose `why` names unknowns that are all CLOSED in the contract is not searched (2026-10-02, ADR-0127): a search has no cache and was paid on every run of a finished project; the skip is said on the terminal and in `discovered[].note`, the dry run previews it, and `--force` runs it. The plan's own URLs are compared with each other by `urlKey` (2026-10-02, break-test pass 3): a page written down in two spellings - `www.`, a trailing slash, a fragment - is queued once, and each other spelling is a `skipped` result naming the spelling that was fetched; until then only search results were compared with the plan, and such a plan paid for the page once per spelling. Since ADR-0027 it drives **two** providers: `searchAdapter` defaults to the fetch adapter, so an un-updated caller is unaffected, and one bounded fallback covers a search-provider outage — reported, never absorbed, because it moves spend back onto the fetch budget. `searchUsage(root)` counts the search meter from the usage log per hour and per month; it **reports and does not block**, because the count is this machine's only and cannot tell a free cached repeat from a billed search, so a guard built on it would refuse work on a number wrong in the blocking direction. **The funnel is `limit` then `perQuery`, and it only works when they differ**: `limit` is how many results the search is asked for, `perQuery` how many survive `selectCandidates`' ranked slice. The collector workflow set both to `max_pages`, so the slice kept everything the search returned and `rankCandidate` - the preferred-domain bonus, the docs/terms bonuses, the blog and forum penalty - could not affect a dispatched collection at all. Measured on job-0922h: 8 results, 8 kept, 7 off topic. Ranking had been inert since `perQuery` was raised to `max_pages` to fix a breadth bug, a change that widened the intake and destroyed the selection in one line. Widening `limit` costs nothing, because a search is billed per search and not per result. **Failures are counted per provider** (`searchFailuresOn`) as well as in total, because the total mixes meters: `searchSummaryLine` prints `searches 0 on serpapi` with "N attempts failed" beside it when the reported provider was asked and failed - on 2026-09-22 the bare zero read as "not used" after a SerpAPI timeout the merge had survived. A clean run prints the line it always did. **`prefer` can name a path on a shared host** (2026-09-26, ADR-0044): `parsePreference` reads an entry as host plus optional path; a bare host covers the host and its subdomains as before, and a host with a path covers that path and below, on exactly that host, at a segment boundary. `prefer: github.com` had given every GitHub page the owner's bonus, and collect run 36279879229 captured a stranger's run page with it. The input descriptions in `collect.yml`, `collect-remote --help` and the MCP tool show the path form, pinned by test. A `prefer` value may be a list or a string (`preferList`, 2026-09-27), at the top level and per query: a string is split on commas and whitespace, as `collect.yml` splits its input. It had been dropped at the top level and spread into single characters per query - silently, both times. **One page, one fetch** (2026-09-27, `urlKey`): "already have this page" compares identities - host case, `www.`, scheme, a trailing slash and the fragment dropped; path case and query kept - in the seen set, in `selectCandidates` (which also keeps one spelling per list) and in decompose's candidate list. Plan URLs enter the seen set before any search runs, and the single-provider path now records its picks as the merged path did. Collect run 36283114657 had paid twice for `.../ui-and-api` and `.../ui-and-api/`. **A page the budget leaves out is said, not only recorded** (2026-09-27): each is logged `skipped ... budget exhausted`, the result carries `overBudget`, and `research.mjs` ends its summary with `left N over the budget`; `--status` states the budget the run will use, min(maxScrapes, the depth's cap), where it quoted the depth's cap alone. A plan naming five pages at maxScrapes 2 had printed two lines and nothing about the other three. The result also carries `maxScrapes`, and the summary's next line says **which cap bound the run** and what lifts it (2026-09-28): the depth's (naming the `--depth` that allows more), the plan's `maxScrapes`, or both at once. A scaffolded plan says depth quick with maxScrapes 10, and a MoonAliza run stopped at 4 without saying the 10 needs a deeper tier; the template's `notes` now state the tiers too. A **dry run names each query** it would search and the provider(s) whose meter it would spend - it had previewed nothing for a plan made of queries. `bin/research.mjs` refuses a plan with no queries and no urls before choosing a transport - exit 2, naming the plan and the shape of a query and a url (2026-09-27): the scaffold's plan is empty, and new-project's step 3 ran on it, printed `collected 0` and exited 0. Before that it refuses a plan file that is missing or does not parse, naming which (the parse error used to read as "no queries and no urls"), and a `--only` that matches no query. Then `planProblems(plan)` (research-run) names every value it cannot use - a depth outside `DEPTHS`, a non-integer count, a query without `q`, a URL that is not http(s) - and research refuses them all at once (2026-09-27): `readPlan` defaulted each in silence, so a query written `{"query": ...}` ran nothing and exited 0. Search credits an adapter estimates are summed as `searchCreditsEstimate` into the usage row and the run's summary line, and `searchUsage` reports `creditsEstimate` for `--status`, labelled as an estimate (2026-09-27, ADR-0065). Vendor-neutral: it sums what adapters report. Every search logs its outcome (`found N` / `search found nothing`), and an empty one is recorded as `op: search-empty` in `.failures.jsonl` (2026-09-27, ADR-0066). One `noteOutcome` reports empty and off-topic searches for both the single-provider and the merged path; the merged path had reported neither. **A result must carry the query before it costs a scrape** (2026-09-27, ADR-0067): `selectCandidates` takes the query and keeps a result only if `matchesQuery` finds `min(2, half)` of its distinctive terms (`queryTerms`; ADR-0068 replaced ADR-0067's "half", which kept 2 of 38 real search results) in the title, snippet or URL, or `isPreferred` puts it on a `prefer` domain. A search whose results all miss scrapes nothing, logs `no result matched`, and records `op: search-off-topic`. A possessive counts as its word (2026-09-29): `Ollama's` had become the term `ollama s`, which no page carries, so a product's own name stopped counting toward the floor. Searches are counted **per paying provider** (`searchesOn`, 2026-09-28): a merged query was logged as two searches under the SerpAPI name and charged to its free-plan meter. The usage row carries `searchesOn`, `searchUsage` counts only the named provider's share of a row that has it (older rows as before), and the summary line names each meter. **A run survives exhausted credits** (2026-09-28, ADR-0086): given a `fallbackAdapter`, the first search or fetch whose failure the adapter's `creditsExhausted` recognises switches every later call that would have gone to it - merged searches and the degraded path included - to the fallback, once, and re-fetches the refused page; the refusal stays a `fail` ledger entry but not a failed page, and the switch is logged, recorded as `op: credits-exhausted`, and returned as `fellBack`. **A page that is gone stays gone** (2026-09-29, ADR-0100): a URL whose last ledger entry is a 404 or 410 inside the refresh window is set aside before the budget is counted and logged `gone`; `--force` retries. `bin/research.mjs` passes `http-keyless` when the chosen adapter can say what exhaustion looks like, announces it before the run, reports `fell back` in the summary, and takes `--no-fallback`. A dry run counts the searches it would make (`wouldSearch`), and `bin/research.mjs` prints the most the plan could spend - page fetches up to the budget, the searches, and on Firecrawl the credit rate - as one `at most` line (2026-09-29): it listed each search and page but never the total. `bin/research.mjs` and `bin/decompose.mjs` refuse, exit 2, when the selection reports the search provider `notReady` and the run would spend (2026-09-30): research passed the provider module rather than the selection, so the `searchAdapter?.notReady` check here never fired, and decompose had no check. An explicit provider with no key or instance failed each search, and decompose degraded to the fetch provider's search, which on Firecrawl spends credits. **The opt-in witness** (2026-09-30, ADR-0106): with `witness`, `witnessCapture` asks `witness.mjs` (or the injected `witnessLookup`) once per newly collected page - never for a cached page, a failed fetch or a dry run - and appends `{ at, url, raw, ...record }` to `research/witnesses.jsonl` (`PATHS.witnesses`), outside `raw/` and the ledger; a throwing lookup is recorded as `witnessed: false` and the capture stands. `bin/research.mjs` passes `--witness`. `rankCandidate` matches the first two words of a reason as plain substrings (`mentionsWhy`, 2026-09-30); it had compiled them into a regular expression, so `a(b` threw a `SyntaxError` and a blank reason built `|`, which matched every URL and gave every candidate the bonus. `bin/research.mjs` refuses, exit 2, a plan with a duplicate object key (`duplicateKey`, 2026-09-30) before it plans or spends: `JSON.parse` keeps the last value, so the url the operator pasted was not the one collected, and the gate that names the duplicate runs only after collection. `canSearch` (in `transport.mjs`, beside `isSearchOnly`): a fetch-only transport is never asked to search - `searchPatiently` names it a failed search, a failed search degrades only to a provider that can search, and an out-of-credits switch hands searches only to a fallback that can (2026-10-01). `isWebUrl`: `selectCandidates` queues only an http(s) search result with a host - a vendor's `javascript:` or `file:` row never reaches a fetch adapter (2026-10-01). `canSearch` is false for a Firecrawl CLI with no key, so no fallback asks it to search (2026-10-01). **Refuses a ledger it cannot record into before the first fetch** (ADR-0122): `assertAppendable(root)` runs beside the vendor-CLI and search-provider refusals, dry runs included, so a broken chain costs nothing - the F12 refusal at the record had let the first fetch be paid for. Without a `fallbackAdapter` (the default on a local run, ADR-0129, 2026-10-02) exhaustion STOPS the run: the first refusal is recognised either way now, every later search is not run and every later page is reported `uncollected` (never failed, never spent), `stopped` carries `{ reason, provider, during, uncollected, skippedSearches }`, and the failure log's `credits-exhausted` row says `action: "stopped"`. `bin/research.mjs` then prints the two ways forward and exits 2; `--fallback` opts into the ADR-0086 switch, which `collect.yml` and `live-collection.yml` pass because nobody is there to decide. `canSearch` moved to `transport.mjs` beside `isSearchOnly` (2026-10-03, architecture pass): a transport's own capability, asked of any provider, had lived in the coordinator; `decompose.mjs` and one test import it from its new home. **The asking of the search providers moved to `search-session.mjs`** (2026-10-03, ADR-0135): `runResearch` builds one session beside its credits policy, refuses through `session.assertReady()`, asks `session.ask(text)` per query, maps the session's failure rows to the failure log's `op: "search"` rows, and reads `session.meters()` into the usage row and the result; what it does with an answer - `noteOutcome`, `selectCandidates`, the target row (`from: 'search'` under a merge, `from: 'query: ... (via ...)'` alone) - stays here. Two counts changed with the move, deliberately: a failed first ask's reported usage is counted whether or not the degrade answers, and a plain failure counts in `searchFailures` as it already did in `searchFailuresOn`. | ADR-0027, ADR-0044, ADR-0086, ADR-0100, ADR-0135 | -| `firecrawl.mjs` | **All** vendor knowledge: the CLI transport (`exec` seam, which carries an **argv array** and spawns with `shell: false` — no URL or query is ever concatenated into a shell string), the Windows `.cmd`-shim resolution (`resolveProgramPath`/`resolveInvocation`: full path off PATH+PATHEXT, a real `.exe` spawned directly, a `.cmd`/`.bat` run through cmd.exe as argv and only after every argument passes `cmdSafeArg` — a **denylist** of cmd.exe metacharacters (`CMD_UNSAFE_CHARS`), refused and never re-quoted — a **denylist**, so an ordinary query with a space passes while an interpreter character does not. `unsafeCmdArgs(argv)` is the plural reporting form and the **single owner** of "which arguments would be refused": `resolveInvocation` calls it rather than filtering inline, so the refusal and the message naming the offenders cannot drift apart. Measured against the allowlist the ADR-0029 source tree still uses, the two agree on every character that can break out of cmd quoting — `"` `%` `&` `|` `<` `>` `^` `(` `)` `` ` `` tab newline — and differ on `'` `{` `}` `\` and **any non-ASCII character**, which an allowlist of ASCII punctuation refuses by construction. That is why `é` is a permitted argument here and a hostile one there, and why that tree's hostile-argv corpus could not be ported), payload normalisation (`normalizeSearch`/`normalizeScrape`/`normalizeMap` — `normalizeSearch` reads the CLI's real `{success, data: {web: […]}, creditsUsed}` object, pinned at `test/fixtures/firecrawl-search-1.23.3.json`), status parsing (`--status`, the flag; `firecrawl status` is not a command), `map`/`search`/`scrape` adapters. `command(args)` survives as a **display-only** rendering (dry-run output, the ledger's `cmd` annotation) that nothing executes. Stamps `transport: 'firecrawl-cli'` — narrowed to `firecrawl-cli-anonymous` by `transport.mjs` when the CLI has no credential — and **grades** completeness at the 1500-character bar rather than asserting `full`. **Tested CLI: 1.25.2** (from 1.24.6, 2026-10-02, `docs/decisions/2026-10-02-firecrawl-cli-1-25`: its `--status`, search and scrape output captured for three credits and read unchanged by the parsers, `test/fixtures/*-1.25.2.*` - the scrape receipt goes to stderr, `data.tools` is populated and reaches no result; before that from 1.23.3, 2026-09-26): 1.23.3's bundled axios 1.15.2 fails behind an HTTPS proxy that requires CONNECT tunnels (405 on every call); 1.24.6 bundles axios 1.18.0, and its status, search and scrape output were read by these parsers before the move - search's new fields are additive `search` reports `searchesUsed: 1` and a `creditsEstimate` of 2 per 10 results, rounded up, at least 2 (2026-09-27, ADR-0065, E-03/E-14): Firecrawl searches had cost credits the kit never recorded. The CLI runs with `maxBuffer: CHILD_OUTPUT_LIMIT` and names an overflow (2026-09-28, ADR-0082): under the 1 MiB default a large answer was lost to ENOBUFS. **`creditsExhausted(text)`** (2026-09-28, ADR-0086) recognises the API's 402 text, "Insufficient credits ...", which is all the CLI passes on - the status never reaches the kit. `normalizeSearch` and `normalizeMap` drop a row that is not an object (2026-09-29, Arena break test): one null row in the CLI's stdout threw on `row.url` and took every valid result with it - the shape `serpapi.normalizeSearch` was hardened against on 2026-09-28. A field of the wrong type - url, title, description, body, status - reads as empty in normalizeScrape and normalizeSearch, as in serpapi and searxng; one hostile shape is fed to all three parsers (2026-09-30). `rateLimitWaitMs` caps a vendor-stated wait at `MAX_RATE_LIMIT_WAIT_MS` (60 s): the number is the vendor's text, and it is slept with the corpus lock held (2026-10-01). The payload is the **first complete JSON value** in stdout (`parsePayload`: each bracket tried as a start, the value ending where its brackets balance outside strings, at most 64 starts), and `scrape` refuses an exit of 0 that printed nothing, no readable JSON, or a page with no text - each a named failure, never a capture (2026-10-02, break-test pass 3: a banner with a bracket, a receipt line after the JSON, a cut payload and an empty stdout had each become a "collected" capture of 0 characters). | ADR-0005, ADR-0020, ADR-0086 | +| `firecrawl.mjs` | **All** vendor knowledge: the CLI transport (`exec` seam, which carries an **argv array** and spawns with `shell: false` — no URL or query is ever concatenated into a shell string), the Windows `.cmd`-shim resolution (`resolveProgramPath`/`resolveInvocation`: full path off PATH+PATHEXT, a real `.exe` spawned directly, a `.cmd`/`.bat` run through cmd.exe as argv and only after every argument passes `cmdSafeArg` — a **denylist** of cmd.exe metacharacters (`CMD_UNSAFE_CHARS`), refused and never re-quoted — a **denylist**, so an ordinary query with a space passes while an interpreter character does not. `unsafeCmdArgs(argv)` is the plural reporting form and the **single owner** of "which arguments would be refused": `resolveInvocation` calls it rather than filtering inline, so the refusal and the message naming the offenders cannot drift apart. Measured against the allowlist the ADR-0029 source tree still uses, the two agree on every character that can break out of cmd quoting — `"` `%` `&` `|` `<` `>` `^` `(` `)` `` ` `` tab newline — and differ on `'` `{` `}` `\` and **any non-ASCII character**, which an allowlist of ASCII punctuation refuses by construction. That is why `é` is a permitted argument here and a hostile one there, and why that tree's hostile-argv corpus could not be ported), payload normalisation (`normalizeSearch`/`normalizeScrape`/`normalizeMap` — `normalizeSearch` reads the CLI's real `{success, data: {web: […]}, creditsUsed}` object, pinned at `test/fixtures/firecrawl-search-1.23.3.json`), status parsing (`--status`, the flag; `firecrawl status` is not a command), `map`/`search`/`scrape` adapters. `command(args)` survives as a **display-only** rendering (dry-run output, the ledger's `cmd` annotation) that nothing executes. Stamps `transport: 'firecrawl-cli'` — narrowed to `firecrawl-cli-anonymous` by `transport.mjs` when the CLI has no credential — and **grades** completeness at the 1500-character bar rather than asserting `full`. **Tested CLI: 1.25.3** (from 1.25.2, 2026-10-03: the release is one fix - `--objective` optional for alexandria feedback, firecrawl/cli#301 - with nothing on scrape, search or `--status`, and its output, captured for three credits, reads unchanged, `test/fixtures/*-1.25.3.*`; from 1.24.6 on 2026-10-02, `docs/decisions/2026-10-02-firecrawl-cli-1-25`: its `--status`, search and scrape output captured for three credits and read unchanged by the parsers, `test/fixtures/*-1.25.2.*` - the scrape receipt goes to stderr, `data.tools` is populated and reaches no result; before that from 1.23.3, 2026-09-26): 1.23.3's bundled axios 1.15.2 fails behind an HTTPS proxy that requires CONNECT tunnels (405 on every call); 1.24.6 bundles axios 1.18.0, and its status, search and scrape output were read by these parsers before the move - search's new fields are additive `search` reports `searchesUsed: 1` and a `creditsEstimate` of 2 per 10 results, rounded up, at least 2 (2026-09-27, ADR-0065, E-03/E-14): Firecrawl searches had cost credits the kit never recorded. The CLI runs with `maxBuffer: CHILD_OUTPUT_LIMIT` and names an overflow (2026-09-28, ADR-0082): under the 1 MiB default a large answer was lost to ENOBUFS. **`creditsExhausted(text)`** (2026-09-28, ADR-0086) recognises the API's 402 text, "Insufficient credits ...", which is all the CLI passes on - the status never reaches the kit. `normalizeSearch` and `normalizeMap` drop a row that is not an object (2026-09-29, Arena break test): one null row in the CLI's stdout threw on `row.url` and took every valid result with it - the shape `serpapi.normalizeSearch` was hardened against on 2026-09-28. A field of the wrong type - url, title, description, body, status - reads as empty in normalizeScrape and normalizeSearch, as in serpapi and searxng; one hostile shape is fed to all three parsers (2026-09-30). `rateLimitWaitMs` caps a vendor-stated wait at `MAX_RATE_LIMIT_WAIT_MS` (60 s): the number is the vendor's text, and it is slept with the corpus lock held (2026-10-01). The payload is the **first complete JSON value** in stdout (`parsePayload`: each bracket tried as a start, the value ending where its brackets balance outside strings, at most 64 starts), and `scrape` refuses an exit of 0 that printed nothing, no readable JSON, or a page with no text - each a named failure, never a capture (2026-10-02, break-test pass 3: a banner with a bracket, a receipt line after the JSON, a cut payload and an empty stdout had each become a "collected" capture of 0 characters). | ADR-0005, ADR-0020, ADR-0086 | | `bin/mcp-server.mjs` | The stdio entrypoint an MCP client launches as a subprocess. No flags: a stdio server's configuration is its environment. Calls `honourEnvProxy()` before touching either stream, because where only the startup flag exists the server starts again on the same pipes (ADR-0047). **stdout carries messages and nothing else** - a stray `console.log` corrupts the stream and surfaces as a malformed response to whatever request was in flight - so every diagnostic goes to stderr, including the one line reporting whether a credential was found. It calls `tolerateClosedStdout()` before its first write (2026-09-30): a client that disconnects mid-response made the kernel's EPIPE an unhandled `error` event, a raw stack and exit 1 that read as a server defect. A real pipe test (`| head -c 100`) and a source-order pin cover it. | ADR-0034 | | `bin/measure.mjs` | The judge-free citation report for the current project (ADR-0089); `--json` for machines. A report, never a gate; exit 2 outside a research project. | ADR-0089 | | `bin/export-warc.mjs` | The corpus as one WARC 1.1 `.warc.gz` for archive tools (ADR-0090); `--out` (default `research-corpus.warc.gz`), written whole with `writeBytes` after `readWarc` re-reads it; a refused write is named by `writeFailure`, exit 2 (2026-09-29). A copy, never a gate; exit 2 outside a research project. | ADR-0090 | diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 5d27c455..08dcebdd 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -38,7 +38,7 @@ repository. The retention rule is **ADR-0102**: | [2026-09-29-perspective-discovery](2026-09-29-perspective-discovery/) | STORM-style perspective discovery without a language model | ADR-0091 | active | | [2026-09-30-searxng-search](2026-09-30-searxng-search/) | SearXNG's JSON Search API as a keyless search transport | ADR-0104; `lib/searxng.mjs` | active | | [2026-09-30-wayback-witness](2026-09-30-wayback-witness/) | The Wayback Machine as a third-party witness for captured pages | ADR-0106; `lib/witness.mjs` | active | -| [2026-10-02-firecrawl-cli-1-25](2026-10-02-firecrawl-cli-1-25/) | Whether the tested Firecrawl CLI pin should move from 1.24.6 to 1.25.2 - it did, on 2026-10-02, after the brief's first step (a fixture capture of 1.25.2's real output, read unchanged by the parsers; `research-kit/test/fixtures/*-1.25.2.*`) | `lib/firecrawl.mjs` (`TESTED_CLI_VERSION`) | active | +| [2026-10-02-firecrawl-cli-1-25](2026-10-02-firecrawl-cli-1-25/) | Whether the tested Firecrawl CLI pin should move from 1.24.6 to 1.25.2 - it did, on 2026-10-02, after the brief's first step (a fixture capture of 1.25.2's real output, read unchanged by the parsers; `research-kit/test/fixtures/*-1.25.2.*`); the pin moved on to 1.25.3 on 2026-10-03 by the same first step | `lib/firecrawl.mjs` (`TESTED_CLI_VERSION`) | active | | [2026-10-02-similar-tools-survey](2026-10-02-similar-tools-survey/) | Which similar agent-research and evidence-provenance tools on GitHub and Hugging Face do something this kit should take - none as code; two mechanisms weighed and deferred in its brief (a signature on ledger entries, anchoring the chain head with a third party) | the freeze (ADR-0117) stands; the brief's deferred items name their triggers | active | | [2026-10-02-github-plain-fetch-refusal](2026-10-02-github-plain-fetch-refusal/) | Why a plain fetch of github.com answered 403 and which free route to name - the block was this sandbox's egress proxy, not GitHub's (its rate-limit 403 carries `x-ratelimit-remaining: 0`, this one did not); `raw.githubusercontent.com` and the browser transport are the measured free routes; the one change is that the keyless failure quotes the server's own reason | `lib/http-transport.mjs` (`serverReason`) | active | | [2026-10-02-licence-and-captured-pages](2026-10-02-licence-and-captured-pages/) | Which licence the kit should publish under so that people and agents may install and run it, and whether the cached third-party pages under every `research/raw/` may be redistributed - raised by an outside review that found LICENSE (all rights reserved) contradicting the README's install instruction. The brief recommends PolyForm Shield 1.0.0 (use and modify, no competing product) or Apache-2.0 if adoption outweighs control, and for the captures a NOTICE in the archives' words with `research/raw/` excluded from the code licence via REUSE; the owner chose Shield on 2026-10-02 | ADR-0130; `LICENSE`, `NOTICE`, `REUSE.toml` | active | diff --git a/research-kit/README.md b/research-kit/README.md index 73b39edb..77512c7b 100644 --- a/research-kit/README.md +++ b/research-kit/README.md @@ -453,7 +453,7 @@ node research-kit/bin/selftest.mjs # all of it node research-kit/bin/selftest.mjs gate hook # just these files ``` -1512 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means +1515 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means the assertions settled (ADR-0021), and each test is raced against a watchdog (`RESEARCH_KIT_TEST_TIMEOUT`, default 60s). diff --git a/research-kit/lib/firecrawl.mjs b/research-kit/lib/firecrawl.mjs index e6d191c5..dc02200f 100644 --- a/research-kit/lib/firecrawl.mjs +++ b/research-kit/lib/firecrawl.mjs @@ -455,8 +455,15 @@ export function cliVersion({ execFn = exec, ...opts } = {}) { * no `url`, so none reaches a result) beside `data.web`; `--status` is the same text. The * bundle is unchanged (SDK 4.40.0, node >= 22), so the proxy behaviour that forced the * previous move is not in question. + * + * Moved from 1.25.2 to 1.25.3 on 2026-10-03. The release is one commit, "fix: make --objective + * optional for alexandria feedback" (firecrawl/cli#301), with nothing on scrape, search or + * `--status`; the bundle is unchanged (SDK 4.40.0, axios 1.18.0, node >= 22). Captured anyway, + * the same way (test/fixtures/*-1.25.3.*): the three outputs read unchanged by these parsers - + * the receipt on stderr, `data.tools` populated and reaching no result, `--status` the same + * text with a 1,500-credit cycle behind it. */ -export const TESTED_CLI_VERSION = '1.25.2'; +export const TESTED_CLI_VERSION = '1.25.3'; export const SUPPORTED_CLI_MAJOR = 1; /** diff --git a/research-kit/test/fixtures/firecrawl-scrape-1.25.3.json b/research-kit/test/fixtures/firecrawl-scrape-1.25.3.json new file mode 100644 index 00000000..d69f53c7 --- /dev/null +++ b/research-kit/test/fixtures/firecrawl-scrape-1.25.3.json @@ -0,0 +1 @@ +{"markdown":"> ## Documentation Index\n>\n> Fetch the complete documentation index at: [/llms.txt](https://docs.firecrawl.dev/llms.txt)\n>\n> Use this file to discover all available pages before exploring further.\n\n[Skip to main content](https://docs.firecrawl.dev/sdks/cli#content-area)\n\nSearch, scrape, interact, crawl, map, and run agent jobs directly from the terminal. The Firecrawl CLI works standalone or with skills that AI coding agents like Codex, Claude Code, Cursor, and OpenCode can discover and use automatically.\n\n## [​](https://docs.firecrawl.dev/sdks/cli\\#installation) Installation\n\nIf you are using an AI agent like Codex, Claude Code, Cursor, or OpenCode, you can install the Firecrawl skills below and the agent will set them up for you.\n\n```\nnpx -y firecrawl-cli@latest init --all --browser\n```\n\n- `--all` skips agent selection and initializes every detected agent\n- `--browser` opens the browser for Firecrawl authentication automatically\n\nAfter installing the skills, restart your agent for it to discover them.\n\nYou can also manually install the Firecrawl CLI globally using npm:\n\nCLI\n\n```\n# Install globally with npm\nnpm install -g firecrawl-cli\n```\n\n## [​](https://docs.firecrawl.dev/sdks/cli\\#authentication) Authentication\n\nBefore using the CLI, you need to authenticate with your Firecrawl API key.\n\n**Some CLI commands work without logging in.** With no API key configured, supported commands fall back to the keyless free tier — free, but rate-limited per IP. See [Rate Limits](https://docs.firecrawl.dev/rate-limits#keyless-no-api-key) for the current keyless command list and caveats. [Sign up for a free key](https://firecrawl.dev/) for 1,000 credits and higher limits; the CLI uses it automatically once configured.\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#login) Login\n\nCLI\n\n```\n# Interactive login (opens browser or prompts for API key)\nfirecrawl login\n\n# Login with browser authentication (recommended for agents)\nfirecrawl login --browser\n\n# Login with API key directly\nfirecrawl login --api-key fc-YOUR-API-KEY\n\n# Or set via environment variable\nexport FIRECRAWL_API_KEY=fc-YOUR-API-KEY\n```\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#view-configuration) View Configuration\n\nCLI\n\n```\n# View current configuration and authentication status\nfirecrawl view-config\n```\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#logout) Logout\n\nCLI\n\n```\n# Clear stored credentials\nfirecrawl logout\n```\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#connect-the-cli-to-self-hosted-firecrawl) Connect the CLI to self-hosted Firecrawl\n\nFirst, get one scrape working with the [self-hosting guide](https://docs.firecrawl.dev/contributing/self-host). Then point the CLI at that API with `--api-url` or `FIRECRAWL_API_URL`:\n\nCLI\n\n```\n# Use a local Firecrawl instance (no API key required)\nfirecrawl --api-url http://localhost:3002 scrape https://example.com\n\n# Or set via environment variable\nexport FIRECRAWL_API_URL=http://localhost:3002\nfirecrawl scrape https://example.com\n\n# Configure and persist the custom API URL\nfirecrawl config --api-url http://localhost:3002\n```\n\nWhen you use a custom API URL instead of `https://api.firecrawl.dev`, the CLI skips Firecrawl Cloud API-key authentication. That matches the trusted-network quickstart, where `USE_DB_AUTHENTICATION=false`.\n\nKeep an unauthenticated API on a trusted network. If you add an authentication\nproxy or another access-control layer, verify that the CLI can send the\ncredentials that layer requires before depending on this path.\n\nThe CLI can call only the capabilities enabled in your deployment. Check [self-hosted feature support](https://docs.firecrawl.dev/contributing/self-host#self-hosted-feature-support) before using Cloud-only or provider-dependent commands.\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#check-status) Check Status\n\nVerify installation, authentication, and view rate limits:\n\nCLI\n\n```\nfirecrawl --status\n```\n\nOutput when ready:\n\n```\n 🔥 firecrawl cli v1.16.2\n\n ● Authenticated via FIRECRAWL_API_KEY\n Concurrency: 0/100 jobs (parallel scrape limit)\n Credits: 500,000 remaining\n```\n\n- **Concurrency**: Maximum parallel jobs. Run parallel operations close to this limit but not above.\n- **Credits**: Remaining API credits. Each scrape/crawl consumes credits.\n\n## [​](https://docs.firecrawl.dev/sdks/cli\\#commands) Commands\n\nThe hidden `firecrawl browser` command is deprecated for agent workflows. Use `firecrawl scrape ` first, then `firecrawl interact ...` with the resulting scrape session.\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#scrape) Scrape\n\nScrape a single URL and extract its content in various formats.\n\nUse `--only-main-content` to get clean output without navigation, footers, and\nads. This is recommended for most use cases where you want just the article or\nmain page content.\n\nCLI\n\n```\n# Scrape a URL (default: markdown output)\nfirecrawl https://example.com\n\n# Or use the explicit scrape command\nfirecrawl scrape https://example.com\n\n# Recommended: use --only-main-content for clean output without nav/footer\nfirecrawl https://example.com --only-main-content\n```\n\n#### [​](https://docs.firecrawl.dev/sdks/cli\\#output-formats) Output Formats\n\nCLI\n\n```\n# Get HTML output\nfirecrawl https://example.com --html\n\n# Multiple formats (returns JSON)\nfirecrawl https://example.com --format markdown,links\n\n# Get images from a page\nfirecrawl https://example.com --format images\n\n# Get a summary of the page content\nfirecrawl https://example.com --format summary\n\n# Track changes on a page\nfirecrawl https://example.com --format changeTracking\n\n# Available formats: markdown, html, rawHtml, links, screenshot, json, images, summary, changeTracking, attributes, branding, product\n```\n\n#### [​](https://docs.firecrawl.dev/sdks/cli\\#scrape-options) Scrape Options\n\nCLI\n\n```\n# Extract only main content (removes navs, footers)\nfirecrawl https://example.com --only-main-content\n\n# Wait for JavaScript rendering\nfirecrawl https://example.com --wait-for 3000\n\n# Take a screenshot\nfirecrawl https://example.com --screenshot\n\n# Extract structured JSON with a schema\nfirecrawl https://example.com --format json --schema '{\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}'\n\n# Run lightweight scrape actions before extraction\nfirecrawl https://example.com --actions '[{\"type\":\"wait\",\"milliseconds\":1000}]'\n\n# Select proxy mode\nfirecrawl https://example.com --proxy basic\n\n# Redact personally identifiable information\nfirecrawl https://example.com --redact-pii\n\n# Include/exclude specific HTML tags\nfirecrawl https://example.com --include-tags article,main\nfirecrawl https://example.com --exclude-tags nav,footer\n\n# Save output to file\nfirecrawl https://example.com -o output.md\n\n# Pretty print JSON output\nfirecrawl https://example.com --format markdown,links --pretty\n\n# Force JSON output even with single format\nfirecrawl https://example.com --json\n\n# Show request timing information\nfirecrawl https://example.com --timing\n```\n\n**Available Options:**\n\n| Option | Short | Description |\n| --- | --- | --- |\n| `--url ` | `-u` | URL to scrape (alternative to positional argument) |\n| `--format ` | `-f` | Output formats (comma-separated): `markdown`, `html`, `rawHtml`, `links`, `screenshot`, `json`, `images`, `summary`, `changeTracking`, `attributes`, `branding` |\n| `--html` | `-H` | Shortcut for `--format html` |\n| `--only-main-content` | | Extract only main content |\n| `--wait-for ` | | Wait time in milliseconds for JS rendering |\n| `--screenshot` | | Take a screenshot |\n| `--full-page-screenshot` | | Take a full page screenshot |\n| `--include-tags ` | | HTML tags to include (comma-separated) |\n| `--exclude-tags ` | | HTML tags to exclude (comma-separated) |\n| `--schema ` | | JSON schema for structured extraction |\n| `--schema-file ` | | Path to JSON schema file |\n| `--actions ` | | JSON actions array to run during scrape |\n| `--actions-file ` | | Path to JSON actions file |\n| `--proxy ` | | Proxy mode for scraping (for example, `auto` or `basic`) |\n| `--redact-pii` | | Redact personally identifiable information from returned content |\n| `--output ` | `-o` | Save output to file |\n| `--json` | | Force JSON output even with single format |\n| `--pretty` | | Pretty print JSON output |\n| `--timing` | | Show request timing and other useful information |\n| `--alexandria ` | | Run an [Alexandria](https://docs.firecrawl.dev/features/alexandria) tool, given as `provider/capability`, instead of scraping a URL. Repeat to run more than one tool |\n| `--options ` | | JSON input for each Alexandria tool, in the same order |\n| `--request-id ` | | Request ID for an Alexandria run. Use the same ID to recover the same run |\n| `--domain-tools` | | Also return Alexandria tools that match the scraped URL. The tools do not run |\n\n* * *\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#search) Search\n\nSearch the web and optionally scrape the results.\n\nCLI\n\n```\n# Search the web\nfirecrawl search \"web scraping tutorials\"\n\n# Limit results\nfirecrawl search \"AI news\" --limit 10\n\n# Pretty print results\nfirecrawl search \"machine learning\" --pretty\n```\n\n#### [​](https://docs.firecrawl.dev/sdks/cli\\#search-options) Search Options\n\nCLI\n\n```\n# Search specific sources\nfirecrawl search \"AI\" --sources web,news,images\n\n# Search with category filters\nfirecrawl search \"react hooks\" --categories developer\nfirecrawl search \"machine learning\" --categories research,pdf\n\n# Time-based filtering\nfirecrawl search \"tech news\" --tbs qdr:h # Last hour\nfirecrawl search \"tech news\" --tbs qdr:d # Last day\nfirecrawl search \"tech news\" --tbs qdr:w # Last week\nfirecrawl search \"tech news\" --tbs qdr:m # Last month\nfirecrawl search \"tech news\" --tbs qdr:y # Last year\n\n# Location-based search\nfirecrawl search \"restaurants\" --location \"Berlin,Germany\" --country DE\n\n# Search and scrape results\nfirecrawl search \"documentation\" --scrape --scrape-formats markdown\n\n# Save to file\nfirecrawl search \"firecrawl\" --pretty -o results.json\n```\n\n**Available Options:**\n\n| Option | Description |\n| --- | --- |\n| `--limit ` | Maximum results (default: 5, max: 100) |\n| `--sources ` | Sources to search: `web`, `images`, `news`, `alexandria` (comma-separated, default: `web,alexandria`). Use `web` to omit Alexandria tools |\n| `--domain-tools` | Include Alexandria tools for the domains in web results (on by default with `alexandria`) |\n| `--no-domain-tools` | Turn off domain matching for Alexandria tools |\n| `--tool-detail ` | Alexandria tool detail: `compact` (default), `summary`, or `full` (includes contracts) |\n| `--categories ` | Filter by category: `research`, `pdf`, `developer` (comma-separated) |\n| `--tbs ` | Time filter: `qdr:h` (hour), `qdr:d` (day), `qdr:w` (week), `qdr:m` (month), `qdr:y` (year) |\n| `--location ` | Geo-targeting (e.g., “Berlin,Germany”) |\n| `--country ` | ISO country code (default: US) |\n| `--timeout ` | Timeout in milliseconds (default: 60000) |\n| `--ignore-invalid-urls` | Exclude URLs invalid for other Firecrawl endpoints |\n| `--scrape` | Scrape search results |\n| `--scrape-formats ` | Formats for scraped content (default: markdown) |\n| `--only-main-content` | Include only main content when scraping (default: true) |\n| `--json` | Output as JSON |\n| `--output ` | Save output to file |\n| `--pretty` | Pretty print JSON output |\n\n* * *\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#alexandria) Alexandria\n\nFind and run data tools from [Alexandria](https://docs.firecrawl.dev/features/alexandria), the Firecrawl catalogue of third-party data providers. Discovery is free. A tool run costs the price listed on the tool.\n\nCLI\n\n```\n# Find tools for a task (tools only, no web results)\nfirecrawl search alexandria \"podcast conversations about AI agents\"\n\n# Browse categories, then providers and their tools\nfirecrawl alexandria list\nfirecrawl list particle\n\n# Read the inputs and response of one tool\nfirecrawl find-tools --options '{\"providers\":[\"particle\"],\"capabilities\":[\"podcasts/episodes/search\"],\"level\":\"tools\",\"expand\":[\"options\",\"response\"]}' --json\n\n# Run the tool\nfirecrawl scrape particle/podcasts/episodes/search --options '{\"semantic_search\":\"AI agents\",\"limit\":2}'\n```\n\n`firecrawl scrape provider/capability` is a short form of `firecrawl scrape --alexandria provider/capability`. Tool addresses go directly to Alexandria. They never fall back to a URL scrape.\n\n#### [​](https://docs.firecrawl.dev/sdks/cli\\#provider-terms) Provider terms\n\nWhen a provider returns `THIRD_PARTY_DATA_TERMS_REQUIRED`, read its terms first:\n\nCLI\n\n```\nfirecrawl alexandria terms show particle --pretty\n```\n\nTo accept, give the exact version and digest that you reviewed. Acceptance applies to the organization of your API key:\n\nCLI\n\n```\nfirecrawl alexandria terms accept particle \\\n --terms-version '' --digest '' --confirm\n```\n\nAn agent must not accept terms on its own. It must show the terms to the user, get explicit approval, and wait. An org admin can also accept terms at [Data sources settings](https://www.firecrawl.dev/app/settings?tab=data-sources). After acceptance, run the original command again.Use `firecrawl alexandria feedback` to report issues or request capabilities. Run `firecrawl alexandria feedback --help` for options.\n\n* * *\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#developer) Developer\n\nSearch the [Developer Index](https://docs.firecrawl.dev/features/developer) — issues, merged pull requests, and READMEs from public code repositories, alongside curated documentation sites.\n\nCLI\n\n```\nfirecrawl developer \"how do I configure retries\" --limit 10\n```\n\n**Available Options:**\n\n| Option | Description |\n| --- | --- |\n| `--limit ` | Number of results to return (default: 10, max: 100) |\n| `--skills-only` | Search only indexed agent-skill files (default: false) |\n| `--json` | Output as compact JSON |\n| `--output ` | Save output to file |\n| `--pretty` | Pretty print JSON output |\n\n* * *\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#map) Map\n\nDiscover all URLs on a website quickly.\n\nCLI\n\n```\n# Discover all URLs on a website\nfirecrawl map https://example.com\n\n# Output as JSON\nfirecrawl map https://example.com --json\n\n# Limit number of URLs\nfirecrawl map https://example.com --limit 500\n```\n\n#### [​](https://docs.firecrawl.dev/sdks/cli\\#map-options) Map Options\n\nCLI\n\n```\n# Filter URLs by search query\nfirecrawl map https://example.com --search \"blog\"\n\n# Include subdomains\nfirecrawl map https://example.com --include-subdomains\n\n# Control sitemap usage\nfirecrawl map https://example.com --sitemap include # Use sitemap\nfirecrawl map https://example.com --sitemap skip # Skip sitemap\nfirecrawl map https://example.com --sitemap only # Only use sitemap\n\n# Ignore query parameters (dedupe URLs)\nfirecrawl map https://example.com --ignore-query-parameters\n\n# Wait for map to complete with timeout\nfirecrawl map https://example.com --wait --timeout 60\n\n# Save to file\nfirecrawl map https://example.com -o urls.txt\nfirecrawl map https://example.com --json --pretty -o urls.json\n```\n\n**Available Options:**\n\n| Option | Description |\n| --- | --- |\n| `--url ` | URL to map (alternative to positional argument) |\n| `--limit ` | Maximum URLs to discover |\n| `--search ` | Filter URLs by search query |\n| `--sitemap ` | Sitemap handling: `include`, `skip`, `only` |\n| `--include-subdomains` | Include subdomains |\n| `--ignore-query-parameters` | Treat URLs with different params as same |\n| `--wait` | Wait for map to complete |\n| `--timeout ` | Timeout in seconds |\n| `--json` | Output as JSON |\n| `--output ` | Save output to file |\n| `--pretty` | Pretty print JSON output |\n\n* * *\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#interact) Interact\n\nScrape a page, then interact with it using natural language or code. Interact uses the most recent scrape by default, or you can pass a specific scrape ID.\n\nCLI\n\n```\n# 1. Scrape Amazon's homepage (scrape ID is saved automatically)\nfirecrawl scrape https://www.amazon.com\n\n# 2. Interact — search for a product and get its price\nfirecrawl interact \"Search for iPhone 16 Pro Max\"\nfirecrawl interact \"Click on the first result and tell me the price\"\n\n# 3. Stop the session\nfirecrawl interact stop\n```\n\n**Available Options:**\n\n| Option | Description |\n| --- | --- |\n| `-p, --prompt ` | AI prompt (alternative to positional argument) |\n| `-c, --code ` | Code to execute in the live page session |\n| `-s, --scrape-id ` | Scrape job ID (default: last scrape) |\n| `--python` | Execute code as Python/Playwright |\n| `--node` | Execute code as Node.js/Playwright (default) |\n| `--bash` | Execute code as Bash |\n| `--timeout ` | Timeout in seconds (1-300, default: 30) |\n| `--output ` | Save output to file |\n| `--json` | Output as JSON format |\n\n* * *\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#crawl) Crawl\n\nCrawl an entire website starting from a URL.\n\nCLI\n\n```\n# Start a crawl (returns job ID immediately)\nfirecrawl crawl https://example.com\n\n# Wait for crawl to complete\nfirecrawl crawl https://example.com --wait\n\n# Wait with progress indicator\nfirecrawl crawl https://example.com --wait --progress\n```\n\n#### [​](https://docs.firecrawl.dev/sdks/cli\\#check-crawl-status) Check Crawl Status\n\nCLI\n\n```\n# Check crawl status using job ID\nfirecrawl crawl \n\n# Example with a real job ID\nfirecrawl crawl 550e8400-e29b-41d4-a716-446655440000\n```\n\n#### [​](https://docs.firecrawl.dev/sdks/cli\\#crawl-options) Crawl Options\n\nCLI\n\n```\n# Limit crawl depth and pages\nfirecrawl crawl https://example.com --limit 100 --max-depth 3 --wait\n\n# Include only specific paths\nfirecrawl crawl https://example.com --include-paths /blog,/docs --wait\n\n# Exclude specific paths\nfirecrawl crawl https://example.com --exclude-paths /admin,/login --wait\n\n# Include subdomains\nfirecrawl crawl https://example.com --allow-subdomains --wait\n\n# Crawl entire domain\nfirecrawl crawl https://example.com --crawl-entire-domain --wait\n\n# Rate limiting\nfirecrawl crawl https://example.com --delay 1000 --max-concurrency 2 --wait\n\n# Pass scrape options to each crawled page\nfirecrawl crawl https://example.com --scrape-options '{\"formats\":[\"markdown\"],\"onlyMainContent\":true}'\n\n# Send crawl completion events to a webhook\nfirecrawl crawl https://example.com --webhook '{\"url\":\"https://example.com/webhook\",\"events\":[\"completed\"]}'\n\n# Cancel an active crawl\nfirecrawl crawl --cancel\n\n# Custom polling interval and timeout\nfirecrawl crawl https://example.com --wait --poll-interval 10 --timeout 300\n\n# Save results to file\nfirecrawl crawl https://example.com --wait --pretty -o results.json\n```\n\n**Available Options:**\n\n| Option | Description |\n| --- | --- |\n| `--url ` | URL to crawl (alternative to positional argument) |\n| `--wait` | Wait for crawl to complete |\n| `--progress` | Show progress indicator while waiting |\n| `--poll-interval ` | Polling interval (default: 5) |\n| `--timeout ` | Timeout when waiting |\n| `--status` | Check status of existing crawl job |\n| `--limit ` | Maximum pages to crawl |\n| `--max-depth ` | Maximum crawl depth |\n| `--include-paths ` | Paths to include (comma-separated) |\n| `--exclude-paths ` | Paths to exclude (comma-separated) |\n| `--sitemap ` | Sitemap handling: `include`, `skip`, `only` |\n| `--allow-subdomains` | Include subdomains |\n| `--allow-external-links` | Follow external links |\n| `--crawl-entire-domain` | Crawl entire domain |\n| `--ignore-query-parameters` | Treat URLs with different params as same |\n| `--delay ` | Delay between requests |\n| `--max-concurrency ` | Maximum concurrent requests |\n| `--scrape-options ` | JSON scrape options passed to each page |\n| `--scrape-options-file ` | Path to scrape options JSON file |\n| `--webhook ` | Webhook URL or configuration |\n| `--cancel` | Cancel an active crawl job by job ID |\n| `--output ` | Save output to file |\n| `--pretty` | Pretty print JSON output |\n\n* * *\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#monitor) Monitor\n\nCreate recurring scrapes or crawls that diff each run against the previous snapshot. Add a goal when you want Firecrawl to judge which changed pages are meaningful for your use case.\n\nCLI\n\n```\nfirecrawl monitor create --name \"Hacker News AI\" \\\n --schedule \"every 30 minutes\" \\\n --goal \"Alert when a new Hacker News story related to AI enters the top 10. Ignore changes to stories that are not about AI. Do not alert on changes outside the top 10.\" \\\n --page https://news.ycombinator.com\n\nfirecrawl monitor run \nfirecrawl monitor checks --limit 10\nfirecrawl monitor check --page-status changed\nfirecrawl monitor update \\\n --goal \"Alert when a new Hacker News story related to AI enters the top 10. Do not alert on changes outside the top 10.\"\nfirecrawl monitor delete \n```\n\nMonitor goals should stay short and faithful to the user’s intent: say what should trigger an alert, restate any stated scope, and include exclusions only when they are obvious or explicitly requested. If the user asks for “any change”, keep the goal broad.**Available Options:**\n\n| Option | Description |\n| --- | --- |\n| `--name ` | Monitor name |\n| `--goal ` | Goal for meaningful-change judging |\n| `--cron ` | Cron schedule, for example `*/30 * * * *` |\n| `--schedule ` | Natural-language schedule, for example `hourly` |\n| `--timezone ` | Schedule timezone, default `UTC` |\n| `--page ` | Single page URL to scrape on each check |\n| `--scrape-urls ` | Comma-separated page URLs to scrape on each check |\n| `--crawl-url ` | Root URL for a crawl target |\n| `--webhook-url ` | Webhook destination |\n| `--webhook-events ` | Comma-separated monitor events |\n| `--email ` | Comma-separated email recipients |\n| `--retention-days ` | Snapshot retention window |\n| `--page-status ` | Filter pages on `monitor check` |\n| `--state ` | Set monitor state on `monitor update`: active/paused |\n\n* * *\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#agent) Agent\n\nSearch and gather data from the web using natural language prompts.\n\nCLI\n\n```\n# Basic usage - URLs are optional\nfirecrawl agent \"Find the top 5 AI startups and their funding amounts\" --wait\n\n# Focus on specific URLs\nfirecrawl agent \"Compare pricing plans\" --urls https://slack.com/pricing,https://teams.microsoft.com/pricing --wait\n\n# Use a schema for structured output\nfirecrawl agent \"Get company information\" --urls https://example.com --schema '{\"type\":\"object\",\"properties\":{\"name\":{\"type\":\"string\"},\"founded\":{\"type\":\"number\"}}}' --wait\n\n# Use schema from a file\nfirecrawl agent \"Get product details\" --urls https://example.com --schema-file schema.json --wait\n```\n\n#### [​](https://docs.firecrawl.dev/sdks/cli\\#agent-options) Agent Options\n\nCLI\n\n```\n# Spark 2 is the default — every run executes on it\nfirecrawl agent \"Competitive analysis across multiple domains\" --model spark-2 --wait\n\n# Deprecated: Spark 1 model names are still accepted, but route to spark-2.\n\n# Set max credits to limit costs\nfirecrawl agent \"Gather contact information from company websites\" --max-credits 100 --wait\n\n# Check status of an existing job\nfirecrawl agent --status\n\n# Send agent events to a webhook\nfirecrawl agent \"Extract product details\" --urls https://example.com --webhook '{\"url\":\"https://example.com/webhook\",\"events\":[\"completed\",\"failed\"]}'\n\n# Cancel an active agent job\nfirecrawl agent --cancel\n\n# Custom polling interval and timeout\nfirecrawl agent \"Summarize recent blog posts\" --wait --poll-interval 10 --timeout 300\n\n# Save output to file\nfirecrawl agent \"Find pricing information\" --urls https://example.com --wait -o pricing.json --pretty\n```\n\n**Available Options:**\n\n| Option | Description |\n| --- | --- |\n| `--urls ` | Optional list of URLs to focus the agent on (comma-separated) |\n| `--model ` | Model to use. Defaults to `spark-2`, the model every run executes on. Spark 1 models are deprecated and route to `spark-2` |\n| `--schema ` | JSON schema for structured output (inline JSON string) |\n| `--schema-file ` | Path to JSON schema file for structured output |\n| `--max-credits ` | Maximum credits to spend (job fails if limit reached) |\n| `--webhook ` | Webhook URL or configuration |\n| `--status` | Check status of existing agent job |\n| `--cancel` | Cancel an active agent job by job ID |\n| `--wait` | Wait for agent to complete before returning results |\n| `--poll-interval ` | Polling interval when waiting (default: 5) |\n| `--timeout ` | Timeout when waiting (default: no timeout) |\n| `--output ` | Save output to file |\n| `--json` | Output as JSON format |\n\n* * *\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#credit-usage) Credit Usage\n\nCheck your team’s credit balance and usage.\n\nCLI\n\n```\n# View credit usage\nfirecrawl credit-usage\n\n# Output as JSON\nfirecrawl credit-usage --json --pretty\n```\n\n* * *\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#version) Version\n\nDisplay the CLI version.\n\nCLI\n\n```\nfirecrawl version\n# or\nfirecrawl --version\n```\n\n## [​](https://docs.firecrawl.dev/sdks/cli\\#global-options) Global Options\n\nThese options are available for all commands:\n\n| Option | Short | Description |\n| --- | --- | --- |\n| `--status` | | Show version, auth, concurrency, and credits |\n| `--api-key ` | `-k` | Override stored API key for this command |\n| `--api-url ` | | Use custom API URL (for self-hosted/local development) |\n| `--help` | `-h` | Show help for a command |\n| `--version` | `-V` | Show CLI version |\n\n`init` also accepts `--skip-auth`, `--skip-install`, `--skip-skills`, and `--agent `. See `firecrawl init --help`.\n\n## [​](https://docs.firecrawl.dev/sdks/cli\\#output-handling) Output Handling\n\nThe CLI outputs to stdout by default, making it easy to pipe or redirect:\n\nCLI\n\n```\n# Pipe markdown to another command\nfirecrawl https://example.com | head -50\n\n# Redirect to a file\nfirecrawl https://example.com > output.md\n\n# Save JSON with pretty formatting\nfirecrawl https://example.com --format markdown,links --pretty -o data.json\n```\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#format-behavior) Format Behavior\n\n- **Single format**: Outputs raw content (markdown text, HTML, etc.)\n- **Multiple formats**: Outputs JSON with all requested data\n\nCLI\n\n```\n# Raw markdown output\nfirecrawl https://example.com --format markdown\n\n# JSON output with multiple formats\nfirecrawl https://example.com --format markdown,links\n```\n\n## [​](https://docs.firecrawl.dev/sdks/cli\\#examples) Examples\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#quick-scrape) Quick Scrape\n\nCLI\n\n```\n# Get markdown content from a URL (use --only-main-content for clean output)\nfirecrawl https://docs.firecrawl.dev --only-main-content\n\n# Get HTML content\nfirecrawl https://example.com --html -o page.html\n```\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#full-site-crawl) Full Site Crawl\n\nCLI\n\n```\n# Crawl a docs site with limits\nfirecrawl crawl https://docs.example.com --limit 50 --max-depth 2 --wait --progress -o docs.json\n```\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#site-discovery) Site Discovery\n\nCLI\n\n```\n# Find all blog posts\nfirecrawl map https://example.com --search \"blog\" -o blog-urls.txt\n```\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#research-workflow) Research Workflow\n\nCLI\n\n```\n# Search and scrape results for research\nfirecrawl search \"machine learning best practices 2024\" --scrape --scrape-formats markdown --pretty\n```\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#agent-2) Agent\n\nCLI\n\n```\n# URLs are optional\nfirecrawl agent \"Find the top 5 AI startups and their funding amounts\" --wait\n\n# Focus on specific URLs\nfirecrawl agent \"Compare pricing plans\" --urls https://slack.com/pricing,https://teams.microsoft.com/pricing --wait\n```\n\n### [​](https://docs.firecrawl.dev/sdks/cli\\#combine-with-other-tools) Combine with Other Tools\n\nCLI\n\n```\n# Extract URLs from search results\njq -r '.data.web[].url' search-results.json\n\n# Get titles from search results\njq -r '.data.web[] | \"\\(.title): \\(.url)\"' search-results.json\n\n# Extract links as JSON and print one URL per line\nfirecrawl https://example.com --format links --json | jq -r '.links[]'\n\n# Count URLs from map\nfirecrawl map https://example.com | wc -l\n```\n\n## [​](https://docs.firecrawl.dev/sdks/cli\\#telemetry) Telemetry\n\nThe CLI collects anonymous usage data during authentication to help improve the product:\n\n- CLI version, OS, and Node.js version\n- Development tool detection (e.g., Cursor, VS Code, Claude Code)\n\n**No command data, URLs, or file contents are collected via the CLI.**To disable telemetry, set the environment variable:\n\nCLI\n\n```\nexport FIRECRAWL_NO_TELEMETRY=1\n```\n\n## [​](https://docs.firecrawl.dev/sdks/cli\\#open-source) Open Source\n\nThe Firecrawl CLI and all three skill segments are open source on GitHub:\n\n- [`firecrawl/cli`](https://github.com/firecrawl/cli) — the CLI and CLI skills (live web work)\n- [`firecrawl/skills`](https://github.com/firecrawl/skills) — build skills (integrate Firecrawl into application code)\n- [`firecrawl/firecrawl-workflows`](https://github.com/firecrawl/firecrawl-workflows) — workflow skills (repeatable deliverables such as research briefs, SEO audits, lead lists, and design clones)\n\n> Are you an AI agent that needs a Firecrawl API key? See [firecrawl.dev/agent-onboarding/SKILL.md](https://www.firecrawl.dev/agent-onboarding/SKILL.md) for automated onboarding instructions.\n\n[Suggest edits](https://github.com/firecrawl/firecrawl-docs/edit/main/sdks/cli.mdx) [Raise issue](https://github.com/firecrawl/firecrawl-docs/issues/new?title=Issue%20on%20docs&body=Path:%20/sdks/cli)\n\n✕","metadata":{"og:description":"Firecrawl skills are an easy way for AI agents to use Firecrawl through the CLI. AI agents can get web data through a better and more context efficient interface","twitter:image:width":"1200","og:site_name":"Firecrawl Docs","description":"Firecrawl skills are an easy way for AI agents such as Codex, Claude Code, Cursor, and OpenCode to use Firecrawl through the CLI.","og:image":"https://firecrawl.mintlify.app/mintlify-assets/_next/image?url=%2F_mintlify%2Fapi%2Fog%3Fdivision%3DGet%2BStarted%26title%3DCLI%26description%3DFirecrawl%2Bskills%2Bare%2Ban%2Beasy%2Bway%2Bfor%2BAI%2Bagents%2Bsuch%2Bas%2BCodex%252C%2BClaude%2BCode%252C%2BCursor%252C%2Band%2BOpenCode%2Bto%2Buse%2BFirecrawl%2Bthrough%2Bthe%2BCLI.%26theme%3Dc0b74c4d51b170cbe77b5559&w=1200&q=100","og:title":"CLI | Firecrawl","og:image:width":"1200","twitter:description":"Firecrawl skills are an easy way for AI agents to use Firecrawl through the CLI. AI agents can get web data through a better and more context efficient interface","generator":"Mintlify","og:image:height":"630","google-site-verification":"Y83UZz1dXMKuT59lnLY_7umo6LJorqIwuP_2rFX6_eE","application-name":"Firecrawl Docs","og:url":"https://docs.firecrawl.dev/sdks/cli","og:type":"website","viewport":"width=device-width, initial-scale=1, viewport-fit=cover","msapplication-config":"/mintlify-assets/_mintlify/favicons/firecrawl/XqnL63rz4lpKQF4P/_generated/favicon/browserconfig.xml","twitter:image:height":"630","language":"en","twitter:title":"CLI | Firecrawl","msapplication-TileColor":"#000","apple-mobile-web-app-title":"Firecrawl Docs","ogDescription":"Firecrawl skills are an easy way for AI agents to use Firecrawl through the CLI. AI agents can get web data through a better and more context efficient interface","ogTitle":"CLI | Firecrawl","title":"CLI | Firecrawl","twitter:card":"summary_large_image","ogImage":"https://firecrawl.mintlify.app/mintlify-assets/_next/image?url=%2F_mintlify%2Fapi%2Fog%3Fdivision%3DGet%2BStarted%26title%3DCLI%26description%3DFirecrawl%2Bskills%2Bare%2Ban%2Beasy%2Bway%2Bfor%2BAI%2Bagents%2Bsuch%2Bas%2BCodex%252C%2BClaude%2BCode%252C%2BCursor%252C%2Band%2BOpenCode%2Bto%2Buse%2BFirecrawl%2Bthrough%2Bthe%2BCLI.%26theme%3Dc0b74c4d51b170cbe77b5559&w=1200&q=100","twitter:image":"https://firecrawl.mintlify.app/mintlify-assets/_next/image?url=%2F_mintlify%2Fapi%2Fog%3Fdivision%3DGet%2BStarted%26title%3DCLI%26description%3DFirecrawl%2Bskills%2Bare%2Ban%2Beasy%2Bway%2Bfor%2BAI%2Bagents%2Bsuch%2Bas%2BCodex%252C%2BClaude%2BCode%252C%2BCursor%252C%2Band%2BOpenCode%2Bto%2Buse%2BFirecrawl%2Bthrough%2Bthe%2BCLI.%26theme%3Dc0b74c4d51b170cbe77b5559&w=1200&q=100","ogUrl":"https://docs.firecrawl.dev/sdks/cli","ogSiteName":"Firecrawl Docs","favicon":"https://docs.firecrawl.dev/mintlify-assets/_mintlify/favicons/firecrawl/XqnL63rz4lpKQF4P/_generated/favicon/android-chrome-192x192.png","scrapeId":"01a10321-2ecb-73a9-9ec2-ddcd9c51dcc5","sourceURL":"https://docs.firecrawl.dev/sdks/cli","url":"https://docs.firecrawl.dev/sdks/cli","statusCode":200,"contentType":"text/html; charset=utf-8","proxyUsed":"basic","cacheState":"hit","cachedAt":"2026-10-03T11:26:29.902Z","creditsUsed":1,"concurrencyLimited":false}} diff --git a/research-kit/test/fixtures/firecrawl-scrape-1.25.3.stderr.txt b/research-kit/test/fixtures/firecrawl-scrape-1.25.3.stderr.txt new file mode 100644 index 00000000..daf77fb8 --- /dev/null +++ b/research-kit/test/fixtures/firecrawl-scrape-1.25.3.stderr.txt @@ -0,0 +1 @@ +Scrape ID: 01a10321-2ecb-73a9-9ec2-ddcd9c51dcc5 diff --git a/research-kit/test/fixtures/firecrawl-search-1.25.3.json b/research-kit/test/fixtures/firecrawl-search-1.25.3.json new file mode 100644 index 00000000..5a19c77a --- /dev/null +++ b/research-kit/test/fixtures/firecrawl-search-1.25.3.json @@ -0,0 +1 @@ +{"success":true,"data":{"tools":[{"provider":"github-com","capability":"repositories/contributors","description":"One page of a repository's contributors ordered by commit count on the default branch, with login, account type and contribution count. Anonymous (email-only) contributors are not listed. GitHub refuses this listing for repositories with very large histories (for example torvalds/linux), which surfaces as an error naming the reason."},{"provider":"github-com","capability":"repositories/issue_counts","description":"Exact issue counts for a repository (pull requests excluded, unlike the repository's open_issues_count): the total for the given state plus one count per requested label, from GitHub's issue search. Makes 1 + labels.length search requests; at most 5 labels per call because GitHub allows 10 keyless searches per minute per IP."},{"provider":"github-com","capability":"repositories/issues","description":"One page of a repository's issues filtered by state and labels (all given labels must match), sorted by created/updated/comments. Pull requests are excluded unless include_pull_requests is true, so a page may hold fewer than per_page records; pull_requests_excluded says how many were dropped. Each record carries title, state, labels, assignees, milestone, comment and reaction counts, body and timestamps."}],"web":[{"url":"https://docs.firecrawl.dev/sdks/cli","title":"CLI - Firecrawl Docs","description":"## Documentation Index\nThe Firecrawl CLI works standalone or with skills that AI coding agents like Codex, Claude Code, Cursor, and OpenCode can discover and use automatically.\n\n## [​](https://docs.firecrawl.dev/sdks/cli#installation) Installation\n```\n# Install globally with npm\nnpm install -g firecrawl-cli\n```\n\n## [​](https://docs.firecrawl.dev/sdks/cli#commands) Commands\n### [​](https://docs.firecrawl.dev/sdks/cli#scrape) Scrape\n#### [​](https://docs.firecrawl.dev/sdks/cli#scrape-options) Scrape Options\n```\n# Extract only main content (removes navs, footers)\nfirecrawl https://example.com --only-main-content\n\n# Wait for JavaScript rendering\nfirecrawl https://example.com --wait-for 3000\n\n# Take a screenshot\nfirecrawl https://example.com --screenshot\n\n# Extract structured JSON with a schema\nfirecrawl https://example.com --format json --schema '{\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}'\n\n# Run lightweight scrape actions before extraction\nfirecrawl https://example.com --actions '[{\"type\":\"wait\",\"milliseconds\":1000}]'\n\n# Select proxy mode\nfirecrawl https://example.com --proxy basic\n\n# Redact personally identifiable information\nfirecrawl https://example.com --redact-pii\n\n# Include/exclude specific HTML tags\nfirecrawl https://example.com --include-tags article,main\nfirecrawl https://example.com --exclude-tags nav,footer\n\n# Save output to file\nfirecrawl https://example.com -o output.md\n\n# Pretty print JSON output\nfirecrawl https://example.com --format markdown,links --pretty\n\n# Force JSON output even with single format\nfirecrawl https://example.com --json\n\n# Show request timing information\nfirecrawl https://example.com --timing\n```\n\n### [​](https://docs.firecrawl.dev/sdks/cli#crawl) Crawl\n#### [​](https://docs.firecrawl.dev/sdks/cli#crawl-options) Crawl Options\nCLI\n\n```\n# Limit crawl depth and pages\nfirecrawl crawl https://example.com --limit 100 --max-depth 3 --wait\n\n# Include only specific paths\nfirecrawl crawl https://example.com --include-paths /blog,/docs --wait\n\n# Exclude specific paths\nfirecrawl crawl https://example.com --exclude-paths /admin,/login --wait\n\n# Include subdomains\nfirecrawl crawl https://example.com --allow-subdomains --wait\n\n# Crawl entire domain\nfirecrawl crawl https://example.com --crawl-entire-domain --wait\n\n# Rate limiting\nfirecrawl crawl https://example.com --delay 1000 --max-concurrency 2 --wait\n\n# Pass scrape options to each crawled page\nfirecrawl crawl https://example.com --scrape-options '{\"formats\":[\"markdown\"],\"onlyMainContent\":true}'\n\n# Send crawl completion events to a webhook\nfirecrawl crawl https://example.com --webhook '{\"url\":\"https://example.com/webhook\",\"events\":[\"completed\"]}'\n\n# Cancel an active crawl\nfirecrawl crawl --cancel\n\n# Custom polling interval and timeout\nfirecrawl crawl https://example.com --wait --poll-interval 10 --timeout 300\n\n# Save results to file\nfirecrawl crawl https://example.com --wait --pretty -o results.json\n```\n\n## [​](https://docs.firecrawl.dev/sdks/cli#examples) Examples\n### [​](https://docs.firecrawl.dev/sdks/cli#full-site-crawl) Full Site Crawl\nCLI\n\n## [​](https://docs.firecrawl.dev/sdks/cli#open-source) Open Source\nThe Firecrawl CLI and all three skill segments are open source on GitHub:\n\n- [`firecrawl/cli`](https://github.com/firecrawl/cli) — the CLI and CLI skills (live web work)","position":1},{"url":"https://github.com/firecrawl/cli","title":"CLI and Agent Skill for Firecrawl - Add scrape, search, and ... - GitHub","description":"# 🔥 Firecrawl CLI\n## Installation\n```\nnpm install -g firecrawl-cli\n```\n\n## Quick Start\n```\nfirecrawl https://example.com\n```\n\n## Global Options\n### Check Status\n```\n 🔥 firecrawl cli v1.20.0\n\n ● Authenticated via stored credentials\n Concurrency: 0/100 jobs (parallel scrape limit)\n Credits: 500,000 / 1,000,000 (50% left this cycle)\n```\n\n## Tips & Tricks\n[Permalink: Tips & Tricks](https://github.com/firecrawl/cli#tips--tricks)\n\n### CI/CD Usage\n[Permalink: CI/CD Usage](https://github.com/firecrawl/cli#cicd-usage)\n\n```\n# Set API key via environment\nexport FIRECRAWL_API_KEY=${{ secrets.FIRECRAWL_API_KEY }}\nfirecrawl crawl https://docs.example.com --wait -o docs.json\n\n# Use self-hosted instance\nexport FIRECRAWL_API_URL=${{ secrets.FIRECRAWL_API_URL }}\nfirecrawl scrape https://example.com -o output.md\n```\n\n## Documentation\nFor more details, visit the [Firecrawl Documentation](https://docs.firecrawl.dev/).\n\n## Languages\n- [TypeScript98.3%](https://github.com/firecrawl/cli/search?l=typescript)\n- [Shell1.1%](https://github.com/firecrawl/cli/search?l=shell)\n- [Other0.6%](https://github.com/firecrawl/cli/search?l=Other)\n\nYou can’t perform that action at this time.","position":2,"category":"github"},{"url":"https://www.firecrawl.dev/blog/introducing-firecrawl-skill-and-cli","title":"Introducing Firecrawl Skill and CLI: The Complete Web Data Toolkit for ...","description":"# Introducing Firecrawl Skill and CLI: The Complete Web Data Toolkit for Agents\n## Meet Firecrawl CLI\n[Firecrawl CLI](https://docs.firecrawl.dev/sdks/cli) is a unified command-line interface designed for agents and developers who need **reliable web data.**\n\n### Context-Efficient Data for Agents\nFirecrawl CLI uses a file-based approach for context management and bash methods for efficient search and retrieval.\n\n### Industry-Leading Reliability\nIt handles JavaScript rendering, dynamic content, authentication flows, and complex page structures that break other scrapers.\n\n## Get Started\nOr use the CLI directly:\n\n```\n$ npm install -g firecrawl-cli\n$ firecrawl auth --api-key fc-YOUR_KEY\n$ firecrawl scrape https://example.com\n```","position":3}]},"id":"01a10321-273e-74bd-821f-5c1339008379","creditsUsed":2} diff --git a/research-kit/test/fixtures/firecrawl-status-1.25.3.txt b/research-kit/test/fixtures/firecrawl-status-1.25.3.txt new file mode 100644 index 00000000..15d91ca0 --- /dev/null +++ b/research-kit/test/fixtures/firecrawl-status-1.25.3.txt @@ -0,0 +1,9 @@ + + 🔥 firecrawl cli v1.25.3 + + ● Authenticated via FIRECRAWL_API_KEY + Concurrency: 0/2 jobs (parallel scrape limit) + Credits: 1,175 / 1,500 (78% left this cycle) + .firecrawl: not found - no local cache + .gitignore: missing - add .firecrawl/ to ignore cache + diff --git a/research-kit/test/transport.test.mjs b/research-kit/test/transport.test.mjs index 91b5813d..5c1ba5c5 100644 --- a/research-kit/test/transport.test.mjs +++ b/research-kit/test/transport.test.mjs @@ -656,7 +656,7 @@ test('O-4: the status parser reads the CLI\'s actual output', () => { }); // Captured from firecrawl-cli v1.24.6 on 2026-09-26, the version TESTED_CLI_VERSION moved to -// that day (and on from, 2026-10-02 - the 1.25.2 fixtures below). +// that day (and on from, 2026-10-02 and 2026-10-03 - the 1.25.2 and 1.25.3 fixtures below). // Why it moved: 1.23.3 bundles axios 1.15.2, which sends plain proxied requests that an // HTTPS proxy requiring CONNECT tunnels refuses with 405 - every call failed behind one // (research/BRIEF.md, 2026-09-26 addendum). 1.24.6 bundles axios 1.18.0. @@ -698,10 +698,10 @@ const REAL_SEARCH_1252 = readText(path.join(KIT_ROOT, 'test', 'fixtures', 'firec const REAL_SCRAPE_1252 = readText(path.join(KIT_ROOT, 'test', 'fixtures', 'firecrawl-scrape-1.25.2.json')); const REAL_SCRAPE_1252_STDERR = readText(path.join(KIT_ROOT, 'test', 'fixtures', 'firecrawl-scrape-1.25.2.stderr.txt')); -test('O-4: the status parser reads the TESTED version\'s actual output (1.25.2)', () => { +test('O-4: the status parser reads 1.25.2\'s actual output, the previously tested version', () => { assert.ok(REAL_STATUS_1252, 'the fixture must exist'); const s = firecrawl.parseStatus(REAL_STATUS_1252); - assert.equal(s.version, firecrawl.TESTED_CLI_VERSION, 'the fixture and the tested version must be the same release'); + assert.equal(s.version, '1.25.2', 'the fixture is the release its name says'); assert.equal(s.authenticated, true); assert.equal(s.credits, 154); assert.equal(s.creditLimit, 1000); @@ -732,6 +732,50 @@ test('O-4: 1.25.2 scrape stdout is one JSON value - the receipt is on stderr - a assert.equal(page.transport, 'firecrawl-cli'); }); +// Captured from firecrawl-cli v1.25.3 on 2026-10-03, the same way (a scratch prefix, the key +// per command, three credits, stdout and stderr apart). The release is one fix - `--objective` +// optional for alexandria feedback (firecrawl/cli#301) - with nothing on scrape, search or +// `--status`; the capture says so rather than the note: the same fields come back. +const REAL_STATUS_1253 = readText(path.join(KIT_ROOT, 'test', 'fixtures', 'firecrawl-status-1.25.3.txt')); +const REAL_SEARCH_1253 = readText(path.join(KIT_ROOT, 'test', 'fixtures', 'firecrawl-search-1.25.3.json')); +const REAL_SCRAPE_1253 = readText(path.join(KIT_ROOT, 'test', 'fixtures', 'firecrawl-scrape-1.25.3.json')); +const REAL_SCRAPE_1253_STDERR = readText(path.join(KIT_ROOT, 'test', 'fixtures', 'firecrawl-scrape-1.25.3.stderr.txt')); + +test('O-4: the status parser reads the TESTED version\'s actual output (1.25.3)', () => { + assert.ok(REAL_STATUS_1253, 'the fixture must exist'); + const s = firecrawl.parseStatus(REAL_STATUS_1253); + assert.equal(s.version, firecrawl.TESTED_CLI_VERSION, 'the fixture and the tested version must be the same release'); + assert.equal(s.authenticated, true); + assert.equal(s.credits, 1175); + assert.equal(s.creditLimit, 1500, 'the plan behind the capture allows 1,500 a cycle; the parser reads the comma'); + assert.equal(s.concurrencyLimit, 2); +}); + +test('O-4: 1.25.3 search output normalises as 1.25.2\'s did - data.tools still reaches no result', () => { + const raw = JSON.parse(REAL_SEARCH_1253); + assert.ok(Array.isArray(raw.data.web) && raw.data.web.length, 'the fixture must carry real results'); + assert.ok(Array.isArray(raw.data.tools) && raw.data.tools.length, 'the fixture must carry the populated tools list'); + const rows = firecrawl.normalizeSearch(REAL_SEARCH_1253); + assert.equal(rows.length, raw.data.web.length); + assert.deepEqual(rows.map((r) => r.url), raw.data.web.map((r) => r.url)); + assert.deepEqual(rows.map((r) => r.position), [1, 2, 3]); + for (const row of rows) assert.match(row.url, /^https?:\/\//); + assert.equal(JSON.stringify(rows).includes('capability'), false, 'a tool entry leaked into the results'); + assert.equal(firecrawl.creditsUsed(REAL_SEARCH_1253), 2, 'the vendor says what the search cost'); +}); + +test('O-4: 1.25.3 scrape stdout is one JSON value - the receipt is on stderr - and normalises whole', () => { + assert.doesNotThrow(() => JSON.parse(REAL_SCRAPE_1253), 'stdout holds nothing before or after the payload'); + assert.match(REAL_SCRAPE_1253_STDERR, /^Scrape ID: [0-9a-f-]+\s*$/, 'the receipt went to stderr'); + const page = firecrawl.normalizeScrape(REAL_SCRAPE_1253, 'https://docs.firecrawl.dev/sdks/cli'); + assert.equal(page.url, 'https://docs.firecrawl.dev/sdks/cli'); + assert.equal(page.title, 'CLI | Firecrawl'); + assert.equal(page.statusCode, 200); + assert.ok(page.markdown.length > 20000, `the whole page text is kept (${page.markdown.length} chars)`); + assert.equal(page.completeness, 'full'); + assert.equal(page.transport, 'firecrawl-cli'); +}); + test('O-4: stripAnsi removes the ESCAPE, not just the bracket sequence', () => { const ESC = String.fromCharCode(27); assert.ok(REAL_STATUS.includes(ESC), 'the fixture carries real escapes');