From c8e26d8e37172d1ce0c8d1e44a5ca2c4de8e49d2 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 13:13:22 +0000 Subject: [PATCH 1/2] docs: update agent source-of-truth files to latest SDK versions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update all 6 agent quickstart files with parameters and features from the latest SDK source code and v2 OpenAPI spec: - Node.js: v4.18.2 → v4.41.0 - Python: v4.22.1 → v4.44.0 - Rust: v2.0.0 → v2.21.0 - Java: v1.2.0 → v1.18.0 - Elixir: v1.0.0 → v1.11.0 - curl: aligned with current v2-openapi.json New parameters across SDKs: lockdown, redactPII, auditMetadata, domainTools, toolDetail, threatProtection, includeDomains, excludeDomains, country, highlights, origin. New format types: product, menu. Updated defaults from OpenAPI spec. Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_015dTb7RqrvM3oqd7xgvnPgU --- agent-source-of-truth/curl.mdx | 90 +++++++++++++++++++++++++++----- agent-source-of-truth/elixir.mdx | 36 ++++++++++++- agent-source-of-truth/java.mdx | 56 ++++++++++++++++++-- agent-source-of-truth/node.mdx | 87 ++++++++++++++++++++++++++++-- agent-source-of-truth/python.mdx | 85 ++++++++++++++++++++++++++++-- agent-source-of-truth/rust.mdx | 82 ++++++++++++++++++++++++++--- 6 files changed, 403 insertions(+), 33 deletions(-) diff --git a/agent-source-of-truth/curl.mdx b/agent-source-of-truth/curl.mdx index b7826e1c9..c474948d9 100644 --- a/agent-source-of-truth/curl.mdx +++ b/agent-source-of-truth/curl.mdx @@ -76,6 +76,7 @@ curl -X POST "https://api.firecrawl.dev/v2/search" \ Successful responses include `success`, `data`, optional `warning`, `id`, and `creditsUsed`. - `data.web`, `data.images`, `data.news`: result arrays; which keys appear depends on `sources` (by default only `data.web` is populated). +- `data.tools`: array of `DiscoveredTool` objects; present when `domainTools` is enabled or an `alexandria` source is included. - Web and news items include fields such as `title`, `url`, and (when `scrapeOptions` / formats request it) `markdown`, `html`, `rawHtml`, `links`, `screenshot`, `audio`, `video`, and `metadata`. - Image items include fields such as `imageUrl`, `url`, and dimensions when available. - `warning`: optional human-readable notice. @@ -96,6 +97,7 @@ Successful responses include `success`, `data`, optional `warning`, `id`, and `c - `{ "type": "web" }` with optional per-source `tbs` and `location` - `{ "type": "news" }` - `{ "type": "images" }` + - `{ "type": "alexandria" }` (free; returns tool contracts instead of web results) - `categories` - Type: array of typed category objects @@ -106,7 +108,7 @@ Successful responses include `success`, `data`, optional `warning`, `id`, and `c - `{ "type": "pdf" }` - `limit` - - Type: integer (minimum 1, maximum 100, default 5) + - Type: integer (minimum 1, maximum 100, default 10) - Use when: you want to cap results. - `tbs` @@ -136,6 +138,33 @@ Successful responses include `success`, `data`, optional `warning`, `id`, and `c - `"zdr"`: end-to-end zero data retention - `"anon"`: anonymized zero data retention +- `includeDomains` + - Type: array of strings (hostname format) + - Use when: you want to restrict results to specific domains. + - Notes: cannot be used together with `excludeDomains`. + +- `excludeDomains` + - Type: array of strings (hostname format) + - Use when: you want to exclude specific domains from results. + - Notes: cannot be used together with `includeDomains`. + +- `highlights` + - Type: boolean (default true) + - Use when: you want query-relevant highlights in search results. + +- `safe` + - Type: boolean + - Use when: you want to filter explicit content (SafeSearch). + +- `domainTools` + - Type: boolean + - Use when: you want to include tool contracts matching result domains. + +- `threatProtection` + - Type: ThreatProtectionOverride object + - Use when: you need per-request threat protection settings. + - Fields: `mode` (`"off"` | `"normal"`), `riskScoreThreshold` (0-100), `blacklist` (string[]), `whitelist` (string[]), `blockedTlds` (string[]), `failurePolicy` (`"open"` | `"closed"`). + - `scrapeOptions` - Type: object - Use when: you want to scrape each search result (see Scrape parameters for fields). @@ -221,6 +250,7 @@ Successful responses include `success` and `data`. Common `data` fields (dependi - `"markdown"`: markdown content - `"html"`: cleaned HTML - `"rawHtml"`: raw HTML + - `"rawBase64"`: base64-encoded response body - `"links"`: page links - `"images"`: image URLs - `"screenshot"`: screenshot output @@ -228,13 +258,17 @@ Successful responses include `success` and `data`. Common `data` fields (dependi - `"changeTracking"`: change tracking output - `"json"`: JSON extraction - `"branding"`: branding profile output + - `"product"`: product profile output + - `"menu"`: menu profile output - `"audio"`: audio extraction - `"video"`: video extraction - - Format object fields: + - Format object forms: - `type`: one of the format strings above - `prompt`, `schema`: JSON extraction options for `type: "json"` - `modes`, `schema`, `prompt`, `tag`: change tracking options for `type: "changeTracking"` - `fullPage`, `quality`, `viewport`: screenshot options for `type: "screenshot"` + - `{ "type": "question", "question": "..." }`: question-answering format (required `question`, max 10000 chars) + - `{ "type": "highlights", "query": "..." }`: highlight extraction format (required `query`, max 10000 chars) - `headers` - Type: object @@ -249,11 +283,11 @@ Successful responses include `success` and `data`. Common `data` fields (dependi - Use when: you want to exclude specific HTML tags. - `onlyMainContent` - - Type: boolean + - Type: boolean (default true) - Use when: you want to strip nav, footer, and other boilerplate. - `timeout` - - Type: number + - Type: number (default 60000, minimum 1000, maximum 300000) - Use when: you need a timeout in milliseconds. - `waitFor` @@ -265,9 +299,9 @@ Successful responses include `success` and `data`. Common `data` fields (dependi - Use when: you want a mobile viewport. - `parsers` - - Type: array of objects + - Type: array of objects (default `["pdf"]`) - Use when: you need file parsing controls. - - Confirmed shape: `{ "type": "pdf", "mode": "fast" | "auto" | "ocr", "maxPages": number }` (`type` required; other fields optional with defaults per spec) + - Confirmed shape: `{ "type": "pdf", "mode": "fast" | "auto" | "ocr", "maxPages": number, "pages": boolean, "blocks": boolean, "pageMarkers": boolean }` (`type` required; other fields optional with defaults per spec) - `actions` - Type: array of action objects @@ -288,24 +322,24 @@ Successful responses include `success` and `data`. Common `data` fields (dependi - Use when: you need geo or language-aware scraping. - `skipTlsVerification` - - Type: boolean + - Type: boolean (default true) - Use when: you need to skip TLS verification. - `removeBase64Images` - - Type: boolean + - Type: boolean (default true) - Use when: you want to drop base64 images from markdown output. - `blockAds` - - Type: boolean + - Type: boolean (default true) - Use when: you want ad and cookie popup blocking. - `proxy` - - Type: string + - Type: string (default `"auto"`) - Use when: you need proxy control. - Confirmed values: `"basic"`, `"enhanced"`, `"auto"` - `maxAge` - - Type: number + - Type: number (default 172800000, i.e. 2 days) - Use when: you want cached data up to a maximum age (milliseconds). - `minAge` @@ -313,7 +347,7 @@ Successful responses include `success` and `data`. Common `data` fields (dependi - Use when: you want cached data only if it is at least this old (milliseconds). - `storeInCache` - - Type: boolean + - Type: boolean (default true) - Use when: you want Firecrawl to cache the result. - `profile` @@ -324,6 +358,38 @@ Successful responses include `success` and `data`. Common `data` fields (dependi - Type: boolean - Use when: you want zero data retention for this scrape. +- `lockdown` + - Type: boolean (default false) + - Use when: you want to serve from cache only, never making an outbound request. ZDR. 5 credits on cache hit, 1 on miss. + +- `redactPII` + - Type: boolean or RedactPIIOptions object (default false) + - Use when: you want to redact PII from returned markdown. + - Notes: `true` for defaults. Object form: `{ "mode": "accurate" | "aggressive" | "fast", "entities": ["PERSON", "EMAIL", "PHONE", "LOCATION", "FINANCIAL", "SECRET"], "replaceStyle": "tag" | "mask" | "remove" }`. + +- `auditMetadata` + - Type: object + - Use when: you need user attribution for SIEM logging. + - Fields: `username` (string, max 1024 chars). + +- `domainTools` + - Type: boolean (default false) + - Use when: you want to discover Alexandria tool contracts for the scraped domain. + +- `threatProtection` + - Type: ThreatProtectionOverride object + - Use when: you need per-request threat protection settings. + - Fields: `mode` (`"off"` | `"normal"`), `riskScoreThreshold` (0-100), `blacklist` (string[]), `whitelist` (string[]), `blockedTlds` (string[]), `failurePolicy` (`"open"` | `"closed"`). + +- `alexandria` + - Type: AlexandriaCall or array of AlexandriaCall (1-10) + - Use when: you want to execute catalogued provider tools instead of scraping a URL. + - Each call: `{ "provider": string, "capability": string, "options": object }`. + +- `onlyCleanContent` + - Type: boolean (default false, beta) + - Use when: you want an LLM-based pass to remove residual boilerplate from output. + ## Interact ### Why use it diff --git a/agent-source-of-truth/elixir.mdx b/agent-source-of-truth/elixir.mdx index 584b20d96..1143b451d 100644 --- a/agent-source-of-truth/elixir.mdx +++ b/agent-source-of-truth/elixir.mdx @@ -10,7 +10,7 @@ Canonical Firecrawl Elixir source of truth for agents. Generated from SDK source Add to `mix.exs`: ```elixir -{:firecrawl, "~> 1.0.0"} +{:firecrawl, "~> 1.11"} ``` ## Authenticate @@ -23,6 +23,15 @@ config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") {:ok, res} = Firecrawl.search_and_scrape([query: "site:docs.firecrawl.dev webhook retries"], api_key: "fc-your-api-key") ``` +No key is required — the SDK falls back to a keyless free tier (rate-limited per IP). Every request body includes an `"origin"` field set to `"elixir-sdk@"` for SDK telemetry. + +### Common options + +These options are available on all functions: + +- `:api_key` -- override the API key for this request +- `:base_url` -- override the default base URL (`https://api.firecrawl.dev/v2`) + ## When To Use What - `search`: use when you start with a query and need discovery. @@ -132,6 +141,19 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or - Type: keyword list - Use when: you want to scrape each search result (see Scrape parameters for fields). +- `include_domains` + - Type: list of strings + - Use when: you want to restrict results to specific domains. + +- `exclude_domains` + - Type: list of strings + - Use when: you want to exclude specific domains. + +- `highlights` + - Type: boolean + - Use when: you want query-relevant highlights. + - Notes: defaults to true. + ## Scrape ### Why use it @@ -302,6 +324,18 @@ Use scrape when you already have a URL and want structured content in one or mor - Type: boolean - Use when: you want zero data retention for this scrape. +- `audit_metadata` + - Type: keyword list (keys: `username: :string`, required) + - Use when: you need SIEM logging attribution. + +- `lockdown` + - Type: boolean + - Use when: you want to serve only cached results, never scrape. + +- `redact_pii` + - Type: boolean + - Use when: you want to redact PII from content. + ## Interact ### Why use it diff --git a/agent-source-of-truth/java.mdx b/agent-source-of-truth/java.mdx index c61d03c6e..55fee91ab 100644 --- a/agent-source-of-truth/java.mdx +++ b/agent-source-of-truth/java.mdx @@ -13,14 +13,14 @@ Maven: com.firecrawl firecrawl-java - 1.2.0 + 1.18.0 ``` Gradle: ```gradle -implementation("com.firecrawl:firecrawl-java:1.2.0") +implementation("com.firecrawl:firecrawl-java:1.18.0") ``` ## Authenticate @@ -33,6 +33,21 @@ FirecrawlClient client = FirecrawlClient.builder() .build(); ``` +### Builder options + +- `apiKey` — Your API key. Also read from the `FIRECRAWL_API_KEY` environment variable or the `firecrawl.apiKey` system property. When omitted entirely the client falls back to the keyless free tier. +- `apiUrl` — Base URL. Defaults to `"https://api.firecrawl.dev"`. Overridable via the `FIRECRAWL_API_URL` environment variable. +- `timeoutMs` — Request timeout in milliseconds. Default: `300_000`. +- `maxRetries` — Maximum number of retries. Default: `3`. +- `backoffFactor` — Exponential back-off multiplier. Default: `0.5`. + +### Factory method + +```java +// Reads FIRECRAWL_API_KEY env var or firecrawl.apiKey system property +FirecrawlClient client = FirecrawlClient.fromEnv(); +``` + ## When To Use What - `search`: use when you start with a query and need discovery. @@ -122,7 +137,8 @@ SearchData results = client.search("site:docs.firecrawl.dev crawl webhooks", opt - `"developer"`: Developer Index results - `"research"`: research and academic results - `"pdf"`: PDF-focused results - - `{type: "developer" | "research" | "pdf"}`: typed category map form + - `"github"`: GitHub results + - `{type: "developer" | "research" | "pdf" | "github"}`: typed category map form - `options.limit` - Type: Integer @@ -152,6 +168,22 @@ SearchData results = client.search("site:docs.firecrawl.dev crawl webhooks", opt - Type: String - Use when: the API expects an integration identifier on the request. +- `options.includeDomains` + - Type: List<String> + - Use when: you want to restrict results to specific domains. + +- `options.excludeDomains` + - Type: List<String> + - Use when: you want to exclude specific domains from results. + +- `options.country` + - Type: String + - Use when: you need a country code to geo-target results. + +- `options.highlights` + - Type: Boolean + - Use when: you want query-relevant highlights in results. Defaults to true. + ## Scrape ### Why use it @@ -165,7 +197,7 @@ Use scrape when you already have a URL and want structured content in one or mor ### Return value -`scrape` returns `Document`. Typical getters include `getMarkdown()`, `getHtml()`, `getRawHtml()`, `getJson()`, `getMetadata()`, `getLinks()`, `getAudio()`, `getVideo()`, and additional fields when the corresponding formats are requested. +`scrape` returns `Document`. Typical getters include `getMarkdown()`, `getHtml()`, `getRawHtml()`, `getJson()`, `getMetadata()`, `getLinks()`, `getAudio()`, `getVideo()`, `getProduct()` (Product), `getMenu()` (Menu), `getPages()` (List<PdfPage>), `getBlocks()` (List<PdfPageBlocks>), and additional fields when the corresponding formats are requested. ### Simple Example @@ -327,6 +359,18 @@ Document doc = client.scrape("https://example.com/pricing", options); - Type: String - Use when: the API expects an integration identifier on the request. +- `options.lockdown` + - Type: Boolean + - Use when: you want to serve only previously cached results. + +- `options.redactPII` + - Type: Boolean + - Use when: you want to redact PII from content. + +- `options.auditMetadata` + - Type: AuditMetadata (field: `username`: String) + - Use when: you need SIEM logging attribution. + ## Interact ### Why use it @@ -411,10 +455,12 @@ BrowserDeleteResponse stopped = client.stopInteractiveBrowser(""); - `origin` - Type: String - - Use when: you need an optional origin label on the request. Prefer omitting unless your integration requires it. + - Default: `"java-sdk@1.18.0"` + - Use when: you need an optional origin label on the request for attribution. Prefer omitting unless your integration requires it. ## Notes +- **Async variants:** All main methods (`search`, `scrape`, `interact`, `stopInteractiveBrowser`) have `*Async` variants (e.g. `searchAsync`, `scrapeAsync`) that return `CompletableFuture<>`. - Deprecated aliases: `scrapeExecute` → `interact`, `deleteScrapeBrowser` → `stopInteractiveBrowser` (and the corresponding `*Async` helpers). - The Java SDK exposes code-based interactions only: there is no `prompt` parameter on `interact` (unlike some other language SDKs). diff --git a/agent-source-of-truth/node.mdx b/agent-source-of-truth/node.mdx index 5881a3427..50ae2166d 100644 --- a/agent-source-of-truth/node.mdx +++ b/agent-source-of-truth/node.mdx @@ -3,7 +3,7 @@ title: "Node.js Source of Truth" description: "Canonical Firecrawl Node.js source of truth for agents using key endpoints like search, scrape, and interact." --- -Canonical Firecrawl Node.js source of truth for agents. Aligned with `firecrawl` **v4.18.2** (`firecrawl/apps/js-sdk/firecrawl`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. +Canonical Firecrawl Node.js source of truth for agents. Aligned with `firecrawl` **v4.41.0** (`firecrawl/apps/js-sdk/firecrawl`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. ## Install @@ -13,12 +13,21 @@ npm install firecrawl ## Authenticate +The constructor accepts either a plain string API key or an options object: + ```ts import { Firecrawl } from "firecrawl"; +// Plain string API key +const client = new Firecrawl(process.env.FIRECRAWL_API_KEY); + +// Options object const client = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY, - // apiUrl: "https://api.firecrawl.dev" // optional; falls back to FIRECRAWL_API_URL or cloud default + apiUrl: "https://api.firecrawl.dev", // optional; falls back to FIRECRAWL_API_URL or cloud default + timeoutMs: 30000, // optional per-request timeout + maxRetries: 3, // optional + backoffFactor: 2, // optional }); ``` @@ -78,6 +87,7 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { - `web`: web index hits - `news`: news hits - `images`: image hits +- `tools?: DiscoveredTool[]`: discovered tools when `domainTools` is enabled **Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Do not access `result.data`. Web results are in `result.web`, news in `result.news`, images in `result.images`. @@ -98,7 +108,8 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { - `"web"`: web index results - `"news"`: news results - `"images"`: image results - - `{ type: "web" | "news" | "images" }`: typed source object form + - `"alexandria"`: Alexandria knowledge base results + - `{ type: "web" | "news" | "images" | "alexandria" }`: typed source object form - `options.categories` - Type: array of category names or typed category objects @@ -107,7 +118,8 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { - `"developer"`: Developer Index results - `"research"`: research and academic results - `"pdf"`: PDF-focused results - - `{ type: "developer" | "research" | "pdf" }`: typed category object form + - `"github"`: GitHub-focused results + - `{ type: "developer" | "research" | "pdf" | "github" }`: typed category object form - `options.limit` - Type: number @@ -133,6 +145,38 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { - Type: `ScrapeOptions` - Use when: you want to scrape each search result (see Scrape parameters for fields). The SDK runs the same validation as for `scrape` (for example plain string `"json"` in `formats` is rejected). +- `options.domainTools` + - Type: boolean + - Use when: you want domain-matched tool discovery. + +- `options.toolDetail` + - Type: `"compact" | "summary" | "full"` + - Use when: you want to control tool contract verbosity. + +- `options.includeDomains` + - Type: array of strings + - Use when: you want to restrict results to specific domains. + +- `options.excludeDomains` + - Type: array of strings + - Use when: you want to exclude specific domains. Cannot be used with `includeDomains`. + +- `options.country` + - Type: string + - Use when: you want ISO 3166-1 alpha-2 geo targeting. + +- `options.highlights` + - Type: boolean + - Use when: you want query-relevant highlights. Defaults to `true`. + +- `options.enterprise` + - Type: array of `"default" | "anon" | "zdr"` + - Use when: you need enterprise search controls. + +- `options.threatProtection` + - Type: `ThreatProtectionOptions` + - Use when: you need threat protection controls. Fields: `mode` (`"off"` | `"manual-only"` | `"normal"` | `"zscaler"`), `riskScoreThreshold`, `blacklist`, `whitelist`, `blockedTlds`, `failurePolicy`. + ## Scrape ### Why use it @@ -208,6 +252,8 @@ const doc = await client.scrape("https://example.com/pricing", { - `"branding"`: branding profile output - `"audio"`: audio extraction - `"video"`: video extraction + - `"product"`: product data extraction + - `"menu"`: menu/navigation extraction - Object-only format types (at minimum `type` as shown): - `{ type: "json", prompt?: string, schema?: JSON schema or Zod schema }`: at least one of `prompt` or `schema` is required (SDK validation). - `{ type: "question", question: string }`: question-answer style extraction. @@ -215,6 +261,7 @@ const doc = await client.scrape("https://example.com/pricing", { - `{ type: "screenshot", fullPage?, quality?, viewport? }`: same options as the string form but as an object. - `{ type: "changeTracking", modes: ("git-diff" | "json")[], schema?, prompt?, tag? }`: `modes` is required. - `{ type: "attributes", selectors: Array<{ selector, attribute }> }` + - `{ type: "query", prompt: string, mode?: "freeform" | "directQuote" }`: **(deprecated)** query-based extraction. - Shared object fields where applicable: - `schema`: JSON schema or Zod schema for `json`, or for `changeTracking` in `json` mode (Zod is converted to JSON Schema by the SDK). - `modes`: for `changeTracking` only: `"git-diff"` and/or `"json"`. @@ -316,6 +363,34 @@ const doc = await client.scrape("https://example.com/pricing", { - Type: object with `name` and optional `saveChanges` - Use when: you want a persistent browser profile shared across scrapes and interactions. +- `options.lockdown` + - Type: boolean + - Use when: you only want to serve previously cached results, never scrape live. + +- `options.redactPII` + - Type: boolean or `RedactPIIOptions` + - Use when: you want to redact personally identifiable information from output. + +- `options.auditMetadata` + - Type: `{ username: string }` + - Use when: you need SIEM logging attribution. + +- `options.domainTools` + - Type: boolean + - Use when: you want to discover Alexandria tools for the URL being scraped. + +- `options.toolDetail` + - Type: `"compact" | "summary" | "full"` + - Use when: you want to control tool contract verbosity. + +- `options.origin` + - Type: string + - Use when: you need request attribution. Auto-set by the SDK. + +- `options.threatProtection` + - Type: `ThreatProtectionOptions` + - Use when: you need threat protection controls. Fields: `mode` (`"off"` | `"manual-only"` | `"normal"` | `"zscaler"`), `riskScoreThreshold`, `blacklist`, `whitelist`, `blockedTlds`, `failurePolicy`. + ## Interact ### Why use it @@ -374,6 +449,10 @@ const result = await client.interact("", { - Type: number - Use when: you need an execution timeout in seconds. +- `args.origin` + - Type: string + - Use when: you need request attribution. Auto-set by the SDK. + ### Return value (`interact`) `ScrapeExecuteResponse` matches `BrowserExecuteResponse`. Confirmed fields include: diff --git a/agent-source-of-truth/python.mdx b/agent-source-of-truth/python.mdx index 710430435..e7b375f06 100644 --- a/agent-source-of-truth/python.mdx +++ b/agent-source-of-truth/python.mdx @@ -3,7 +3,7 @@ title: "Python Source of Truth" description: "Canonical Firecrawl Python source of truth for agents using key endpoints like search, scrape, and interact." --- -Canonical Firecrawl Python source of truth for agents. Generated from SDK source (`firecrawl-py` / `firecrawl` **4.22.1**) and the v2 OpenAPI spec. Method names, parameters, and return types match the v2 client in `firecrawl/v2/client.py` unless noted. +Canonical Firecrawl Python source of truth for agents. Generated from SDK source (`firecrawl-py` / `firecrawl` **4.44.0**) and the v2 OpenAPI spec. Method names, parameters, and return types match the v2 client in `firecrawl/v2/client.py` unless noted. ## Install @@ -19,8 +19,14 @@ from firecrawl import Firecrawl client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) # client = Firecrawl(api_key="fc-...", api_url="https://api.firecrawl.dev") +# client = Firecrawl(api_key="fc-...", timeout=30.0, max_retries=5, backoff_factor=1.0) ``` +The constructor also accepts: +- `timeout` (float): Request timeout in seconds. +- `max_retries` (int): Maximum number of retries for failed requests (default `3`). +- `backoff_factor` (float): Exponential backoff factor for retries, e.g. `0.5` means wait 0.5s, then 1s, then 2s between retries (default `0.5`). + ## When To Use What - `search`: use when you start with a query and need discovery. @@ -46,6 +52,7 @@ Returns a `SearchData` model with optional lists: - `web` — web hits (`SearchResultWeb` or full `Document` when `scrape_options` hydrates content) - `news` — news hits (`SearchResultNews` or `Document`) - `images` — image hits (`SearchResultImages` or `Document`) +- `tools` — discovered tools (`Optional[List[DiscoveredTool]]`), present when `domain_tools` is enabled Omitted buckets are `None` when the API did not return that key. @@ -115,7 +122,8 @@ results = client.search( - `"developer"`: Developer Index results - `"research"`: research and academic results - `"pdf"`: PDF-focused results - - `Category(type="developer" | "research" | "pdf")`: typed category object form + - `"github"`: restrict web results to github.com (a `site:` filter) + - `Category(type="developer" | "research" | "pdf" | "github")`: typed category object form - `limit` - Type: int @@ -143,6 +151,38 @@ results = client.search( - Type: `ScrapeOptions` - Use when: you want to scrape each search result (see Scrape parameters for fields). +- `domain_tools` + - Type: bool + - Use when: domain-matched tool discovery. + +- `tool_detail` + - Type: Literal["compact", "summary", "full"] + - Use when: control tool contract verbosity. + +- `include_domains` + - Type: list of str + - Use when: restrict results to specific domains. + +- `exclude_domains` + - Type: list of str + - Use when: exclude specific domains. Mutually exclusive with `include_domains`. + +- `country` + - Type: str + - Use when: ISO 3166-1 alpha-2 geo targeting. + +- `highlights` + - Type: bool + - Use when: query-relevant highlights. Defaults to true. + +- `enterprise` + - Type: list of str + - Use when: enterprise search controls. + +- `threat_protection` + - Type: `ThreatProtectionOptions` + - Use when: you need threat protection controls (see Scrape parameters for fields). + ## Scrape ### Why use it @@ -214,10 +254,14 @@ doc = client.scrape( - `"branding"`: branding profile output - `"audio"`: audio extraction - `"video"`: video extraction + - `"product"`: structured product data extraction + - `"menu"`: structured menu data extraction + - `"query"`: deprecated alias — use `{"type": "highlights", ...}` instead - Object-only format types: - `{"type": "json", ...}`: JSON extraction. Use an object, not the plain string `"json"`. - `{"type": "question", "question": "..."}`: question-answer output. - `{"type": "highlights", "query": "..."}`: relevant source-text output. + - Deprecated: `QueryFormat` — use `{"type": "highlights", ...}` instead. - Format object fields: - `type`: one of the format strings above, or `"json"`, `"question"`, or `"highlights"` for object-only formats - `question`: for `type: "question"` @@ -320,6 +364,36 @@ doc = client.scrape( - Type: dict with `name` and optional `save_changes` or `saveChanges` - Use when: you want a persistent browser profile shared across scrapes and interactions. +- `auto_resume` + - Type: bool + - Use when: you want the SDK to auto-resume timed-out scrapes (default behavior). + +- `lockdown` + - Type: bool + - Use when: you only want to serve previously cached results, never scrape live. Returns 404 `SCRAPE_LOCKDOWN_CACHE_MISS` on cache miss. + +- `redact_pii` + - Type: Union[bool, `RedactPIIOptions`] + - Use when: you want to redact PII from output. + - Notes: set on `ScrapeOptions`. `RedactPIIOptions` fields: `mode` (Literal["accurate", "aggressive", "fast"]), `entities` (list of entity types), `replace_style` (Literal["tag", "mask", "remove"]). + +- `audit_metadata` + - Type: `AuditMetadata` (has `username: str`) + - Use when: you need SIEM logging attribution. + +- `domain_tools` + - Type: bool + - Use when: you want to discover Alexandria tools for the URL being scraped. + +- `tool_detail` + - Type: Literal["compact", "summary", "full"] + - Use when: you want to control tool contract verbosity. + +- `threat_protection` + - Type: `ThreatProtectionOptions` + - Use when: you need threat protection controls. + - Fields: `mode` (Literal["off", "manual-only", "normal", "zscaler"]), `risk_score_threshold` (int, 0-100), `blacklist` (list of str), `whitelist` (list of str), `blocked_tlds` (list of str), `failure_policy` (Literal["open", "closed"]). + ## Interact ### Why use it @@ -328,7 +402,7 @@ Use interact when a page requires browser actions or code execution after a scra ### Preferred SDK method -`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None)` +`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None, origin=None)` `prompt` is keyword-only: call `client.interact(job_id, prompt="...")` or `client.interact(job_id, code="...")`. At least one of `code` or `prompt` must be non-empty (validated in `firecrawl/v2/methods/scrape.py`). @@ -382,6 +456,11 @@ result = client.interact( - Type: int - Use when: you need an execution timeout in seconds. +- `origin` + - Type: str + - Use when: request attribution. + - Notes: auto-set by SDK. + ### Return value Returns `BrowserExecuteResponse`: `success`, optional `live_view_url`, `interactive_live_view_url`, `output`, `stdout`, `result`, `stderr`, `exit_code`, `killed`, `error` (API camelCase is normalized to snake_case on the model). diff --git a/agent-source-of-truth/rust.mdx b/agent-source-of-truth/rust.mdx index a862bfa16..790d86e9e 100644 --- a/agent-source-of-truth/rust.mdx +++ b/agent-source-of-truth/rust.mdx @@ -11,7 +11,7 @@ Canonical Firecrawl Rust source of truth for agents. Generated from SDK source a cargo add firecrawl ``` -Crate: **`firecrawl`** on crates.io. The current SDK version is **2.0.0** (verify the latest release on crates.io before pinning). +Crate: **`firecrawl`** on crates.io. The current SDK version is **2.21.0** (verify the latest release on crates.io before pinning). ## Authenticate @@ -106,12 +106,12 @@ let results = client - `options.sources` - Type: `Vec` - Use when: you want to control which sources are searched. - - Confirmed values: `Web`, `News`, `Images` + - Confirmed values: `Web`, `News`, `Images`, `Alexandria` - `options.categories` - Type: `Vec` - Use when: you want to filter results by category. - - Confirmed values: `Research`, `Pdf` + - Confirmed values: `Research`, `Pdf`, `Github` - `options.limit` - Type: u32 @@ -137,11 +137,48 @@ let results = client - Type: `ScrapeOptions` - Use when: you want to scrape each search result (see Scrape parameters for fields). +- `options.domain_tools` + - Type: `Option` + - Use when: you want domain tool discovery. + +- `options.tool_detail` + - Type: `Option` + - Use when: you want to control tool detail level. + - Confirmed values: `Compact`, `Summary`, `Full` + - Notes: defaults to `Compact`. + +- `options.include_domains` + - Type: `Option>` + - Use when: you want to restrict results to specific domains. + +- `options.exclude_domains` + - Type: `Option>` + - Use when: you want to exclude specific domains from results. + +- `options.country` + - Type: `Option` + - Use when: you need country code geo targeting. + +- `options.highlights` + - Type: `Option` + - Use when: you want highlighted snippets in results. + - Notes: defaults to `true`. + +- `options.origin` + - Type: `Option` + - Use when: origin label for telemetry. + - Notes: auto-set by the SDK. + - `options.integration` - Type: `Option` - Use when: you need an integration identifier for server-side tracking. - Notes: omit in agent-oriented examples unless your product intentionally sets it. +### SearchData update + +`SearchData` now includes: +- `tools: Option>` — discovered tools when `domain_tools` is enabled. + ## Scrape ### Why use it @@ -225,7 +262,7 @@ let doc = client - `options.formats` - Type: `Vec` - Use when: you want multiple output formats. - - Confirmed values: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Audio`, `Video` + - Confirmed values: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Audio`, `Video`, `Product`, `Menu`, `Question(...)`, `Highlights(...)`, `Query(...)` - `options.headers` - Type: `HashMap` @@ -260,7 +297,7 @@ let doc = client - Use when: you need file parsing controls. - Confirmed values: - `ParserConfig::Simple("pdf".to_string())` - - `ParserConfig::Pdf { parser_type: "pdf", max_pages: Some(n) }` + - `ParserConfig::Pdf { parser_type: "pdf", max_pages: Some(n), mode: Option, pages: Option, blocks: Option, page_markers: Option }` - `options.actions` - Type: `Vec` @@ -327,7 +364,7 @@ let doc = client - `options.json_options` - Type: `JsonOptions` - Use when: you want to configure JSON extraction. - - Confirmed fields: `schema`, `system_prompt`, `prompt` + - Confirmed fields: `schema`, `system_prompt`, `prompt`, `check_prompt_injection` - `options.screenshot_options` - Type: `ScreenshotOptions` @@ -344,6 +381,33 @@ let doc = client - Use when: you want attribute extraction output. - Confirmed fields: `selector`, `attribute` +- `options.origin` + - Type: `Option` + - Use when: origin label for telemetry. + - Notes: auto-set to `"rust-sdk@{version}"` by the SDK. + +- `options.lockdown` + - Type: `Option` + - Use when: you want to serve only cached results. + +- `options.redact_pii` + - Type: `Option` + - Use when: you want PII redaction. + - Notes: serializes as `redactPII`. + +- `options.audit_metadata` + - Type: `Option` + - Use when: you need SIEM logging attribution. + +- `options.domain_tools` + - Type: `Option` + - Use when: you want Alexandria tool discovery. + +- `options.tool_detail` + - Type: `Option` + - Use when: you want to control tool detail level. + - Confirmed values: `Compact`, `Summary`, `Full` + ## Interact ### Why use it @@ -423,8 +487,8 @@ let stopped = client.stop_interaction("").await?; - `options.origin` - Type: `Option` - - Use when: you need an optional origin label for execution telemetry. - - Notes: omit in agent-oriented examples unless your product intentionally sets it. + - Use when: origin label for execution telemetry. + - Notes: auto-set by the SDK. Omit in agent-oriented examples unless your product intentionally overrides it. At least one of `options.code` or `options.prompt` must be non-empty; otherwise the SDK returns `FirecrawlError::Misuse` before calling the API. @@ -442,6 +506,8 @@ The v2 OpenAPI spec currently models the interact request body with `code` as re - Deprecated aliases: `scrape_execute`, `stop_interactive_browser`, and `delete_scrape_browser` map to `interact` and `stop_interaction`. - `ScrapeOptions` includes dedicated `json_options`, `screenshot_options`, and `change_tracking_options` for advanced formats. - `search_and_scrape(query, limit)` is a convenience helper: it calls `search` with default `ScrapeOptions` and returns `Vec` built from `SearchResultOrDocument::Document` entries in `data.web` (see `search.rs`). +- `scrape_with_schema(url, schema, prompt)` is a convenience method for JSON extraction with a schema and prompt. +- `scrape_alexandria(calls, options)` executes Alexandria tool calls. - v2 SDK exports all types at the crate root: `use firecrawl::Client` (not `use firecrawl::v2::Client`). - Error types simplified in v2: `CrawlJobFailed(String, CrawlStatus)` → `JobFailed(String)`, `Missuse` → `Misuse`. From 4a6b0bbe17aac1cb481973ceb908184993201319 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 13:14:32 +0000 Subject: [PATCH 2/2] docs: add origin param and response fields to curl interact section Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_015dTb7RqrvM3oqd7xgvnPgU --- agent-source-of-truth/curl.mdx | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/agent-source-of-truth/curl.mdx b/agent-source-of-truth/curl.mdx index c474948d9..e15e40520 100644 --- a/agent-source-of-truth/curl.mdx +++ b/agent-source-of-truth/curl.mdx @@ -443,9 +443,13 @@ curl -X POST "https://api.firecrawl.dev/v2/scrape//interact" \ - Type: integer (seconds; minimum 1, maximum 300, default 30) - Use when: you need an execution timeout. +- `origin` (JSON body) + - Type: string (optional) + - Use when: you want to attach an origin label to the interaction. + ### Response -Successful responses include `success` plus execution fields such as `stdout`, `result` (alias of stdout), `stderr`, `exitCode`, `killed`, and `error` (nullable). +Successful responses include `success` plus execution fields such as `stdout`, `result` (alias of stdout), `stderr`, `exitCode`, `killed`, `error` (nullable), `cdpUrl`, and `interactiveLiveViewUrl`. ### DELETE /scrape/{jobId}/interact