diff --git a/agent-source-of-truth/elixir.mdx b/agent-source-of-truth/elixir.mdx index 584b20d96..330da5419 100644 --- a/agent-source-of-truth/elixir.mdx +++ b/agent-source-of-truth/elixir.mdx @@ -3,16 +3,18 @@ title: "Elixir Source of Truth" description: "Canonical Firecrawl Elixir source of truth for agents using key endpoints like search, scrape, and interact." --- -Canonical Firecrawl Elixir source of truth for agents. Generated from SDK source and the v2 OpenAPI spec. +Canonical Firecrawl Elixir source of truth for agents. Generated from SDK source (`firecrawl` hex package **v1.12.0**) and the v2 OpenAPI spec. ## Install Add to `mix.exs`: ```elixir -{:firecrawl, "~> 1.0.0"} +{:firecrawl, "~> 1.12.0"} ``` +Requires Elixir ~> 1.15. + ## Authenticate ```elixir @@ -23,6 +25,14 @@ config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") {:ok, res} = Firecrawl.search_and_scrape([query: "site:docs.firecrawl.dev webhook retries"], api_key: "fc-your-api-key") ``` +All functions accept an optional keyword list as the last argument (`opts`) with: + +- `:api_key` — override the API key for this request. Nil/empty = keyless free tier (rate-limited per IP). +- `:base_url` — override the default API base URL (default: `"https://api.firecrawl.dev/v2"`; useful for self-hosted instances). +- Any other options are passed through to `Req`. + +The SDK automatically sends an `origin` field with every request body set to `"elixir-sdk@"`. + ## When To Use What - `search`: use when you start with a query and need discovery. @@ -39,6 +49,8 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or `Firecrawl.search_and_scrape(params \\ [], opts \\ [])` +Bang variant: `Firecrawl.search_and_scrape!/2` raises on error. + ### Simple Example ```elixir @@ -57,7 +69,10 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or location: "San Francisco,California,United States", country: "US", ignore_invalid_urls: true, + highlights: true, timeout: 60000, + include_domains: ["docs.firecrawl.dev"], + tool_detail: :summary, scrape_options: [ formats: [ "markdown", @@ -75,7 +90,7 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or ### Parameters - `query` - - Type: string + - Type: string (required) - Use when: you need a search query. - Notes: use `site:example.com` to limit results to a domain. @@ -86,7 +101,16 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or - `"web"` or `:web` - `"news"` or `:news` - `"images"` or `:images` - - `%{type: "web" | "news" | "images"}` + - `"alexandria"` or `:alexandria` + - `%{type: "web" | "news" | "images" | "alexandria"}` + +- `domain_tools` + - Type: boolean + - Use when: you want domain-matched Alexandria tools alongside semantic matches. + +- `tool_detail` + - Type: atom or string (`:compact`, `:summary`, `:full`) + - Use when: you want to control tool discovery detail level. - `categories` - Type: list of atoms, strings, or maps @@ -97,6 +121,14 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or - `"pdf"` or `:pdf` - `%{type: "developer" | "research" | "pdf"}` +- `include_domains` + - Type: list of strings + - Use when: you want to restrict results to specific domains. + +- `exclude_domains` + - Type: list of strings + - Use when: you want to exclude specific domains from results. + - `limit` - Type: integer - Use when: you want to cap results. @@ -113,10 +145,18 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or - Type: string - Use when: you want ISO 3166-1 alpha-2 targeting (for example `"US"`). +- `safe` + - Type: boolean + - Use when: you want SafeSearch filtering of explicit content. + - `ignore_invalid_urls` - Type: boolean - Use when: you want to drop URLs that cannot be scraped by other endpoints. +- `highlights` + - Type: boolean + - Use when: you want query-relevant highlights for search results. Defaults to true. + - `timeout` - Type: integer - Use when: you need a request timeout in milliseconds. @@ -128,6 +168,10 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or - `"zdr"`: end-to-end zero data retention - `"anon"`: anonymized zero data retention +- `threat_protection` + - Type: keyword list + - Use when: you need enterprise per-request threat protection override. + - `scrape_options` - Type: keyword list - Use when: you want to scrape each search result (see Scrape parameters for fields). @@ -142,6 +186,8 @@ Use scrape when you already have a URL and want structured content in one or mor `Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` +Bang variant: `Firecrawl.scrape_and_extract_from_url!/2` raises on error. + ### Simple Example ```elixir @@ -177,7 +223,6 @@ Use scrape when you already have a URL and want structured content in one or mor block_ads: true, proxy: :auto, max_age: 86400000, - min_age: 1, store_in_cache: true, profile: [name: "docs-session", save_changes: true], zero_data_retention: false @@ -187,7 +232,7 @@ Use scrape when you already have a URL and want structured content in one or mor ### Parameters - `url` - - Type: string + - Type: string (required) - Use when: you want to scrape a specific page. - `formats` @@ -204,6 +249,8 @@ Use scrape when you already have a URL and want structured content in one or mor - `"changeTracking"`: change tracking output - `"json"`: JSON extraction - `"branding"`: branding profile output + - `"product"`: product profile output + - `"menu"`: menu profile output - `"audio"`: audio extraction - `"video"`: video extraction - Format map fields: @@ -284,7 +331,7 @@ Use scrape when you already have a URL and want structured content in one or mor - `max_age` - Type: integer - - Use when: you want cached data up to a maximum age (milliseconds). + - Use when: you want cached data up to a maximum age (milliseconds). Set to 0 to bypass index reuse. - `min_age` - Type: integer @@ -294,10 +341,34 @@ Use scrape when you already have a URL and want structured content in one or mor - Type: boolean - Use when: you want Firecrawl to cache the result. +- `lockdown` + - Type: boolean + - Use when: you want to serve only previously cached results and never make outbound requests. + +- `check_prompt_injection` + - Type: boolean + - Use when: you want to scan page content for prompt injection before LLM-backed formats run. Adds 4 credits. + +- `redact_pii` + - Type: any (boolean or options map) + - Use when: you want to redact PII from returned markdown. + +- `audit_metadata` + - Type: keyword list with `username:` (required, string) + - Use when: you need user attribution included with SIEM logging events. + - `profile` - Type: keyword list with `name:` and optional `save_changes:` (or `saveChanges:`) - Use when: you want a persistent browser profile shared across scrapes and interactions. +- `domain_tools` + - Type: boolean + - Use when: you want domain-matched Alexandria tools for this URL included in `tools`. + +- `threat_protection` + - Type: keyword list + - Use when: you need enterprise per-request threat protection override. + - `zero_data_retention` - Type: boolean - Use when: you want zero data retention for this scrape. @@ -308,9 +379,13 @@ Use scrape when you already have a URL and want structured content in one or mor Use interact when a page requires browser actions or code execution after a scrape starts. -### Preferred SDK method +### Scrape-bound interact + +**Preferred SDK method:** `Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` -`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` +Bang variant: `Firecrawl.interact_with_scrape_browser_session!/3` + +This interacts with the browser session tied to a scrape job via `POST /scrape/{jobId}/interact`. ### Simple Example @@ -334,14 +409,14 @@ Use interact when a page requires browser actions or code execution after a scra ) ``` -### Parameters +### Parameters (scrape-bound interact) - `job_id` - Type: string - Use when: you have a scrape job ID. - `code` - - Type: string + - Type: string (required) - Use when: you want to run code in the browser session. - `language` @@ -357,19 +432,36 @@ Use interact when a page requires browser actions or code execution after a scra - Type: string - Use when: you want an optional origin label for execution telemetry. -### Stop interactive session +### Stop scrape-bound session + +`Firecrawl.stop_interactive_scrape_browser_session(job_id, opts \\ [])` issues `DELETE /scrape/{jobId}/interact` and tears down the browser session for that scrape job. -`Firecrawl.stop_interactive_scrape_browser_session(job_id, opts \\ [])` issues `DELETE /scrape/{jobId}/interact` and tears down the browser session for that scrape job. A bang variant `stop_interactive_scrape_browser_session!/2` is also available. +Bang variant: `Firecrawl.stop_interactive_scrape_browser_session!/2` ```elixir {:ok, res} = Firecrawl.stop_interactive_scrape_browser_session("") ``` +### Standalone interact sessions + +The Elixir SDK also exposes standalone interact sessions (not tied to a scrape job): + +- `Firecrawl.create_browser_session(params, opts)` — `POST /interact`. Creates a new browser session. + - Params: `activity_ttl`, `block_ads`, `location`, `profile`, `stream_web_view`, `ttl` +- `Firecrawl.execute_browser_code(session_id, params, opts)` — `POST /interact/{sessionId}/execute`. Executes code in a session. + - Params: `code` (required), `language`, `timeout` +- `Firecrawl.list_browser_sessions(params, opts)` — `GET /interact`. Lists sessions. + - Params: `status` (`:active` or `:destroyed`) +- `Firecrawl.delete_browser_session(session_id, opts)` — `DELETE /interact/{sessionId}`. Deletes a session. + ## Notes - The Elixir client is OpenAPI-shaped; function names and parameter keys are generated from the spec. - Each public function has a bang (`!`) variant that raises on error instead of returning `{:error, _}`. +- Return type for all functions: `{:ok, %Req.Response{}} | {:error, Exception.t() | Firecrawl.Error.t()}`. - This SDK exposes code-based interactions only (no `prompt` parameter on `interact_with_scrape_browser_session`). +- No deprecated aliases — the SDK is auto-generated from the OpenAPI spec. +- A nil/empty API key is allowed: scrape, search, and interact fall back to the keyless free tier. ## Source Of Truth diff --git a/agent-source-of-truth/java.mdx b/agent-source-of-truth/java.mdx index d3d8cf3b2..07a765b9a 100644 --- a/agent-source-of-truth/java.mdx +++ b/agent-source-of-truth/java.mdx @@ -3,7 +3,7 @@ title: "Java Source of Truth" description: "Canonical Firecrawl Java source of truth for agents using key endpoints like search, scrape, and interact." --- -Canonical Firecrawl Java source of truth for agents. Generated from SDK source and the v2 OpenAPI spec. +Canonical Firecrawl Java source of truth for agents. Generated from SDK source (`com.firecrawl:firecrawl-java` **v1.18.4**) and the v2 OpenAPI spec. ## Install @@ -13,16 +13,18 @@ Maven: com.firecrawl firecrawl-java - 1.18.0 + 1.18.4 ``` Gradle: ```gradle -implementation("com.firecrawl:firecrawl-java:1.18.0") +implementation("com.firecrawl:firecrawl-java:1.18.4") ``` +Java 11+ required. + ## Authenticate ```java @@ -31,8 +33,22 @@ import com.firecrawl.client.FirecrawlClient; FirecrawlClient client = FirecrawlClient.builder() .apiKey(System.getenv("FIRECRAWL_API_KEY")) .build(); + +// Or from environment: +FirecrawlClient client = FirecrawlClient.fromEnv(); +// Reads FIRECRAWL_API_KEY env var, then firecrawl.apiKey system property ``` +Builder options: + +- `apiKey(String)` — falls back to `FIRECRAWL_API_KEY` env var, then `firecrawl.apiKey` system property. Null/blank key is allowed (keyless free tier for scrape/search/interact). +- `apiUrl(String)` — defaults to `"https://api.firecrawl.dev"`. Falls back to `FIRECRAWL_API_URL` env var. +- `timeoutMs(long)` — HTTP request timeout in ms (default 300000). Ignored when custom `httpClient` is provided. +- `maxRetries(int)` — max automatic retries for transient failures (default 3). +- `backoffFactor(double)` — exponential backoff factor in seconds (default 0.5). +- `asyncExecutor(Executor)` — custom executor for async operations (default `ForkJoinPool.commonPool()`). +- `httpClient(OkHttpClient)` — pre-configured OkHttpClient instance. + ## When To Use What - `search`: use when you start with a query and need discovery. @@ -52,6 +68,8 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or - `client.search(query)` - `client.search(query, options)` +Async: `client.searchAsync(query, options)` → `CompletableFuture` + ### Return value `search` returns `SearchData`. Read result buckets with `getWeb()`, `getNews()`, and `getImages()` (each is `List>` and may be null). Do not treat `SearchData` as a directly iterable list of hits. @@ -73,7 +91,6 @@ List> web = results.getWeb(); import com.firecrawl.models.SearchOptions; import com.firecrawl.models.ScrapeOptions; import com.firecrawl.models.JsonFormat; -import com.firecrawl.models.LocationConfig; SearchOptions options = SearchOptions.builder() .sources(List.of("web", "news")) @@ -81,8 +98,11 @@ SearchOptions options = SearchOptions.builder() .limit(10) .tbs("qdr:m") .location("San Francisco,California,United States") + .country("US") .ignoreInvalidURLs(true) + .highlights(true) .timeout(60000) + .includeDomains(List.of("docs.firecrawl.dev")) .scrapeOptions( ScrapeOptions.builder() .formats(List.of( @@ -119,10 +139,18 @@ SearchData results = client.search("site:docs.firecrawl.dev crawl webhooks", opt - Type: List of category strings or maps - Use when: you want to filter results by category. - Confirmed values: - - `"developer"`: Developer Index results + - `"github"`: site:github.com filter - `"research"`: research and academic results - `"pdf"`: PDF-focused results - - `{type: "developer" | "research" | "pdf"}`: typed category map form + - `{type: "github" | "research" | "pdf"}`: typed category map form + +- `options.includeDomains` + - Type: `List` + - Use when: you want to restrict results to specific domains. + +- `options.excludeDomains` + - Type: `List` + - Use when: you want to exclude specific domains from results. - `options.limit` - Type: Integer @@ -136,10 +164,18 @@ SearchData results = client.search("site:docs.firecrawl.dev crawl webhooks", opt - Type: String - Use when: you want localized results. +- `options.country` + - Type: String + - Use when: you want ISO 3166-1 alpha-2 geo-targeting for search results. + - `options.ignoreInvalidURLs` - Type: Boolean - Use when: you want to drop URLs that cannot be scraped by other endpoints. +- `options.highlights` + - Type: Boolean + - Use when: you want query-relevant highlights for search results. Defaults to true (server-side). + - `options.timeout` - Type: Integer - Use when: you need a request timeout in milliseconds. @@ -163,9 +199,11 @@ Use scrape when you already have a URL and want structured content in one or mor - `client.scrape(url)` - `client.scrape(url, options)` +Async: `client.scrapeAsync(url, options)` → `CompletableFuture` + ### Return value -`scrape` returns `Document`. Typical getters include `getMarkdown()`, `getHtml()`, `getRawHtml()`, `getJson()`, `getMetadata()`, `getLinks()`, `getAudio()`, `getVideo()`, and additional fields when the corresponding formats are requested. +`scrape` returns `Document`. Typical getters include `getMarkdown()`, `getHtml()`, `getRawHtml()`, `getJson()`, `getMetadata()`, `getLinks()`, `getAudio()`, `getVideo()`, `getProduct()`, `getMenu()`, `getPages()`, `getBlocks()`, and additional fields when the corresponding formats are requested. ### Simple Example @@ -181,6 +219,9 @@ Document doc = client.scrape( ```java import com.firecrawl.models.ScrapeOptions; import com.firecrawl.models.JsonFormat; +import com.firecrawl.models.PdfParser; +import com.firecrawl.models.LocationConfig; +import com.firecrawl.models.AuditMetadata; List> actions = List.of( Map.of("type", "click", "selector", "#accept"), @@ -202,7 +243,7 @@ ScrapeOptions options = ScrapeOptions.builder() )) .onlyMainContent(true) .waitFor(1000) - .parsers(List.of(Map.of("type", "pdf", "maxPages", 5))) + .parsers(List.of(PdfParser.builder().mode("auto").maxPages(5).build())) .actions(actions) .location(location) .removeBase64Images(true) @@ -222,7 +263,7 @@ Document doc = client.scrape("https://example.com/pricing", options); - Use when: you want to scrape a specific page. - `options.formats` - - Type: List of format strings or format maps + - Type: List of format strings or format objects - Use when: you want multiple output formats. - Confirmed format strings: - `"markdown"`: markdown content @@ -236,9 +277,15 @@ Document doc = client.scrape("https://example.com/pricing", options); - `"json"`: JSON extraction - `"attributes"`: attribute extraction - `"branding"`: branding profile output + - `"product"`: product profile output + - `"menu"`: menu profile output - `"audio"`: audio extraction - `"video"`: video extraction - - Format object fields: + - Format builder types: + - `JsonFormat.builder().prompt("...").schema(Map).build()`: JSON extraction + - `QuestionFormat.builder().question("...").build()`: question-answer output + - `HighlightsFormat.builder().query("...").build()`: relevant source-text output + - Format object fields (when using maps): - `type`: one of the format strings above - `prompt`, `schema`: JSON extraction options for `type: "json"` - `modes`, `schema`, `prompt`, `tag`: change tracking options for `type: "changeTracking"` @@ -278,7 +325,7 @@ Document doc = client.scrape("https://example.com/pricing", options); - Use when: you need file parsing controls. - Confirmed values: - `"pdf"` - - `{type: "pdf", maxPages: number}` + - `PdfParser.builder().mode("fast" | "auto" | "ocr").maxPages(n).pages(bool).blocks(bool).pageMarkers(bool).build()` - `options.actions` - Type: `List>` @@ -317,12 +364,28 @@ Document doc = client.scrape("https://example.com/pricing", options); - `options.maxAge` - Type: Long - - Use when: you want cached data up to a maximum age (milliseconds). + - Use when: you want cached data up to a maximum age (milliseconds). Set to 0 to bypass index reuse. - `options.storeInCache` - Type: Boolean - Use when: you want Firecrawl to cache the result. +- `options.lockdown` + - Type: Boolean + - Use when: you want to serve only previously cached results and never make outbound requests. + +- `options.checkPromptInjection` + - Type: Boolean + - Use when: you want to scan page content for prompt injection before LLM-backed formats run. Adds 4 credits. + +- `options.redactPII` + - Type: Boolean + - Use when: you want to redact personally identifiable information from returned content. + +- `options.auditMetadata` + - Type: `AuditMetadata` with `username` (String, required) + - Use when: you need user attribution included with SIEM logging events. + - `options.integration` - Type: String - Use when: the API expects an integration identifier on the request. @@ -339,6 +402,8 @@ Use interact when a page requires browser actions or code execution after a scra - `client.interact(jobId, code, language, timeout)` — `timeout` is seconds (1–300), or null to omit and use the API default (30 seconds) - `client.interact(jobId, code, language, timeout, origin)` — optional `origin` string is sent only when non-null (request attribution) +Async: `client.interactAsync(...)` → `CompletableFuture` + ### Simple Example ```java @@ -369,6 +434,8 @@ End the scrape-bound browser session when finished. **Preferred SDK method:** `client.stopInteractiveBrowser(jobId)` +Async: `client.stopInteractiveBrowserAsync(jobId)` → `CompletableFuture` + ```java import com.firecrawl.models.BrowserDeleteResponse; @@ -417,6 +484,8 @@ BrowserDeleteResponse stopped = client.stopInteractiveBrowser(""); - Deprecated aliases: `scrapeExecute` → `interact`, `deleteScrapeBrowser` → `stopInteractiveBrowser` (and the corresponding `*Async` helpers). - The Java SDK exposes code-based interactions only: there is no `prompt` parameter on `interact` (unlike some other language SDKs). +- `QueryFormat` is deprecated; use `QuestionFormat` or `HighlightsFormat` instead. +- No API key is required; scrape, search, and interact fall back to the keyless free tier. ## Source Of Truth @@ -426,7 +495,11 @@ BrowserDeleteResponse stopped = client.stopInteractiveBrowser(""); - `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` - `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchData.java` - `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/JsonFormat.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/QuestionFormat.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/HighlightsFormat.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/PdfParser.java` - `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/LocationConfig.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/AuditMetadata.java` - `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/Document.java` - `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/BrowserExecuteResponse.java` - `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/BrowserDeleteResponse.java` diff --git a/agent-source-of-truth/node.mdx b/agent-source-of-truth/node.mdx index 0ecb22279..eb8f27454 100644 --- a/agent-source-of-truth/node.mdx +++ b/agent-source-of-truth/node.mdx @@ -3,7 +3,7 @@ title: "Node.js Source of Truth" description: "Canonical Firecrawl Node.js source of truth for agents using key endpoints like search, scrape, and interact." --- -Canonical Firecrawl Node.js source of truth for agents. Aligned with `firecrawl` **v4.18.2** (`firecrawl/apps/js-sdk/firecrawl`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. +Canonical Firecrawl Node.js source of truth for agents. Aligned with `@mendable/firecrawl-js` **v4.46.0** (`firecrawl/apps/js-sdk/firecrawl`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. ## Install @@ -22,6 +22,14 @@ const client = new Firecrawl({ }); ``` +Constructor accepts a string (API key) or a `FirecrawlClientOptions` object: + +- `apiKey`: string — falls back to `FIRECRAWL_API_KEY` env var. Omit for keyless free tier (rate-limited per IP). +- `apiUrl`: string — falls back to `FIRECRAWL_API_URL` env var, then `"https://api.firecrawl.dev"`. +- `timeoutMs`: number — per-request timeout in milliseconds. +- `maxRetries`: number — max automatic retries for transient failures. +- `backoffFactor`: number — exponential backoff factor for retries. + ## When To Use What - `search`: use when you start with a query and need discovery. @@ -55,8 +63,11 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { limit: 10, tbs: "qdr:m", location: "San Francisco,California,United States", + country: "US", ignoreInvalidURLs: true, + highlights: true, timeout: 60000, + includeDomains: ["docs.firecrawl.dev"], scrapeOptions: { formats: [ "markdown", @@ -78,8 +89,11 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { - `web`: web index hits - `news`: news hits - `images`: image hits +- `tools`: discovered tools (when `domainTools` or `sources` includes `"alexandria"`) +- `warning`: optional warning string +- `agent_hints`: optional hints array -**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Do not access `result.data`. Web results are in `result.web`, news in `result.news`, images in `result.images`. +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Accessing `.data` throws an error. Web results are in `result.web`, news in `result.news`, images in `result.images`. ### Parameters @@ -98,7 +112,8 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { - `"web"`: web index results - `"news"`: news results - `"images"`: image results - - `{ type: "web" | "news" | "images" }`: typed source object form + - `"alexandria"`: Alexandria source results + - `{ type: "web" | "news" | "images" | "alexandria" }`: typed source object form - `options.categories` - Type: array of category names or typed category objects @@ -106,12 +121,21 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { - Confirmed values: - `"developer"`: Developer Index results - `"gov"`: Firecrawl Government Index results - - `"research"`: research and academic results - - `"pdf"`: PDF-focused results - - `{ type: "developer" | "gov" | "research" | "pdf" }`: typed category object form + - `"research"`: research and academic results (~14 academic website domains, NOT the paper index) + - `"pdf"`: PDF-focused results (filetype:pdf filter) + - `"github"`: site:github.com filter + - `{ type: "developer" | "gov" | "research" | "pdf" | "github" }`: typed category object form + +- `options.includeDomains` + - Type: array of strings + - Use when: you want to restrict results to specific domains. Cannot be combined with `excludeDomains`. + +- `options.excludeDomains` + - Type: array of strings + - Use when: you want to exclude specific domains from results. Cannot be combined with `includeDomains`. - `options.limit` - - Type: number + - Type: number (must be positive) - Use when: you want to cap results. - `options.tbs` @@ -122,17 +146,51 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { - Type: string - Use when: you want localized results. +- `options.country` + - Type: string + - Use when: you want ISO 3166-1 alpha-2 geo-targeting for search results. + - `options.ignoreInvalidURLs` - Type: boolean - Use when: you want to drop URLs that cannot be scraped by other endpoints. +- `options.highlights` + - Type: boolean + - Use when: you want query-relevant highlights for search results. + - Notes: defaults to `true`. + - `options.timeout` - - Type: number + - Type: number (must be positive) - Use when: you need a request timeout in milliseconds. - `options.scrapeOptions` - - Type: `ScrapeOptions` - - Use when: you want to scrape each search result (see Scrape parameters for fields). The SDK runs the same validation as for `scrape` (for example plain string `"json"` in `formats` is rejected). + - Type: `ScrapeOptions` (minus `toolDetail`) + - Use when: you want to scrape each search result (see Scrape parameters for fields). + +- `options.domainTools` + - Type: boolean + - Use when: you want domain-matched Alexandria tools alongside semantic matches. + +- `options.toolDetail` + - Type: `"compact" | "summary" | "full"` + - Use when: you want to control tool discovery detail level. Compact by default; summary adds metadata, full includes contracts. + +- `options.enterprise` + - Type: array of `"default" | "anon" | "zdr"` + - Use when: you need enterprise search options. `"zdr"` = zero data retention, `"anon"` = anonymized search. Must be enabled for your team. + +- `options.threatProtection` + - Type: `ThreatProtectionOptions` + - Use when: you need enterprise per-request threat protection override. + - Fields: `mode` (`"off" | "manual-only" | "normal" | "zscaler"`), `riskScoreThreshold` (0-100), `blacklist`, `whitelist`, `blockedTlds`, `failurePolicy` (`"open" | "closed"`). + +- `options.integration` + - Type: string + - Use when: you need an integration identifier for server-side tracking. + +- `options.origin` + - Type: string + - Use when: you need an origin identifier for request attribution. ## Scrape @@ -181,7 +239,6 @@ const doc = await client.scrape("https://example.com/pricing", { blockAds: true, proxy: "auto", maxAge: 86400000, - minAge: 1, storeInCache: true, profile: { name: "docs-session", saveChanges: true } }); @@ -207,6 +264,8 @@ const doc = await client.scrape("https://example.com/pricing", { - `"changeTracking"`: change tracking output (for options like `modes`, use `{ type: "changeTracking", modes: [...] }` — `modes` is required on that object in typings) - `"attributes"`: attribute extraction (use `{ type: "attributes", selectors: [...] }` when passing selectors) - `"branding"`: branding profile output + - `"product"`: product profile output + - `"menu"`: menu profile output - `"audio"`: audio extraction - `"video"`: video extraction - Object-only format types (at minimum `type` as shown): @@ -255,7 +314,8 @@ const doc = await client.scrape("https://example.com/pricing", { - Use when: you need file parsing controls (for example PDF parsing). - Confirmed values: - `"pdf"`: enable PDF parsing - - `{ type: "pdf", mode?: "fast" | "auto" | "ocr", maxPages?: number }`: PDF parser options + - `{ type: "pdf", mode?: "fast" | "auto" | "ocr", maxPages?: number, pages?: boolean, pageMarkers?: boolean, blocks?: boolean }`: PDF parser options. `pages` returns per-page markdown; `blocks` returns per-page typed layout blocks. + - `{ type: "image" }`: image parser - `options.actions` - Type: array of action objects @@ -303,7 +363,7 @@ const doc = await client.scrape("https://example.com/pricing", { - `options.maxAge` - Type: number - - Use when: you want cached data up to a maximum age (milliseconds). + - Use when: you want cached data up to a maximum age (milliseconds). Set to 0 to bypass index reuse. - `options.minAge` - Type: number @@ -313,10 +373,48 @@ const doc = await client.scrape("https://example.com/pricing", { - Type: boolean - Use when: you want Firecrawl to cache the result. +- `options.lockdown` + - Type: boolean + - Use when: you want to serve only previously cached results and never make outbound requests. Returns 404 on cache miss. + +- `options.checkPromptInjection` + - Type: boolean + - Use when: you want to scan page content for prompt injection before LLM-backed formats run. Adds 4 credits. + +- `options.redactPII` + - Type: boolean or `RedactPIIOptions` + - Use when: you want to redact personally identifiable information from output. + - `RedactPIIOptions` fields: `mode` (`"accurate" | "aggressive" | "fast"`), `entities` (array of `"PERSON" | "EMAIL" | "PHONE" | "LOCATION" | "FINANCIAL" | "SECRET"`), `replaceStyle` (`"tag" | "mask" | "remove"`). + +- `options.threatProtection` + - Type: `ThreatProtectionOptions` + - Use when: you need enterprise per-request threat protection override. + +- `options.auditMetadata` + - Type: `AuditMetadata` with `username: string` + - Use when: you need metadata included in SIEM logging events. + - `options.profile` - Type: object with `name` and optional `saveChanges` - Use when: you want a persistent browser profile shared across scrapes and interactions. +- `options.domainTools` + - Type: boolean + - Use when: you want domain-matched Alexandria tools for this URL included in `tools`. + +- `options.toolDetail` + - Type: `"compact" | "summary" | "full"` + - Use when: you want to control tool discovery detail level. + +- `options.autoResume` + - Type: boolean (SDK-only, never sent to API) + - Use when: you want to control automatic resumption for large documents that outlive the request window. Bounded to at most 5 resumes / 20 minutes total wait. Set `false` to surface timeout error immediately. + - Notes: defaults to `true` implicitly. + +- `options.integration` + - Type: string + - Use when: you need an integration identifier for server-side tracking. + ## Interact ### Why use it @@ -375,6 +473,10 @@ const result = await client.interact("", { - Type: number - Use when: you need an execution timeout in seconds. +- `args.origin` + - Type: string + - Use when: you need an origin label for execution telemetry. + ### Return value (`interact`) `ScrapeExecuteResponse` matches `BrowserExecuteResponse`. Confirmed fields include: @@ -386,6 +488,8 @@ const result = await client.interact("", { - `stderr`: string - `exitCode`: number - `killed`: boolean +- `truncated`: boolean +- `cdpUrl`: string - `liveViewUrl`: string - `interactiveLiveViewUrl`: string - `error`: string @@ -395,7 +499,7 @@ const result = await client.interact("", { `client.stopInteraction(jobId)` → `Promise` - `jobId`: scrape job id (same id used for `interact`). -- Response fields in typings include `success`, `sessionDurationMs`, `creditsBilled`, `error`. +- Response fields in typings include `success`, `status`, `sessionDurationMs`, `creditsBilled`, `error`. ## Ask (Agentic Debugging) @@ -527,10 +631,11 @@ console.log(result.answer); ## Notes -- Deprecated client aliases: `scrapeExecute` → `interact`; `stopInteractiveBrowser` and `deleteScrapeBrowser` → `stopInteraction`. +- Deprecated client aliases: `scrapeUrl` → `scrape`; `scrapeExecute` → `interact`; `stopInteractiveBrowser` and `deleteScrapeBrowser` → `stopInteraction`. - The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`. - Zod schemas passed to `formats` (for `json` or `changeTracking`) are converted to JSON Schema by the SDK. - The package declares **Node.js >= 22** in `engines`. +- No API key is required; scrape, search, and interact fall back to the keyless free tier (rate-limited per IP). ## Source Of Truth @@ -538,7 +643,6 @@ console.log(result.answer); - `firecrawl/apps/js-sdk/firecrawl/src/index.ts` - `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` - `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` -- `firecrawl/apps/js-sdk/firecrawl/src/v2/utils/validation.ts` - `firecrawl/apps/js-sdk/firecrawl/src/v2/methods/search.ts` - `firecrawl/apps/js-sdk/firecrawl/src/v2/methods/scrape.ts` - `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-source-of-truth/python.mdx b/agent-source-of-truth/python.mdx index 75d0827f9..14fcc8bcf 100644 --- a/agent-source-of-truth/python.mdx +++ b/agent-source-of-truth/python.mdx @@ -3,7 +3,7 @@ title: "Python Source of Truth" description: "Canonical Firecrawl Python source of truth for agents using key endpoints like search, scrape, and interact." --- -Canonical Firecrawl Python source of truth for agents. Generated from SDK source (`firecrawl-py` / `firecrawl` **4.22.1**) and the v2 OpenAPI spec. Method names, parameters, and return types match the v2 client in `firecrawl/v2/client.py` unless noted. +Canonical Firecrawl Python source of truth for agents. Generated from SDK source (`firecrawl-py` **4.50.0**) and the v2 OpenAPI spec. Method names, parameters, and return types match the v2 client in `firecrawl/v2/client.py` unless noted. ## Install @@ -21,6 +21,17 @@ client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) # client = Firecrawl(api_key="fc-...", api_url="https://api.firecrawl.dev") ``` +Constructor parameters: + +- `api_key`: str — falls back to `FIRECRAWL_API_KEY` env var. Omit for keyless free tier (rate-limited per IP). +- `api_url`: str — defaults to `"https://api.firecrawl.dev"`. +- `timeout`: float — default request timeout in seconds for all HTTP requests. +- `max_retries`: int — maximum retries for failed requests (default 3). +- `backoff_factor`: float — exponential backoff factor (default 0.5). +- `origin`: str — attribution string stamped into API request payloads (defaults to `python-sdk@`). + +Async variant: `from firecrawl import AsyncFirecrawl`. + ## When To Use What - `search`: use when you start with a query and need discovery. @@ -46,6 +57,9 @@ Returns a `SearchData` model with optional lists: - `web` — web hits (`SearchResultWeb` or full `Document` when `scrape_options` hydrates content) - `news` — news hits (`SearchResultNews` or `Document`) - `images` — image hits (`SearchResultImages` or `Document`) +- `tools` — discovered tools (when `domain_tools` or `sources` includes `"alexandria"`) +- `warning` — optional warning string +- `agent_hints` — optional hints list Omitted buckets are `None` when the API did not return that key. @@ -71,8 +85,11 @@ results = client.search( limit=10, tbs="qdr:m", location="San Francisco,California,United States", + country="US", ignore_invalid_urls=True, + highlights=True, timeout=300000, + include_domains=["docs.firecrawl.dev"], scrape_options=ScrapeOptions( formats=[ "markdown", @@ -83,11 +100,6 @@ results = client.search( include_tags=["main", "article"], exclude_tags=["nav", "footer"], wait_for=1000, - actions=[ - {"type": "click", "selector": "button#accept"}, - {"type": "wait", "milliseconds": 750}, - {"type": "scrape"}, - ], ), ) ``` @@ -106,7 +118,16 @@ results = client.search( - `"web"`: web index results - `"news"`: news results - `"images"`: image results - - `Source(type="web" | "news" | "images")`: typed source object form + - `"alexandria"`: Alexandria source results + - `Source(type="web" | "news" | "images" | "alexandria")`: typed source object form + +- `domain_tools` + - Type: bool + - Use when: you want domain-matched Alexandria tools alongside semantic matches. + +- `tool_detail` + - Type: `"compact" | "summary" | "full"` + - Use when: you want to control tool discovery detail level. Compact by default; summary adds metadata, full includes contracts. - `categories` - Type: list of category names or `Category` objects @@ -116,7 +137,16 @@ results = client.search( - `"gov"`: Firecrawl Government Index results - `"research"`: research and academic results - `"pdf"`: PDF-focused results - - `Category(type="developer" | "gov" | "research" | "pdf")`: typed category object form + - `"github"`: site:github.com filter + - `Category(type="developer" | "gov" | "research" | "pdf" | "github")`: typed category object form + +- `include_domains` + - Type: list of str + - Use when: you want to restrict results to specific domains. Cannot be combined with `exclude_domains`. + +- `exclude_domains` + - Type: list of str + - Use when: you want to exclude specific domains from results. Cannot be combined with `include_domains`. - `limit` - Type: int @@ -131,10 +161,19 @@ results = client.search( - Type: str - Use when: you want localized results. +- `country` + - Type: str + - Use when: you want ISO 3166-1 alpha-2 geo-targeting for search results. + - `ignore_invalid_urls` - Type: bool - Use when: you want to drop URLs that cannot be scraped by other endpoints. +- `highlights` + - Type: bool + - Use when: you want query-relevant highlights for search results. + - Notes: defaults to `true`. + - `timeout` - Type: int - Use when: you need a request timeout in milliseconds. @@ -144,6 +183,19 @@ results = client.search( - Type: `ScrapeOptions` - Use when: you want to scrape each search result (see Scrape parameters for fields). +- `enterprise` + - Type: list of str + - Use when: you need enterprise search options. `["zdr"]` for zero data retention, `["anon"]` for anonymized search. Must be enabled for your team. + +- `threat_protection` + - Type: `ThreatProtectionOptions` + - Use when: you need enterprise per-request threat protection override. + - Fields: `mode` (`"off" | "manual-only" | "normal" | "zscaler"`), `risk_score_threshold` (0-100), `blacklist`, `whitelist`, `blocked_tlds`, `failure_policy` (`"open" | "closed"`). + +- `integration` + - Type: str + - Use when: you need an integration identifier for server-side tracking. + ## Scrape ### Why use it @@ -213,6 +265,8 @@ doc = client.scrape( - `"changeTracking"` or `"change_tracking"`: change tracking output - `"attributes"`: attribute extraction - `"branding"`: branding profile output + - `"product"`: product profile output + - `"menu"`: menu profile output - `"audio"`: audio extraction - `"video"`: video extraction - Object-only format types: @@ -263,7 +317,8 @@ doc = client.scrape( - Use when: you need file parsing controls (for example PDF parsing). - Confirmed values: - `"pdf"`: enable PDF parsing - - `{ "type": "pdf", "mode": "fast" | "auto" | "ocr", "max_pages": int }`: PDF parser options + - `{"type": "pdf", "mode": "fast" | "auto" | "ocr", "max_pages": int, "pages": bool, "blocks": bool, "page_markers": bool}`: PDF parser options. `pages` returns per-page markdown; `blocks` returns per-page typed layout blocks. + - `{"type": "image"}`: image parser - `actions` - Type: list of action objects @@ -306,21 +361,52 @@ doc = client.scrape( - `max_age` - Type: int - - Use when: you want cached data up to a maximum age (milliseconds). - -- `min_age` - - Type: int - - Use when: you want cached data only if it is at least this old (milliseconds). - - Notes: only on `ScrapeOptions` passed as `scrape_options` to `client.search`. The `client.scrape` method does not accept `min_age`. + - Use when: you want cached data up to a maximum age (milliseconds). Set to 0 to bypass index reuse. - `store_in_cache` - Type: bool - Use when: you want Firecrawl to cache the result. +- `lockdown` + - Type: bool + - Use when: you want to serve only previously cached results and never make outbound requests. Returns 404 on cache miss. + +- `check_prompt_injection` + - Type: bool + - Use when: you want to scan page content for prompt injection before LLM-backed formats run. Adds 4 credits. + +- `redact_pii` + - Type: bool or `RedactPIIOptions` + - Use when: you want to redact personally identifiable information from output. + +- `threat_protection` + - Type: `ThreatProtectionOptions` + - Use when: you need enterprise per-request threat protection override. + +- `audit_metadata` + - Type: `AuditMetadata` with `username: str` (max 1024 chars) + - Use when: you need metadata included in SIEM logging events. + - `profile` - Type: dict with `name` and optional `save_changes` or `saveChanges` - Use when: you want a persistent browser profile shared across scrapes and interactions. +- `domain_tools` + - Type: bool + - Use when: you want domain-matched Alexandria tools for this URL included in `tools`. + +- `tool_detail` + - Type: `"compact" | "summary" | "full"` + - Use when: you want to control tool discovery detail level. + +- `auto_resume` + - Type: bool (SDK-only, never sent to API) + - Use when: you want to control automatic resumption for large documents. + +- `integration` + - Type: str + - Use when: you need an integration identifier for server-side tracking. + ## Interact ### Why use it @@ -381,11 +467,15 @@ result = client.interact( - `timeout` - Type: int - - Use when: you need an execution timeout in seconds. + - Use when: you need an execution timeout in seconds (1-300). + +- `origin` + - Type: str + - Use when: you need an origin label for execution telemetry. ### Return value -Returns `BrowserExecuteResponse`: `success`, optional `live_view_url`, `interactive_live_view_url`, `output`, `stdout`, `result`, `stderr`, `exit_code`, `killed`, `error` (API camelCase is normalized to snake_case on the model). +Returns `BrowserExecuteResponse`: `success`, optional `cdp_url`, `live_view_url`, `interactive_live_view_url`, `output`, `stdout`, `result`, `stderr`, `exit_code`, `killed`, `truncated`, `error` (API camelCase is normalized to snake_case on the model). ## Ask (Agentic Debugging) @@ -508,9 +598,10 @@ print(response.json()["answer"]) ## Notes -- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. +- Deprecated aliases: `scrape_url` → `scrape`; `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. - The top-level `Firecrawl` client exposes v2 methods directly; v1 remains under `client.v1`. -- The bundled v2 OpenAPI snippet for `POST /v2/scrape/{jobId}/interact` may only document `code`; the Python SDK and server accept either `code` or `prompt` for this endpoint. +- Class aliases: `FirecrawlApp = Firecrawl`, `AsyncFirecrawlApp = AsyncFirecrawl` (backward-compatible). +- No API key is required; scrape, search, and interact fall back to the keyless free tier (rate-limited per IP). ## Source Of Truth @@ -521,5 +612,4 @@ print(response.json()["answer"]) - `firecrawl/apps/python-sdk/firecrawl/v2/types.py` - `firecrawl/apps/python-sdk/firecrawl/v2/methods/search.py` - `firecrawl/apps/python-sdk/firecrawl/v2/methods/scrape.py` -- `firecrawl/apps/python-sdk/firecrawl/v2/utils/validation.py` - `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-source-of-truth/rust.mdx b/agent-source-of-truth/rust.mdx index a862bfa16..738d31590 100644 --- a/agent-source-of-truth/rust.mdx +++ b/agent-source-of-truth/rust.mdx @@ -3,7 +3,7 @@ title: "Rust Source of Truth" description: "Canonical Firecrawl Rust source of truth for agents using key endpoints like search, scrape, and interact." --- -Canonical Firecrawl Rust source of truth for agents. Generated from SDK source and the v2 OpenAPI spec. +Canonical Firecrawl Rust source of truth for agents. Generated from SDK source (`firecrawl` crate **v2.21.5**) and the v2 OpenAPI spec. ## Install @@ -11,7 +11,7 @@ Canonical Firecrawl Rust source of truth for agents. Generated from SDK source a cargo add firecrawl ``` -Crate: **`firecrawl`** on crates.io. The current SDK version is **2.0.0** (verify the latest release on crates.io before pinning). +Crate: **`firecrawl`** on crates.io. ## Authenticate @@ -21,8 +21,13 @@ use firecrawl::Client; let client = Client::new("fc-your-api-key")?; // Self-hosted example: // let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +// No auth (keyless free tier): +// let client = Client::new_selfhosted("http://localhost:3002", None::<&str>)?; ``` +- `Client::new(api_key)` — cloud client at `https://api.firecrawl.dev`. Returns error if API key is empty. +- `Client::new_selfhosted(api_url, api_key)` — self-hosted client. `api_key` is optional; omit for keyless free tier (rate-limited per IP). + ## When To Use What - `search`: use when you start with a query and need discovery. @@ -56,6 +61,7 @@ let results = client ```rust use firecrawl::{ Client, SearchOptions, SearchSource, SearchCategory, ScrapeOptions, Format, JsonOptions, + ToolDetail, }; let options = SearchOptions { @@ -64,8 +70,11 @@ let options = SearchOptions { limit: Some(10), tbs: Some("qdr:m".to_string()), location: Some("San Francisco,California,United States".to_string()), + country: Some("US".to_string()), ignore_invalid_urls: Some(true), + highlights: Some(true), timeout: Some(60000), + include_domains: Some(vec!["docs.firecrawl.dev".to_string()]), scrape_options: Some(ScrapeOptions { formats: Some(vec![Format::Markdown, Format::Links, Format::Json]), json_options: Some(JsonOptions { @@ -92,26 +101,44 @@ let results = client - `data: SearchData` - `warning: Option` - `SearchData` - - `web: Option>` — each item is either `SearchResultOrDocument::WebResult(SearchResultWeb)` (url, title, description, …) or `SearchResultOrDocument::Document(Document)` when the API returned scraped content for that row (detected when markdown, html, rawHtml, or metadata is present). + - `web: Option>` — each item is either `SearchResultOrDocument::WebResult(SearchResultWeb)` (url, title, description, …) or `SearchResultOrDocument::Document(Document)` when the API returned scraped content for that row. - `news: Option>` - `images: Option>` + - `tools: Option>` ### Parameters - `query` - Type: `impl AsRef` - - Use when: you need a search query. + - Use when: you need a search query. Cannot be empty (returns `FirecrawlError::Misuse`). - Notes: use `site:example.com` to limit results to a domain. - `options.sources` - Type: `Vec` - Use when: you want to control which sources are searched. - - Confirmed values: `Web`, `News`, `Images` + - Confirmed values: `Web`, `News`, `Images`, `Alexandria` + +- `options.domain_tools` + - Type: bool + - Use when: you want domain-matched Alexandria tools alongside semantic matches. + +- `options.tool_detail` + - Type: `ToolDetail` + - Use when: you want to control tool discovery detail level. + - Confirmed values: `Compact`, `Summary`, `Full` - `options.categories` - Type: `Vec` - Use when: you want to filter results by category. - - Confirmed values: `Research`, `Pdf` + - Confirmed values: `Github`, `Research`, `Pdf` + +- `options.include_domains` + - Type: `Vec` + - Use when: you want to restrict results to specific domains. + +- `options.exclude_domains` + - Type: `Vec` + - Use when: you want to exclude specific domains from results. - `options.limit` - Type: u32 @@ -125,10 +152,18 @@ let results = client - Type: String - Use when: you want localized results. +- `options.country` + - Type: String + - Use when: you want ISO 3166-1 alpha-2 geo-targeting for search results. + - `options.ignore_invalid_urls` - Type: bool - Use when: you want to drop URLs that cannot be scraped by other endpoints. +- `options.highlights` + - Type: bool + - Use when: you want query-relevant highlights for search results. Defaults to true. + - `options.timeout` - Type: u32 - Use when: you need a request timeout in milliseconds. @@ -140,7 +175,6 @@ let results = client - `options.integration` - Type: `Option` - Use when: you need an integration identifier for server-side tracking. - - Notes: omit in agent-oriented examples unless your product intentionally sets it. ## Scrape @@ -170,7 +204,7 @@ let doc = client ```rust use firecrawl::{ Client, ScrapeOptions, Format, JsonOptions, ScreenshotOptions, ChangeTrackingOptions, - ChangeTrackingMode, AttributeSelector, Action, ParserConfig, ProxyType, + ChangeTrackingMode, AttributeSelector, Action, ParserConfig, ProxyType, ProfileConfig, }; let doc = client @@ -203,7 +237,11 @@ let doc = client }]), parsers: Some(vec![ParserConfig::Pdf { parser_type: "pdf".to_string(), + mode: Some("auto".to_string()), max_pages: Some(5), + pages: None, + blocks: None, + page_markers: None, }]), actions: Some(vec![ Action::Click { selector: "#accept".to_string() }, @@ -211,6 +249,10 @@ let doc = client Action::Scrape, ]), proxy: Some(ProxyType::Auto), + profile: Some(ProfileConfig { + name: "docs-session".to_string(), + save_changes: Some(true), + }), ..Default::default() }) .await?; @@ -225,7 +267,10 @@ let doc = client - `options.formats` - Type: `Vec` - Use when: you want multiple output formats. - - Confirmed values: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Audio`, `Video` + - String variants: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video` + - Object variants: + - `Format::Question(QuestionFormat { question })`: question-answer extraction + - `Format::Highlights(HighlightsFormat { query })`: relevant source-text extraction - `options.headers` - Type: `HashMap` @@ -260,7 +305,7 @@ let doc = client - Use when: you need file parsing controls. - Confirmed values: - `ParserConfig::Simple("pdf".to_string())` - - `ParserConfig::Pdf { parser_type: "pdf", max_pages: Some(n) }` + - `ParserConfig::Pdf { parser_type: "pdf", mode, max_pages, pages, blocks, page_markers }`: `pages` returns per-page markdown; `blocks` returns per-page typed layout blocks. - `options.actions` - Type: `Vec` @@ -274,7 +319,7 @@ let doc = client - `Scroll`: `direction` (`Up` or `Down`) required, `selector` optional - `Scrape`: no additional fields - `ExecuteJavascript`: `script` required - - `Pdf`: `format` (A0, A1, A2, A3, A4, A5, A6, Letter, Legal, Tabloid, Ledger), `landscape`, `scale` optional + - `Pdf`: `format`, `landscape`, `scale` optional - `options.location` - Type: `LocationConfig` @@ -304,7 +349,7 @@ let doc = client - `options.max_age` - Type: u32 - - Use when: you want cached data up to a maximum age (milliseconds). + - Use when: you want cached data up to a maximum age (milliseconds). Set to 0 to bypass index reuse. - `options.min_age` - Type: u32 @@ -314,15 +359,38 @@ let doc = client - Type: bool - Use when: you want Firecrawl to cache the result. +- `options.lockdown` + - Type: bool + - Use when: you want to serve only previously cached results and never make outbound requests. + +- `options.check_prompt_injection` + - Type: bool + - Use when: you want to scan page content for prompt injection before LLM-backed formats run. Adds 4 credits. + +- `options.redact_pii` + - Type: bool + - Use when: you want to redact personally identifiable information from returned content. + +- `options.audit_metadata` + - Type: `AuditMetadata` + - Use when: you need user attribution included with SIEM logging events. + - `options.profile` - Type: `ProfileConfig` - Use when: you want a persistent browser profile shared across scrapes and interactions. - Confirmed fields: `name`, `save_changes` +- `options.domain_tools` + - Type: bool + - Use when: you want domain-matched Alexandria tools for this URL included in `tools`. + +- `options.tool_detail` + - Type: `ToolDetail` + - Use when: you want to control tool discovery detail level (`Compact`, `Summary`, `Full`). + - `options.integration` - Type: `Option` - Use when: you need an integration identifier for server-side tracking. - - Notes: omit in agent-oriented examples unless your product intentionally sets it. - `options.json_options` - Type: `JsonOptions` @@ -423,8 +491,7 @@ let stopped = client.stop_interaction("").await?; - `options.origin` - Type: `Option` - - Use when: you need an optional origin label for execution telemetry. - - Notes: omit in agent-oriented examples unless your product intentionally sets it. + - Use when: you need an origin label for execution telemetry. Auto-set to SDK version if omitted. At least one of `options.code` or `options.prompt` must be non-empty; otherwise the SDK returns `FirecrawlError::Misuse` before calling the API. @@ -435,15 +502,13 @@ At least one of `options.code` or `options.prompt` must be non-empty; otherwise - `stop_interaction` → `Result` - `success: bool` is always present; `session_duration_ms`, `credits_billed`, and `error` are optional. -The v2 OpenAPI spec currently models the interact request body with `code` as required; the Rust SDK matches server behavior by accepting **either** `code` **or** `prompt` (see `ScrapeExecuteOptions` in `scrape.rs`). - ## Notes - Deprecated aliases: `scrape_execute`, `stop_interactive_browser`, and `delete_scrape_browser` map to `interact` and `stop_interaction`. - `ScrapeOptions` includes dedicated `json_options`, `screenshot_options`, and `change_tracking_options` for advanced formats. -- `search_and_scrape(query, limit)` is a convenience helper: it calls `search` with default `ScrapeOptions` and returns `Vec` built from `SearchResultOrDocument::Document` entries in `data.web` (see `search.rs`). +- `search_and_scrape(query, limit)` is a convenience helper: it calls `search` with default `ScrapeOptions` and returns `Vec` built from `SearchResultOrDocument::Document` entries in `data.web`. - v2 SDK exports all types at the crate root: `use firecrawl::Client` (not `use firecrawl::v2::Client`). -- Error types simplified in v2: `CrawlJobFailed(String, CrawlStatus)` → `JobFailed(String)`, `Missuse` → `Misuse`. +- All source files live directly under `src/` (no `v2/` subdirectory). ## Source Of Truth @@ -452,9 +517,6 @@ The v2 OpenAPI spec currently models the interact request body with `code` as re - `firecrawl/apps/rust-sdk/src/client.rs` - `firecrawl/apps/rust-sdk/src/scrape.rs` - `firecrawl/apps/rust-sdk/src/search.rs` -- `firecrawl/apps/rust-sdk/src/crawl.rs` -- `firecrawl/apps/rust-sdk/src/map.rs` -- `firecrawl/apps/rust-sdk/src/batch_scrape.rs` -- `firecrawl/apps/rust-sdk/src/agent.rs` - `firecrawl/apps/rust-sdk/src/types.rs` +- `firecrawl/apps/rust-sdk/src/tools.rs` - `firecrawl-docs/api-reference/v2-openapi.json`