diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..dd924c893 --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,261 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl via the official Elixir SDK. It is generated from SDK source and the OpenAPI spec. The Elixir SDK is auto-generated from the OpenAPI spec and uses function names that mirror operation IDs. + +## Install + +Add to your `mix.exs` dependencies: + +```elixir +defp deps do + [ + {:firecrawl, "~> 1.12"} + ] +end +``` + +## Authenticate + +Configure in `config/config.exs`: + +```elixir +config :firecrawl, api_key: "fc-YOUR-API-KEY" +``` + +Or pass `api_key` as an option to any function call: + +```elixir +Firecrawl.search_and_scrape([query: "example"], api_key: "fc-YOUR-API-KEY") +``` + +## When To Use What + +- **`search_and_scrape`** — Use when you start with a query and need to discover relevant pages. Returns web results, news, images, and optionally scraped page content. +- **`scrape_and_extract_from_url`** — Use when you already have a URL and want structured page content (markdown, HTML, JSON extraction, screenshots, etc.). +- **`interact_with_scrape_browser_session`** / **`execute_browser_code`** — Use when a page needs browser automation — clicks, form fills, or post-scrape dynamic content extraction. + +## Search + +### Why use it + +Search the web for a query and get results with titles, URLs, descriptions, and optionally full page content. Supports filtering by domain, time range, category, and source type. + +### Preferred SDK function + +``` +Firecrawl.search_and_scrape(params, opts \\ []) +``` + +### Example + +```elixir +{:ok, response} = Firecrawl.search_and_scrape( + query: "firecrawl web scraping", + limit: 5, + scrape_options: [formats: ["markdown"]] +) + +for result <- response.body["data"]["web"] do + IO.puts("#{result["title"]} #{result["url"]}") +end +``` + +### Parameters + +All parameters are passed as a keyword list. Keys use snake_case and are automatically mapped to camelCase for the API. + +| Parameter | Type | Description | +|---|---|---| +| `query` | `:string` | The search query (max 500 chars). **Required.** | +| `limit` | `:integer` | Max results to return per source type. Default: `10` (API). Range: 1–100. | +| `sources` | `list` | Source types: `"web"`, `"news"`, `"images"`, `"alexandria"`. Default: `["web"]`. | +| `categories` | `list` | Filter by category: `"github"`, `"research"`, `"pdf"`, `"developer"`, `"gov"`. | +| `include_domains` | `list(string)` | Restrict results to these domains. Cannot combine with `exclude_domains`. | +| `exclude_domains` | `list(string)` | Exclude results from these domains. | +| `tbs` | `:string` | Time-based filter. E.g. `"qdr:d"` for past day, `"sbd:1,qdr:w"` for sorted by date in past week. | +| `location` | `:string` | Location for search results. | +| `country` | `:string` | ISO 3166-1 alpha-2 country code. | +| `highlights` | `:boolean` | Return query-relevant highlights. | +| `timeout` | `:integer` | Timeout in milliseconds. | +| `ignore_invalid_urls` | `:boolean` | Skip invalid URLs instead of failing. | +| `safe` | `:boolean` | Enable safe search. | +| `scrape_options` | `:keyword_list` | Apply scrape options to each result page. | +| `domain_tools` | `:boolean` | Include Alexandria tool contracts matched to result domains. | +| `tool_detail` | `:compact \| :summary \| :full` | Detail level for tool contracts. Default: `"compact"`. | +| `enterprise` | `list(string)` | Enterprise options: `"default"`, `"anon"`, `"zdr"`. | + +### Bang variant + +`Firecrawl.search_and_scrape!(params, opts)` — raises on error instead of returning `{:error, ...}`. + +## Scrape + +### Why use it + +Scrape a URL and get its content in your chosen format — markdown, HTML, JSON (with schema-based extraction), screenshots, and more. Supports browser actions, caching, proxies, and PDF parsing. + +### Preferred SDK function + +``` +Firecrawl.scrape_and_extract_from_url(params, opts \\ []) +``` + +### Example + +```elixir +{:ok, response} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown", "links"], + only_main_content: true +) + +IO.puts(response.body["data"]["markdown"]) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `:string` | The URL to scrape. **Required.** | +| `formats` | `list` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"json"`, `"summary"`, `"audio"`, `"video"`, `"changeTracking"`, `"branding"`, `"product"`, `"menu"`. Default: `["markdown"]`. Some accept config maps. | +| `only_main_content` | `:boolean` | Strip headers, navs, footers. Default: `true` (API). | +| `only_clean_content` | `:boolean` | LLM-based boilerplate removal. Default: `false`. | +| `include_tags` | `list(string)` | Only include content from these HTML tags. | +| `exclude_tags` | `list(string)` | Exclude content from these HTML tags. | +| `headers` | `any` | Custom HTTP headers. | +| `wait_for` | `:integer` | Extra delay in ms before fetching content. Default: `0`. | +| `timeout` | `:integer` | Request timeout in ms. Default: 60000, max: 300000. | +| `mobile` | `:boolean` | Emulate a mobile device. Default: `false`. | +| `max_age` | `:integer` | Use cached result if younger than this (ms). Default: 172800000 (2 days). | +| `min_age` | `:integer` | Cache-only mode. | +| `store_in_cache` | `:boolean` | Cache this scrape result. | +| `actions` | `list` | Browser actions before scraping. | +| `location` | `:keyword_list` | Geolocation config. | +| `parsers` | `list` | File parsers. Default: `["pdf"]`. | +| `proxy` | `:basic \| :enhanced \| :auto \| string` | Proxy type. | +| `skip_tls_verification` | `:boolean` | Skip TLS verification. Default: `true` (API). | +| `remove_base64_images` | `:boolean` | Strip base64 images from output. | +| `block_ads` | `:boolean` | Block ads and cookie popups. | +| `lockdown` | `:boolean` | Serve only cached results. | +| `check_prompt_injection` | `:boolean` | Scan for prompt injection (adds 4 credits). | +| `redact_pii` | `any` | Redact PII. | +| `profile` | `:keyword_list` | Persistent browser profile. | +| `zero_data_retention` | `:boolean` | Enable zero data retention. | +| `alexandria` | `any` | Alexandria tool execution. | +| `domain_tools` | `:boolean` | Return Alexandria tool contracts. | + +### Bang variant + +`Firecrawl.scrape_and_extract_from_url!(params, opts)` — raises on error. + +## Interact + +### Why use it + +The Elixir SDK supports two interact patterns: + +1. **Post-scrape interaction** — After scraping a page, interact with its browser session using `interact_with_scrape_browser_session/3`. +2. **Standalone browser session** — Create an independent browser session with `create_browser_session/2`, then execute code with `execute_browser_code/3`. + +### Preferred SDK functions + +**Post-scrape interaction:** +``` +Firecrawl.interact_with_scrape_browser_session(job_id, params, opts \\ []) +``` + +**Standalone browser session:** +``` +Firecrawl.create_browser_session(params, opts \\ []) +Firecrawl.execute_browser_code(session_id, params, opts \\ []) +``` + +### Example — Post-scrape interaction + +```elixir +{:ok, scrape} = Firecrawl.scrape_and_extract_from_url(url: "https://example.com") +scrape_id = scrape.body["data"]["metadata"]["scrapeId"] + +{:ok, response} = Firecrawl.interact_with_scrape_browser_session(scrape_id, + code: "document.title", + language: :node +) + +IO.puts(response.body["data"]["result"]) + +Firecrawl.stop_interactive_scrape_browser_session(scrape_id) +``` + +### Example — Standalone browser session + +```elixir +{:ok, session} = Firecrawl.create_browser_session(ttl: 300) +session_id = session.body["data"]["sessionId"] + +{:ok, response} = Firecrawl.execute_browser_code(session_id, + code: ~s(await page.goto("https://example.com"); document.title), + language: :node +) + +IO.puts(response.body["data"]["result"]) + +Firecrawl.delete_browser_session(session_id) +``` + +### Post-scrape interaction parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `String.t()` | The scrape job ID from a previous scrape. **Required (positional).** | +| `code` | `:string` | Code to execute in the browser sandbox. **Required.** Max 100,000 chars. | +| `language` | `:python \| :node \| :bash` | Language for code execution. Default: `"node"` (API). | +| `timeout` | `:integer` | Execution timeout in seconds. Min: 1, default: 30 (API), max: 300. | +| `origin` | `:string` | Origin label for telemetry. | + +### Standalone browser session parameters + +**`create_browser_session`:** + +| Parameter | Type | Description | +|---|---|---| +| `ttl` | `:integer` | Total session TTL in seconds. Default: 300. Range: 30–3600. | +| `activity_ttl` | `:integer` | Inactivity timeout in seconds. Range: 10–3600. | +| `stream_web_view` | `:boolean` | Stream a live browser view. Default: `true`. | +| `block_ads` | `:boolean` | Block ads and cookie popups. Default: `true`. | +| `profile` | `:keyword_list` | Persistent storage profile. | +| `location` | `:keyword_list` | Country: `[country: "GB"]`. | + +**`execute_browser_code`:** + +| Parameter | Type | Description | +|---|---|---| +| `session_id` | `String.t()` | The session ID. **Required (positional).** | +| `code` | `:string` | Code to execute. **Required.** | +| `language` | `:python \| :node \| :bash` | Language. Default: `"node"` (API). | +| `timeout` | `:integer` | Timeout in seconds. | + +### Stopping sessions + +- Post-scrape: `Firecrawl.stop_interactive_scrape_browser_session(job_id)` +- Standalone: `Firecrawl.delete_browser_session(session_id)` + +## Notes + +- **No deprecated aliases** exist in the Elixir SDK. +- **Naming style:** All parameters use snake_case. The SDK auto-maps to camelCase for the API. +- **Function names are OpenAPI-derived.** They mirror operation IDs exactly (e.g. `scrape_and_extract_from_url`, not `scrape`). +- **Return type:** All functions return `{:ok, Req.Response.t()} | {:error, Exception.t()}`. Use bang variants (`!`) to raise on error. +- **Package name:** `:firecrawl` (Hex.pm). +- **Base URL:** Defaults to `https://api.firecrawl.dev/v2`. Override with `base_url` option. +- **Dependencies:** `req ~> 0.5`, `nimble_options ~> 1.1`. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..5c3f50f40 --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,232 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl via the official Java SDK. It is generated from SDK source and the OpenAPI spec. + +## Install + +### Maven + +```xml + + com.firecrawl + firecrawl + 1.18.4 + +``` + +### Gradle + +```groovy +implementation 'com.firecrawl:firecrawl:1.18.4' +``` + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey("fc-YOUR-API-KEY") + .build(); +``` + +Or set the `FIRECRAWL_API_KEY` environment variable: + +```java +FirecrawlClient client = FirecrawlClient.builder().build(); +``` + +## When To Use What + +- **`search`** — Use when you start with a query and need to discover relevant pages. Returns web results, news, images, and optionally scraped page content. +- **`scrape`** — Use when you already have a URL and want structured page content (markdown, HTML, JSON extraction, screenshots, etc.). +- **`interact`** — Use when a page needs browser automation after scraping — clicks, form fills, or post-scrape dynamic content extraction. + +## Search + +### Why use it + +Search the web for a query and get results with titles, URLs, descriptions, and optionally full page content. Supports filtering by domain, time range, category, and source type. + +### Preferred SDK method + +``` +client.search(query, options) +``` + +### Example + +```java +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.SearchData; + +SearchData results = client.search("firecrawl web scraping", + SearchOptions.builder() + .limit(5) + .build()); + +for (var result : results.getWeb()) { + System.out.println(result.get("title") + " " + result.get("url")); +} +``` + +### Parameters + +`SearchOptions` builder methods (all nullable, default `null` unless noted): + +| Method | Type | Description | +|---|---|---| +| `sources(List)` | `List` | Source types: `"web"`, `"news"`, `"images"`. Default: `["web"]`. | +| `categories(List)` | `List` | Filter by category: `"github"`, `"research"`, `"pdf"`. | +| `includeDomains(List)` | `List` | Restrict results to these domains. Cannot combine with `excludeDomains`. | +| `excludeDomains(List)` | `List` | Exclude results from these domains. | +| `limit(Integer)` | `Integer` | Max results to return. Default: `10` (API). Range: 1–100. | +| `tbs(String)` | `String` | Time-based filter. E.g. `"qdr:d"` for past day. | +| `location(String)` | `String` | Location for search results. | +| `country(String)` | `String` | ISO 3166-1 alpha-2 country code. | +| `highlights(Boolean)` | `Boolean` | Return query-relevant highlights. Default: `true`. | +| `timeout(Integer)` | `Integer` | Timeout in milliseconds. | +| `ignoreInvalidURLs(Boolean)` | `Boolean` | Skip invalid URLs instead of failing. | +| `scrapeOptions(ScrapeOptions)` | `ScrapeOptions` | Apply scrape options to each result page. | +| `integration(String)` | `String` | Integration identifier. | + +### Overloads + +- `client.search(query)` — Search with defaults. +- `client.search(query, options)` — Search with options. +- `client.searchAsync(query, options)` — Returns `CompletableFuture`. + +## Scrape + +### Why use it + +Scrape a URL and get its content in your chosen format — markdown, HTML, JSON (with schema-based extraction), screenshots, and more. Supports browser actions, caching, proxies, and PDF parsing. + +### Preferred SDK method + +``` +client.scrape(url, options) +``` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.Document; +import java.util.List; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder() + .formats(List.of("markdown", "links")) + .onlyMainContent(true) + .build()); + +System.out.println(doc.getMarkdown()); +``` + +### Parameters + +`ScrapeOptions` builder methods (all nullable, default `null` unless noted): + +| Method | Type | Description | +|---|---|---| +| `formats(List)` | `List` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"json"`, `"summary"`, `"audio"`, `"video"`. Also accepts config objects for `json`, `screenshot`, etc. Default: `["markdown"]`. | +| `onlyMainContent(Boolean)` | `Boolean` | Strip headers, navs, footers. Default: `true` (API). | +| `includeTags(List)` | `List` | Only include content from these HTML tags. | +| `excludeTags(List)` | `List` | Exclude content from these HTML tags. | +| `headers(Map)` | `Map` | Custom HTTP headers. | +| `waitFor(Integer)` | `Integer` | Extra delay in ms before fetching content. Default: `0`. | +| `timeout(Integer)` | `Integer` | Request timeout in ms. Default: 60000, max: 300000. | +| `mobile(Boolean)` | `Boolean` | Emulate a mobile device. Default: `false`. | +| `maxAge(Long)` | `Long` | Use cached result if younger than this (ms). Default: 172800000 (2 days). | +| `storeInCache(Boolean)` | `Boolean` | Cache this scrape result. | +| `actions(List>)` | `List>` | Browser actions before scraping (wait, click, write, scroll, etc.). | +| `location(LocationConfig)` | `LocationConfig` | Geolocation: `LocationConfig.builder().country("US").build()`. | +| `parsers(List)` | `List` | File parsers. Default: `["pdf"]`. | +| `proxy(String)` | `String` | Proxy type: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `skipTlsVerification(Boolean)` | `Boolean` | Skip TLS verification. Default: `true` (API). | +| `removeBase64Images(Boolean)` | `Boolean` | Strip base64 images from output. | +| `blockAds(Boolean)` | `Boolean` | Block ads and cookie popups. | +| `lockdown(Boolean)` | `Boolean` | Serve only cached results. | +| `checkPromptInjection(Boolean)` | `Boolean` | Scan content for prompt injection (adds 4 credits). | +| `redactPII(Boolean)` | `Boolean` | Redact PII. | +| `auditMetadata(AuditMetadata)` | `AuditMetadata` | User attribution for SIEM logging. | +| `integration(String)` | `String` | Integration identifier. | + +### Overloads + +- `client.scrape(url)` — Scrape with defaults. +- `client.scrape(url, options)` — Scrape with options. +- `client.scrapeAsync(url, options)` — Returns `CompletableFuture`. + +## Interact + +### Why use it + +After scraping a page, interact with it in a live browser session — execute code to click buttons, fill forms, navigate, and extract dynamic content. + +### Preferred SDK method + +``` +client.interact(jobId, code) +``` + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; + +Document doc = client.scrape("https://example.com"); +String scrapeId = (String) doc.getMetadata().get("scrapeId"); + +BrowserExecuteResponse response = client.interact(scrapeId, "document.title"); +System.out.println(response.getResult()); + +BrowserExecuteResponse nav = client.interact(scrapeId, + "document.querySelector('a').click()", "node", 60); +System.out.println(nav.getStdout()); + +client.stopInteractiveBrowser(scrapeId); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `String` | The scrape job ID from a previous scrape's `metadata.scrapeId`. **Required.** | +| `code` | `String` | Code to execute in the browser sandbox. **Required.** Max 100,000 chars. | +| `language` | `String` | Language: `"python"`, `"node"`, `"bash"`. Default: `"node"`. | +| `timeout` | `Integer` | Execution timeout in seconds. Min: 1, default: 30 (API), max: 300. | +| `origin` | `String` | Origin label (auto-set to `"java-sdk@{version}"`). | + +### Overloads + +- `client.interact(jobId, code)` — Execute with defaults (language: `"node"`). +- `client.interact(jobId, code, language, timeout)` — Specify language and timeout. +- `client.interact(jobId, code, language, timeout, origin)` — Full control. +- All have corresponding `interactAsync(...)` variants returning `CompletableFuture`. + +### Stopping an interaction + +Call `client.stopInteractiveBrowser(jobId)` to close the browser session and stop billing. + +## Notes + +- **Deprecated aliases:** `scrapeExecute` → `interact`, `deleteScrapeBrowser` → `stopInteractiveBrowser`. Always use the preferred names. +- **Parameter naming:** All parameters use camelCase (Java convention). +- **Builder pattern:** `ScrapeOptions` and `SearchOptions` use the builder pattern. +- **Package coordinates:** `com.firecrawl:firecrawl` (Maven Central). +- **Constants:** Default API URL: `https://api.firecrawl.dev`, default timeout: 300,000ms, default max retries: 3. +- **Note:** The Java SDK `interact` method requires `code` as a positional parameter. Natural-language `prompt`-based interaction is not yet supported in the Java SDK — use `code` to automate browser actions. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..2e5a9a8fa --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,205 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl via the official Node.js SDK. It is generated from SDK source and the OpenAPI spec. + +## Install + +```bash +npm install firecrawl +``` + +## Authenticate + +```javascript +import { Firecrawl } from "firecrawl"; + +const app = new Firecrawl({ apiKey: "fc-YOUR-API-KEY" }); +``` + +Or set the `FIRECRAWL_API_KEY` environment variable and omit `apiKey`: + +```javascript +const app = new Firecrawl(); +``` + +## When To Use What + +- **`search`** — Use when you start with a query and need to discover relevant pages. Returns web results, news, images, and optionally scraped page content. +- **`scrape`** — Use when you already have a URL and want structured page content (markdown, HTML, JSON extraction, screenshots, etc.). +- **`interact`** — Use when a page needs browser automation after scraping — clicks, form fills, or post-scrape dynamic content extraction. + +## Search + +### Why use it + +Search the web for a query and get results with titles, URLs, descriptions, and optionally full page content. Supports filtering by domain, time range, category, and source type. + +### Preferred SDK method + +``` +app.search(query, options?) +``` + +### Example + +```javascript +const results = await app.search("firecrawl web scraping", { + limit: 5, + scrapeOptions: { formats: ["markdown"] }, +}); + +for (const result of results.web) { + console.log(result.title, result.url); + console.log(result.markdown); // available when scrapeOptions is set +} +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | The search query (max 500 chars). **Required.** | +| `limit` | `number` | Max results to return per source type. Default: `10`. Range: 1–100. | +| `sources` | `Array` | Source types to search: `"web"`, `"news"`, `"images"`, `"alexandria"`. Default: `["web"]`. | +| `categories` | `Array` | Filter results by category: `"github"`, `"research"`, `"pdf"`, `"developer"`, `"gov"`. | +| `includeDomains` | `string[]` | Restrict results to these domains (hostnames only). Cannot combine with `excludeDomains`. | +| `excludeDomains` | `string[]` | Exclude results from these domains. Cannot combine with `includeDomains`. | +| `tbs` | `string` | Time-based filter. Predefined: `qdr:h` (hour), `qdr:d` (day), `qdr:w` (week), `qdr:m` (month), `qdr:y` (year). Custom: `cdr:1,cd_min:MM/DD/YYYY,cd_max:MM/DD/YYYY`. Sort by date: `sbd:1`. Combinable. | +| `location` | `string` | Location for search results. | +| `country` | `string` | ISO 3166-1 alpha-2 country code. | +| `highlights` | `boolean` | Return query-relevant highlights. Default: `true`. | +| `timeout` | `number` | Timeout in milliseconds. | +| `ignoreInvalidURLs` | `boolean` | Skip invalid URLs instead of failing. | +| `scrapeOptions` | `ScrapeOptions` | Apply scrape options to each search result page to get full content. | +| `domainTools` | `boolean` | Include Alexandria tool contracts matched to result domains. | +| `toolDetail` | `"compact" \| "summary" \| "full"` | Detail level for returned tool contracts. Default: `"compact"`. | +| `integration` | `string` | Integration identifier. | +| `origin` | `string` | Origin label for telemetry. | + +## Scrape + +### Why use it + +Scrape a URL and get its content in your chosen format — markdown, HTML, JSON (with schema-based extraction), screenshots, and more. Supports browser actions, caching, proxies, and PDF parsing. + +### Preferred SDK method + +``` +app.scrape(url, options?) +``` + +### Example + +```javascript +const result = await app.scrape("https://example.com", { + formats: ["markdown", "links"], + onlyMainContent: true, +}); + +console.log(result.markdown); +console.log(result.links); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | The URL to scrape. **Required.** | +| `formats` | `FormatOption[]` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"json"`, `"summary"`, `"audio"`, `"video"`, `"changeTracking"`, `"branding"`, `"product"`, `"menu"`, `"question"`, `"highlights"`. Default: `["markdown"]`. Some accept config objects. | +| `onlyMainContent` | `boolean` | Strip headers, navs, footers. Deterministic HTML filter, no LLM. Default: `true`. | +| `onlyCleanContent` | `boolean` | Additional LLM-based boilerplate removal pass. Default: `false`. | +| `includeTags` | `string[]` | Only include content from these HTML tags. | +| `excludeTags` | `string[]` | Exclude content from these HTML tags. | +| `headers` | `Record` | Custom HTTP headers (cookies, user-agent, etc.). | +| `waitFor` | `number` | Extra delay in ms before fetching content, on top of smart-wait. Default: `0`. | +| `timeout` | `number` | Request timeout in ms. Min: 1000, default: 60000, max: 300000. | +| `mobile` | `boolean` | Emulate a mobile device. Default: `false`. | +| `maxAge` | `number` | Use cached result if younger than this (ms). Default: 172800000 (2 days). Set `0` to bypass cache. | +| `minAge` | `number` | Cache-only mode. Returns cached data if it exists, 404 otherwise. Set `1` to accept any cached data. | +| `storeInCache` | `boolean` | Cache this scrape result. | +| `actions` | `ActionOption[]` | Browser actions to perform before scraping (wait, click, type, scroll, screenshot, etc.). | +| `location` | `LocationConfig` | Geolocation: `{ country: string, languages?: string[] }`. | +| `parsers` | `Array` | File parsers. Default: `["pdf"]`. PDF parser supports `mode` (`"fast"`, `"auto"`, `"ocr"`), `maxPages`, `pages`, `blocks`, `pageMarkers`. | +| `proxy` | `string` | Proxy type: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or custom. | +| `skipTlsVerification` | `boolean` | Skip TLS cert verification. Default: `true`. | +| `removeBase64Images` | `boolean` | Strip base64 images from output. | +| `blockAds` | `boolean` | Block ads and cookie popups. | +| `fastMode` | `boolean` | Enable fast scraping mode. | +| `lockdown` | `boolean` | Serve only cached results. | +| `checkPromptInjection` | `boolean` | Scan content for prompt injection (adds 4 credits). | +| `redactPII` | `boolean \| RedactPIIOptions` | Redact personally identifiable information. | +| `profile` | `{ name: string, saveChanges?: boolean }` | Persistent browser profile for storage across sessions. | +| `domainTools` | `boolean` | Return Alexandria tool contracts matched to the scraped domain. | +| `toolDetail` | `"compact" \| "summary" \| "full"` | Detail level for tool contracts. | +| `integration` | `string` | Integration identifier. | +| `origin` | `string` | Origin label for telemetry. | + +## Interact + +### Why use it + +After scraping a page, interact with it in a live browser session — execute code or natural-language prompts to click buttons, fill forms, navigate, and extract dynamic content. + +### Preferred SDK method + +``` +app.interact(jobId, args) +``` + +### Example + +```javascript +const scrapeResult = await app.scrape("https://www.amazon.com", { + formats: ["markdown"], +}); +const scrapeId = scrapeResult.metadata?.scrapeId; + +const response = await app.interact(scrapeId, { + prompt: "Search for iPhone 16 Pro Max", +}); +console.log(response.output); + +const result = await app.interact(scrapeId, { + code: "document.title", + language: "node", +}); +console.log(result.result); + +await app.stopInteraction(scrapeId); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `string` | The scrape job ID from a previous scrape's `metadata.scrapeId`. **Required.** | +| `code` | `string` | Code to execute in the browser sandbox. Provide `code` or `prompt`. Max 100,000 chars. | +| `prompt` | `string` | Natural-language instruction for the browser agent. Provide `code` or `prompt`. | +| `language` | `"python" \| "node" \| "bash"` | Language for code execution. Default: `"node"`. | +| `timeout` | `number` | Execution timeout in seconds. Min: 1, default: 30, max: 300. | +| `origin` | `string` | Origin label for telemetry. | + +### Stopping an interaction + +Call `app.stopInteraction(jobId)` to close the browser session and stop billing. + +## Notes + +- **Deprecated aliases:** `scrapeUrl` → `scrape`, `scrapeExecute` → `interact`, `stopInteractiveBrowser` / `deleteScrapeBrowser` → `stopInteraction`. Always use the preferred names. +- **Parameter naming:** All parameters use camelCase. +- **Package name:** `firecrawl` (npm). +- **Import:** `import { Firecrawl } from "firecrawl"`. +- **Base URL:** Defaults to `https://api.firecrawl.dev`. Override with `apiUrl` in the constructor for self-hosted instances. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/methods/scrape.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/methods/search.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..e4a22b9ce --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,190 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl via the official Python SDK. It is generated from SDK source and the OpenAPI spec. + +## Install + +```bash +pip install firecrawl +``` + +## Authenticate + +```python +from firecrawl import Firecrawl + +app = Firecrawl(api_key="fc-YOUR-API-KEY") +``` + +Or set the `FIRECRAWL_API_KEY` environment variable and omit `api_key`: + +```python +app = Firecrawl() +``` + +## When To Use What + +- **`search`** — Use when you start with a query and need to discover relevant pages. Returns web results, news, images, and optionally scraped page content. +- **`scrape`** — Use when you already have a URL and want structured page content (markdown, HTML, JSON extraction, screenshots, etc.). +- **`interact`** — Use when a page needs browser automation after scraping — clicks, form fills, or post-scrape dynamic content extraction. + +## Search + +### Why use it + +Search the web for a query and get results with titles, URLs, descriptions, and optionally full page content. Supports filtering by domain, time range, category, and source type. + +### Preferred SDK method + +``` +app.search(query, **kwargs) +``` + +### Example + +```python +results = app.search("firecrawl web scraping", limit=5, scrape_options={"formats": ["markdown"]}) + +for result in results.web: + print(result.title, result.url) + print(result.markdown) # available when scrape_options is set +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `str` | The search query (max 500 chars). **Required.** | +| `limit` | `int` | Max results to return per source type. Default: `5` (SDK), `10` (API). Range: 1–100. | +| `sources` | `list[str \| Source]` | Source types: `"web"`, `"news"`, `"images"`, `"alexandria"`. Default: `["web"]`. | +| `categories` | `list[str \| Category]` | Filter by category: `"github"`, `"research"`, `"pdf"`, `"developer"`, `"gov"`. | +| `include_domains` | `list[str]` | Restrict results to these domains (hostnames only). Cannot combine with `exclude_domains`. | +| `exclude_domains` | `list[str]` | Exclude results from these domains. Cannot combine with `include_domains`. | +| `tbs` | `str` | Time-based filter. Predefined: `qdr:h` (hour), `qdr:d` (day), `qdr:w` (week), `qdr:m` (month), `qdr:y` (year). Custom: `cdr:1,cd_min:MM/DD/YYYY,cd_max:MM/DD/YYYY`. Sort by date: `sbd:1`. Combinable. | +| `location` | `str` | Location for search results (city/region name). | +| `country` | `str` | ISO 3166-1 alpha-2 country code. | +| `highlights` | `bool` | Return query-relevant highlights. | +| `timeout` | `int` | Timeout in milliseconds. Default: `300000`. | +| `ignore_invalid_urls` | `bool` | Skip invalid URLs instead of failing. | +| `scrape_options` | `ScrapeOptions \| dict` | Apply scrape options to each search result page to get full content. | +| `domain_tools` | `bool` | Include Alexandria tool contracts matched to result domains. | +| `tool_detail` | `"compact" \| "summary" \| "full"` | Detail level for returned tool contracts. Default: `"compact"`. | +| `integration` | `str` | Integration identifier. | +| `enterprise` | `list[str]` | Enterprise options: `"default"`, `"anon"`, `"zdr"`. | + +## Scrape + +### Why use it + +Scrape a URL and get its content in your chosen format — markdown, HTML, JSON (with schema-based extraction), screenshots, and more. Supports browser actions, caching, proxies, and PDF parsing. + +### Preferred SDK method + +``` +app.scrape(url, **kwargs) +``` + +### Example + +```python +result = app.scrape("https://example.com", formats=["markdown", "links"], only_main_content=True) + +print(result.markdown) +print(result.links) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `str` | The URL to scrape. **Required.** | +| `formats` | `list[FormatOption]` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"json"`, `"summary"`, `"audio"`, `"video"`, `"changeTracking"`, `"branding"`, `"product"`, `"menu"`, `"question"`, `"highlights"`. Default: `["markdown"]`. Some accept config dicts. | +| `only_main_content` | `bool` | Strip headers, navs, footers. Deterministic HTML filter, no LLM. Default: `true` (API). | +| `only_clean_content` | `bool` | Additional LLM-based boilerplate removal pass. Default: `false`. | +| `include_tags` | `list[str]` | Only include content from these HTML tags. | +| `exclude_tags` | `list[str]` | Exclude content from these HTML tags. | +| `headers` | `dict[str, str]` | Custom HTTP headers (cookies, user-agent, etc.). | +| `wait_for` | `int` | Extra delay in ms before fetching content, on top of smart-wait. Default: `0`. | +| `timeout` | `int` | Request timeout in ms. Min: 1000, default: 60000, max: 300000. | +| `mobile` | `bool` | Emulate a mobile device. Default: `false`. | +| `max_age` | `int` | Use cached result if younger than this (ms). Default: 172800000 (2 days). Set `0` to bypass cache. | +| `min_age` | `int` | Cache-only mode. Returns cached data if it exists, 404 otherwise. Set `1` to accept any cached data. | +| `store_in_cache` | `bool` | Cache this scrape result. | +| `actions` | `list[Action]` | Browser actions to perform before scraping (wait, click, type, scroll, screenshot, etc.). | +| `location` | `Location` | Geolocation: `Location(country="US", languages=["en"])`. | +| `parsers` | `list[str \| PDFParser]` | File parsers. Default: `["pdf"]`. PDF parser supports `mode` (`"fast"`, `"auto"`, `"ocr"`), `max_pages`, `pages`, `blocks`, `page_markers`. | +| `proxy` | `str` | Proxy type: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `skip_tls_verification` | `bool` | Skip TLS cert verification. Default: `true` (API). | +| `remove_base64_images` | `bool` | Strip base64 images from output. | +| `block_ads` | `bool` | Block ads and cookie popups. | +| `fast_mode` | `bool` | Enable fast scraping mode. | +| `lockdown` | `bool` | Serve only cached results. | +| `check_prompt_injection` | `bool` | Scan content for prompt injection (adds 4 credits). | +| `redact_pii` | `bool \| RedactPIIOptions` | Redact personally identifiable information. | +| `profile` | `dict` | Persistent browser profile: `{"name": "my-profile", "save_changes": True}`. | +| `domain_tools` | `bool` | Return Alexandria tool contracts matched to the scraped domain. | +| `tool_detail` | `"compact" \| "summary" \| "full"` | Detail level for tool contracts. | +| `integration` | `str` | Integration identifier. | + +## Interact + +### Why use it + +After scraping a page, interact with it in a live browser session — execute code or natural-language prompts to click buttons, fill forms, navigate, and extract dynamic content. + +### Preferred SDK method + +``` +app.interact(job_id, code=None, *, prompt=None, language="node", timeout=None) +``` + +### Example + +```python +result = app.scrape("https://www.amazon.com", formats=["markdown"]) +scrape_id = result.metadata.get("scrapeId") + +response = app.interact(scrape_id, prompt="Search for iPhone 16 Pro Max") +print(response.output) + +result = app.interact(scrape_id, code="document.title", language="node") +print(result.result) + +app.stop_interaction(scrape_id) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `str` | The scrape job ID from a previous scrape's `metadata["scrapeId"]`. **Required.** | +| `code` | `str` | Code to execute in the browser sandbox. Provide `code` or `prompt`. Max 100,000 chars. | +| `prompt` | `str` | Natural-language instruction for the browser agent. Provide `code` or `prompt`. | +| `language` | `"python" \| "node" \| "bash"` | Language for code execution. Default: `"node"`. | +| `timeout` | `int` | Execution timeout in seconds. Min: 1, default: 30 (API), max: 300. | +| `origin` | `str` | Origin label for telemetry. | + +### Stopping an interaction + +Call `app.stop_interaction(job_id)` to close the browser session and stop billing. + +## Notes + +- **Deprecated aliases:** `scrape_url` → `scrape`, `scrape_execute` → `interact`, `stop_interactive_browser` / `delete_scrape_browser` → `stop_interaction`. Always use the preferred names. +- **Backward-compat class names:** `FirecrawlApp` is an alias for `Firecrawl`. Use `Firecrawl`. +- **Parameter naming:** All parameters use snake_case. The SDK automatically converts to camelCase for the API. +- **Package name:** `firecrawl` (PyPI). +- **Async support:** Use `AsyncFirecrawl` for async/await patterns. +- **Base URL:** Defaults to `https://api.firecrawl.dev`. Override with `api_url` in the constructor for self-hosted instances. + +## Source Of Truth + +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..9cc3de714 --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,223 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl via the official Rust SDK. It is generated from SDK source and the OpenAPI spec. + +## Install + +Add to your `Cargo.toml`: + +```toml +[dependencies] +firecrawl = "1" +tokio = { version = "1", features = ["full"] } +``` + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-YOUR-API-KEY")?; +``` + +Or set the `FIRECRAWL_API_KEY` environment variable and use: + +```rust +let client = Client::from_env()?; +``` + +## When To Use What + +- **`search`** — Use when you start with a query and need to discover relevant pages. Returns web results, news, images, and optionally scraped page content. +- **`scrape`** — Use when you already have a URL and want structured page content (markdown, HTML, JSON extraction, screenshots, etc.). +- **`interact`** — Use when a page needs browser automation after scraping — clicks, form fills, or post-scrape dynamic content extraction. + +## Search + +### Why use it + +Search the web for a query and get results with titles, URLs, descriptions, and optionally full page content. Supports filtering by domain, time range, category, and source type. + +### Preferred SDK method + +``` +client.search(query, options).await +``` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, ScrapeOptions}; + +let client = Client::new("fc-YOUR-API-KEY")?; +let response = client.search("firecrawl web scraping", SearchOptions { + limit: Some(5), + scrape_options: Some(ScrapeOptions::default()), + ..Default::default() +}).await?; + +if let Some(web) = response.data.web { + for result in web { + println!("{:?}", result); + } +} +``` + +### Parameters + +`SearchOptions` fields (all `Option`, default `None` unless noted): + +| Field | Type | Description | +|---|---|---| +| `limit` | `Option` | Max results to return. Default: `5`. Max: 20. | +| `sources` | `Option>` | Source types: `Web`, `News`, `Images`, `Alexandria`. Default: `[Web]`. | +| `categories` | `Option>` | Filter by category: `Github`, `Research`, `Pdf`. | +| `include_domains` | `Option>` | Restrict results to these domains. Cannot combine with `exclude_domains`. | +| `exclude_domains` | `Option>` | Exclude results from these domains. | +| `tbs` | `Option` | Time-based filter. E.g. `"qdr:d"` for past day, `"sbd:1,qdr:w"` for sorted by date in past week. | +| `location` | `Option` | Location for search results. | +| `country` | `Option` | ISO 3166-1 alpha-2 country code. | +| `highlights` | `Option` | Return query-relevant highlights. Default: `true`. | +| `timeout` | `Option` | Timeout in milliseconds. | +| `ignore_invalid_urls` | `Option` | Skip invalid URLs instead of failing. | +| `scrape_options` | `Option` | Apply scrape options to each result page. | +| `domain_tools` | `Option` | Include Alexandria tool contracts matched to result domains. | +| `tool_detail` | `Option` | Detail level: `Compact`, `Summary`, `Full`. Default: `Compact`. | +| `integration` | `Option` | Integration identifier. | +| `origin` | `Option` | Origin label (auto-set to `"rust-sdk@{version}"`). | + +## Scrape + +### Why use it + +Scrape a URL and get its content in your chosen format — markdown, HTML, JSON (with schema-based extraction), screenshots, and more. Supports browser actions, caching, proxies, and PDF parsing. + +### Preferred SDK method + +``` +client.scrape(url, options).await +``` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format}; + +let client = Client::new("fc-YOUR-API-KEY")?; +let doc = client.scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links]), + only_main_content: Some(true), + ..Default::default() +}).await?; + +println!("{}", doc.markdown.unwrap_or_default()); +``` + +### Parameters + +`ScrapeOptions` fields (all `Option`, default `None` unless noted): + +| Field | Type | Description | +|---|---|---| +| `formats` | `Option>` | Output formats: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Json`, `Summary`, `Audio`, `Video`, `ChangeTracking`, `Branding`, `Product`, `Menu`, `Question(QuestionFormat)`, `Highlights(HighlightsFormat)`. Default: `[Markdown]`. | +| `only_main_content` | `Option` | Strip headers, navs, footers. Default: `true` (API). | +| `include_tags` | `Option>` | Only include content from these HTML tags. | +| `exclude_tags` | `Option>` | Exclude content from these HTML tags. | +| `headers` | `Option>` | Custom HTTP headers. | +| `wait_for` | `Option` | Extra delay in ms before fetching content. Default: `0`. | +| `timeout` | `Option` | Request timeout in ms. Default: 60000, max: 300000. | +| `mobile` | `Option` | Emulate a mobile device. Default: `false`. | +| `max_age` | `Option` | Use cached result if younger than this (ms). Default: 172800000 (2 days). | +| `min_age` | `Option` | Cache-only mode. | +| `store_in_cache` | `Option` | Cache this scrape result. | +| `actions` | `Option>` | Browser actions before scraping. | +| `location` | `Option` | Geolocation: `LocationConfig { country, languages }`. | +| `parsers` | `Option>` | File parsers. Default: `["pdf"]`. | +| `proxy` | `Option` | Proxy: `Basic`, `Stealth`, `Enhanced`, `Auto`. | +| `skip_tls_verification` | `Option` | Skip TLS verification. Default: `true` (API). | +| `remove_base64_images` | `Option` | Strip base64 images from output. | +| `block_ads` | `Option` | Block ads and cookie popups. | +| `fast_mode` | `Option` | Enable fast scraping mode. | +| `lockdown` | `Option` | Serve only cached results. | +| `check_prompt_injection` | `Option` | Scan content for prompt injection (adds 4 credits). | +| `redact_pii` | `Option` | Redact PII. | +| `profile` | `Option` | Persistent browser profile. | +| `domain_tools` | `Option` | Return Alexandria tool contracts for the domain. | +| `tool_detail` | `Option` | Tool contract detail level. | +| `integration` | `Option` | Integration identifier. | +| `json_options` | `Option` | JSON extraction schema and prompt. | +| `screenshot_options` | `Option` | Screenshot quality and viewport settings. | +| `origin` | `Option` | Origin label (auto-set to `"rust-sdk@{version}"`). | + +## Interact + +### Why use it + +After scraping a page, interact with it in a live browser session — execute code or natural-language prompts to click buttons, fill forms, navigate, and extract dynamic content. + +### Preferred SDK method + +``` +client.interact(job_id, options).await +``` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, ScrapeExecuteOptions}; + +let client = Client::new("fc-YOUR-API-KEY")?; + +let doc = client.scrape("https://example.com", None).await?; +let scrape_id = doc.metadata.as_ref() + .and_then(|m| m.get("scrapeId")) + .and_then(|v| v.as_str()) + .unwrap(); + +let response = client.interact(scrape_id, ScrapeExecuteOptions { + prompt: Some("Click the first link".to_string()), + ..Default::default() +}).await?; + +println!("{}", response.output.unwrap_or_default()); + +client.stop_interaction(scrape_id).await?; +``` + +### Parameters + +`ScrapeExecuteOptions` fields (all `Option` except where noted): + +| Field | Type | Description | +|---|---|---| +| `code` | `Option` | Code to execute in the browser sandbox. Provide `code` or `prompt`. Max 100,000 chars. | +| `prompt` | `Option` | Natural-language instruction for the browser agent. Provide `code` or `prompt`. | +| `language` | `Option` | Language: `Python`, `Node`, `Bash`. Default: `Node`. | +| `timeout` | `Option` | Execution timeout in seconds. Min: 1, default: 30 (API), max: 300. | +| `origin` | `Option` | Origin label (auto-set to `"rust-sdk@{version}"`). | + +### Stopping an interaction + +Call `client.stop_interaction(job_id).await?` to close the browser session and stop billing. + +## Notes + +- **Deprecated aliases:** `scrape_execute` → `interact`, `stop_interactive_browser` / `delete_scrape_browser` → `stop_interaction`. Always use the preferred names. +- **Naming style:** All fields use snake_case (Rust convention). Serialized to camelCase for the API via serde. +- **Crate name:** `firecrawl` (crates.io). +- **Async runtime:** All methods are async and require a Tokio runtime. +- **Convenience method:** `client.search_and_scrape(query, limit)` performs a search with scrape options and returns `Vec` directly. +- **JSON extraction:** Use `client.scrape_with_schema(url, schema, prompt)` for schema-based extraction. +- **Base URL:** Defaults to `https://api.firecrawl.dev`. Override with `Client::new_with_url(api_key, url)`. + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/docs.json b/docs.json index 40fc33ccc..126cc8275 100755 --- a/docs.json +++ b/docs.json @@ -290,6 +290,16 @@ } ] }, + { + "group": "Agent Quickstarts", + "pages": [ + "agent-quickstart/node", + "agent-quickstart/python", + "agent-quickstart/rust", + "agent-quickstart/java", + "agent-quickstart/elixir" + ] + }, { "group": "Developer Guides", "pages": [