diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..197f900aa --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,201 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl` hex package v1.11.0) and the v2 OpenAPI spec. The Elixir client is auto-generated from the OpenAPI spec; function names match the spec operation IDs. + +## Install + +Add to `mix.exs`: + +```elixir +{:firecrawl, "~> 1.11"} +``` + +## Authenticate + +```elixir +# config/runtime.exs +config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") + +# Or pass per-call: +{:ok, res} = Firecrawl.search_and_scrape([query: "example"], api_key: "fc-your-api-key") +``` + +Every function accepts a trailing `opts` keyword list with `:api_key` and `:base_url` (default `https://api.firecrawl.dev/v2`) overrides. Scrape, search, and interact work without an API key (keyless free tier, rate-limited by IP). + +## When To Use What + +- **search**: use when you start with a query and need discovery. +- **scrape**: use when you already have a URL and want page content. +- **interact**: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.search_and_scrape( + query: "site:docs.firecrawl.dev webhook retries", + sources: [:web, :news], + limit: 10, + scrape_options: [ + formats: ["markdown"], + only_main_content: true + ] +) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | Search query. Required. | +| `sources` | `list` | `:web`, `:news`, `:images` (atoms or strings or maps). Default `["web"]`. | +| `categories` | `list` | `:developer`, `:research`, `:pdf` (atoms or strings or maps). | +| `include_domains` | `list(string)` | Restrict to these domains. | +| `exclude_domains` | `list(string)` | Exclude these domains. | +| `limit` | `integer` | Max results. | +| `tbs` | `string` | Time-based filter (e.g. `qdr:d`, `qdr:w`). | +| `location` | `string` | Localized results. | +| `country` | `string` | ISO country code for geo-targeting. | +| `ignore_invalid_urls` | `boolean` | Drop invalid URLs. | +| `timeout` | `integer` | Timeout in milliseconds. | +| `highlights` | `boolean` | Query-relevant highlights. Default `true`. | +| `enterprise` | `list(string)` | `["zdr"]` or `["anon"]` for zero data retention. | +| `scrape_options` | `keyword list` | Scrape each result. See Scrape parameters. | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com/pricing", + formats: [ + "markdown", + "links", + %{type: "json", prompt: "Extract plan names and prices."} + ], + only_main_content: true, + wait_for: 1000, + actions: [ + %{type: "click", selector: "#accept"}, + %{type: "wait", milliseconds: 750}, + %{type: "scrape"} + ] +) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | URL to scrape. Required. | +| `formats` | `list` | Format strings or format maps. See format types below. | +| `headers` | `map` | Custom request headers. | +| `include_tags` | `list(string)` | Only include specific HTML tags. | +| `exclude_tags` | `list(string)` | Exclude specific HTML tags. | +| `only_main_content` | `boolean` | Strip nav, footer, boilerplate. | +| `timeout` | `integer` | Timeout in milliseconds. Default 60000. | +| `wait_for` | `integer` | Wait for page render (milliseconds). | +| `mobile` | `boolean` | Mobile viewport. | +| `parsers` | `list` | `"pdf"` or `%{type: "pdf", mode: "auto", maxPages: 5}`. | +| `actions` | `list(map)` | Pre-scrape actions: `wait`, `screenshot`, `click`, `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. | +| `location` | `keyword list` | `[country: "US", languages: ["en-US"]]`. | +| `skip_tls_verification` | `boolean` | Skip TLS verification. | +| `remove_base64_images` | `boolean` | Drop base64 images from markdown. | +| `block_ads` | `boolean` | Ad and cookie popup blocking. | +| `proxy` | `atom` | `:basic`, `:enhanced`, `:auto`. | +| `max_age` | `integer` | Max age of cached data (milliseconds). Default 2 days. | +| `min_age` | `integer` | Minimum age of cached data (milliseconds). | +| `store_in_cache` | `boolean` | Cache the result. | +| `lockdown` | `boolean` | Serve only cached results. | +| `redact_pii` | `boolean` | Redact PII from content. | +| `profile` | `keyword list` | `[name: "session-name", save_changes: true]`. Persistent browser profile. | +| `audit_metadata` | `keyword list` | `[username: "..."]` for SIEM logging. | +| `zero_data_retention` | `boolean` | Enable zero data retention. | + +**Format strings:** `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"branding"`, `"audio"`, `"video"`. + +**Format maps:** +- `%{type: "json", prompt: "...", schema: %{...}}` +- `%{type: "question", question: "..."}` +- `%{type: "highlights", query: "..."}` +- `%{type: "screenshot", fullPage: true, quality: 80, viewport: %{width: 1280, height: 720}}` +- `%{type: "changeTracking", modes: ["git-diff"], tag: "..."}` + +## Interact + +### Why use it + +Execute code in the browser session tied to a scrape job. The Elixir SDK exposes code-based interactions only (no `prompt` parameter). + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, scrape_res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"] +) + +job_id = scrape_res.body["data"]["metadata"]["scrapeId"] + +{:ok, res} = Firecrawl.interact_with_scrape_browser_session( + job_id, + code: "console.log(await page.title());", + language: :node, + timeout: 60 +) + +# Stop the session when done: +{:ok, _} = Firecrawl.stop_interactive_scrape_browser_session(job_id) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `string` | Scrape job ID (positional first argument). | +| `code` | `string` | Code to execute in the browser session. Required. | +| `language` | `atom` | `:python`, `:node`, `:bash`. | +| `timeout` | `integer` | Execution timeout in seconds. | +| `origin` | `string` | Optional origin label for telemetry. | + +## Notes + +- The Elixir client is auto-generated from the OpenAPI spec via `mix run generate.exs`. +- Every function has a bang (`!`) variant: `scrape_and_extract_from_url!/2` raises on error instead of returning `{:error, _}`. +- Parameters are **snake_case keyword lists**, auto-converted to camelCase for the JSON body. +- Atoms are accepted for enum values (`:basic`, `:node`, etc.) and auto-stringified. +- `origin` is auto-injected as `"elixir-sdk@"` when not set. +- This SDK exposes **code-based interactions only** — there is no `prompt` parameter on the interact function (unlike Node.js, Python, and Rust SDKs). + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..a677ecfad --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,233 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl-java` v1.18.0) and the v2 OpenAPI spec. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.18.0 + +``` + +Gradle: + +```gradle +implementation("com.firecrawl:firecrawl-java:1.18.0") +``` + +Requires Java 11+. + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey(System.getenv("FIRECRAWL_API_KEY")) + .build(); + +// Or from environment directly: +// FirecrawlClient client = FirecrawlClient.fromEnv(); +``` + +Builder options: `apiKey` (falls back to `FIRECRAWL_API_KEY` env var or `firecrawl.apiKey` system property), `apiUrl` (default `https://api.firecrawl.dev`), `timeoutMs` (default 300000), `maxRetries` (default 3), `backoffFactor` (default 0.5), `asyncExecutor`, `httpClient`. Scrape, search, and interact work without an API key (keyless free tier, rate-limited by IP). + +## When To Use What + +- **search**: use when you start with a query and need discovery. +- **scrape**: use when you already have a URL and want page content. +- **interact**: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`. + +### Preferred SDK method + +- `client.search(query)` → `SearchData` +- `client.search(query, options)` → `SearchData` + +### Example + +```java +import com.firecrawl.models.SearchData; +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.ScrapeOptions; + +SearchOptions options = SearchOptions.builder() + .sources(List.of("web", "news")) + .limit(10) + .scrapeOptions( + ScrapeOptions.builder() + .formats(List.of("markdown")) + .onlyMainContent(true) + .build() + ) + .build(); + +SearchData results = client.search("site:docs.firecrawl.dev webhook retries", options); +List> web = results.getWeb(); +``` + +`SearchData` has `.getWeb()`, `.getNews()`, `.getImages()` (each `List>`, may be null). Do not treat it as a directly iterable list. + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `String` | Search query. Use `site:example.com` to limit to a domain. | +| `options.sources` | `List` | `"web"`, `"news"`, `"images"` as strings or `{type: ...}` maps. | +| `options.categories` | `List` | `"github"`, `"research"`, `"pdf"`. | +| `options.includeDomains` | `List` | Restrict to these domains. | +| `options.excludeDomains` | `List` | Exclude these domains. | +| `options.limit` | `Integer` | Max results. | +| `options.tbs` | `String` | Time-based filter (e.g. `qdr:d`). | +| `options.location` | `String` | Localized results. | +| `options.country` | `String` | ISO country code. | +| `options.ignoreInvalidURLs` | `Boolean` | Drop invalid URLs. | +| `options.timeout` | `Integer` | Timeout in milliseconds. | +| `options.highlights` | `Boolean` | Query-relevant highlights. Default `true` server-side. | +| `options.scrapeOptions` | `ScrapeOptions` | Scrape each result. See Scrape parameters. | +| `options.integration` | `String` | Integration identifier. | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +- `client.scrape(url)` → `Document` +- `client.scrape(url, options)` → `Document` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.JsonFormat; +import com.firecrawl.models.Document; + +ScrapeOptions options = ScrapeOptions.builder() + .formats(List.of( + "markdown", + "links", + JsonFormat.builder().prompt("Extract plan names and prices.").build() + )) + .onlyMainContent(true) + .waitFor(1000) + .actions(List.of( + Map.of("type", "click", "selector", "#accept"), + Map.of("type", "wait", "milliseconds", 750), + Map.of("type", "scrape") + )) + .build(); + +Document doc = client.scrape("https://example.com/pricing", options); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `String` | URL to scrape. | +| `options.formats` | `List` | Format strings or format config objects (`JsonFormat`, `QuestionFormat`, `HighlightsFormat`). | +| `options.headers` | `Map` | Custom request headers. | +| `options.includeTags` | `List` | Only include specific HTML tags. | +| `options.excludeTags` | `List` | Exclude specific HTML tags. | +| `options.onlyMainContent` | `Boolean` | Strip nav, footer, boilerplate. | +| `options.timeout` | `Integer` | Timeout in milliseconds. | +| `options.waitFor` | `Integer` | Wait for page render (milliseconds). | +| `options.mobile` | `Boolean` | Mobile viewport. | +| `options.parsers` | `List` | `"pdf"` or `PdfParser` object (`{maxPages, pages, blocks, pageMarkers}`). | +| `options.actions` | `List>` | Pre-scrape actions: `wait`, `screenshot`, `click`, `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. | +| `options.location` | `LocationConfig` | `{country, languages}`. | +| `options.skipTlsVerification` | `Boolean` | Skip TLS verification. | +| `options.removeBase64Images` | `Boolean` | Drop base64 images from markdown. | +| `options.blockAds` | `Boolean` | Ad and cookie popup blocking. | +| `options.proxy` | `String` | `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or custom URL. | +| `options.maxAge` | `Long` | Max age of cached data (milliseconds). | +| `options.storeInCache` | `Boolean` | Cache the result. | +| `options.lockdown` | `Boolean` | Serve only cached results. | +| `options.redactPII` | `Boolean` | Redact PII from content. | +| `options.auditMetadata` | `AuditMetadata` | `{username}` for SIEM logging. | +| `options.integration` | `String` | Integration identifier. | + +**Format strings:** `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"screenshot"`, `"json"`, `"summary"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"audio"`, `"video"`. + +**Format config objects:** +- `JsonFormat.builder().prompt("...").schema(jsonSchema).build()` +- `QuestionFormat.builder().question("...").build()` +- `HighlightsFormat.builder().query("...").build()` + +## Interact + +### Why use it + +Execute code in the browser session tied to a scrape job. The Java SDK supports code-based interactions only (no `prompt` parameter). + +### Preferred SDK method + +- `client.interact(jobId, code)` — uses default language `"node"` +- `client.interact(jobId, code, language, timeout)` — timeout in seconds (1–300) + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build()); + +String jobId = ((Map) doc.getMetadata()).get("scrapeId").toString(); + +BrowserExecuteResponse result = client.interact( + jobId, + "console.log(await page.title());", + "node", + 60 +); + +// Stop the session when done: +client.stopInteractiveBrowser(jobId); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `String` | Scrape job ID. | +| `code` | `String` | Code to execute in the browser session. Required. | +| `language` | `String` | `"python"`, `"node"`, or `"bash"`. Default `"node"`. | +| `timeout` | `Integer` | Execution timeout in seconds (1–300). Server default 30 when omitted. | +| `origin` | `String` | Optional origin label (5-arg overload only). | + +All sync methods have `...Async` `CompletableFuture` counterparts. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `deleteScrapeBrowser` → `stopInteractiveBrowser`. Async equivalents follow the same pattern. +- The Java SDK exposes **code-based interactions only** — there is no `prompt` parameter on `interact` (unlike Node.js, Python, and Rust SDKs). +- Options fields are `List` / `Map` based — callers pass raw strings, maps, or typed model objects (`JsonFormat`, `PdfParser`, etc.). +- `origin` is auto-injected as `"java-sdk@"` via `putIfAbsent`. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchData.java` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..31cdce004 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,218 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl` v4.41.0) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```bash +npm install firecrawl +``` + +Requires Node.js >= 22. + +## Authenticate + +```ts +import Firecrawl from "firecrawl"; + +const client = new Firecrawl({ + apiKey: process.env.FIRECRAWL_API_KEY, +}); +``` + +Constructor options: `apiKey` (falls back to `FIRECRAWL_API_KEY` env var), `apiUrl` (falls back to `FIRECRAWL_API_URL`, default `https://api.firecrawl.dev`), `timeoutMs`, `maxRetries`, `backoffFactor`. Scrape, search, and interact work without an API key (keyless free tier, rate-limited by IP). + +## When To Use What + +- **search**: use when you start with a query and need discovery. +- **scrape**: use when you already have a URL and want page content. +- **interact**: use when the page needs clicks, forms, or post-scrape browser actions. For multi-step interactive flows, prefer `interact` over scrape-time `actions`. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options?)` → `Promise` + +### Example + +```ts +const results = await client.search("site:docs.firecrawl.dev webhook retries", { + sources: ["web", "news"], + limit: 10, + scrapeOptions: { + formats: ["markdown"], + onlyMainContent: true, + }, +}); + +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Web results are in `result.web`, news in `result.news`, images in `result.images`. + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | Search query. Use `site:example.com` to limit to a domain. | +| `options.sources` | `Array<"web" \| "news" \| "images" \| "alexandria" \| {type: ...}>` | Sources to search. | +| `options.categories` | `Array<"github" \| "research" \| "pdf" \| "developer" \| {type: ...}>` | Filter by category. `"research"` is a web-domain filter, not the paper index. | +| `options.includeDomains` | `string[]` | Restrict to these domains. Cannot combine with `excludeDomains`. | +| `options.excludeDomains` | `string[]` | Exclude these domains. Cannot combine with `includeDomains`. | +| `options.limit` | `number` | Max results. Must be > 0. | +| `options.tbs` | `string` | Time-based filter (e.g. `qdr:d`, `qdr:w`, `sbd:1,qdr:m`). | +| `options.location` | `string` | Localized results (e.g. `"San Francisco,California,United States"`). | +| `options.country` | `string` | ISO 3166-1 alpha-2 country code. | +| `options.ignoreInvalidURLs` | `boolean` | Drop URLs that cannot be scraped by other endpoints. | +| `options.timeout` | `number` | Timeout in milliseconds. | +| `options.highlights` | `boolean` | Query-relevant highlights. Defaults to `true` server-side. | +| `options.domainTools` | `boolean` | Include Alexandria domain-matched tools in results. | +| `options.toolDetail` | `"compact" \| "summary" \| "full"` | Tool metadata verbosity. Default `"compact"`. | +| `options.scrapeOptions` | `ScrapeOptions` | Scrape each search result. See Scrape parameters. | +| `options.enterprise` | `Array<"default" \| "anon" \| "zdr">` | Enterprise ZDR options. | +| `options.threatProtection` | `ThreatProtectionOptions` | Enterprise per-request threat protection override. | +| `options.integration` | `string` | Integration identifier. | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options?)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com/pricing", { + formats: [ + "markdown", + "links", + { type: "json", prompt: "Extract plan names and prices." }, + ], + onlyMainContent: true, + waitFor: 1000, + actions: [ + { type: "click", selector: "#accept" }, + { type: "wait", milliseconds: 750 }, + { type: "scrape" }, + ], +}); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | URL to scrape. | +| `options.formats` | `FormatOption[]` | Output formats. See format types below. | +| `options.headers` | `Record` | Custom request headers. | +| `options.includeTags` | `string[]` | Only include specific HTML tags. | +| `options.excludeTags` | `string[]` | Exclude specific HTML tags. | +| `options.onlyMainContent` | `boolean` | Strip nav, footer, boilerplate. | +| `options.timeout` | `number` | Timeout in milliseconds. | +| `options.waitFor` | `number` | Wait for page render (milliseconds). | +| `options.mobile` | `boolean` | Mobile viewport. | +| `options.parsers` | `Array` | File parsing controls. `PDFParser`: `{type: "pdf", mode?, maxPages?, pages?, blocks?, pageMarkers?}`. | +| `options.actions` | `ActionOption[]` | Pre-scrape actions: `wait`, `screenshot`, `click`, `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. | +| `options.location` | `{country?, languages?}` | Geo/language-aware scraping. | +| `options.skipTlsVerification` | `boolean` | Skip TLS verification. | +| `options.removeBase64Images` | `boolean` | Drop base64 images from markdown. | +| `options.fastMode` | `boolean` | Faster scrapes with reduced fidelity. | +| `options.blockAds` | `boolean` | Ad and cookie popup blocking. | +| `options.proxy` | `"basic" \| "stealth" \| "enhanced" \| "auto" \| string` | Proxy control. | +| `options.maxAge` | `number` | Max age of cached data (milliseconds). | +| `options.minAge` | `number` | Minimum age of cached data (milliseconds). | +| `options.storeInCache` | `boolean` | Cache the result. | +| `options.lockdown` | `boolean` | Serve only cached results, never hit target URL. | +| `options.redactPII` | `boolean \| RedactPIIOptions` | Redact PII from markdown. | +| `options.profile` | `{name, saveChanges?}` | Persistent browser profile. | +| `options.domainTools` | `boolean` | Include Alexandria domain-matched tools. | +| `options.toolDetail` | `"compact" \| "summary" \| "full"` | Tool metadata verbosity. Default `"summary"` for scrape. | +| `options.auditMetadata` | `{username}` | SIEM logging attribution. | +| `options.threatProtection` | `ThreatProtectionOptions` | Enterprise threat protection override. | +| `options.integration` | `string` | Integration identifier. | + +**Format types:** + +String formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. + +Object formats (require `type` plus additional fields): +- `{type: "json", prompt?, schema?}` — at least one of `prompt` or `schema` required. Zod schemas auto-converted. +- `{type: "question", question}` — question-answer extraction. +- `{type: "highlights", query}` — relevant source-text extraction. +- `{type: "screenshot", fullPage?, quality?, viewport?}` — screenshot with options. +- `{type: "changeTracking", modes, schema?, prompt?, tag?}` — `modes` required: `"git-diff"` and/or `"json"`. +- `{type: "attributes", selectors}` — `selectors`: `Array<{selector, attribute}>`. + +## Interact + +### Why use it + +Control the browser session tied to a scrape job. Supports code execution or natural-language prompts. Requires a `scrapeId` from a prior scrape. + +### Preferred SDK method + +`client.interact(jobId, args)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { formats: ["markdown"] }); +const jobId = doc.metadata?.scrapeId; + +const result = await client.interact(jobId, { + prompt: "Click the pricing tab and summarize the plans.", +}); + +// Or use code execution: +const codeResult = await client.interact(jobId, { + code: "console.log(await page.title());", + language: "node", + timeout: 60, +}); + +// Stop the session when done: +await client.stopInteraction(jobId); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `string` | Scrape job ID from `document.metadata.scrapeId`. | +| `args.code` | `string` | Code to run in the browser session. | +| `args.prompt` | `string` | Natural-language instruction for the browser agent. | +| `args.language` | `"python" \| "node" \| "bash"` | Runtime language. Default `"node"`. | +| `args.timeout` | `number` | Execution timeout in seconds. | + +At least one of `code` or `prompt` must be provided. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `stopInteractiveBrowser` and `deleteScrapeBrowser` → `stopInteraction`. +- The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`. +- Zod schemas passed in `formats` (for `json` or `changeTracking`) are auto-converted to JSON Schema. +- `search()` does not return `.data` — access `.web`, `.news`, `.images`, `.tools` instead. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/package.json` +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..6f38a5150 --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,216 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl-py`) and the v2 OpenAPI spec. Method names, parameters, and types match the v2 client. + +## Install + +```bash +pip install firecrawl-py +``` + +Requires Python >= 3.8. + +## Authenticate + +```python +import os +from firecrawl import Firecrawl + +client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) +``` + +Constructor parameters: `api_key` (falls back to `FIRECRAWL_API_KEY` env var), `api_url` (default `https://api.firecrawl.dev`), `timeout`, `max_retries` (default 3), `backoff_factor` (default 0.5). An async client is also available: `from firecrawl import AsyncFirecrawl`. Scrape, search, and interact work without an API key (keyless free tier, rate-limited by IP). + +## When To Use What + +- **search**: use when you start with a query and need discovery. +- **scrape**: use when you already have a URL and want page content. +- **interact**: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, **options)` → `SearchData` + +### Example + +```python +results = client.search( + "site:docs.firecrawl.dev webhook retries", + sources=["web", "news"], + limit=10, + scrape_options=ScrapeOptions( + formats=["markdown"], + only_main_content=True, + ), +) + +for item in results.web or []: + print(getattr(item, "url", None), getattr(item, "title", None)) +``` + +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Web results are in `result.web`, news in `result.news`, images in `result.images`. + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `str` | Search query. Use `site:example.com` to limit to a domain. | +| `sources` | `list` | Sources to search: `"web"`, `"news"`, `"images"`, or `Source` objects. | +| `categories` | `list` | Filter by category: `"github"`, `"research"`, `"pdf"`, `"developer"`, or `Category` objects. | +| `include_domains` | `list[str]` | Restrict to these domains. Cannot combine with `exclude_domains`. | +| `exclude_domains` | `list[str]` | Exclude these domains. Cannot combine with `include_domains`. | +| `limit` | `int` | Max results. SDK model default `5`. | +| `tbs` | `str` | Time-based filter (e.g. `qdr:d`, `qdr:w`). | +| `location` | `str` | Localized results. | +| `country` | `str` | ISO country code for geo-targeting. | +| `ignore_invalid_urls` | `bool` | Drop invalid URLs. | +| `timeout` | `int` | Timeout in milliseconds. SDK model default `300000`. | +| `highlights` | `bool` | Query-relevant highlights. Default `True` server-side. | +| `domain_tools` | `bool` | Include Alexandria domain-matched tools. | +| `tool_detail` | `"compact" \| "summary" \| "full"` | Tool metadata verbosity. | +| `scrape_options` | `ScrapeOptions` | Scrape each result. See Scrape parameters. | +| `enterprise` | `list[str]` | Enterprise ZDR: `["zdr"]` or `["anon"]`. | +| `threat_protection` | `ThreatProtectionOptions` | Enterprise per-request override. | +| `integration` | `str` | Integration identifier. | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`client.scrape(url, **options)` → `Document` + +### Example + +```python +doc = client.scrape( + "https://example.com/pricing", + formats=[ + "markdown", + "links", + {"type": "json", "prompt": "Extract plan names and prices."}, + ], + only_main_content=True, + wait_for=1000, + actions=[ + {"type": "click", "selector": "#accept"}, + {"type": "wait", "milliseconds": 750}, + {"type": "scrape"}, + ], +) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `str` | URL to scrape. | +| `formats` | `list` | Output formats. See format types below. | +| `headers` | `dict[str, str]` | Custom request headers. | +| `include_tags` | `list[str]` | Only include specific HTML tags. | +| `exclude_tags` | `list[str]` | Exclude specific HTML tags. | +| `only_main_content` | `bool` | Strip nav, footer, boilerplate. | +| `timeout` | `int` | Timeout in milliseconds. | +| `wait_for` | `int` | Wait for page render (milliseconds). | +| `mobile` | `bool` | Mobile viewport. | +| `parsers` | `list` | File parsing controls. E.g. `"pdf"` or `{"type": "pdf", "mode": "auto", "max_pages": 5}`. | +| `actions` | `list[dict]` | Pre-scrape actions: `wait`, `screenshot`, `click`, `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. | +| `location` | `dict` | `{"country": "US", "languages": ["en-US"]}`. | +| `skip_tls_verification` | `bool` | Skip TLS verification. | +| `remove_base64_images` | `bool` | Drop base64 images from markdown. | +| `fast_mode` | `bool` | Faster scrapes with reduced fidelity. | +| `block_ads` | `bool` | Ad and cookie popup blocking. | +| `proxy` | `str` | `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `max_age` | `int` | Max age of cached data (milliseconds). | +| `min_age` | `int` | Minimum age of cached data (milliseconds). Only on `ScrapeOptions` within `scrape_options` for search. | +| `store_in_cache` | `bool` | Cache the result. | +| `lockdown` | `bool` | Serve only cached results, never hit target URL. | +| `profile` | `dict` | `{"name": "session-name", "save_changes": True}`. Persistent browser profile. | +| `domain_tools` | `bool` | Include Alexandria domain-matched tools. | +| `tool_detail` | `"compact" \| "summary" \| "full"` | Tool metadata verbosity. Default `"summary"` for scrape. | +| `audit_metadata` | `AuditMetadata` | `{"username": "..."}` for SIEM logging. | +| `threat_protection` | `ThreatProtectionOptions` | Enterprise threat protection override. | +| `integration` | `str` | Integration identifier. | + +**Format types:** + +String formats: `"markdown"`, `"html"`, `"rawHtml"` (or `"raw_html"`), `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"` (or `"change_tracking"`), `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. + +Object formats: +- `{"type": "json", "prompt": "...", "schema": {...}}` — at least one of `prompt` or `schema` required. +- `{"type": "question", "question": "..."}` — question-answer extraction. +- `{"type": "highlights", "query": "..."}` — relevant source-text extraction. +- `{"type": "screenshot", "full_page": True, "quality": 80, "viewport": {"width": 1280, "height": 720}}`. +- `{"type": "changeTracking", "modes": ["git-diff"], "tag": "..."}` — `modes` required. +- `{"type": "attributes", "selectors": [{"selector": "a", "attribute": "href"}]}`. + +## Interact + +### Why use it + +Control the browser session tied to a scrape job. Supports code execution or natural-language prompts. Requires a `scrape_id` from a prior scrape. + +### Preferred SDK method + +`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None)` + +### Example + +```python +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.scrape_id if doc.metadata else None + +result = client.interact(job_id, prompt="Click the pricing tab and summarize the plans.") + +# Or use code execution: +code_result = client.interact( + job_id, + code="print(await page.title())", + language="python", + timeout=60, +) + +# Stop the session when done: +client.stop_interaction(job_id) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `str` | Scrape job ID from `document.metadata.scrape_id`. | +| `code` | `str` | Code to run in the browser session. | +| `prompt` | `str` | Natural-language instruction for the browser agent. Keyword-only. | +| `language` | `str` | `"python"`, `"node"`, or `"bash"`. Default `"node"`. | +| `timeout` | `int` | Execution timeout in seconds. | + +At least one of `code` or `prompt` must be provided. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. +- The top-level `Firecrawl` client exposes v2 methods directly; v1 remains under `client.v1`. +- Parameters accept both camelCase and snake_case (e.g. `"rawHtml"` or `"raw_html"` in formats). +- `search()` does not return `.data` — access `.web`, `.news`, `.images` instead. + +## Source Of Truth + +- `firecrawl/apps/python-sdk/pyproject.toml` +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..221f743dd --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,232 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl` crate v2.21.0) and the v2 OpenAPI spec. + +## Install + +```bash +cargo add firecrawl +``` + +Or add to `Cargo.toml`: + +```toml +[dependencies] +firecrawl = "2.21" +``` + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-your-api-key")?; + +// Self-hosted: +// let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +``` + +Scrape, search, and interact work without an API key (keyless free tier, rate-limited by IP). + +## When To Use What + +- **search**: use when you start with a query and need discovery. +- **scrape**: use when you already have a URL and want page content. +- **interact**: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`. + +### Preferred SDK method + +`client.search(query, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, SearchSource, ScrapeOptions, Format}; + +let results = client + .search("site:docs.firecrawl.dev webhook retries", SearchOptions { + sources: Some(vec![SearchSource::Web, SearchSource::News]), + limit: Some(10), + scrape_options: Some(ScrapeOptions { + formats: Some(vec![Format::Markdown]), + only_main_content: Some(true), + ..Default::default() + }), + ..Default::default() + }) + .await?; + +if let Some(web) = &results.data.web { + for item in web { + // Each item is SearchResultOrDocument::WebResult or ::Document + } +} +``` + +### Parameters + +| Field | Type | Description | +|---|---|---| +| `query` | `impl AsRef` | Search query. Use `site:example.com` to limit to a domain. | +| `options.sources` | `Option>` | `Web`, `News`, `Images`, `Alexandria`. | +| `options.categories` | `Option>` | `Github`, `Research`, `Pdf`. | +| `options.include_domains` | `Option>` | Restrict to these domains. | +| `options.exclude_domains` | `Option>` | Exclude these domains. | +| `options.limit` | `Option` | Max results. Default 5, max 20. | +| `options.tbs` | `Option` | Time-based filter (e.g. `qdr:d`). | +| `options.location` | `Option` | Localized results. | +| `options.country` | `Option` | ISO country code. | +| `options.ignore_invalid_urls` | `Option` | Drop invalid URLs. | +| `options.timeout` | `Option` | Timeout in milliseconds. | +| `options.highlights` | `Option` | Query-relevant highlights. Default `true`. | +| `options.domain_tools` | `Option` | Include Alexandria domain-matched tools. | +| `options.tool_detail` | `Option` | `Compact`, `Summary`, or `Full`. | +| `options.scrape_options` | `Option` | Scrape each result. See Scrape parameters. | +| `options.integration` | `Option` | Integration identifier. | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, JsonOptions, Action}; + +let doc = client + .scrape("https://example.com/pricing", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links, Format::Json]), + json_options: Some(JsonOptions { + prompt: Some("Extract plan names and prices.".to_string()), + ..Default::default() + }), + only_main_content: Some(true), + wait_for: Some(1000), + actions: Some(vec![ + Action::Click { selector: "#accept".to_string() }, + Action::Wait { milliseconds: Some(750), selector: None }, + Action::Scrape, + ]), + ..Default::default() + }) + .await?; +``` + +### Parameters + +| Field | Type | Description | +|---|---|---| +| `url` | `impl AsRef` | URL to scrape. | +| `options.formats` | `Option>` | Output formats: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`. Also `Question(QuestionFormat)` and `Highlights(HighlightsFormat)`. | +| `options.headers` | `Option>` | Custom request headers. | +| `options.include_tags` | `Option>` | Only include specific HTML tags. | +| `options.exclude_tags` | `Option>` | Exclude specific HTML tags. | +| `options.only_main_content` | `Option` | Strip nav, footer, boilerplate. | +| `options.timeout` | `Option` | Timeout in milliseconds. | +| `options.wait_for` | `Option` | Wait for page render (milliseconds). | +| `options.mobile` | `Option` | Mobile viewport. | +| `options.parsers` | `Option>` | `ParserConfig::Simple("pdf")` or `ParserConfig::Pdf { .. }`. | +| `options.actions` | `Option>` | Pre-scrape actions: `Wait`, `Screenshot`, `Click`, `Write`, `Press`, `Scroll`, `Scrape`, `ExecuteJavascript`, `Pdf`. | +| `options.location` | `Option` | `{country, languages}`. | +| `options.skip_tls_verification` | `Option` | Skip TLS verification. | +| `options.remove_base64_images` | `Option` | Drop base64 images from markdown. | +| `options.fast_mode` | `Option` | Faster scrapes with reduced fidelity. | +| `options.block_ads` | `Option` | Ad and cookie popup blocking. | +| `options.proxy` | `Option` | `Basic`, `Stealth`, `Enhanced`, `Auto`. | +| `options.max_age` | `Option` | Max age of cached data (milliseconds). | +| `options.min_age` | `Option` | Minimum age of cached data (milliseconds). | +| `options.store_in_cache` | `Option` | Cache the result. | +| `options.lockdown` | `Option` | Serve only cached results. | +| `options.redact_pii` | `Option` | Redact PII from content. | +| `options.profile` | `Option` | `{name, save_changes}`. Persistent browser profile. | +| `options.domain_tools` | `Option` | Include Alexandria domain-matched tools. | +| `options.tool_detail` | `Option` | Tool metadata verbosity. | +| `options.audit_metadata` | `Option` | SIEM logging attribution. | +| `options.integration` | `Option` | Integration identifier. | +| `options.json_options` | `Option` | `{schema, system_prompt, prompt}`. For `Format::Json`. | +| `options.screenshot_options` | `Option` | `{full_page, quality, viewport}`. For `Format::Screenshot`. | +| `options.change_tracking_options` | `Option` | `{modes, schema, prompt, tag}`. Modes: `GitDiff`, `Json`. | +| `options.attribute_selectors` | `Option>` | `{selector, attribute}`. For `Format::Attributes`. | + +## Interact + +### Why use it + +Control the browser session tied to a scrape job. Supports code execution or natural-language prompts. Requires a `scrapeId` from a prior scrape. + +### Preferred SDK method + +`client.interact(job_id, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, ScrapeExecuteOptions, Format}; + +let doc = client.scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() +}).await?; + +let job_id = doc.metadata + .and_then(|m| m.scrape_id) + .expect("Missing scrapeId"); + +let result = client + .interact(&job_id, ScrapeExecuteOptions { + prompt: Some("Click the pricing tab and summarize the plans.".to_string()), + ..Default::default() + }) + .await?; + +// Stop the session when done: +client.stop_interaction(&job_id).await?; +``` + +### Parameters + +| Field | Type | Description | +|---|---|---| +| `job_id` | `impl AsRef` | Scrape job ID from `document.metadata.scrape_id`. | +| `options.code` | `Option` | Code to run in the browser session. | +| `options.prompt` | `Option` | Natural-language instruction for the browser agent. | +| `options.language` | `Option` | `Python`, `Node`, `Bash`. Default `Node`. | +| `options.timeout` | `Option` | Execution timeout in seconds. | + +At least one of `code` or `prompt` must be provided (SDK returns `FirecrawlError::Misuse` otherwise). + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. +- All option structs use `..Default::default()` as the idiomatic builder pattern (all fields are `Option`). +- `options` parameters accept `None`, a bare struct, or `Some(struct)` via `impl Into>`. +- `origin` is auto-injected as `"rust-sdk@"` when not set. +- `search_and_scrape(query, limit)` is a convenience helper that calls `search` with default `ScrapeOptions` and returns `Vec`. +- Types are exported at crate root: `use firecrawl::Client` (not `use firecrawl::v2::Client`). + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl/apps/rust-sdk/src/lib.rs` +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl/apps/rust-sdk/src/types.rs` +- `firecrawl-docs/api-reference/v2-openapi.json`