Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 83 additions & 13 deletions agent-source-of-truth/curl.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ curl -X POST "https://api.firecrawl.dev/v2/search" \
Successful responses include `success`, `data`, optional `warning`, `id`, and `creditsUsed`.

- `data.web`, `data.images`, `data.news`: result arrays; which keys appear depends on `sources` (by default only `data.web` is populated).
- `data.tools`: array of `DiscoveredTool` objects; present when `domainTools` is enabled or an `alexandria` source is included.
- Web and news items include fields such as `title`, `url`, and (when `scrapeOptions` / formats request it) `markdown`, `html`, `rawHtml`, `links`, `screenshot`, `audio`, `video`, and `metadata`.
- Image items include fields such as `imageUrl`, `url`, and dimensions when available.
- `warning`: optional human-readable notice.
Expand All @@ -96,6 +97,7 @@ Successful responses include `success`, `data`, optional `warning`, `id`, and `c
- `{ "type": "web" }` with optional per-source `tbs` and `location`
- `{ "type": "news" }`
- `{ "type": "images" }`
- `{ "type": "alexandria" }` (free; returns tool contracts instead of web results)

- `categories`
- Type: array of typed category objects
Expand All @@ -106,7 +108,7 @@ Successful responses include `success`, `data`, optional `warning`, `id`, and `c
- `{ "type": "pdf" }`

- `limit`
- Type: integer (minimum 1, maximum 100, default 5)
- Type: integer (minimum 1, maximum 100, default 10)
- Use when: you want to cap results.

- `tbs`
Expand Down Expand Up @@ -136,6 +138,33 @@ Successful responses include `success`, `data`, optional `warning`, `id`, and `c
- `"zdr"`: end-to-end zero data retention
- `"anon"`: anonymized zero data retention

- `includeDomains`
- Type: array of strings (hostname format)
- Use when: you want to restrict results to specific domains.
- Notes: cannot be used together with `excludeDomains`.

- `excludeDomains`
- Type: array of strings (hostname format)
- Use when: you want to exclude specific domains from results.
- Notes: cannot be used together with `includeDomains`.

- `highlights`
- Type: boolean (default true)
- Use when: you want query-relevant highlights in search results.

- `safe`
- Type: boolean
- Use when: you want to filter explicit content (SafeSearch).

- `domainTools`
- Type: boolean
- Use when: you want to include tool contracts matching result domains.

- `threatProtection`
- Type: ThreatProtectionOverride object
- Use when: you need per-request threat protection settings.
- Fields: `mode` (`"off"` | `"normal"`), `riskScoreThreshold` (0-100), `blacklist` (string[]), `whitelist` (string[]), `blockedTlds` (string[]), `failurePolicy` (`"open"` | `"closed"`).

- `scrapeOptions`
- Type: object
- Use when: you want to scrape each search result (see Scrape parameters for fields).
Expand Down Expand Up @@ -221,20 +250,25 @@ Successful responses include `success` and `data`. Common `data` fields (dependi
- `"markdown"`: markdown content
- `"html"`: cleaned HTML
- `"rawHtml"`: raw HTML
- `"rawBase64"`: base64-encoded response body
- `"links"`: page links
- `"images"`: image URLs
- `"screenshot"`: screenshot output
- `"summary"`: summary output
- `"changeTracking"`: change tracking output
- `"json"`: JSON extraction
- `"branding"`: branding profile output
- `"product"`: product profile output
- `"menu"`: menu profile output
- `"audio"`: audio extraction
- `"video"`: video extraction
- Format object fields:
- Format object forms:
- `type`: one of the format strings above
- `prompt`, `schema`: JSON extraction options for `type: "json"`
- `modes`, `schema`, `prompt`, `tag`: change tracking options for `type: "changeTracking"`
- `fullPage`, `quality`, `viewport`: screenshot options for `type: "screenshot"`
- `{ "type": "question", "question": "..." }`: question-answering format (required `question`, max 10000 chars)
- `{ "type": "highlights", "query": "..." }`: highlight extraction format (required `query`, max 10000 chars)

- `headers`
- Type: object
Expand All @@ -249,11 +283,11 @@ Successful responses include `success` and `data`. Common `data` fields (dependi
- Use when: you want to exclude specific HTML tags.

- `onlyMainContent`
- Type: boolean
- Type: boolean (default true)
- Use when: you want to strip nav, footer, and other boilerplate.

- `timeout`
- Type: number
- Type: number (default 60000, minimum 1000, maximum 300000)
- Use when: you need a timeout in milliseconds.

- `waitFor`
Expand All @@ -265,9 +299,9 @@ Successful responses include `success` and `data`. Common `data` fields (dependi
- Use when: you want a mobile viewport.

- `parsers`
- Type: array of objects
- Type: array of objects (default `["pdf"]`)
- Use when: you need file parsing controls.
- Confirmed shape: `{ "type": "pdf", "mode": "fast" | "auto" | "ocr", "maxPages": number }` (`type` required; other fields optional with defaults per spec)
- Confirmed shape: `{ "type": "pdf", "mode": "fast" | "auto" | "ocr", "maxPages": number, "pages": boolean, "blocks": boolean, "pageMarkers": boolean }` (`type` required; other fields optional with defaults per spec)

- `actions`
- Type: array of action objects
Expand All @@ -288,32 +322,32 @@ Successful responses include `success` and `data`. Common `data` fields (dependi
- Use when: you need geo or language-aware scraping.

- `skipTlsVerification`
- Type: boolean
- Type: boolean (default true)
- Use when: you need to skip TLS verification.

- `removeBase64Images`
- Type: boolean
- Type: boolean (default true)
- Use when: you want to drop base64 images from markdown output.

- `blockAds`
- Type: boolean
- Type: boolean (default true)
- Use when: you want ad and cookie popup blocking.

- `proxy`
- Type: string
- Type: string (default `"auto"`)
- Use when: you need proxy control.
- Confirmed values: `"basic"`, `"enhanced"`, `"auto"`

- `maxAge`
- Type: number
- Type: number (default 172800000, i.e. 2 days)
- Use when: you want cached data up to a maximum age (milliseconds).

- `minAge`
- Type: number
- Use when: you want cached data only if it is at least this old (milliseconds).

- `storeInCache`
- Type: boolean
- Type: boolean (default true)
- Use when: you want Firecrawl to cache the result.

- `profile`
Expand All @@ -324,6 +358,38 @@ Successful responses include `success` and `data`. Common `data` fields (dependi
- Type: boolean
- Use when: you want zero data retention for this scrape.

- `lockdown`
- Type: boolean (default false)
- Use when: you want to serve from cache only, never making an outbound request. ZDR. 5 credits on cache hit, 1 on miss.

- `redactPII`
- Type: boolean or RedactPIIOptions object (default false)
- Use when: you want to redact PII from returned markdown.
- Notes: `true` for defaults. Object form: `{ "mode": "accurate" | "aggressive" | "fast", "entities": ["PERSON", "EMAIL", "PHONE", "LOCATION", "FINANCIAL", "SECRET"], "replaceStyle": "tag" | "mask" | "remove" }`.

- `auditMetadata`
- Type: object
- Use when: you need user attribution for SIEM logging.
- Fields: `username` (string, max 1024 chars).

- `domainTools`
- Type: boolean (default false)
- Use when: you want to discover Alexandria tool contracts for the scraped domain.

- `threatProtection`
- Type: ThreatProtectionOverride object
- Use when: you need per-request threat protection settings.
- Fields: `mode` (`"off"` | `"normal"`), `riskScoreThreshold` (0-100), `blacklist` (string[]), `whitelist` (string[]), `blockedTlds` (string[]), `failurePolicy` (`"open"` | `"closed"`).

- `alexandria`
- Type: AlexandriaCall or array of AlexandriaCall (1-10)
- Use when: you want to execute catalogued provider tools instead of scraping a URL.
- Each call: `{ "provider": string, "capability": string, "options": object }`.

- `onlyCleanContent`
- Type: boolean (default false, beta)
- Use when: you want an LLM-based pass to remove residual boilerplate from output.

## Interact

### Why use it
Expand Down Expand Up @@ -377,9 +443,13 @@ curl -X POST "https://api.firecrawl.dev/v2/scrape/<jobId>/interact" \
- Type: integer (seconds; minimum 1, maximum 300, default 30)
- Use when: you need an execution timeout.

- `origin` (JSON body)
- Type: string (optional)
- Use when: you want to attach an origin label to the interaction.

### Response

Successful responses include `success` plus execution fields such as `stdout`, `result` (alias of stdout), `stderr`, `exitCode`, `killed`, and `error` (nullable).
Successful responses include `success` plus execution fields such as `stdout`, `result` (alias of stdout), `stderr`, `exitCode`, `killed`, `error` (nullable), `cdpUrl`, and `interactiveLiveViewUrl`.

### DELETE /scrape/{jobId}/interact

Expand Down
36 changes: 35 additions & 1 deletion agent-source-of-truth/elixir.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Canonical Firecrawl Elixir source of truth for agents. Generated from SDK source
Add to `mix.exs`:

```elixir
{:firecrawl, "~> 1.0.0"}
{:firecrawl, "~> 1.11"}
```

## Authenticate
Expand All @@ -23,6 +23,15 @@ config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY")
{:ok, res} = Firecrawl.search_and_scrape([query: "site:docs.firecrawl.dev webhook retries"], api_key: "fc-your-api-key")
```

No key is required — the SDK falls back to a keyless free tier (rate-limited per IP). Every request body includes an `"origin"` field set to `"elixir-sdk@<version>"` for SDK telemetry.

### Common options

These options are available on all functions:

- `:api_key` -- override the API key for this request
- `:base_url` -- override the default base URL (`https://api.firecrawl.dev/v2`)

## When To Use What

- `search`: use when you start with a query and need discovery.
Expand Down Expand Up @@ -132,6 +141,19 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or
- Type: keyword list
- Use when: you want to scrape each search result (see Scrape parameters for fields).

- `include_domains`
- Type: list of strings
- Use when: you want to restrict results to specific domains.

- `exclude_domains`
- Type: list of strings
- Use when: you want to exclude specific domains.

- `highlights`
- Type: boolean
- Use when: you want query-relevant highlights.
- Notes: defaults to true.

## Scrape

### Why use it
Expand Down Expand Up @@ -302,6 +324,18 @@ Use scrape when you already have a URL and want structured content in one or mor
- Type: boolean
- Use when: you want zero data retention for this scrape.

- `audit_metadata`
- Type: keyword list (keys: `username: :string`, required)
- Use when: you need SIEM logging attribution.

- `lockdown`
- Type: boolean
- Use when: you want to serve only cached results, never scrape.

- `redact_pii`
- Type: boolean
- Use when: you want to redact PII from content.

## Interact

### Why use it
Expand Down
56 changes: 51 additions & 5 deletions agent-source-of-truth/java.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,14 @@ Maven:
<dependency>
<groupId>com.firecrawl</groupId>
<artifactId>firecrawl-java</artifactId>
<version>1.2.0</version>
<version>1.18.0</version>
</dependency>
```

Gradle:

```gradle
implementation("com.firecrawl:firecrawl-java:1.2.0")
implementation("com.firecrawl:firecrawl-java:1.18.0")
```

## Authenticate
Expand All @@ -33,6 +33,21 @@ FirecrawlClient client = FirecrawlClient.builder()
.build();
```

### Builder options

- `apiKey` — Your API key. Also read from the `FIRECRAWL_API_KEY` environment variable or the `firecrawl.apiKey` system property. When omitted entirely the client falls back to the keyless free tier.
- `apiUrl` — Base URL. Defaults to `"https://api.firecrawl.dev"`. Overridable via the `FIRECRAWL_API_URL` environment variable.
- `timeoutMs` — Request timeout in milliseconds. Default: `300_000`.
- `maxRetries` — Maximum number of retries. Default: `3`.
- `backoffFactor` — Exponential back-off multiplier. Default: `0.5`.

### Factory method

```java
// Reads FIRECRAWL_API_KEY env var or firecrawl.apiKey system property
FirecrawlClient client = FirecrawlClient.fromEnv();
```

## When To Use What

- `search`: use when you start with a query and need discovery.
Expand Down Expand Up @@ -122,7 +137,8 @@ SearchData results = client.search("site:docs.firecrawl.dev crawl webhooks", opt
- `"developer"`: Developer Index results
- `"research"`: research and academic results
- `"pdf"`: PDF-focused results
- `{type: "developer" | "research" | "pdf"}`: typed category map form
- `"github"`: GitHub results
- `{type: "developer" | "research" | "pdf" | "github"}`: typed category map form

- `options.limit`
- Type: Integer
Expand Down Expand Up @@ -152,6 +168,22 @@ SearchData results = client.search("site:docs.firecrawl.dev crawl webhooks", opt
- Type: String
- Use when: the API expects an integration identifier on the request.

- `options.includeDomains`
- Type: List&lt;String&gt;
- Use when: you want to restrict results to specific domains.

- `options.excludeDomains`
- Type: List&lt;String&gt;
- Use when: you want to exclude specific domains from results.

- `options.country`
- Type: String
- Use when: you need a country code to geo-target results.

- `options.highlights`
- Type: Boolean
- Use when: you want query-relevant highlights in results. Defaults to true.

## Scrape

### Why use it
Expand All @@ -165,7 +197,7 @@ Use scrape when you already have a URL and want structured content in one or mor

### Return value

`scrape` returns `Document`. Typical getters include `getMarkdown()`, `getHtml()`, `getRawHtml()`, `getJson()`, `getMetadata()`, `getLinks()`, `getAudio()`, `getVideo()`, and additional fields when the corresponding formats are requested.
`scrape` returns `Document`. Typical getters include `getMarkdown()`, `getHtml()`, `getRawHtml()`, `getJson()`, `getMetadata()`, `getLinks()`, `getAudio()`, `getVideo()`, `getProduct()` (Product), `getMenu()` (Menu), `getPages()` (List&lt;PdfPage&gt;), `getBlocks()` (List&lt;PdfPageBlocks&gt;), and additional fields when the corresponding formats are requested.

### Simple Example

Expand Down Expand Up @@ -327,6 +359,18 @@ Document doc = client.scrape("https://example.com/pricing", options);
- Type: String
- Use when: the API expects an integration identifier on the request.

- `options.lockdown`
- Type: Boolean
- Use when: you want to serve only previously cached results.

- `options.redactPII`
- Type: Boolean
- Use when: you want to redact PII from content.

- `options.auditMetadata`
- Type: AuditMetadata (field: `username`: String)
- Use when: you need SIEM logging attribution.

## Interact

### Why use it
Expand Down Expand Up @@ -411,10 +455,12 @@ BrowserDeleteResponse stopped = client.stopInteractiveBrowser("<scrapeJobId>");

- `origin`
- Type: String
- Use when: you need an optional origin label on the request. Prefer omitting unless your integration requires it.
- Default: `"java-sdk@1.18.0"`
- Use when: you need an optional origin label on the request for attribution. Prefer omitting unless your integration requires it.

## Notes

- **Async variants:** All main methods (`search`, `scrape`, `interact`, `stopInteractiveBrowser`) have `*Async` variants (e.g. `searchAsync`, `scrapeAsync`) that return `CompletableFuture<>`.
- Deprecated aliases: `scrapeExecute` → `interact`, `deleteScrapeBrowser` → `stopInteractiveBrowser` (and the corresponding `*Async` helpers).
- The Java SDK exposes code-based interactions only: there is no `prompt` parameter on `interact` (unlike some other language SDKs).

Expand Down
Loading
Loading