From a9aa67c49c5d0670010eca81e61aaa1e7491ce4b Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 16 Sep 2026 20:33:30 +0000 Subject: [PATCH 1/4] docs: add Safe Mode enterprise page Safe Mode had no documentation anywhere in this repo. A repo-wide search for "safe mode", "safeMode", "safe-mode" and "SAFE_MODE_BLOCKED" returned zero hits, so callers hitting a 403 had nothing to read. Add features/safe-mode.mdx, modelled on features/threat-protection.mdx. It covers the organization-level posture and the tri-state safeMode request parameter, the 12 boolean controls and the allowlist with their strict defaults, what is refused with the real messages, what changes silently (the proxy downgrade, credential stripping, forced robots and the endpoints that degrade without an error), lockdown, the three error codes with their real statuses, and the credit exposure from forced threat protection. Add the nav entry under Documentation > Get Started > Enterprise in the v2 and v1 trees of the en block only. The locale trees are owned by the translation pipeline. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01GGZeJ3KNUyVqcyPGdHmKvD --- docs.json | 2 + features/safe-mode.mdx | 171 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 173 insertions(+) create mode 100644 features/safe-mode.mdx diff --git a/docs.json b/docs.json index 5d52fe578..5eaaf911d 100755 --- a/docs.json +++ b/docs.json @@ -147,6 +147,7 @@ "enterprise", "features/ip-restrictions", "features/key-restrictions", + "features/safe-mode", "features/threat-protection", "features/siem" ] @@ -710,6 +711,7 @@ "enterprise", "features/ip-restrictions", "features/key-restrictions", + "features/safe-mode", "features/threat-protection", "features/siem" ] diff --git a/features/safe-mode.mdx b/features/safe-mode.mdx new file mode 100644 index 000000000..14f7f866d --- /dev/null +++ b/features/safe-mode.mdx @@ -0,0 +1,171 @@ +--- +title: "Safe Mode" +description: "An organization-level posture that stops Firecrawl from working around a site's defenses or reaching content behind a login. Enforced server-side." +og:title: "Safe Mode | Firecrawl" +og:description: "An organization-level posture that stops Firecrawl from working around a site's defenses or reaching content behind a login. Enforced server-side." +--- + +Safe Mode restricts how Firecrawl fetches a page. It stops Firecrawl from working around a site's defenses, and it stops Firecrawl from reaching content behind a login. Firecrawl honors robots.txt, uses only basic proxies, presents itself as an automated browser, and drops any credential you send. + +Safe Mode is a posture set on the organization, not an option you choose per request. It applies to every API key and every endpoint automatically. A request cannot turn Safe Mode on, and a request cannot turn Safe Mode off unless an administrator allows per-request opt-outs. + + +Safe Mode is an enterprise feature and is gated per organization. Contact your Firecrawl account team or support@firecrawl.com to have it enabled for your account. There is no self-serve API, route, or dashboard page for the flag today. + + +## The `safeMode` request parameter + +`safeMode` is an optional boolean on the request. It has three states: + +| Value | Result | +| --- | --- | +| Omitted | Your organization's setting applies. This is the normal case. | +| `true` | Accepted if Safe Mode is on for your organization. Refused with a `403` if it is not. | +| `false` | Refused with a `403`, unless an administrator allows per-request opt-outs. | + +Send the parameter at the top level on `scrape` and `batch scrape`. Send it inside `scrapeOptions` on `crawl` and `search`. `map` and `extract` do not read it. + +```json +{ + "url": "https://example.com", + "safeMode": true +} +``` + +The parameter cannot weaken your posture on its own. Only an administrator can grant the bypass that makes `safeMode: false` work. + +## Controls + +Safe Mode has 12 boolean controls and one allowlist. An organization with no configuration gets every control at its strict default, so the whole posture is on. Each control relaxes on its own: set one to `false` and that single restriction lifts while the rest stay strict. + +| Control | Strict default | What it does | +| --- | --- | --- | +| `lockdown` | `false` | Serves cached data only and never makes an outbound request. See [Lockdown](#lockdown) below. | +| `domainControls` | `true` | Forces [Threat Protection](/features/threat-protection) on, even if your own policy is set to Off. | +| `enforceRobots` | `true` | Always honors robots.txt, and refuses the `ignoreRobotsTxt` parameter. | +| `disableStealthProxy` | `true` | Refuses the stealth and enhanced proxies, and downgrades an inherited proxy choice to `basic`. | +| `disableAuthentication` | `true` | Refuses and strips credentials: credential headers, browser profiles, login-capable actions, and credentials in the URL. | +| `disableSiteHandling` | `true` | Returns a site's own restriction instead of working around it. | +| `exposeWebdriver` | `true` | The browser advertises itself as automated. | +| `useHeadlessUserAgent` | `true` | The browser sends a headless user agent string. | +| `disablePlatformSelection` | `true` | No platform selection. | +| `disableCountrySelection` | `true` | No country selection. | +| `disableAutomaticReferrer` | `true` | No synthetic referrer. | +| `allowBypassSafeMode` | absent | When set, a request may send `safeMode: false` and opt out. | +| `allowlist` | absent | Domains that run with most controls relaxed. See below. | + +The `allowlist` is matched per URL, by domain. A match clears 8 of the controls for that URL. `lockdown` and `domainControls` still apply to an allowlisted URL. + +Ask your account team to change any of these. They are stored on the organization and are not readable or writable through the API. + +## What is refused + +These requests fail with a `403` and the error code `SAFE_MODE_BLOCKED`. The `error` string names the option that caused the refusal. + +```json +{ + "success": false, + "code": "SAFE_MODE_BLOCKED", + "error": "Safe Mode: browser profiles are not allowed for your organization (nothing behind a login)." +} +``` + +| Request option | Message | +| --- | --- | +| `proxy: "stealth"` or `proxy: "enhanced"` | `Safe Mode: stealth and enhanced proxies are not allowed for your organization. Remove the proxy option or use proxy: "basic".` | +| `ignoreRobotsTxt` | `Safe Mode: robots.txt is always honored for your organization; the ignoreRobotsTxt parameter is not allowed.` | +| A browser `profile` | `Safe Mode: browser profiles are not allowed for your organization (nothing behind a login).` | +| A `write`, `press`, or `executeJavascript` action | `Safe Mode: the write action is not allowed for your organization (nothing behind a login).` The message names the action type. | +| An `authorization`, `cookie`, or `proxy-authorization` header | `Safe Mode: the Cookie header is not allowed for your organization (nothing behind a login).` The match is case-insensitive. The message echoes your own casing. | +| `safeMode: true` on an organization without the feature | `Safe Mode is not enabled for your organization. Contact support@firecrawl.com to enable this feature.` | +| `safeMode: false` without an administrator opt-out | `Requests are not allowed to disable Safe Mode for your organization. An organization admin can allow per-request opt-outs from the Safe Mode settings.` | + + +Only a literal `"stealth"` or `"enhanced"` is refused. A request that omits `proxy` is not refused, because `proxy` defaults to `"auto"` before the check runs. That request is downgraded to `basic` in silence instead. See [What changes silently](#what-changes-silently). + + +Two more surfaces are refused outright, and their bodies carry **no `code` field**: + +- **Interactive browser sessions.** `/v2/browser` create and execute, and `/v2/scrape/{id}/interact`, return `403` with `{ "success": false, "error": "Safe Mode is enabled for your organization, which does not support interactive browser sessions." }`. +- **The whole v0 API.** v0 `scrape`, `crawl`, and `search` return `403` with `{ "error": "Safe Mode is enabled for your organization, which is not supported on the v0 API. Please update your code to use the v1 or v2 API." }`. This body has no `success` field either. Move to [v1 or v2](/migrating-from-v0). + +## What changes silently + +Some restrictions do not produce an error. Your request succeeds, and the result is different from what you asked for. This is the part callers get wrong most often, so read it before you debug a result. + +- **The proxy is downgraded to `basic`.** This applies to `auto`, `stealth`, and `enhanced`. A request that names stealth or enhanced directly is refused at the endpoint, so the downgrade reaches inherited options instead: crawl children, batch children, and search or extract sub-scrapes. The only observable signal is `metadata.proxyUsed`, which reads `"basic"`. +- **Credential headers are stripped.** `authorization`, `cookie`, and `proxy-authorization` are removed before the fetch. A request that sends one directly is refused. A job that inherits one has it stripped. +- **Credentials in the URL are stripped.** A `user:pass@host` URL loses its userinfo. Firecrawl strips it from the URL it fetches and from `unnormalizedSourceURL`, so the credential is never returned to you and never persisted. +- **Login-capable actions are dropped.** A `write`, `press`, or `executeJavascript` action is filtered out of the actions array. A browser `profile` is cleared. +- **robots.txt is always honored.** Firecrawl checks robots.txt on every non-lockdown scrape. `enforceRobots` also overrides a team `ignoreRobots` setting of `forced` on a crawl. +- **Site handling is turned off.** Firecrawl sends behavior overrides on every scrape: no site-specific workarounds, an automated browser that advertises itself, a headless user agent, no platform selection, no country selection, and no synthetic referrer. Mobile proxies and persistent browser storage are turned off. +- **Threat Protection is forced on.** `domainControls` resolves your policy with `force: true`, even if your own configuration reads Off. A blocked URL returns a `403` with `code: unsafe_domain_blocked`. See [Billing](#billing) for what this costs. + +### Endpoints that are restricted without an error + +`scrape`, `crawl`, `batch scrape`, and `map` are checked at the endpoint, so a restricted option there returns a `403`. Three paths are not: + +- **`extract`** is not checked at the endpoint. Its sub-scrapes are restricted by the worker instead. +- **`search` without `scrapeOptions`** is not checked. The check runs only when a request asks for formats and supplies `scrapeOptions`. +- **[Monitoring](/features/monitoring) runs** follow the same pattern as extract. + +These callers get a `200` with degraded results and no signal that anything was restricted. Plan for that: compare against a known baseline rather than waiting for an error. + +## Lockdown + +`lockdown` turns your organization into a cache-only reader. It supersedes every other control. A lockdown organization that sends `proxy: "stealth"`, a browser profile, and a `Cookie` header does not get `SAFE_MODE_BLOCKED`. It gets the lockdown cache result instead, because the other gates are skipped. + +- **Cache only.** Firecrawl never makes an outbound request to the target. +- **`maxAge` is forced to 2 years** when you do not set it, so older cached pages stay eligible. +- **A cache miss is an error.** See `SCRAPE_LOCKDOWN_CACHE_MISS` in the [error reference](#error-reference). +- **Crawl skips discovery.** The crawler skips both robots.txt and sitemap discovery. +- **Map is forced to index only.** +- **Search is forced to the `zdr` kind**, unless a separate `searchZDR` setting forces another kind. +- **Zero data retention applies**, including on a request that tried to bypass Safe Mode and was refused. + +[Lockdown Mode](/features/lockdown) documents the per-request `lockdown: true` parameter on `/v2/scrape`. Safe Mode can force the same behavior organization-wide, with no request parameter. + +## Error reference + +| Status | Code | When | +| --- | --- | --- | +| `403` | `SAFE_MODE_BLOCKED` | A request used a restricted option, or tried to turn Safe Mode on or off. See [What is refused](#what-is-refused). | +| `500` | `SCRAPE_SITE_RESTRICTION_BLOCKED` | The site restricts automated access and `disableSiteHandling` is on, so Firecrawl returns the site's restriction instead of working around it. Message: `This site restricts automated access to the requested content. Safe Mode is enabled for your organization, so the site's restriction is returned instead of being worked around.` | +| `404` on v2, `500` on v1 | `SCRAPE_LOCKDOWN_CACHE_MISS` | Lockdown is on and no cached copy exists. v2 `scrape` returns `404`. v1 `scrape` returns `500` for the same condition. Match on the code, not on the status. | + +`SCRAPE_SITE_RESTRICTION_BLOCKED` is terminal. It stops engine fallback, so there is nothing to retry. + + +Do not switch on `code` alone. Four Safe Mode refusals return a `403` with no `code` field at all: a v0 endpoint, an interactive browser endpoint, a `/v2/scrape/{id}/interact` call, and a per-request `threatProtection` opt-out that `domainControls` refuses. A client that reads only `code` sees `undefined`. The v0 body has no `success` field either. Read the `error` string as well as the status. + + +Safe Mode never appears in a successful response. No `safeMode` field is returned in any response body or document metadata. `metadata.proxyUsed` is the only observable signal. + +## Billing + +Safe Mode itself adds no credit charges. Forced Threat Protection is the one thing to check. + +**Most organizations pay nothing extra.** If your own Threat Protection configuration reads **Off**, Safe Mode does not charge you the +2 credits per URL scan fee. `domainControls` raises an Off policy only to list enforcement: your own whitelist, blacklist, and blocked-TLD rules are applied, and everything else is allowed with no provider call. The scan fee is charged only when a provider is consulted, so this path never reaches the charge. You still get enforcement: a blacklisted domain is still blocked. + +**One case does start paying.** If you stored a Threat Protection mode of **Normal** while the Threat Protection feature was disabled for your team, that configuration was inert and cost you nothing. Safe Mode bypasses the feature check and activates the stored mode. From then on, that organization pays **+2 credits per URL scanned**. **Zscaler** mode stays exempt, because classification runs against your own tenant. + +If this applies to you, set your Threat Protection mode to Off before Safe Mode is turned on, or budget for the scan fee. See [Threat Protection billing](/features/threat-protection#billing) and [Billing](/billing). + +## What Safe Mode does not change + +Safe Mode restricts how a page is fetched. It does not restrict what you do with the result. These are unaffected: + +- Output formats, [document parsing](/features/document-parsing), and PDF parsing. +- [JSON and LLM extraction](/features/llm-extract). +- Webhooks. +- `blockAds` and `skipTlsVerification`. +- Benign actions such as `scroll` and `wait`, and benign headers such as `user-agent`. + +`executeJavascript` is refused as one of the three login-capable actions. Page JavaScript still runs, so JavaScript-rendered content is still scraped. + +## Notes + +- Safe Mode is organization-wide. It applies to every API key and every endpoint automatically, with no way to disable enforcement from within a request. +- Turning Safe Mode on or changing a control is a support request. Contact support@firecrawl.com. +- No SDK carries the `SAFE_MODE_BLOCKED` or `SCRAPE_SITE_RESTRICTION_BLOCKED` constant yet. Compare the `code` string directly. +- Safe Mode is independent of [Key Restrictions](/features/key-restrictions) and [IP Restrictions](/features/ip-restrictions). An organization can use all three at once. From 157466ecae2787a87d82796615bcd2c235babbc2 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 16 Sep 2026 20:38:57 +0000 Subject: [PATCH 2/4] docs(safe-mode): name the controls an allowlist match clears The allowlist section said a match clears 8 controls. It did not name them, and the count was wrong. The code clears nine controls. Source: apps/api/src/lib/safe-mode.ts:229-245. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01GGZeJ3KNUyVqcyPGdHmKvD --- features/safe-mode.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/features/safe-mode.mdx b/features/safe-mode.mdx index 14f7f866d..eb44661ee 100644 --- a/features/safe-mode.mdx +++ b/features/safe-mode.mdx @@ -54,7 +54,7 @@ Safe Mode has 12 boolean controls and one allowlist. An organization with no con | `allowBypassSafeMode` | absent | When set, a request may send `safeMode: false` and opt out. | | `allowlist` | absent | Domains that run with most controls relaxed. See below. | -The `allowlist` is matched per URL, by domain. A match clears 8 of the controls for that URL. `lockdown` and `domainControls` still apply to an allowlisted URL. +The `allowlist` is matched per URL, by domain. A match clears nine controls for that URL: `enforceRobots`, `disableStealthProxy`, `disableAuthentication`, `disableSiteHandling`, `exposeWebdriver`, `useHeadlessUserAgent`, `disablePlatformSelection`, `disableCountrySelection`, and `disableAutomaticReferrer`. `lockdown` and `domainControls` still apply to an allowlisted URL. Ask your account team to change any of these. They are stored on the organization and are not readable or writable through the API. From 695869eb6090d84e08ff404d2cd0136922cd02d7 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 24 Sep 2026 14:36:51 +0000 Subject: [PATCH 3/4] docs(safe-mode): rewrite the page for users Rewrite the Safe Mode page in user-facing terms. The page now tells what Safe Mode is, what changes for requests, how a team admin configures it in Enterprise Controls, and what each error means. - Drop the self-serve availability sentence. - Remove all mentions of "stealth". - Remove the passage on browser session and v0 refusals. - Drop config key names and exact error bodies. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Hjz6hgqrjcrFMszbGrEUqY --- features/safe-mode.mdx | 215 ++++++++++++++++++++--------------------- 1 file changed, 104 insertions(+), 111 deletions(-) diff --git a/features/safe-mode.mdx b/features/safe-mode.mdx index eb44661ee..f6ee85992 100644 --- a/features/safe-mode.mdx +++ b/features/safe-mode.mdx @@ -1,171 +1,164 @@ --- title: "Safe Mode" -description: "An organization-level posture that stops Firecrawl from working around a site's defenses or reaching content behind a login. Enforced server-side." +description: "Keep your organization in Firecrawl's strictest compliance posture. Firecrawl honors each site's rules, never scrapes behind a login, and identifies itself as automated. Enforced server-side." og:title: "Safe Mode | Firecrawl" -og:description: "An organization-level posture that stops Firecrawl from working around a site's defenses or reaching content behind a login. Enforced server-side." +og:description: "Keep your organization in Firecrawl's strictest compliance posture. Firecrawl honors each site's rules, never scrapes behind a login, and identifies itself as automated. Enforced server-side." --- -Safe Mode restricts how Firecrawl fetches a page. It stops Firecrawl from working around a site's defenses, and it stops Firecrawl from reaching content behind a login. Firecrawl honors robots.txt, uses only basic proxies, presents itself as an automated browser, and drops any credential you send. +Safe Mode keeps your organization in Firecrawl's strictest compliance posture. When Safe Mode is on, Firecrawl honors each site's rules and never works around a site's restrictions. Firecrawl never scrapes content behind a login. The browser identifies itself as automated. -Safe Mode is a posture set on the organization, not an option you choose per request. It applies to every API key and every endpoint automatically. A request cannot turn Safe Mode on, and a request cannot turn Safe Mode off unless an administrator allows per-request opt-outs. +Use Safe Mode when your legal, security, or compliance policy requires a conservative scraping posture on every request. Safe Mode applies to the whole organization: every API key and every endpoint. You set it once, and each request follows it automatically. -Safe Mode is an enterprise feature and is gated per organization. Contact your Firecrawl account team or support@firecrawl.com to have it enabled for your account. There is no self-serve API, route, or dashboard page for the flag today. +Safe Mode is an enterprise feature and is gated per organization. Firecrawl turns it on for your organization. Contact your Firecrawl account team or support@firecrawl.com to have it enabled. -## The `safeMode` request parameter +## What changes for your requests -`safeMode` is an optional boolean on the request. It has three states: +Safe Mode contains a set of protections. All protections are on by default. A team admin can turn off each protection separately. See [Configuring Safe Mode](#configuring-safe-mode). -| Value | Result | -| --- | --- | -| Omitted | Your organization's setting applies. This is the normal case. | -| `true` | Accepted if Safe Mode is on for your organization. Refused with a `403` if it is not. | -| `false` | Refused with a `403`, unless an administrator allows per-request opt-outs. | +Some requests are refused with an error. Other requests succeed, but Firecrawl changes how it fetches the page. The sections below tell you which is which. -Send the parameter at the top level on `scrape` and `batch scrape`. Send it inside `scrapeOptions` on `crawl` and `search`. `map` and `extract` do not read it. +### Proxies -```json -{ - "url": "https://example.com", - "safeMode": true -} -``` +Firecrawl uses basic proxies only. It does not use enhanced or mobile proxies, and it does not escalate to an enhanced proxy automatically. -The parameter cannot weaken your posture on its own. Only an administrator can grant the bypass that makes `safeMode: false` work. +- A request that sets `proxy: "enhanced"` is refused. +- A request that omits `proxy`, or sets `proxy: "auto"`, succeeds on a basic proxy. +- Pages that Firecrawl fetches for you inside a crawl, a batch scrape, a search, or an extract also use a basic proxy. -## Controls +To see which proxy Firecrawl used, read `metadata.proxyUsed` in the response. With Safe Mode on, it is `"basic"`. -Safe Mode has 12 boolean controls and one allowlist. An organization with no configuration gets every control at its strict default, so the whole posture is on. Each control relaxes on its own: set one to `false` and that single restriction lifts while the rest stay strict. +### Logins and credentials -| Control | Strict default | What it does | -| --- | --- | --- | -| `lockdown` | `false` | Serves cached data only and never makes an outbound request. See [Lockdown](#lockdown) below. | -| `domainControls` | `true` | Forces [Threat Protection](/features/threat-protection) on, even if your own policy is set to Off. | -| `enforceRobots` | `true` | Always honors robots.txt, and refuses the `ignoreRobotsTxt` parameter. | -| `disableStealthProxy` | `true` | Refuses the stealth and enhanced proxies, and downgrades an inherited proxy choice to `basic`. | -| `disableAuthentication` | `true` | Refuses and strips credentials: credential headers, browser profiles, login-capable actions, and credentials in the URL. | -| `disableSiteHandling` | `true` | Returns a site's own restriction instead of working around it. | -| `exposeWebdriver` | `true` | The browser advertises itself as automated. | -| `useHeadlessUserAgent` | `true` | The browser sends a headless user agent string. | -| `disablePlatformSelection` | `true` | No platform selection. | -| `disableCountrySelection` | `true` | No country selection. | -| `disableAutomaticReferrer` | `true` | No synthetic referrer. | -| `allowBypassSafeMode` | absent | When set, a request may send `safeMode: false` and opt out. | -| `allowlist` | absent | Domains that run with most controls relaxed. See below. | +Firecrawl never scrapes behind a login. These requests are refused: -The `allowlist` is matched per URL, by domain. A match clears nine controls for that URL: `enforceRobots`, `disableStealthProxy`, `disableAuthentication`, `disableSiteHandling`, `exposeWebdriver`, `useHeadlessUserAgent`, `disablePlatformSelection`, `disableCountrySelection`, and `disableAutomaticReferrer`. `lockdown` and `domainControls` still apply to an allowlisted URL. +- A request with a browser `profile`. +- A request with an `Authorization`, `Cookie`, or `Proxy-Authorization` header. +- A request with a `write`, `press`, or `executeJavascript` action. These actions can enter credentials. -Ask your account team to change any of these. They are stored on the organization and are not readable or writable through the API. +Firecrawl also removes credentials from pages that it fetches for you inside a crawl, a batch scrape, a search, or an extract. It removes a user name and password from a URL such as `https://user:pass@example.com`. It never returns or stores those credentials. -## What is refused +`executeJavascript` is refused only as an action. Page JavaScript still runs, so Firecrawl still scrapes content that JavaScript renders. -These requests fail with a `403` and the error code `SAFE_MODE_BLOCKED`. The `error` string names the option that caused the refusal. +### robots.txt -```json -{ - "success": false, - "code": "SAFE_MODE_BLOCKED", - "error": "Safe Mode: browser profiles are not allowed for your organization (nothing behind a login)." -} -``` +Firecrawl honors robots.txt on every request, including single-page scrapes. A request that sets `ignoreRobotsTxt` is refused. -| Request option | Message | -| --- | --- | -| `proxy: "stealth"` or `proxy: "enhanced"` | `Safe Mode: stealth and enhanced proxies are not allowed for your organization. Remove the proxy option or use proxy: "basic".` | -| `ignoreRobotsTxt` | `Safe Mode: robots.txt is always honored for your organization; the ignoreRobotsTxt parameter is not allowed.` | -| A browser `profile` | `Safe Mode: browser profiles are not allowed for your organization (nothing behind a login).` | -| A `write`, `press`, or `executeJavascript` action | `Safe Mode: the write action is not allowed for your organization (nothing behind a login).` The message names the action type. | -| An `authorization`, `cookie`, or `proxy-authorization` header | `Safe Mode: the Cookie header is not allowed for your organization (nothing behind a login).` The match is case-insensitive. The message echoes your own casing. | -| `safeMode: true` on an organization without the feature | `Safe Mode is not enabled for your organization. Contact support@firecrawl.com to enable this feature.` | -| `safeMode: false` without an administrator opt-out | `Requests are not allowed to disable Safe Mode for your organization. An organization admin can allow per-request opt-outs from the Safe Mode settings.` | +### Site restrictions - -Only a literal `"stealth"` or `"enhanced"` is refused. A request that omits `proxy` is not refused, because `proxy` defaults to `"auto"` before the check runs. That request is downgraded to `basic` in silence instead. See [What changes silently](#what-changes-silently). - +When a site restricts automated access, Firecrawl does not work around the restriction. The request returns the `SCRAPE_SITE_RESTRICTION_BLOCKED` error instead of the page content. -Two more surfaces are refused outright, and their bodies carry **no `code` field**: +### Browser identity -- **Interactive browser sessions.** `/v2/browser` create and execute, and `/v2/scrape/{id}/interact`, return `403` with `{ "success": false, "error": "Safe Mode is enabled for your organization, which does not support interactive browser sessions." }`. -- **The whole v0 API.** v0 `scrape`, `crawl`, and `search` return `403` with `{ "error": "Safe Mode is enabled for your organization, which is not supported on the v0 API. Please update your code to use the v1 or v2 API." }`. This body has no `success` field either. Move to [v1 or v2](/migrating-from-v0). +The browser presents itself as an automated tool, not as a person: -## What changes silently +- It sends the standard signal that identifies the browser as automated. +- It sends a user agent that identifies the browser as automated. +- It reports the actual device and operating system of the browser. +- It uses a location only when your request sets one. It never selects a location automatically. +- It sends a referrer header only when your request sets one. -Some restrictions do not produce an error. Your request succeeds, and the result is different from what you asked for. This is the part callers get wrong most often, so read it before you debug a result. +### Threat Protection -- **The proxy is downgraded to `basic`.** This applies to `auto`, `stealth`, and `enhanced`. A request that names stealth or enhanced directly is refused at the endpoint, so the downgrade reaches inherited options instead: crawl children, batch children, and search or extract sub-scrapes. The only observable signal is `metadata.proxyUsed`, which reads `"basic"`. -- **Credential headers are stripped.** `authorization`, `cookie`, and `proxy-authorization` are removed before the fetch. A request that sends one directly is refused. A job that inherits one has it stripped. -- **Credentials in the URL are stripped.** A `user:pass@host` URL loses its userinfo. Firecrawl strips it from the URL it fetches and from `unnormalizedSourceURL`, so the credential is never returned to you and never persisted. -- **Login-capable actions are dropped.** A `write`, `press`, or `executeJavascript` action is filtered out of the actions array. A browser `profile` is cleared. -- **robots.txt is always honored.** Firecrawl checks robots.txt on every non-lockdown scrape. `enforceRobots` also overrides a team `ignoreRobots` setting of `forced` on a crawl. -- **Site handling is turned off.** Firecrawl sends behavior overrides on every scrape: no site-specific workarounds, an automated browser that advertises itself, a headless user agent, no platform selection, no country selection, and no synthetic referrer. Mobile proxies and persistent browser storage are turned off. -- **Threat Protection is forced on.** `domainControls` resolves your policy with `force: true`, even if your own configuration reads Off. A blocked URL returns a `403` with `code: unsafe_domain_blocked`. See [Billing](#billing) for what this costs. +Safe Mode turns on your [Threat Protection](/features/threat-protection) domain lists for every request, even when your Threat Protection mode is Off. Your blocklist, allowlist, and blocked TLDs apply. A request cannot turn Threat Protection off or lower its mode. -### Endpoints that are restricted without an error +### Requests that succeed with restricted results -`scrape`, `crawl`, `batch scrape`, and `map` are checked at the endpoint, so a restricted option there returns a `403`. Three paths are not: +Most endpoints refuse a restricted option with a `403`. These requests do not. They succeed, and Firecrawl applies the restrictions to the pages it fetches: -- **`extract`** is not checked at the endpoint. Its sub-scrapes are restricted by the worker instead. -- **`search` without `scrapeOptions`** is not checked. The check runs only when a request asks for formats and supplies `scrapeOptions`. -- **[Monitoring](/features/monitoring) runs** follow the same pattern as extract. +- `extract` requests. +- `search` requests without `scrapeOptions`. +- [Monitoring](/features/monitoring) runs. -These callers get a `200` with degraded results and no signal that anything was restricted. Plan for that: compare against a known baseline rather than waiting for an error. +These requests return a `200` with no sign that Firecrawl restricted them. If your results look incomplete, compare them with a known good result. Do not wait for an error. -## Lockdown +## Cache-only mode -`lockdown` turns your organization into a cache-only reader. It supersedes every other control. A lockdown organization that sends `proxy: "stealth"`, a browser profile, and a `Cookie` header does not get `SAFE_MODE_BLOCKED`. It gets the lockdown cache result instead, because the other gates are skipped. +Cache-only mode is the strictest Safe Mode setting. It is off by default. When it is on, Firecrawl never contacts the target website: -- **Cache only.** Firecrawl never makes an outbound request to the target. -- **`maxAge` is forced to 2 years** when you do not set it, so older cached pages stay eligible. -- **A cache miss is an error.** See `SCRAPE_LOCKDOWN_CACHE_MISS` in the [error reference](#error-reference). -- **Crawl skips discovery.** The crawler skips both robots.txt and sitemap discovery. -- **Map is forced to index only.** -- **Search is forced to the `zdr` kind**, unless a separate `searchZDR` setting forces another kind. -- **Zero data retention applies**, including on a request that tried to bypass Safe Mode and was refused. +- Firecrawl serves only pages that it already has in its cache. When you do not set `maxAge`, cached pages up to 2 years old are eligible. +- A page that is not in the cache returns the `SCRAPE_LOCKDOWN_CACHE_MISS` error. +- A crawl does not discover pages from robots.txt or sitemaps. +- A map returns results from the index only. +- Zero data retention applies to every request. -[Lockdown Mode](/features/lockdown) documents the per-request `lockdown: true` parameter on `/v2/scrape`. Safe Mode can force the same behavior organization-wide, with no request parameter. +Cache-only mode takes priority over the other protections, because Firecrawl makes no outbound request. Threat Protection domain lists still filter cached results. -## Error reference +For the same behavior on a single request, see [Lockdown Mode](/features/lockdown). + +## Configuring Safe Mode + +Team admins configure Safe Mode from [Enterprise Controls → Safe Mode](https://www.firecrawl.dev/app/enterprise-controls?tab=safe-mode) in the dashboard: + +1. Open **Enterprise Controls → Safe Mode**. +2. Turn on **Cache-only mode** if you need a guarantee that Firecrawl never contacts the target website. +3. Under **Request controls** and **Identity**, turn off any protection that you do not need. The other protections stay on. +4. Under **Governance**, add domains to the **Allowlist**, and choose whether requests can opt out of Safe Mode. +5. Select **Save changes**. + +Only team admins can change Safe Mode settings. -| Status | Code | When | -| --- | --- | --- | -| `403` | `SAFE_MODE_BLOCKED` | A request used a restricted option, or tried to turn Safe Mode on or off. See [What is refused](#what-is-refused). | -| `500` | `SCRAPE_SITE_RESTRICTION_BLOCKED` | The site restricts automated access and `disableSiteHandling` is on, so Firecrawl returns the site's restriction instead of working around it. Message: `This site restricts automated access to the requested content. Safe Mode is enabled for your organization, so the site's restriction is returned instead of being worked around.` | -| `404` on v2, `500` on v1 | `SCRAPE_LOCKDOWN_CACHE_MISS` | Lockdown is on and no cached copy exists. v2 `scrape` returns `404`. v1 `scrape` returns `500` for the same condition. Match on the code, not on the status. | +### Allowlist -`SCRAPE_SITE_RESTRICTION_BLOCKED` is terminal. It stops engine fallback, so there is nothing to retry. +The allowlist contains domains that do not get the proxy, robots.txt, login, site restriction, and browser identity protections. Enter a domain such as `example.com`, or a pattern such as `*.example.com`. The allowlist holds up to 500 entries. - -Do not switch on `code` alone. Four Safe Mode refusals return a `403` with no `code` field at all: a v0 endpoint, an interactive browser endpoint, a `/v2/scrape/{id}/interact` call, and a per-request `threatProtection` opt-out that `domainControls` refuses. A client that reads only `code` sees `undefined`. The v0 body has no `success` field either. Read the `error` string as well as the status. - +Cache-only mode and Threat Protection domain lists still apply to allowlisted domains. -Safe Mode never appears in a successful response. No `safeMode` field is returned in any response body or document metadata. `metadata.proxyUsed` is the only observable signal. +### Per-request opt-outs + +When you turn on **Allow requests to disable Safe Mode**, a single request can opt out of Safe Mode. The request sends `safeMode: false`. Your request logs record each opt-out. + +## The `safeMode` request parameter + +You do not need to send `safeMode`. Your organization's settings apply to every request automatically. The parameter has three states: + +| Value | Result | +| --- | --- | +| Omitted | Your organization's settings apply. This is the normal case. | +| `true` | Accepted when Safe Mode is on for your organization. Refused with a `403` when it is not. | +| `false` | Opts out of Safe Mode for this request. Refused with a `403` unless an admin allows per-request opt-outs. | + +Send `safeMode` at the top level on `scrape` and `batch scrape`. Send it inside `scrapeOptions` on `crawl` and `search`. + +```json +{ + "url": "https://example.com", + "safeMode": true +} +``` ## Billing -Safe Mode itself adds no credit charges. Forced Threat Protection is the one thing to check. +Safe Mode adds no credit charges. + +Safe Mode turns on your Threat Protection policy. If your Threat Protection mode is Off, Firecrawl applies only your own domain lists, and there is no scan fee. If your stored Threat Protection mode is Normal, the Normal mode scan fee of +2 credits per URL applies. Zscaler mode has no scan fee. To prevent the fee, set your Threat Protection mode to Off. See [Threat Protection billing](/features/threat-protection#billing). -**Most organizations pay nothing extra.** If your own Threat Protection configuration reads **Off**, Safe Mode does not charge you the +2 credits per URL scan fee. `domainControls` raises an Off policy only to list enforcement: your own whitelist, blacklist, and blocked-TLD rules are applied, and everything else is allowed with no provider call. The scan fee is charged only when a provider is consulted, so this path never reaches the charge. You still get enforcement: a blacklisted domain is still blocked. +## Error reference -**One case does start paying.** If you stored a Threat Protection mode of **Normal** while the Threat Protection feature was disabled for your team, that configuration was inert and cost you nothing. Safe Mode bypasses the feature check and activates the stored mode. From then on, that organization pays **+2 credits per URL scanned**. **Zscaler** mode stays exempt, because classification runs against your own tenant. +| Status | Code | When | What to do | +| --- | --- | --- | --- | +| `403` | `SAFE_MODE_BLOCKED` | The request uses an option that Safe Mode does not allow. The `error` message names the option. | Remove the option from the request, or ask an admin to change your Safe Mode settings. | +| `403` | `SAFE_MODE_BLOCKED` | The request sends `safeMode: false`, and your organization does not allow opt-outs. | Remove `safeMode: false`, or ask an admin to allow per-request opt-outs. | +| `403` | `SAFE_MODE_BLOCKED` | The request sends `safeMode: true`, and Safe Mode is not on for your organization. | Remove `safeMode`, or contact support@firecrawl.com. | +| `403` | `unsafe_domain_blocked` | Your Threat Protection policy blocks the URL. | See [Threat Protection](/features/threat-protection). | +| `500` | `SCRAPE_SITE_RESTRICTION_BLOCKED` | The site restricts automated access. | Do not retry. The result does not change. To scrape the site, an admin can turn off **Respect site restrictions** or add the domain to the allowlist. | +| `404` on v2, `500` on v1 | `SCRAPE_LOCKDOWN_CACHE_MISS` | Cache-only mode is on, and the page is not in the cache. | The page is not available in cache-only mode. v2 and v1 return different status codes for this error, so check the `code`. | -If this applies to you, set your Threat Protection mode to Off before Safe Mode is turned on, or budget for the scan fee. See [Threat Protection billing](/features/threat-protection#billing) and [Billing](/billing). +Some Safe Mode refusals return a `403` without a `code`. Check the status and the `error` message as well as the `code`. ## What Safe Mode does not change -Safe Mode restricts how a page is fetched. It does not restrict what you do with the result. These are unaffected: +Safe Mode controls how Firecrawl fetches a page. It does not control what you do with the result. These features work as usual: - Output formats, [document parsing](/features/document-parsing), and PDF parsing. - [JSON and LLM extraction](/features/llm-extract). - Webhooks. - `blockAds` and `skipTlsVerification`. -- Benign actions such as `scroll` and `wait`, and benign headers such as `user-agent`. - -`executeJavascript` is refused as one of the three login-capable actions. Page JavaScript still runs, so JavaScript-rendered content is still scraped. +- Actions such as `scroll` and `wait`, and headers such as `User-Agent`. ## Notes -- Safe Mode is organization-wide. It applies to every API key and every endpoint automatically, with no way to disable enforcement from within a request. -- Turning Safe Mode on or changing a control is a support request. Contact support@firecrawl.com. -- No SDK carries the `SAFE_MODE_BLOCKED` or `SCRAPE_SITE_RESTRICTION_BLOCKED` constant yet. Compare the `code` string directly. -- Safe Mode is independent of [Key Restrictions](/features/key-restrictions) and [IP Restrictions](/features/ip-restrictions). An organization can use all three at once. +- Safe Mode applies to every API key and every endpoint in your organization. A request cannot turn off enforcement, unless an admin allows per-request opt-outs. +- The Firecrawl SDKs do not include constants for the Safe Mode error codes yet. Compare the `code` string directly. +- Safe Mode is independent of [Key Restrictions](/features/key-restrictions) and [IP Restrictions](/features/ip-restrictions). An organization can use all three at the same time. From 0268662dc24876e2d856c2ef721bf57b0ccc8bb7 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 18:27:40 +0000 Subject: [PATCH 4/4] docs(safe-mode): condense the page to a high-level overview Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01Hjz6hgqrjcrFMszbGrEUqY --- features/safe-mode.mdx | 156 +++-------------------------------------- 1 file changed, 9 insertions(+), 147 deletions(-) diff --git a/features/safe-mode.mdx b/features/safe-mode.mdx index f6ee85992..1f59161aa 100644 --- a/features/safe-mode.mdx +++ b/features/safe-mode.mdx @@ -5,160 +5,22 @@ og:title: "Safe Mode | Firecrawl" og:description: "Keep your organization in Firecrawl's strictest compliance posture. Firecrawl honors each site's rules, never scrapes behind a login, and identifies itself as automated. Enforced server-side." --- -Safe Mode keeps your organization in Firecrawl's strictest compliance posture. When Safe Mode is on, Firecrawl honors each site's rules and never works around a site's restrictions. Firecrawl never scrapes content behind a login. The browser identifies itself as automated. - -Use Safe Mode when your legal, security, or compliance policy requires a conservative scraping posture on every request. Safe Mode applies to the whole organization: every API key and every endpoint. You set it once, and each request follows it automatically. +Safe Mode keeps your organization in Firecrawl's strictest compliance posture. Turn it on when your legal, security, or compliance policy requires a conservative scraping posture on every request. It applies to every API key and every endpoint in your organization. Safe Mode is an enterprise feature and is gated per organization. Firecrawl turns it on for your organization. Contact your Firecrawl account team or support@firecrawl.com to have it enabled. -## What changes for your requests - -Safe Mode contains a set of protections. All protections are on by default. A team admin can turn off each protection separately. See [Configuring Safe Mode](#configuring-safe-mode). - -Some requests are refused with an error. Other requests succeed, but Firecrawl changes how it fetches the page. The sections below tell you which is which. - -### Proxies - -Firecrawl uses basic proxies only. It does not use enhanced or mobile proxies, and it does not escalate to an enhanced proxy automatically. - -- A request that sets `proxy: "enhanced"` is refused. -- A request that omits `proxy`, or sets `proxy: "auto"`, succeeds on a basic proxy. -- Pages that Firecrawl fetches for you inside a crawl, a batch scrape, a search, or an extract also use a basic proxy. - -To see which proxy Firecrawl used, read `metadata.proxyUsed` in the response. With Safe Mode on, it is `"basic"`. - -### Logins and credentials - -Firecrawl never scrapes behind a login. These requests are refused: - -- A request with a browser `profile`. -- A request with an `Authorization`, `Cookie`, or `Proxy-Authorization` header. -- A request with a `write`, `press`, or `executeJavascript` action. These actions can enter credentials. - -Firecrawl also removes credentials from pages that it fetches for you inside a crawl, a batch scrape, a search, or an extract. It removes a user name and password from a URL such as `https://user:pass@example.com`. It never returns or stores those credentials. - -`executeJavascript` is refused only as an action. Page JavaScript still runs, so Firecrawl still scrapes content that JavaScript renders. - -### robots.txt - -Firecrawl honors robots.txt on every request, including single-page scrapes. A request that sets `ignoreRobotsTxt` is refused. - -### Site restrictions - -When a site restricts automated access, Firecrawl does not work around the restriction. The request returns the `SCRAPE_SITE_RESTRICTION_BLOCKED` error instead of the page content. - -### Browser identity - -The browser presents itself as an automated tool, not as a person: - -- It sends the standard signal that identifies the browser as automated. -- It sends a user agent that identifies the browser as automated. -- It reports the actual device and operating system of the browser. -- It uses a location only when your request sets one. It never selects a location automatically. -- It sends a referrer header only when your request sets one. - -### Threat Protection - -Safe Mode turns on your [Threat Protection](/features/threat-protection) domain lists for every request, even when your Threat Protection mode is Off. Your blocklist, allowlist, and blocked TLDs apply. A request cannot turn Threat Protection off or lower its mode. - -### Requests that succeed with restricted results - -Most endpoints refuse a restricted option with a `403`. These requests do not. They succeed, and Firecrawl applies the restrictions to the pages it fetches: - -- `extract` requests. -- `search` requests without `scrapeOptions`. -- [Monitoring](/features/monitoring) runs. - -These requests return a `200` with no sign that Firecrawl restricted them. If your results look incomplete, compare them with a known good result. Do not wait for an error. - -## Cache-only mode +## What changes -Cache-only mode is the strictest Safe Mode setting. It is off by default. When it is on, Firecrawl never contacts the target website: - -- Firecrawl serves only pages that it already has in its cache. When you do not set `maxAge`, cached pages up to 2 years old are eligible. -- A page that is not in the cache returns the `SCRAPE_LOCKDOWN_CACHE_MISS` error. -- A crawl does not discover pages from robots.txt or sitemaps. -- A map returns results from the index only. -- Zero data retention applies to every request. - -Cache-only mode takes priority over the other protections, because Firecrawl makes no outbound request. Threat Protection domain lists still filter cached results. - -For the same behavior on a single request, see [Lockdown Mode](/features/lockdown). +- **Basic proxies only.** Firecrawl does not use enhanced or mobile proxies. +- **No logins.** Firecrawl never scrapes behind a login and never uses credentials. +- **robots.txt is honored** on every request. +- **Site rules are respected.** Firecrawl does not work around a site's restrictions, and the browser identifies itself as automated. +- **Optional cache-only mode.** Firecrawl serves only cached pages and never contacts the target website. ## Configuring Safe Mode -Team admins configure Safe Mode from [Enterprise Controls → Safe Mode](https://www.firecrawl.dev/app/enterprise-controls?tab=safe-mode) in the dashboard: - -1. Open **Enterprise Controls → Safe Mode**. -2. Turn on **Cache-only mode** if you need a guarantee that Firecrawl never contacts the target website. -3. Under **Request controls** and **Identity**, turn off any protection that you do not need. The other protections stay on. -4. Under **Governance**, add domains to the **Allowlist**, and choose whether requests can opt out of Safe Mode. -5. Select **Save changes**. - -Only team admins can change Safe Mode settings. - -### Allowlist - -The allowlist contains domains that do not get the proxy, robots.txt, login, site restriction, and browser identity protections. Enter a domain such as `example.com`, or a pattern such as `*.example.com`. The allowlist holds up to 500 entries. - -Cache-only mode and Threat Protection domain lists still apply to allowlisted domains. - -### Per-request opt-outs - -When you turn on **Allow requests to disable Safe Mode**, a single request can opt out of Safe Mode. The request sends `safeMode: false`. Your request logs record each opt-out. - -## The `safeMode` request parameter - -You do not need to send `safeMode`. Your organization's settings apply to every request automatically. The parameter has three states: - -| Value | Result | -| --- | --- | -| Omitted | Your organization's settings apply. This is the normal case. | -| `true` | Accepted when Safe Mode is on for your organization. Refused with a `403` when it is not. | -| `false` | Opts out of Safe Mode for this request. Refused with a `403` unless an admin allows per-request opt-outs. | - -Send `safeMode` at the top level on `scrape` and `batch scrape`. Send it inside `scrapeOptions` on `crawl` and `search`. - -```json -{ - "url": "https://example.com", - "safeMode": true -} -``` - -## Billing - -Safe Mode adds no credit charges. - -Safe Mode turns on your Threat Protection policy. If your Threat Protection mode is Off, Firecrawl applies only your own domain lists, and there is no scan fee. If your stored Threat Protection mode is Normal, the Normal mode scan fee of +2 credits per URL applies. Zscaler mode has no scan fee. To prevent the fee, set your Threat Protection mode to Off. See [Threat Protection billing](/features/threat-protection#billing). - -## Error reference - -| Status | Code | When | What to do | -| --- | --- | --- | --- | -| `403` | `SAFE_MODE_BLOCKED` | The request uses an option that Safe Mode does not allow. The `error` message names the option. | Remove the option from the request, or ask an admin to change your Safe Mode settings. | -| `403` | `SAFE_MODE_BLOCKED` | The request sends `safeMode: false`, and your organization does not allow opt-outs. | Remove `safeMode: false`, or ask an admin to allow per-request opt-outs. | -| `403` | `SAFE_MODE_BLOCKED` | The request sends `safeMode: true`, and Safe Mode is not on for your organization. | Remove `safeMode`, or contact support@firecrawl.com. | -| `403` | `unsafe_domain_blocked` | Your Threat Protection policy blocks the URL. | See [Threat Protection](/features/threat-protection). | -| `500` | `SCRAPE_SITE_RESTRICTION_BLOCKED` | The site restricts automated access. | Do not retry. The result does not change. To scrape the site, an admin can turn off **Respect site restrictions** or add the domain to the allowlist. | -| `404` on v2, `500` on v1 | `SCRAPE_LOCKDOWN_CACHE_MISS` | Cache-only mode is on, and the page is not in the cache. | The page is not available in cache-only mode. v2 and v1 return different status codes for this error, so check the `code`. | - -Some Safe Mode refusals return a `403` without a `code`. Check the status and the `error` message as well as the `code`. - -## What Safe Mode does not change - -Safe Mode controls how Firecrawl fetches a page. It does not control what you do with the result. These features work as usual: - -- Output formats, [document parsing](/features/document-parsing), and PDF parsing. -- [JSON and LLM extraction](/features/llm-extract). -- Webhooks. -- `blockAds` and `skipTlsVerification`. -- Actions such as `scroll` and `wait`, and headers such as `User-Agent`. - -## Notes +Team admins turn off individual controls and manage an allowlist of domains in [Enterprise Controls → Safe Mode](https://www.firecrawl.dev/app/enterprise-controls?tab=safe-mode) in the dashboard. -- Safe Mode applies to every API key and every endpoint in your organization. A request cannot turn off enforcement, unless an admin allows per-request opt-outs. -- The Firecrawl SDKs do not include constants for the Safe Mode error codes yet. Compare the `code` string directly. -- Safe Mode is independent of [Key Restrictions](/features/key-restrictions) and [IP Restrictions](/features/ip-restrictions). An organization can use all three at the same time. +A request that Safe Mode does not allow returns a `403` with the error code `SAFE_MODE_BLOCKED`.