Document stopOnContent crawl option

advanced-scraping-guide.mdx

@@ -608,6 +608,7 @@ Patterns use Rust regex (RE2-style) syntax, so look-around and backreferences ar
 | `allowExternalLinks` | `boolean` | `false` | Follow links to external domains. |
 | `allowSubdomains` | `boolean` | `false` | Follow subdomains of the main domain. |
 | `delay` | `number` (s) | — | Delay between scrapes. Setting this forces concurrency to 1. |
+| `stopOnContent` | `string[]` | — | Stop following links from pages whose visible content contains any of these phrases (1-20, up to 500 chars each). Useful for soft-200 "No results found" pagination. Matching pages are still returned. |
 
 #### Sitemap and deduplication
 

api-reference/v2-openapi.json

@@ -1958,6 +1958,17 @@
                     "description": "Do not re-scrape the same path with different (or none) query parameters",
                     "default": false
                   },
+                  "stopOnContent": {
+                    "type": "array",
+                    "minItems": 1,
+                    "maxItems": 20,
+                    "items": {
+                      "type": "string",
+                      "minLength": 1,
+                      "maxLength": 500
+                    },
+                    "description": "Stop discovering links from pages whose visible content contains any of these phrases. Useful for pagination that returns HTTP 200 with an empty-state message (e.g. \"No results found\") instead of a 404. Matching is case-insensitive, collapses whitespace, and ignores script, style, template, and noscript content. A matching page is still scraped, returned, and billed, but its links are not followed; other branches of the crawl continue. Accepts 1-20 phrases of at most 500 characters each."
+                  },
                   "regexOnFullURL": {
                     "type": "boolean",
                     "description": "When true, includePaths and excludePaths regex patterns are matched against the full URL (including query parameters) instead of just the URL pathname. Useful when you need to filter URLs based on query strings.",
@@ -7657,7 +7668,7 @@
               },
               "crawlOptions": {
                 "type": "object",
-                "description": "Crawl options such as `limit`, `maxDepth`, `includePaths`, and `excludePaths`."
+                "description": "Crawl options such as `limit`, `maxDepth`, `includePaths`, `excludePaths`, and `stopOnContent`. Accepts the same crawler options as `POST /crawl`."
               },
               "scrapeOptions": {
                 "$ref": "#/components/schemas/ScrapeOptions"

features/crawl.mdx

@@ -306,6 +306,7 @@ Coverage is bounded by the scope parameters you set, all documented in the [conf
 - **`maxDiscoveryDepth`** caps how many link-discovery hops from the root are followed. Pages at the maximum depth are still scraped, but links found on them are not followed.
 - **`limit`** caps the number of pages, and defaults to `10000`.
 - **`ignoreQueryParameters`** avoids re-scraping the same path with different query parameters.
+- **`stopOnContent`** stops link discovery on pages whose visible content contains one of the given phrases. Use it for pagination that returns `200` with an empty-state message (for example `"No results found"`) instead of a `404`, which would otherwise be followed until `limit` is reached. The matching page is still returned and billed; only its links are not followed.
 - **robots.txt is respected** unless `ignoreRobotsTxt` is enabled (Enterprise only).
 
 `maxConcurrency` defaults to your team's concurrency limit, which is set by your plan — see [Rate limits](/rate-limits).
@@ -340,6 +341,7 @@ The full set of parameters available when submitting a crawl job:
 | `allowExternalLinks` | `boolean` | `false` | Follow links to external websites. External links are followed one hop (their own links are not crawled), and links pointing to an external site's homepage are skipped — see [External links](#external-links). |
 | `sitemap` | `string` | `"include"` | Sitemap handling: `"include"` (default), `"skip"`, or `"only"` |
 | `ignoreQueryParameters` | `boolean` | `false` | Avoid re-scraping the same path with different query parameters |
+| `stopOnContent` | `string[]` | (none) | Phrases (1-20, up to 500 characters each) that stop link discovery on a page whose visible content contains any of them. Case-insensitive and whitespace-normalized; script/style/template/noscript content is ignored. Matching pages are still returned and billed. |
 | `ignoreRobotsTxt` | `boolean` | `false` | Ignore the website's robots.txt rules. **Enterprise only** — contact support@firecrawl.com to enable. |
 | `robotsUserAgent` | `string` | (none) | Custom User-Agent string for robots.txt evaluation. When set, robots.txt is fetched with this User-Agent and rules are matched against it instead of the default. **Enterprise only** — contact support@firecrawl.com to enable. |
 | `delay` | `number` | (none) | Delay in seconds between scrapes to respect rate limits. Setting this forces concurrency to 1. |