From 754dd4dfa68a935ab0a5a67e8a194377d28f2bc3 Mon Sep 17 00:00:00 2001 From: Max Loffgren Date: Wed, 30 Sep 2026 22:58:07 -0500 Subject: [PATCH 1/3] fix(mcp): keep hosted scrape read-only --- README.md | 34 ++--- docs/search-profile.md | 8 ++ src/index.ts | 65 ++++++++-- src/tool-output.ts | 4 +- tests/helpers/alexandria-metadata.mjs | 4 +- tests/helpers/exchange-mcp.mjs | 1 + tests/mcp-alexandria-terms.test.mjs | 31 +++-- tests/mcp-read-only-scrape.test.mjs | 176 ++++++++++++++++++++++++++ tests/mcp-search-profile.test.mjs | 6 +- tests/mcp-smoke.test.mjs | 22 +++- 10 files changed, 298 insertions(+), 53 deletions(-) create mode 100644 tests/mcp-read-only-scrape.test.mjs diff --git a/README.md b/README.md index 5bf08075..9084fac8 100644 --- a/README.md +++ b/README.md @@ -419,6 +419,7 @@ Scrape content from a single URL with advanced options. **Branding format:** Extracts comprehensive brand identity (colors, fonts, typography, spacing, logo, UI components) for design analysis or style replication. **Privacy:** Set `redactPII: true` to return content with personally identifiable information redacted. +**Hosted server:** On the hosted server (`CLOUD_SERVICE=true`) scrape is read-only. It takes no browser `actions`, and a named `profile` loads saved browser state without saving changes to it. To save browser state to a profile, open the page with `firecrawl_interact` (see below). **Returns:** @@ -747,17 +748,17 @@ The agent performs web searches, follows links, reads pages, and gathers data au - `enabled`, `toolkits` (up to 5 provider slugs), `maxCalls` (1 to 30), `requireApproval` (paid calls end the turn with a `pendingApproval`; needs `mode: "chat"` on the same call, even on a follow-up) - `onTermsRequired`: what to do when an Alexandria provider the agent would use needs data terms your team has not accepted. Gated providers are never called in any mode. Omitted on a follow-up keeps the previous turn's value. - `"skip"` (default): answer with accepted providers only. `exchange.skippedProviders` on the status result lists the gated providers that would have helped. - - `"ask"`: the same, plus a terms `pendingApproval` and `exchange.requiresAction` with the exact `terms/show` and `terms/accept` calls for each provider. Each provider's `digest` is always present and is `string | null`; when it is `null`, `terms/show` returns the current digest to send. + - `"ask"`: the same, plus a terms `pendingApproval` and `exchange.requiresAction` with the approval ID and provider requirements. Read terms with `terms/show`; an organization admin accepts them in the Firecrawl dashboard. - `approve`: `{ approvalId, callIds?, always? }` answers yes to the `pendingApproval` the previous turn ended on. `callIds` and `always` apply to paid-call approvals only. - `decline`: `{ approvalId }` answers no. A declined terms offer keeps those providers out of the rest of the thread. - `approve` and `decline` need `threadId`, and only one of them can be sent. -**Provider terms (ask mode):** there is no auto-accept mode. When a turn ends on a terms offer, the status result carries `pendingApproval` (`kind: "terms"`) and `exchange.requiresAction` with the `approvalId` and the exact `terms/show` and `terms/accept` calls. To use the provider: +**Provider terms (ask mode):** there is no auto-accept mode. When a turn ends on a terms offer, the status result carries `pendingApproval` (`kind: "terms"`) and `exchange.requiresAction` with the `approvalId` and provider requirements. Any `terms/accept` descriptor in that API payload is unavailable through MCP. To use the provider: 1. Show the user the terms (`terms/show` through `firecrawl_scrape` with `alexandria`). -2. Get the user's explicit consent to that provider's terms. A data request is not consent. -3. Run the `terms/accept` call through `firecrawl_scrape`. -4. Continue the same thread: call `firecrawl_agent` with the same `threadId` and `exchange.approve: { "approvalId": "..." }`. +2. Direct an organization admin to accept the terms at the provider's URL, or [data sources settings](https://www.firecrawl.dev/app/settings?tab=data-sources). A data request is not consent. +3. Wait for the admin to confirm acceptance in the dashboard. +4. Continue the same thread: call `firecrawl_agent` with the same `threadId` and `exchange.approve: { "approvalId": "..." }`. This resumes research and does not accept terms. If the user says no, call `firecrawl_agent` with the same `threadId` and `exchange.decline: { "approvalId": "..." }` instead. @@ -818,13 +819,13 @@ Then poll with `firecrawl_agent_status` using the returned job ID. } ``` -**Usage Example (continue the thread after the user accepted a provider's terms):** +**Usage Example (continue the thread after an admin confirmed dashboard acceptance):** ```json { "name": "firecrawl_agent", "arguments": { - "prompt": "I accepted the Apollo terms. Continue.", + "prompt": "The admin confirmed acceptance of the Apollo terms in the dashboard. Continue.", "threadId": "0199a1b2-0000-7000-8000-000000000031", "exchange": { "approve": { "approvalId": "0199a1b2-0000-7000-8000-000000000033" } } } @@ -867,6 +868,7 @@ Interact with a fresh URL or with a page that was already opened by `firecrawl_s - Pass `url` to scrape and open a page for interaction in one MCP call. - Pass `scrapeId` to continue interacting with an existing scraped page. - Pass exactly one of `url` or `scrapeId`, plus either `prompt` or `code`. +- To save browser state (cookies, localStorage) to a named profile, pass `url` with `scrapeOptions: { "profile": { "name": "my-profile", "saveChanges": true } }`. The state is saved when `firecrawl_interact_stop` ends the session. **Usage Example:** @@ -1113,16 +1115,14 @@ HTTP 403 and this body: The tool result relays it as an error with `structuredContent` carrying `code`, `status: 403`, `requestId`, the `requiresAction` object unchanged, and `next_actions` (`human_action_required` then `retry_same_request`). Accepting -terms is a legal act. Use the returned `nextTool` call to read the agreement through `firecrawl_scrape` -with `alexandria: [{provider: "firecrawl", capability: "terms/show", options: {provider: ""}}]`. -Present it to the user and obtain explicit authorization to bind their organization -before calling `firecrawl_scrape` with capability `terms/accept` under provider `firecrawl`. -Its options are `provider`, the exact reviewed `version` -and 64-character lowercase hexadecimal `digest`, and `confirmed: true`. A request -for data is not consent. Authority or eligibility errors may require an organization -admin to use `requiresAction.url`. No automatic acceptance or uncertain retries occur. -Send terms calls separately from execution. These are nested capabilities, not top-level MCP tools. -No credits are charged for the blocked retrieval. After confirmed acceptance, call the same +terms is a legal act, so an organization admin accepts them in the Firecrawl dashboard, not +through MCP. Use the returned `nextTool` call to read the agreement through `firecrawl_scrape` +with `alexandria: [{provider: "firecrawl", capability: "terms/show", options: {provider: ""}}]`, +sent separately from provider execution, and present it to the user. An organization admin then +accepts it at `requiresAction.url`, or at https://www.firecrawl.dev/app/settings?tab=data-sources. +`firecrawl_scrape` refuses every other `terms/*` capability, so it makes no account changes. A request +for data is not consent, and no automatic acceptance or uncertain retries occur. +No credits are charged for the blocked retrieval. After the admin confirms acceptance, call the same tool again with the identical payload and `requestId`. ### 16. Credit Usage Tool diff --git a/docs/search-profile.md b/docs/search-profile.md index b841592e..0cc4e432 100644 --- a/docs/search-profile.md +++ b/docs/search-profile.md @@ -67,6 +67,14 @@ names only tools this surface exposes. `firecrawl_find_tools` is registered the same way. Alexandria results on this surface carry no `feedbackTool` pointer, since `firecrawl_feedback` is not registered here. +`firecrawl_scrape` is read-only here (`readOnlyHint: true`): the surface runs in +hosted safe mode, so it takes no browser `actions`, and a named `profile` loads +saved browser state without saving changes to it. Provider terms can be read +with the nested `terms/show` capability. As on the full surface, an organization +admin accepts them in the dashboard: `firecrawl_scrape` refuses every other +`terms/*` capability, and terms errors link to `requiresAction.url` or +https://www.firecrawl.dev/app/settings?tab=data-sources. + ## Alexandria source `sources` entries are source names (`web`, `news`, `images`, `alexandria`) or diff --git a/src/index.ts b/src/index.ts index c11fa946..7ec67baf 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1082,6 +1082,26 @@ type TermsRequiredAction = { type ExchangeErrorContext = { tool: string; requestId?: string; providers?: string[] }; +const DATA_SOURCES_SETTINGS_URL = + 'https://www.firecrawl.dev/app/settings?tab=data-sources'; + +// An organization admin accepts provider terms in the dashboard. Through MCP, +// agents only read them with terms/show, so firecrawl_scrape changes no +// account state. +function isTermsWrite(call: { provider: string; capability: string }): boolean { + const capability = call.capability.trim().toLowerCase(); + return ( + call.provider.trim().toLowerCase() === 'firecrawl' && + capability.startsWith('terms/') && + capability !== 'terms/show' + ); +} + +function termsWriteError(): UserError { + const message = `Provider terms are accepted in the Firecrawl dashboard, not through this connection. Read them with terms/show, then ask an organization admin to accept them at ${DATA_SOURCES_SETTINGS_URL}.`; + return new UserError(message, { code: 'invalid_option', status: 400, message }); +} + function termsRequiredAction(body: unknown): TermsRequiredAction | undefined { const data = body as | { code?: unknown; requiresAction?: unknown } @@ -1116,7 +1136,7 @@ function termsRequiredError( const retryIdentity = requestId ? ` and requestId ${requestId}` : ''; const message = `Alexandria provider terms required. An organization admin must accept the ${action.terms} provider's terms (version ${action.version}) before this request can run.`; return new UserError( - `${message}\n\n1. Read the agreement with firecrawl_scrape using alexandria: {provider: "firecrawl", capability: "terms/show", options: {provider: "${action.terms}"}} and present it to the user. Only after explicit authorization to accept that exact version and digest, use firecrawl_scrape with alexandria: {provider: "firecrawl", capability: "terms/accept", options: {provider: "${action.terms}", version: "", digest: "", confirmed: true}}. Send terms calls separately from provider execution. If authority or eligibility requires a dashboard action, ask an organization admin to visit ${action.url} or https://www.firecrawl.dev/app/settings?tab=data-sources if that page is unavailable. Never infer acceptance from a data request.\n2. After they confirm, call ${context.tool} again with the identical payload${retryIdentity}. If the same terms error persists, stop and ask an organization admin to check access at https://www.firecrawl.dev/app/settings?tab=data-sources.\n\nDo not retry until acceptance is confirmed.`, + `${message}\n\n1. Read the agreement with firecrawl_scrape using alexandria: {provider: "firecrawl", capability: "terms/show", options: {provider: "${action.terms}"}} and present it to the user. Send terms/show separately from provider execution. Terms are accepted in the Firecrawl dashboard, not through this connection: ask an organization admin to accept them at ${action.url}, or ${DATA_SOURCES_SETTINGS_URL} if that page is unavailable. Never infer acceptance from a data request.\n2. After the admin confirms, call ${context.tool} again with the identical payload${retryIdentity}. If the same terms error persists, stop and ask an organization admin to check access at ${DATA_SOURCES_SETTINGS_URL}.\n\nDo not retry until acceptance is confirmed.`, { code: TERMS_REQUIRED_CODE, status: 403, @@ -1203,7 +1223,7 @@ async function relayExchangeError( : undefined; if (disabledProvider) { throw new UserError( - `${message} Read the provider terms and status using nextTool and present them to the user. Never infer acceptance from a data request. Only after explicit authorization for the reviewed version and digest may you call firecrawl_scrape with alexandria:{provider:"firecrawl",capability:"terms/accept",options:{provider:"${disabledProvider}",version:"",digest:"",confirmed:true}}. Send terms calls separately. Acceptance may not restore disabled access; an organization admin may need to review https://www.firecrawl.dev/app/settings?tab=data-sources. Retry the original request only after access is restored.`, + `${message} Read the provider terms and status using nextTool and present them to the user. Never infer acceptance from a data request. Terms are accepted in the Firecrawl dashboard, not through this connection; acceptance may not restore disabled access, and an organization admin can review access at ${DATA_SOURCES_SETTINGS_URL}. Retry the original request only after access is restored.`, { code: typeof data?.code === 'string' ? data.code : 'exchange_error', status: 403, @@ -2246,6 +2266,22 @@ const scrapeParamsSchema = z.object({ .optional(), }); +// In safe mode firecrawl_scrape and firecrawl_search are read-only, so a named +// profile they open loads saved browser state without writing it back. +// firecrawl_interact (url with scrapeOptions.profile) saves profile changes. +const readOnlyProfileSchema = z + .object({ name: z.string() }) + .describe('Loads a saved browser profile without saving changes to it.'); + +function withReadOnlyProfile( + options: Record +): Record { + const profile = options.profile as { name: string } | undefined; + return SAFE_MODE && profile + ? { ...options, profile: { name: profile.name, saveChanges: false } } + : options; +} + // firecrawl_scrape accepts either a page URL or an Exchange batch. The base // schema stays url-required because search, crawl, and monitor reuse it for // nested scrapeOptions, where `alexandria` has no meaning. @@ -2253,6 +2289,7 @@ const ALEXANDRIA_IGNORED_SCRAPE_OPTIONS = new Set(['toolDetail', 'domainTools']) const scrapeToolParamsSchema = scrapeParamsSchema .extend({ + ...(SAFE_MODE ? { profile: readOnlyProfileSchema.optional() } : {}), url: z.string().url().optional(), timeout: z.number().int().positive().optional().describe("Execution timeout in milliseconds."), requestId: z @@ -2698,7 +2735,9 @@ const scrapeTool: RegisteredTool = { name: 'firecrawl_scrape', annotations: { title: 'Firecrawl scrape', - readOnlyHint: false, // Alexandria capabilities can record provider agreement acceptance. + // Hosted scrape omits browser actions, loads profiles without saving, + // and refuses provider terms writes before execution. + readOnlyHint: SAFE_MODE, openWorldHint: true, // Accepts any user-supplied URL on the public web. destructiveHint: false, // Does not modify, delete, or write to external websites. }, @@ -2727,6 +2766,9 @@ Alexandria mode, on an authenticated session with Alexandria access: \`alexandri } & Record; if (alexandria) { assertExchangeCredential(session); + if ((Array.isArray(alexandria) ? alexandria : [alexandria]).some(isTermsWrite)) { + throw termsWriteError(); + } log.info('Executing Alexandria capabilities', { count: Array.isArray(alexandria) ? alexandria.length : 1, }); @@ -2737,7 +2779,7 @@ Alexandria mode, on an authenticated session with Alexandria access: \`alexandri const transformed = transformScrapeParams( options as Record ); - const cleaned = removeEmptyTopLevel(transformed); + const cleaned = withReadOnlyProfile(removeEmptyTopLevel(transformed)); if (cleaned.lockdown) { log.info('Scraping URL (lockdown)'); } else { @@ -2822,7 +2864,7 @@ server.addTool({ _meta: { 'anthropic/alwaysLoad': true }, annotations: { title: 'Firecrawl web search', - readOnlyHint: true, // Runs a web search and returns results; does not modify external sites. + readOnlyHint: SAFE_MODE, // Local scrapeOptions can run browser actions and save profiles. openWorldHint: true, // Searches the open web across arbitrary domains and sources. destructiveHint: false, // Query-only; no destructive side effects on external entities. }, @@ -2841,6 +2883,7 @@ For a programming question, add \`categories: ["developer"]\`; its hits return i ...searchToolBaseFields, scrapeOptions: scrapeParamsSchema .omit({ url: true }) + .extend(SAFE_MODE ? { profile: readOnlyProfileSchema } : {}) .partial() .optional() .describe('Attach page content for web results in the same call. These fetches ignore maxAge, so use firecrawl_scrape when you need a live fetch. scrapeOptions fetches web pages, never Alexandria provider tools.'), @@ -2863,8 +2906,8 @@ For a programming question, add \`categories: ["developer"]\`; its hits return i searchOpts.toolDetail ??= 'compact'; if (searchOpts.scrapeOptions) { - searchOpts.scrapeOptions = transformScrapeParams( - searchOpts.scrapeOptions as Record + searchOpts.scrapeOptions = withReadOnlyProfile( + transformScrapeParams(searchOpts.scrapeOptions as Record) ); } @@ -3008,7 +3051,7 @@ server.addTool(findToolsTool); // (no crawl, map, interact, monitor, parse or feedback references). Registered // on the search surface in place of the module-level tools above. const SEARCH_SURFACE_SCRAPE_DESCRIPTION = ` -Scrape one URL and return its content, or execute catalogued Alexandria capabilities. URL mode returns markdown by default, or HTML, links, screenshots, branding data, a targeted answer, or JSON matching a supplied schema, plus page metadata. Firecrawl may serve recently indexed content; set \`maxAge: 0\` for a live fetch. A successful response does not by itself confirm the page is still current. Browser actions can change the live page when interactive actions are enabled, and a named browser profile can load saved session data and overwrite its stored state. +Scrape one URL and return its content, or execute catalogued Alexandria capabilities. URL mode returns markdown by default, or HTML, links, screenshots, branding data, a targeted answer, or JSON matching a supplied schema, plus page metadata. Firecrawl may serve recently indexed content; set \`maxAge: 0\` for a live fetch. A successful response does not by itself confirm the page is still current. ${SAFE_MODE ? 'A named browser profile loads saved session data without saving changes to it.' : 'Browser actions can change the live page, and a named browser profile can load saved session data and overwrite its stored state.'} \`firecrawl_search\` with \`sources\` unset and \`firecrawl_find_tools\` can discover providers for the same fields across several pages; a matching Alexandria provider returns typed records in one call. @@ -3838,7 +3881,7 @@ const agentExchangeSchema = z }) .optional() .describe( - 'Answer yes to the pendingApproval the previous turn of this thread ended on (its id, also exchange.requiresAction.approvalId). Needs threadId. For a terms offer, send it only after the user explicitly agreed and terms/accept succeeded; callIds and always are ignored on terms offers. For paid calls, callIds picks a subset (default all) and always stops asking for the rest of the thread.' + 'Answer yes to the pendingApproval the previous turn of this thread ended on (its id, also exchange.requiresAction.approvalId). Needs threadId. For a terms offer, send it only after an organization admin confirms acceptance in the Firecrawl dashboard; this does not accept terms. callIds and always are ignored on terms offers. For paid calls, callIds picks a subset (default all) and always stops asking for the rest of the thread.' ), decline: z .strictObject({ approvalId: z.string().uuid() }) @@ -3850,7 +3893,7 @@ const agentExchangeSchema = z .enum(['skip', 'ask']) .optional() .describe( - 'What to do when a provider the agent would use needs data terms the team has not accepted. Gated providers are never called. "skip" (default): answer with accepted providers and list the rest in exchange.skippedProviders. "ask": the same, plus a terms pendingApproval and exchange.requiresAction with the terms/show and terms/accept calls. Each provider digest is string | null and always present; when null, terms/show returns it. There is no auto-accept. Omitted on a follow-up keeps the previous turn\'s value.' + 'What to do when a provider the agent would use needs data terms the team has not accepted. Gated providers are never called. "skip" (default): answer with accepted providers and list the rest in exchange.skippedProviders. "ask": the same, plus a terms pendingApproval and exchange.requiresAction. Read terms with terms/show; an organization admin accepts them in the Firecrawl dashboard. There is no auto-accept. Omitted on a follow-up keeps the previous turn\'s value.' ), }) .describe( @@ -3872,7 +3915,7 @@ This call returns only a job ID, not the research result. Read the job with \`fi The job also returns a \`threadId\`. To continue that thread, pass it with a follow-up \`prompt\`; omitted \`mode\`, \`urls\`, \`schema\` and exchange settings carry over from the previous turn. -The agent only calls Alexandria providers whose data terms the team has accepted. The status result's \`exchange.skippedProviders\` lists gated providers that would have helped. With \`exchange.onTermsRequired\` "ask", a terms offer ends the turn: \`pendingApproval\` (kind "terms") and \`exchange.requiresAction\` carry the \`approvalId\` and the exact terms/show and terms/accept calls. Show the user the terms, get their EXPLICIT consent, run terms/accept through \`firecrawl_scrape\`, then call \`firecrawl_agent\` with the same \`threadId\` and \`exchange.approve: {approvalId}\`. If they decline, send \`exchange.decline: {approvalId}\` instead. Never call terms/accept without that consent; a data request is not consent. +The agent only calls Alexandria providers whose terms the team has accepted. \`exchange.skippedProviders\` lists gated providers. With \`exchange.onTermsRequired\` "ask", a terms \`pendingApproval\` and \`exchange.requiresAction\` carry the \`approvalId\` and provider requirements. Read terms/show through \`firecrawl_scrape\`. An organization admin must accept terms in the Firecrawl dashboard at the provider URL or ${DATA_SOURCES_SETTINGS_URL}. Ignore any terms/accept call in the API response. Only after the admin confirms acceptance, resume with the same \`threadId\` and \`exchange.approve: {approvalId}\`; this does not accept terms. To decline, use \`exchange.decline: {approvalId}\`. Never infer acceptance from a data request. `, outputSchema: agentOutputSchema, parameters: z.object({ diff --git a/src/tool-output.ts b/src/tool-output.ts index 3222882f..b035020e 100644 --- a/src/tool-output.ts +++ b/src/tool-output.ts @@ -248,10 +248,10 @@ export const agentStatusOutputSchema = z message: unknown('The agent\'s reply; in chat mode, the short answer to a follow-up.'), suggestions: unknown('Follow-ups the agent offers; send one as the prompt of the next turn with this threadId.'), exchange: unknown( - 'What the job did with Alexandria providers: onTermsRequired, paidCalls, creditsUsed, skippedProviders (gated providers that would have helped), and requiresAction (approvalId plus the terms/show and terms/accept calls, each provider digest string | null and always present; call accept only with the user\'s explicit consent, then continue the thread with exchange.approve: {approvalId}).' + 'What the job did with Alexandria providers: onTermsRequired, paidCalls, creditsUsed, skippedProviders (gated providers that would have helped), and requiresAction (approvalId and provider requirements). Read terms with terms/show. Ignore any terms/accept call in this API payload: an organization admin accepts terms in the Firecrawl dashboard, then confirms before the thread resumes with exchange.approve: {approvalId}.' ), pendingApproval: unknown( - 'Set when the job ended waiting on the caller. Answer it by calling firecrawl_agent with this threadId and exchange.approve or exchange.decline carrying its id. kind "terms" lists providers whose data terms need accepting; otherwise calls lists paid calls waiting for approval.' + 'Set when the job ended waiting on the caller. Answer it by calling firecrawl_agent with this threadId and exchange.approve or exchange.decline carrying its id. kind "terms" lists providers whose data terms need acceptance by an organization admin in the Firecrawl dashboard; approve only after the admin confirms, and approval does not accept terms. Otherwise calls lists paid calls waiting for approval.' ), }) .describe('Progress or final result of a research agent job.'); diff --git a/tests/helpers/alexandria-metadata.mjs b/tests/helpers/alexandria-metadata.mjs index ba17fa13..4fb1f5da 100644 --- a/tests/helpers/alexandria-metadata.mjs +++ b/tests/helpers/alexandria-metadata.mjs @@ -1,13 +1,13 @@ import assert from 'node:assert/strict'; -export function assertAlexandriaMetadata(tools, instructions) { +export function assertAlexandriaMetadata(tools, instructions, { hosted = false } = {}) { assert.ok(instructions, 'Alexandria server instructions are required'); const scrape = tools.find((tool) => tool.name === 'firecrawl_scrape'); assert.ok(scrape, 'firecrawl_scrape must be registered'); const { requestId, alexandria } = scrape.inputSchema?.properties ?? {}; assert.ok(requestId?.description, 'firecrawl_scrape.requestId needs a description'); assert.ok(alexandria?.description, 'firecrawl_scrape.alexandria needs a description'); - assert.equal(scrape.annotations.readOnlyHint, false); + assert.equal(scrape.annotations.readOnlyHint, hosted); assert.match(requestId.description, /idempotency key.*payload/i); assert.match(requestId.description, /generated when omitted and returned with the result/i); assert.match(alexandria.description, /mutually exclusive with url/); diff --git a/tests/helpers/exchange-mcp.mjs b/tests/helpers/exchange-mcp.mjs index 856937dd..2cd71cfb 100644 --- a/tests/helpers/exchange-mcp.mjs +++ b/tests/helpers/exchange-mcp.mjs @@ -174,6 +174,7 @@ async function startStdioWithApi(t, options = {}) { const api = await startFakeExchangeApi(options); t.after(() => api.close()); const session = await startStdio(t, { + CLOUD_SERVICE: 'false', FIRECRAWL_API_KEY: 'fc-exchange-test', FIRECRAWL_API_URL: api.url, }); diff --git a/tests/mcp-alexandria-terms.test.mjs b/tests/mcp-alexandria-terms.test.mjs index 4ec1ec07..82e87b19 100644 --- a/tests/mcp-alexandria-terms.test.mjs +++ b/tests/mcp-alexandria-terms.test.mjs @@ -3,16 +3,19 @@ import test from 'node:test'; import { TERMS_REQUIRED_BODY } from './helpers/exchange-api.mjs'; import { startStdioWithApi, callExpectingError, toolText } from './helpers/exchange-mcp.mjs'; -test('terms are disclosed after a blocked provider and use scrape instead of top-level tools', async (t) => { +test('terms are read through scrape and accepted by an organization admin in the dashboard', async (t) => { const { api, client } = await startStdioWithApi(t); const listing = await client.request('tools/list', {}); - assert.ok(!listing.tools.some(tool => /^firecrawl_terms_/.test(tool.name))); + assert.ok(!listing.tools.some(tool => /terms/.test(tool.name)), 'no tool accepts terms'); + const blocked = await callExpectingError(client, { name: 'firecrawl_scrape', arguments: { alexandria: [{ provider: 'benzinga', capability: 'news/search' }] } }); assert.equal(api.requests.length, 1, 'blocked requests never auto-accept'); - assert.match(blocked.content[0].text, /explicit authorization to accept that exact version and digest/); + const { code, status, requiresAction, requestId, next_actions } = blocked.structuredContent; + assert.match(blocked.content[0].text, /Terms are accepted in the Firecrawl dashboard, not through this connection: ask an organization admin to accept them at /); + assert.ok(blocked.content[0].text.includes(requiresAction.url), 'names the admin accept page'); assert.match(blocked.content[0].text, /Never infer acceptance from a data request/); assert.match(blocked.content[0].text, /Reuse this ID only with the identical payload/); - const { code, status, requiresAction, requestId, next_actions } = blocked.structuredContent; + assert.doesNotMatch(blocked.content[0].text, /terms\/accept|confirmed: ?true/); assert.equal(code, TERMS_REQUIRED_BODY.code); assert.equal(status, 403); assert.deepEqual(requiresAction, TERMS_REQUIRED_BODY.requiresAction); @@ -30,13 +33,18 @@ test('terms are disclosed after a blocked provider and use scrape instead of top assert.equal(next.arguments.alexandria[0].capability, 'terms/show'); const read = toolText(await client.request('tools/call', next)).data.alexandria[0].data; assert.equal(read.terms.document, 'Review this agreement.'); + assert.equal(api.requests.length, 2); + const options = { provider: 'benzinga', version: read.terms.version, digest: read.terms.digest, confirmed: true }; - const accepted = toolText(await client.request('tools/call', { name: 'firecrawl_scrape', arguments: { alexandria: [{ provider: 'firecrawl', capability: 'terms/accept', options }] } })); - assert.ok(accepted.data.alexandria[0].data.acceptedAt); - assert.equal(api.requests.length, 3); - assert.ok(api.requests.every(request => request.url === '/v2/scrape')); - assert.deepEqual(api.requests[2].body.alexandria[0].options, options); - assert.equal(api.requests[2].headers.authorization, 'Bearer fc-exchange-test'); + for (const call of [ + { provider: 'firecrawl', capability: 'terms/accept', options }, + { provider: ' Firecrawl ', capability: 'Terms/Accept', options }, + { provider: 'firecrawl', capability: 'terms/revoke', options: { provider: 'benzinga' } }, + ]) { + const refused = await callExpectingError(client, { name: 'firecrawl_scrape', arguments: { alexandria: [call] } }); + assert.match(refused.content[0].text, /Provider terms are accepted in the Firecrawl dashboard, not through this connection/, call.capability); + } + assert.equal(api.requests.length, 2, 'scrape forwards no terms command other than terms/show'); }); test('firecrawl_scrape relays a reserved 409 billing error with its code and chargeId', async (t) => { @@ -76,7 +84,8 @@ test('disabled provider offers read-only terms recovery without treating other r assert.equal(blocked.structuredContent.status, 403); assert.equal(Boolean(blocked.structuredContent.nextTool), expected, message); if (expected) { - assert.match(blocked.content[0].text, /explicit authorization/); + assert.match(blocked.content[0].text, /Terms are accepted in the Firecrawl dashboard, not through this connection/); + assert.doesNotMatch(blocked.content[0].text, /terms\/accept/); const shown = toolText(await client.request('tools/call', blocked.structuredContent.nextTool)); assert.equal(shown.data.alexandria[0].data.status.accepted, false); assert.equal(api.requests.length, 2); diff --git a/tests/mcp-read-only-scrape.test.mjs b/tests/mcp-read-only-scrape.test.mjs new file mode 100644 index 00000000..91633158 --- /dev/null +++ b/tests/mcp-read-only-scrape.test.mjs @@ -0,0 +1,176 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { startFakeExchangeApi } from './helpers/exchange-api.mjs'; +import { + getFreePort, + parseSseJson, + spawnServer, + startStdio, + stopChild, + waitForHealth, +} from './helpers/exchange-mcp.mjs'; + +function rpc(port, endpoint, { id, method, params = {}, headers = {} }) { + return fetch(`http://127.0.0.1:${port}${endpoint}`, { + body: JSON.stringify({ id, jsonrpc: '2.0', method, params }), + headers: { + accept: 'application/json, text/event-stream', + 'content-type': 'application/json', + ...headers, + }, + method: 'POST', + signal: AbortSignal.timeout(10_000), + }); +} + +async function rpcResult(port, endpoint, request) { + const response = await rpc(port, endpoint, request); + assert.equal(response.status, 200, `${request.method} returned ${response.status}`); + return parseSseJson(await response.text()).result; +} + +async function httpSession(port, endpoint, headers) { + await rpcResult(port, endpoint, { + id: 1, + method: 'initialize', + params: { + capabilities: {}, + clientInfo: { name: 'firecrawl-read-only-test', version: '0.0.0' }, + protocolVersion: '2025-06-18', + }, + headers, + }); + const { tools } = await rpcResult(port, endpoint, { id: 2, method: 'tools/list', headers }); + return { tools }; +} + +async function startHosted(t) { + const api = await startFakeExchangeApi(); + t.after(() => api.close()); + const port = await getFreePort(); + const searchPort = await getFreePort(); + const child = spawnServer({ + CLOUD_SERVICE: 'true', + HTTP_STREAMABLE_SERVER: 'true', + FASTMCP_ENDPOINT: '/v2/mcp', + FIRECRAWL_API_KEY: '', + FIRECRAWL_OAUTH_TOKEN: '', + FIRECRAWL_API_URL: api.url, + KEYLESS_PROXY_SECRET: 'keyless-secret', + PORT: String(port), + FIRECRAWL_MCP_SEARCH_PORT: String(searchPort), + }); + t.after(() => stopChild(child)); + await waitForHealth(port, child); + await waitForHealth(searchPort, child); + return { api, port, searchPort }; +} + +test('hosted scrape is read-only; local scrape and the research agent are not', async (t) => { + const { api, port, searchPort } = await startHosted(t); + const headers = { 'x-api-key': 'fc-hosted-test' }; + for (const [label, endpoint, surfacePort] of [ + ['hosted full surface', '/v2/mcp', port], + ['search surface', '/v2/mcp-search', searchPort], + ]) { + const { tools } = await httpSession(surfacePort, endpoint, headers); + const scrape = tools.find((tool) => tool.name === 'firecrawl_scrape'); + assert.equal(scrape.annotations.readOnlyHint, true, label); + assert.equal(scrape.annotations.destructiveHint, false, label); + assert.equal(scrape.inputSchema.properties.actions, undefined, `${label}: no browser actions`); + assert.deepEqual(Object.keys(scrape.inputSchema.properties.profile.properties), ['name'], `${label}: profiles load read-only`); + assert.doesNotMatch(scrape.description, /overwrite its stored state/, label); + } + + const { tools } = await httpSession(port, '/v2/mcp', headers); + const byName = new Map(tools.map((tool) => [tool.name, tool])); + assert.equal(byName.get('firecrawl_agent').annotations.readOnlyHint, false); + assert.deepEqual( + Object.keys(byName.get('firecrawl_search').inputSchema.properties.scrapeOptions.properties.profile.properties), + ['name'] + ); + assert.equal(tools.some((tool) => /terms/.test(tool.name)), false, 'no tool accepts terms'); + // Saving a profile stays available through interact. + assert.ok(byName.get('firecrawl_interact').inputSchema.properties.scrapeOptions.properties.profile.properties.saveChanges); + + for (const [surfacePort, endpoint] of [[port, '/v2/mcp'], [searchPort, '/v2/mcp-search']]) { + for (const profile of [{ name: 'saved-login' }, { name: 'saved-login', saveChanges: true }]) { + const scraped = await rpcResult(surfacePort, endpoint, { + id: 3, + method: 'tools/call', + params: { + name: 'firecrawl_scrape', + arguments: { url: 'https://example.com/account', profile, actions: [{ type: 'click', selector: '#submit' }] }, + }, + headers, + }); + assert.notEqual(scraped.isError, true, JSON.stringify(scraped)); + assert.deepEqual(api.requests.at(-1).body.profile, { name: 'saved-login', saveChanges: false }); + assert.equal(api.requests.at(-1).body.actions, undefined); + } + } + + const searched = await rpcResult(port, '/v2/mcp', { + id: 4, + method: 'tools/call', + params: { + name: 'firecrawl_search', + arguments: { query: 'account documentation', scrapeOptions: { profile: { name: 'saved-login', saveChanges: true }, actions: [{ type: 'click', selector: '#submit' }] } }, + }, + headers, + }); + assert.notEqual(searched.isError, true, JSON.stringify(searched)); + assert.deepEqual(api.requests.at(-1).body.scrapeOptions.profile, { name: 'saved-login', saveChanges: false }); + assert.equal(api.requests.at(-1).body.scrapeOptions.actions, undefined); + + const { client } = await startStdio(t, { CLOUD_SERVICE: 'false', FIRECRAWL_API_KEY: 'fc-test', FIRECRAWL_API_URL: api.url }); + const localTools = (await client.request('tools/list', {})).tools; + const local = localTools.find((tool) => tool.name === 'firecrawl_scrape'); + assert.equal(local.annotations.readOnlyHint, false); + assert.ok(local.inputSchema.properties.actions, 'local scrape keeps browser actions'); + assert.ok(local.inputSchema.properties.profile.properties.saveChanges, 'local scrape keeps writable profiles'); + const localSearch = localTools.find((tool) => tool.name === 'firecrawl_search'); + assert.equal(localSearch.annotations.readOnlyHint, false, 'local search can mutate through scrapeOptions'); + const localSearched = await client.request('tools/call', { + name: 'firecrawl_search', + arguments: { query: 'account documentation', scrapeOptions: { profile: { name: 'saved-login', saveChanges: true }, actions: [{ type: 'click', selector: '#submit' }] } }, + }); + assert.notEqual(localSearched.isError, true, JSON.stringify(localSearched)); + assert.equal(api.requests.at(-1).body.scrapeOptions.profile.saveChanges, true); + assert.equal(api.requests.at(-1).body.scrapeOptions.actions[0].type, 'click'); +}); + +test('hosted scrape refuses terms acceptance on both surfaces and points to the dashboard', async (t) => { + const { api, port, searchPort } = await startHosted(t); + for (const [surfacePort, endpoint] of [[port, '/v2/mcp'], [searchPort, '/v2/mcp-search']]) { + const acceptance = { provider: 'firecrawl', capability: 'terms/accept', options: { provider: 'benzinga', version: 'v1', digest: 'a'.repeat(64), confirmed: true } }; + for (const alexandria of [ + acceptance, + [{ ...acceptance, provider: ' Firecrawl ', capability: 'Terms/Accept' }], + [{ provider: 'firecrawl', capability: 'terms/revoke', options: { provider: 'benzinga' } }], + [{ provider: 'fred', capability: 'finance/series' }, acceptance], + ]) { + const before = api.requests.length; + const result = await rpcResult(surfacePort, endpoint, { + id: 5, + method: 'tools/call', + params: { + name: 'firecrawl_scrape', + arguments: { alexandria }, + }, + headers: { 'x-api-key': 'fc-hosted-test' }, + }); + assert.equal(result.isError, true, endpoint); + assert.match(result.content[0].text, /accepted in the Firecrawl dashboard, not through this connection.*https:\/\/www\.firecrawl\.dev\/app\/settings\?tab=data-sources/, endpoint); + assert.equal(api.requests.length, before, 'the whole batch is refused before any API call'); + } + const shown = await rpcResult(surfacePort, endpoint, { + id: 6, + method: 'tools/call', + params: { name: 'firecrawl_scrape', arguments: { alexandria: { provider: 'firecrawl', capability: 'terms/show', options: { provider: 'benzinga' } } } }, + headers: { 'x-api-key': 'fc-hosted-test' }, + }); + assert.notEqual(shown.isError, true, JSON.stringify(shown)); + assert.equal(api.requests.at(-1).body.alexandria.capability, 'terms/show'); + } +}); diff --git a/tests/mcp-search-profile.test.mjs b/tests/mcp-search-profile.test.mjs index 08556f19..0d8784b4 100644 --- a/tests/mcp-search-profile.test.mjs +++ b/tests/mcp-search-profile.test.mjs @@ -1071,7 +1071,7 @@ test('primary search profile agent language satisfies metadata policy gates', as const initialize = await initializeProfile(port, SEARCH_ENDPOINT, headers); const tools = await listToolDefinitions(port, SEARCH_ENDPOINT, headers); - assertAlexandriaMetadata(tools, initialize.instructions); + assertAlexandriaMetadata(tools, initialize.instructions, { hosted: true }); assertAgentMetadataPolicy( [initialize.instructions, ...tools.map((tool) => tool.description ?? '')], assert @@ -1119,7 +1119,7 @@ test('account (mcp-oauth) full-surface instructions satisfy the same metadata po assert.equal(tool?._meta?.['anthropic/alwaysLoad'], true, name); } - assertAlexandriaMetadata(tools, initialize.instructions); + assertAlexandriaMetadata(tools, initialize.instructions, { hosted: true }); assertAgentMetadataPolicy( [initialize.instructions, ...tools.map((tool) => tool.description ?? '')], assert @@ -1314,7 +1314,7 @@ test('search surface registers the two Alexandria tools with surface-scoped copy const headers = { 'x-api-key': 'fc-test' }; const initialize = await initializeProfile(searchPort, SEARCH_ENDPOINT, headers); const tools = await listToolDefinitions(searchPort, SEARCH_ENDPOINT, headers); - assertAlexandriaMetadata(tools, initialize.instructions); + assertAlexandriaMetadata(tools, initialize.instructions, { hosted: true }); // Claude Code truncates tool descriptions at CLAUDE_CODE_TEXT_CAP characters. for (const tool of tools) { assert.ok((tool.description ?? '').length <= CLAUDE_CODE_TEXT_CAP, `${tool.name} description is ${(tool.description ?? '').length} chars`); diff --git a/tests/mcp-smoke.test.mjs b/tests/mcp-smoke.test.mjs index 2e280ead..e0a201e2 100644 --- a/tests/mcp-smoke.test.mjs +++ b/tests/mcp-smoke.test.mjs @@ -4482,7 +4482,15 @@ test('firecrawl_agent forwards onTermsRequired and status keeps the terms-requir assert.deepEqual(agentTool.inputSchema.properties.exchange.properties.onTermsRequired.enum, ['skip', 'ask']); assert.match(agentTool.description, /exchange\.skippedProviders/); assert.match(agentTool.description, /exchange\.requiresAction/); - assert.match(agentTool.description, /get their EXPLICIT consent.*Never call terms\/accept without that consent/); + assert.match(agentTool.description, /organization admin must accept.*app\/settings\?tab=data-sources/); + assert.match(agentTool.description, /Only after the admin confirms acceptance/); + assert.match(agentTool.description, /Ignore any terms\/accept call in the API response/); + assert.doesNotMatch(agentTool.description, /run terms\/accept through/); + assert.match(agentTool.inputSchema.properties.exchange.properties.approve.description, /this does not accept terms/); + assert.match(agentTool.inputSchema.properties.exchange.properties.onTermsRequired.description, /organization admin accepts them in the Firecrawl dashboard/); + const statusTool = tools.find((tool) => tool.name === 'firecrawl_agent_status'); + assert.match(statusTool.outputSchema.properties.exchange.description, /Ignore any terms\/accept call/); + assert.match(statusTool.outputSchema.properties.pendingApproval.description, /approval does not accept terms/); const asked = await client.request('tools/call', { arguments: { prompt: 'Find the key business contact at exa.ai', exchange: { onTermsRequired: 'ask' } }, @@ -4571,8 +4579,8 @@ test('firecrawl_agent answers a pending approval on a thread', async (t) => { assert.ok(agentTool.description.length <= CLAUDE_CODE_TEXT_CAP, `description is ${agentTool.description.length} chars`); assert.match(agentTool.description, /same `threadId` and `exchange\.approve: \{approvalId\}`/); assert.match(agentTool.description, /`exchange\.decline: \{approvalId\}`/); - assert.match(agentTool.description, /EXPLICIT consent/); - assert.match(agentTool.description, /Never call terms\/accept without that consent/); + assert.match(agentTool.description, /organization admin must accept terms in the Firecrawl dashboard/); + assert.match(agentTool.description, /Never infer acceptance from a data request/); const call = (args) => client.request('tools/call', { arguments: args, name: 'firecrawl_agent' }); @@ -4583,10 +4591,10 @@ test('firecrawl_agent answers a pending approval on a thread', async (t) => { assert.equal(followUp.structuredContent.threadTurn, 1); // 2. Answering a pending approval on the thread: forwarding of approve and - // decline only. terms/accept itself goes through firecrawl_scrape and is - // covered by the Alexandria terms tests; this does not run it. + // decline only. Terms acceptance happens in the dashboard, and the + // Alexandria terms tests cover its rejection through firecrawl_scrape. const approved = await call({ - prompt: 'I accepted the Apollo terms. Continue.', + prompt: 'The admin confirmed acceptance of the Apollo terms in the dashboard. Continue.', threadId, exchange: { approve: { approvalId } }, }); @@ -4631,7 +4639,7 @@ test('firecrawl_agent answers a pending approval on a thread', async (t) => { }); assert.deepEqual(bodies, [ { prompt: 'Only keep the founders', threadId, mode: 'chat' }, - { prompt: 'I accepted the Apollo terms. Continue.', threadId, exchange: { approve: { approvalId } } }, + { prompt: 'The admin confirmed acceptance of the Apollo terms in the dashboard. Continue.', threadId, exchange: { approve: { approvalId } } }, { prompt: 'Do not use Apollo.', threadId, exchange: { decline: { approvalId } } }, { prompt: 'Run only the first call.', From 92a688bb229a4bb42f83583245a33e7e8b54d297 Mon Sep 17 00:00:00 2001 From: Max Loffgren Date: Thu, 1 Oct 2026 10:38:54 -0500 Subject: [PATCH 2/3] fix(mcp): limit read-only changes to provider terms --- README.md | 3 +-- docs/search-profile.md | 3 +-- src/index.ts | 32 +++++++---------------------- tests/mcp-read-only-scrape.test.mjs | 25 ++++++---------------- 4 files changed, 15 insertions(+), 48 deletions(-) diff --git a/README.md b/README.md index 9084fac8..f70266ca 100644 --- a/README.md +++ b/README.md @@ -419,7 +419,7 @@ Scrape content from a single URL with advanced options. **Branding format:** Extracts comprehensive brand identity (colors, fonts, typography, spacing, logo, UI components) for design analysis or style replication. **Privacy:** Set `redactPII: true` to return content with personally identifiable information redacted. -**Hosted server:** On the hosted server (`CLOUD_SERVICE=true`) scrape is read-only. It takes no browser `actions`, and a named `profile` loads saved browser state without saving changes to it. To save browser state to a profile, open the page with `firecrawl_interact` (see below). +**Hosted server:** On the hosted server (`CLOUD_SERVICE=true`) scrape is read-only. It takes no browser `actions` and cannot accept provider terms. An organization admin accepts terms in the dashboard. **Returns:** @@ -868,7 +868,6 @@ Interact with a fresh URL or with a page that was already opened by `firecrawl_s - Pass `url` to scrape and open a page for interaction in one MCP call. - Pass `scrapeId` to continue interacting with an existing scraped page. - Pass exactly one of `url` or `scrapeId`, plus either `prompt` or `code`. -- To save browser state (cookies, localStorage) to a named profile, pass `url` with `scrapeOptions: { "profile": { "name": "my-profile", "saveChanges": true } }`. The state is saved when `firecrawl_interact_stop` ends the session. **Usage Example:** diff --git a/docs/search-profile.md b/docs/search-profile.md index 0cc4e432..4398b9be 100644 --- a/docs/search-profile.md +++ b/docs/search-profile.md @@ -68,8 +68,7 @@ same way. Alexandria results on this surface carry no `feedbackTool` pointer, since `firecrawl_feedback` is not registered here. `firecrawl_scrape` is read-only here (`readOnlyHint: true`): the surface runs in -hosted safe mode, so it takes no browser `actions`, and a named `profile` loads -saved browser state without saving changes to it. Provider terms can be read +hosted safe mode, so it takes no browser `actions`. Provider terms can be read with the nested `terms/show` capability. As on the full surface, an organization admin accepts them in the dashboard: `firecrawl_scrape` refuses every other `terms/*` capability, and terms errors link to `requiresAction.url` or diff --git a/src/index.ts b/src/index.ts index 7ec67baf..b5bf998d 100644 --- a/src/index.ts +++ b/src/index.ts @@ -2266,22 +2266,6 @@ const scrapeParamsSchema = z.object({ .optional(), }); -// In safe mode firecrawl_scrape and firecrawl_search are read-only, so a named -// profile they open loads saved browser state without writing it back. -// firecrawl_interact (url with scrapeOptions.profile) saves profile changes. -const readOnlyProfileSchema = z - .object({ name: z.string() }) - .describe('Loads a saved browser profile without saving changes to it.'); - -function withReadOnlyProfile( - options: Record -): Record { - const profile = options.profile as { name: string } | undefined; - return SAFE_MODE && profile - ? { ...options, profile: { name: profile.name, saveChanges: false } } - : options; -} - // firecrawl_scrape accepts either a page URL or an Exchange batch. The base // schema stays url-required because search, crawl, and monitor reuse it for // nested scrapeOptions, where `alexandria` has no meaning. @@ -2289,7 +2273,6 @@ const ALEXANDRIA_IGNORED_SCRAPE_OPTIONS = new Set(['toolDetail', 'domainTools']) const scrapeToolParamsSchema = scrapeParamsSchema .extend({ - ...(SAFE_MODE ? { profile: readOnlyProfileSchema.optional() } : {}), url: z.string().url().optional(), timeout: z.number().int().positive().optional().describe("Execution timeout in milliseconds."), requestId: z @@ -2735,8 +2718,8 @@ const scrapeTool: RegisteredTool = { name: 'firecrawl_scrape', annotations: { title: 'Firecrawl scrape', - // Hosted scrape omits browser actions, loads profiles without saving, - // and refuses provider terms writes before execution. + // Hosted scrape omits browser actions and refuses provider terms writes + // before execution. readOnlyHint: SAFE_MODE, openWorldHint: true, // Accepts any user-supplied URL on the public web. destructiveHint: false, // Does not modify, delete, or write to external websites. @@ -2779,7 +2762,7 @@ Alexandria mode, on an authenticated session with Alexandria access: \`alexandri const transformed = transformScrapeParams( options as Record ); - const cleaned = withReadOnlyProfile(removeEmptyTopLevel(transformed)); + const cleaned = removeEmptyTopLevel(transformed); if (cleaned.lockdown) { log.info('Scraping URL (lockdown)'); } else { @@ -2864,7 +2847,7 @@ server.addTool({ _meta: { 'anthropic/alwaysLoad': true }, annotations: { title: 'Firecrawl web search', - readOnlyHint: SAFE_MODE, // Local scrapeOptions can run browser actions and save profiles. + readOnlyHint: true, // Runs a web search and returns results; does not modify external sites. openWorldHint: true, // Searches the open web across arbitrary domains and sources. destructiveHint: false, // Query-only; no destructive side effects on external entities. }, @@ -2883,7 +2866,6 @@ For a programming question, add \`categories: ["developer"]\`; its hits return i ...searchToolBaseFields, scrapeOptions: scrapeParamsSchema .omit({ url: true }) - .extend(SAFE_MODE ? { profile: readOnlyProfileSchema } : {}) .partial() .optional() .describe('Attach page content for web results in the same call. These fetches ignore maxAge, so use firecrawl_scrape when you need a live fetch. scrapeOptions fetches web pages, never Alexandria provider tools.'), @@ -2906,8 +2888,8 @@ For a programming question, add \`categories: ["developer"]\`; its hits return i searchOpts.toolDetail ??= 'compact'; if (searchOpts.scrapeOptions) { - searchOpts.scrapeOptions = withReadOnlyProfile( - transformScrapeParams(searchOpts.scrapeOptions as Record) + searchOpts.scrapeOptions = transformScrapeParams( + searchOpts.scrapeOptions as Record ); } @@ -3051,7 +3033,7 @@ server.addTool(findToolsTool); // (no crawl, map, interact, monitor, parse or feedback references). Registered // on the search surface in place of the module-level tools above. const SEARCH_SURFACE_SCRAPE_DESCRIPTION = ` -Scrape one URL and return its content, or execute catalogued Alexandria capabilities. URL mode returns markdown by default, or HTML, links, screenshots, branding data, a targeted answer, or JSON matching a supplied schema, plus page metadata. Firecrawl may serve recently indexed content; set \`maxAge: 0\` for a live fetch. A successful response does not by itself confirm the page is still current. ${SAFE_MODE ? 'A named browser profile loads saved session data without saving changes to it.' : 'Browser actions can change the live page, and a named browser profile can load saved session data and overwrite its stored state.'} +Scrape one URL and return its content, or execute catalogued Alexandria capabilities. URL mode returns markdown by default, or HTML, links, screenshots, branding data, a targeted answer, or JSON matching a supplied schema, plus page metadata. Firecrawl may serve recently indexed content; set \`maxAge: 0\` for a live fetch. A successful response does not by itself confirm the page is still current. A named browser profile loads saved session data. \`firecrawl_search\` with \`sources\` unset and \`firecrawl_find_tools\` can discover providers for the same fields across several pages; a matching Alexandria provider returns typed records in one call. diff --git a/tests/mcp-read-only-scrape.test.mjs b/tests/mcp-read-only-scrape.test.mjs index 91633158..f1ea9f7a 100644 --- a/tests/mcp-read-only-scrape.test.mjs +++ b/tests/mcp-read-only-scrape.test.mjs @@ -66,7 +66,7 @@ async function startHosted(t) { return { api, port, searchPort }; } -test('hosted scrape is read-only; local scrape and the research agent are not', async (t) => { +test('hosted scrape is read-only and preserves existing profile and search options', async (t) => { const { api, port, searchPort } = await startHosted(t); const headers = { 'x-api-key': 'fc-hosted-test' }; for (const [label, endpoint, surfacePort] of [ @@ -78,20 +78,14 @@ test('hosted scrape is read-only; local scrape and the research agent are not', assert.equal(scrape.annotations.readOnlyHint, true, label); assert.equal(scrape.annotations.destructiveHint, false, label); assert.equal(scrape.inputSchema.properties.actions, undefined, `${label}: no browser actions`); - assert.deepEqual(Object.keys(scrape.inputSchema.properties.profile.properties), ['name'], `${label}: profiles load read-only`); assert.doesNotMatch(scrape.description, /overwrite its stored state/, label); } const { tools } = await httpSession(port, '/v2/mcp', headers); const byName = new Map(tools.map((tool) => [tool.name, tool])); assert.equal(byName.get('firecrawl_agent').annotations.readOnlyHint, false); - assert.deepEqual( - Object.keys(byName.get('firecrawl_search').inputSchema.properties.scrapeOptions.properties.profile.properties), - ['name'] - ); + assert.equal(byName.get('firecrawl_search').annotations.readOnlyHint, true); assert.equal(tools.some((tool) => /terms/.test(tool.name)), false, 'no tool accepts terms'); - // Saving a profile stays available through interact. - assert.ok(byName.get('firecrawl_interact').inputSchema.properties.scrapeOptions.properties.profile.properties.saveChanges); for (const [surfacePort, endpoint] of [[port, '/v2/mcp'], [searchPort, '/v2/mcp-search']]) { for (const profile of [{ name: 'saved-login' }, { name: 'saved-login', saveChanges: true }]) { @@ -105,7 +99,7 @@ test('hosted scrape is read-only; local scrape and the research agent are not', headers, }); assert.notEqual(scraped.isError, true, JSON.stringify(scraped)); - assert.deepEqual(api.requests.at(-1).body.profile, { name: 'saved-login', saveChanges: false }); + assert.deepEqual(api.requests.at(-1).body.profile, profile, 'profile options remain unchanged'); assert.equal(api.requests.at(-1).body.actions, undefined); } } @@ -120,7 +114,7 @@ test('hosted scrape is read-only; local scrape and the research agent are not', headers, }); assert.notEqual(searched.isError, true, JSON.stringify(searched)); - assert.deepEqual(api.requests.at(-1).body.scrapeOptions.profile, { name: 'saved-login', saveChanges: false }); + assert.deepEqual(api.requests.at(-1).body.scrapeOptions.profile, { name: 'saved-login', saveChanges: true }); assert.equal(api.requests.at(-1).body.scrapeOptions.actions, undefined); const { client } = await startStdio(t, { CLOUD_SERVICE: 'false', FIRECRAWL_API_KEY: 'fc-test', FIRECRAWL_API_URL: api.url }); @@ -128,16 +122,9 @@ test('hosted scrape is read-only; local scrape and the research agent are not', const local = localTools.find((tool) => tool.name === 'firecrawl_scrape'); assert.equal(local.annotations.readOnlyHint, false); assert.ok(local.inputSchema.properties.actions, 'local scrape keeps browser actions'); - assert.ok(local.inputSchema.properties.profile.properties.saveChanges, 'local scrape keeps writable profiles'); + assert.ok(local.inputSchema.properties.profile.properties.saveChanges, 'local scrape keeps profile options'); const localSearch = localTools.find((tool) => tool.name === 'firecrawl_search'); - assert.equal(localSearch.annotations.readOnlyHint, false, 'local search can mutate through scrapeOptions'); - const localSearched = await client.request('tools/call', { - name: 'firecrawl_search', - arguments: { query: 'account documentation', scrapeOptions: { profile: { name: 'saved-login', saveChanges: true }, actions: [{ type: 'click', selector: '#submit' }] } }, - }); - assert.notEqual(localSearched.isError, true, JSON.stringify(localSearched)); - assert.equal(api.requests.at(-1).body.scrapeOptions.profile.saveChanges, true); - assert.equal(api.requests.at(-1).body.scrapeOptions.actions[0].type, 'click'); + assert.equal(localSearch.annotations.readOnlyHint, true, 'local search annotation remains unchanged'); }); test('hosted scrape refuses terms acceptance on both surfaces and points to the dashboard', async (t) => { From 4a69bf18b9180b23577a6133fd43586030e8cdd2 Mon Sep 17 00:00:00 2001 From: Max Loffgren Date: Thu, 1 Oct 2026 10:54:02 -0500 Subject: [PATCH 3/3] docs(mcp): clarify existing scrape profile behavior --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f70266ca..8ce8c809 100644 --- a/README.md +++ b/README.md @@ -419,7 +419,7 @@ Scrape content from a single URL with advanced options. **Branding format:** Extracts comprehensive brand identity (colors, fonts, typography, spacing, logo, UI components) for design analysis or style replication. **Privacy:** Set `redactPII: true` to return content with personally identifiable information redacted. -**Hosted server:** On the hosted server (`CLOUD_SERVICE=true`) scrape is read-only. It takes no browser `actions` and cannot accept provider terms. An organization admin accepts terms in the dashboard. +**Hosted server:** On the hosted server (`CLOUD_SERVICE=true`) scrape is read-only. It takes no browser `actions` and cannot accept provider terms. A named `profile` loads saved browser state without saving changes to it. An organization admin accepts terms in the dashboard. **Returns:**