GroktoCrawl is a self-hosted, MIT-licensed web data platform compatible with the Firecrawl v2 API surface. It combines scraping, crawl and map jobs, search, structured extraction, browser automation, monitors, semantic retrieval, an autonomous research agent, and an MCP server in one Docker deployment.
It is designed as a local-first, observable, model-flexible foundation for organizations that want agentic research and web intelligence without surrendering control of their data, infrastructure, or model choices.
Research memory keeps full artifacts in Valkey and defaults to Qdrant for vector lookup. Deployments can explicitly mirror that index to PostgreSQL/pgvector and migrate it with the research-memory storage runbook; the separate semantic page index also has a PostgreSQL-only mode after its own backfill. These modes are opt-in and do not change existing storage configuration by default.
Choose your path. Both run the same Docker stack defined in docker-compose.yml; they differ in which services and credentials you configure.
The fastest end-to-end smoke test, with no external credentials. The fixture profile starts a local LLM fixture (llm-svc) and two fixture test sites (test-site, tier3-fixture).
Config: copy .env.sample to .env. No credentials are required; the optional direct SlopSearX MCP companion (slopsearx-mcp) is opt-in via the mcp Compose profile and only then needs a non-empty SLOPSEARX_MCP_AUTH_TOKEN.
test -f .env || cp .env.sample .env
docker compose --profile fixture up --build -d
curl http://localhost:8080/health
./groktocrawl scrape https://example.comExpected success output: curl http://localhost:8080/health returns {"status":"ok", "checks":{...}} once all dependencies are up, and ./groktocrawl scrape https://example.com prints example.com rendered as markdown.
The same stack pointed at a real LLM provider, an open-web search backend, and a hardened API — without the fixture-only services.
Config (in .env): an OpenAI-compatible LLM provider (LLM_BASE_URL, LLM_API_KEY, LLM_MODEL), BRAVE_API_KEY for web search, and API_KEY for API authentication. llm-svc, test-site, and tier3-fixture are fixture-only and must not be started in production; omit the fixture profile. The optional direct SlopSearX MCP companion (slopsearx-mcp) is opt-in via the mcp Compose profile and then requires a non-empty SLOPSEARX_MCP_AUTH_TOKEN.
Timeouts are configurable: raise LLM_CALL_TIMEOUT (idle-timeout seconds, default 120 — e.g. set 300 for reasoning models), and tune QDRANT_QUERY_TIMEOUT / QDRANT_CLIENT_TIMEOUT if the semantic/Qdrant path needs different bounds.
test -f .env || cp .env.sample .env # set LLM_BASE_URL/LLM_API_KEY/LLM_MODEL, BRAVE_API_KEY, API_KEY
docker compose up --build -d
curl http://localhost:8080/health
./groktocrawl scrape https://example.comExpected success output: curl http://localhost:8080/health returns {"status":"ok", "checks":{...}}, and ./groktocrawl scrape https://example.com prints example.com as markdown. Set API_KEY before exposing the API outside a trusted network.
This exercises the async job path end to end and shows why the durability boundary matters. The CLI crawl command is documented in the CLI guide; the curl forms below show the underlying API.
# Create an async crawl job with a completion webhook
curl -X POST http://localhost:8080/v2/crawl \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com","maxDepth":2,"limit":20,"webhook":{"url":"https://your-host.example/hook"}}'
# → {"id":"crawl_<job_id>","status":"processing", ...}
# Poll status until the crawl finishes
curl http://localhost:8080/v2/crawl/crawl_<job_id>
# → {"status":"completed","completed":N,"total":N,"data":[ ... ], ...}
# Or stream live progress inline (stream: true runs the crawl in the response)
curl -N -X POST http://localhost:8080/v2/crawl \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com","maxDepth":2,"limit":20,"stream":true}'
# → data: {"type":"page","url":...,"markdown":...}
# data: {"type":"done","id":...,"status":"completed","pages":N,...}limit is honored as a page cap (must be ≥ 1); omit it for an unlimited crawl.
Expected success output: the webhook fires a crawl.started event before scraping, a crawl.page event after each page, and a crawl.completed event at the end — each with a unique webhookId (omit an events filter to receive all of them). The SSE stream emits flat page events and a terminal done event.
Background work has a fail-closed restart boundary. Async jobs (crawl, agent, extract, batch-scrape, llmstxt, plan execution) still run in-process. New jobs that are durably recorded but not yet claimed can be dispatched after restart. If a worker had started, GroktoCrawl does not replay it: after its Valkey lease expires, the job becomes
failedwithOUTCOME_UNKNOWN, because an external side effect may already have happened. Pre-marker records follow the same rule. Terminal webhook intents are persisted and retried with the samewebhookId(delivery is at-least-once), while partial artifacts are not rolled back. See Job durability and recovery, the Interrupted Jobs runbook, and ADR-0072.
The /experimental/research/v1 endpoint can execute an objective through the server-configured search, scraper, and model, then retain source bytes, checked knowledge, and three report layers in PostgreSQL. Enable all four research flags and schema 16; leave the development-only FEATURE_EXPERIMENTAL_RESEARCH_FIXTURE_RUNS=false. Setting that flag true explicitly selects scripted demonstration runs. Existing retained records keep their readers.
This durable endpoint plans up to three searches and retains up to eight sources within 256,000 source bytes and a 180-second deadline. Known capture failures remain visible as insufficient coverage; ambiguous external dispatch stops as outcome_unknown. Completed call receipts are reused after restart without another paid call. as_of is a timezone-aware, nonfuture evidence evaluation target: it does not retrieve an archived web page, and a new capture cannot establish past validity by itself. Optional completion, failure and cancellation webhooks use a durable outbox with three attempts and one stable event ID; receivers must deduplicate that ID. See the live execution contract and ADR-0070. The remaining 1.0.1 acceptance work is tracked in #666.
Research-memory artifacts remain in Valkey; its vector index defaults to Qdrant and can be migrated to PostgreSQL/pgvector after a verified backfill. The semantic page index has separate Qdrant, shadow, and PostgreSQL-only modes. Qdrant remains available for rollback until operators verify a reverse copy and end their rollback window. The /experimental/research/v1 path remains stable for compatibility. See ADR-0069, ADR-0071, the research-memory storage runbook, and the deployment migration runbook. The protected capture topology is described in the protected Compose override. Historical experiment decisions and their scope are recorded in the GroktoCrawl X ADR archive.
| Area | Capabilities |
|---|---|
| Web data | Scrape, batch scrape, map, crawl, parse, browser sessions, and llms.txt generation; the scraper recovers low-yield (card-style) page bodies, warns on partial extractions, and detects bot-challenge pages (e.g. Fastly JS challenges) so barrier content is never served as article content |
| Search and retrieval | SlopSearX search (DuckDuckGo engine falls back to its lite endpoint when the primary frontend is bot-blocked; results best-effort), rich/deep research modes, semantic index, and similarity search; vector-search backend failures surface as errors rather than silent empty results |
| Research | Grounded answers, streaming agent research, plans, sessions, citations, and reusable research memory |
| Operations | Monitors, webhooks, health probes, Prometheus metrics, cache controls, and politeness controls |
| Integrations | Portal UI, Model Context Protocol server, and site adapters for code, publishing, commerce, and security sources |
- API guide — authentication, jobs, SSE, webhooks, examples, and compatibility.
- CLI guide — commands, global flags, and streaming/JSON output.
- Deployment and configuration — services, profiles, configuration, security, and operations.
- Feature guides — scraping, crawl, search, research, sessions, browser, monitors, parse, portal, and MCP.
- Architecture — current service and data-flow design.
- Contributor guide — contribution intake, local development, tests, API/CLI parity, and ADRs.
- Roadmap — Now / Next / Later priorities.
- Public surface inventory — validated route, CLI, compose, and configuration indexes.
When the stack is running, FastAPI publishes the canonical request/response schema at Swagger UI and OpenAPI JSON. The Markdown guides explain behavior and workflows; OpenAPI is authoritative for wire schemas.
agent-svc is the public API and coordinator. It persists job state and research-memory artifacts in Valkey, calls scraper-svc and SlopSearX, uses semantic-svc for page-vector retrieval, and delegates synthesis to an OpenAI-compatible LLM. The semantic index defaults to Qdrant and can use PostgreSQL/pgvector in its explicit PostgreSQL-only mode. Research-memory vectors have their own Qdrant/shadow/pgvector modes. Supporting services provide browser sessions, document parsing, the portal, scheduled monitors, and MCP access. See the architecture guide for the service graph and boundaries.
Site adapters run before the generic scraper pipeline and fall back safely to it when their specialized extraction fails. Supported categories include GitHub, YouTube, Bluesky, Substack, Gutenberg, Greenhouse, AshbyHQ (official posting API with SSR fallback), Shopify, and security/threat-intelligence sources such as NVD, CVE.org, AbuseIPDB, Shodan, VirusTotal, and VulnCheck. Configuration and extension guidance are in the scraping guide.
Set API_KEY for authentication, restrict network exposure, and review outbound proxy and robots/politeness settings before production use. The service emits an X-Security-Warning header while authentication is disabled. See deployment and configuration for the operational baseline.
Self-hosted CI is hardened against untrusted fork PRs: workflow edits on fork PRs fail closed (defense in depth — the platform-level control is authoritative), and deleted-fork PRs are excluded from the self-hosted runner lane. scripts/enforce-branch-protection.py gains a read-only --verify-rulesets mode that asserts both main rulesets are active and diff-clean. See self-hosted runner fork protection and SECURITY.md; the org-level runner-group settings remain deferred with residual risk accepted.
Core Firecrawl-compatible workflows and GroktoCrawl extensions are actively developed. Review the roadmap for Now / Next / Later priorities, the changelog, ADRs, and issues for change history and planned work. Bugs and feature ideas are filed through the contribution intake route.
Merges to main require the Code Quality Gate and Runtime Gate checks to pass and at least one approving review for non-automation changes (stale approvals are dismissed and open review conversations block merge). dependabot[bot] skips the review requirement only — it must still pass the required checks; the sole maintainer can merge their own PRs without an approving review (review bypass only — required checks still bind); release-please PRs require a human approving review. See ADR-0046 for the full policy and Emergency Branch Protection Bypass for the audited emergency exception path.