SlopSearX

Cloud-native, stateless, AI-agent-first meta search engine. Drop-in SearXNG replacement for the GroktoCrawl stack.

---

SlopSearX is a horizontally scalable, stateless meta search engine designed for AI agent consumption. It replaces SearXNG in the GroktoCrawl stack with:

• HTML output by default — SearXNG-compatible browser responses with JSON/CSV/RSS negotiation
• Human search portal — a responsive, server-rendered landing and results experience with Dark / Darker themes
• YAML+Markdown native output — structured + readable for AI agent contexts via format=yaml
• SearXNG-compatible API — drop-in replacement for existing consumers
• Plugin engine adapters — one file per engine, @register_engine, zero orchestrator changes
• Category routing — SearXNG-compatible taxonomy with sub-categories and env-var overrides
• Distributed rate limiting — Valkey-backed sliding windows, correct at 50+ replicas
• Response caching — Valkey-backed, category-aware TTL, 150x speedup on cache hits
• OpenMetrics observability — /metrics endpoint, per-engine counters + latency + status
• Stateless, cloud-native — no local volumes, no persistent DB, all replicas interchangeable
• Hybrid config — env vars for secrets, optional mounted file for engine tuning

API

Endpoint	Description
GET /search?q=...	SearXNG-compatible HTML (default; Accept-aware)
GET /search?q=...&format=json	SearXNG-compatible JSON
GET /search?q=...&format=csv	CSV results when enabled in search.formats
GET /search?q=...&format=rss	RSS results when enabled in search.formats
GET /search?q=...&format=yaml	YAML+Markdown agent-native output
GET /	Human search portal landing page (HTML; machine formats retain the missing-query error)
GET /search?q=...&categories=science,news	Filter by category (OR semantics)
GET /search?q=...&engines=brave,wikipedia	Explicit engine selection
GET /health	Per-engine health check with status
GET /metrics	OpenMetrics for Prometheus scraping
GET /config	Categories→engines mapping for runtime discovery

Europe PMC is opt-in; see configuration, attribution, and evaluation.

Engines (51)

The table below is maintained to match the live adapter registry (51 registered adapters as of this writing). The MCP server's slopsearx_list_capabilities tool and the slopsearx://capabilities resource are generated from that same registry at runtime — treat them as authoritative.

General / Web

Engine	Type	Auth	Categories
Brave Search	API	ENGINE_BRAVE_API_KEY	general, news, science, images
DuckDuckGo	Scrape	None	general, news, images
Google	Scrape	None	general, news
Hacker News	API	None	general, news
Reddit	API	Optional OAuth bearer token: ENGINE_REDDIT_API_KEY	general, social, reddit:subreddit
Wikipedia	API	None	general, science, reference

Developer / Package Registries

Engine	Type	Auth	Categories
Crates.io	API	None	it, reference, packages
Docker Hub	API	None	it, reference, packages
GitHub	API	Optional for public repository/issues; ENGINE_GITHUB_API_KEY required for code search	reference, github:code, github:issues, github:prs
npm	API	None	it, reference, packages
PyPI	API	None	it, reference, packages
RubyGems	API	None	it, reference, packages
Stack Exchange	API	Optional app key: ENGINE_STACKEXCHANGE_API_KEY	general, reference, science, stackexchange:code, stackexchange:serverfault

GitHub authentication is sub-category-specific: public repository and issue/PR searches may run without a token, while code search requires ENGINE_GITHUB_API_KEY. The capability catalog reports authentication at the engine level, so its single auth.class cannot express this distinction; the adapter behavior is authoritative for the selected sub-category.

Science & Research

Engine	Type	Auth	Categories
arXiv	API	None	science, reference
HuggingFace	API	HF_TOKEN or ENGINE_HUGGINGFACE_API_KEY (optional)	science, huggingface:datasets, huggingface:papers
OpenAlex	API	None	science, reference
Open Library	API	None	books, reference
Semantic Scholar	API	Optional	science, reference
UniProt	API	None	science, reference, biology, medical
Internet Archive	API	None	reference, web:archive, historical

Medical / Health

Engine	Type	Auth	Categories
ClinicalTrials.gov	API	None	medical, health, science
openFDA	API	None	medical, health, science, government
PubChem	API	None	science, reference, chemistry, medical
Europe PMC	API	None; opt-in	science, reference, medical, health
PubMed	API	None	science, reference, medical, health

Security / Threat Intelligence

Engine	Type	Auth	Categories
AbuseIPDB	API	ENGINE_ABUSEIPDB_API_KEY	security, threat-intel
AlienVault OTX	API	ENGINE_OTX_API_KEY	security, threat-intel
Censys	API	ENGINE_CENSYS_API_KEY + _API_SECRET	it, security
CRT.sh	API	None	it, security
CVE Program (MITRE)	API	None	it, security
DeHashed	API	ENGINE_DEHASHED_API_KEY	security, threat-intel
Exploit-DB	Scrape	None	security, exploit
FIRST EPSS	API	None	security, threat-intel
GreyNoise	API	ENGINE_GREYNOISE_API_KEY (optional)	security, threat-intel
Have I Been Pwned	API	ENGINE_HIBP_API_KEY	security, reference
IntelX	API	ENGINE_INTELX_API_KEY	security, threat-intel
MITRE ATT&CK	API	None	security, reference
NVD (NIST)	API	ENGINE_NVD_API_KEY (optional)	it, security
Shodan	API	ENGINE_SHODAN_API_KEY	it, security
URLhaus	API	ENGINE_URLHAUS_API_KEY	security, threat-intel
VirusTotal	API	ENGINE_VIRUSTOTAL_API_KEY	security, malware
VulnCheck	API	ENGINE_VULNCHECK_API_KEY	security, threat-intel

Finance / Economics

Engine	Type	Auth	Categories
FRED	API	ENGINE_FRED_API_KEY	finance, reference, economics
SEC EDGAR	API	None	finance, reference

Media & Entertainment

Engine	Type	Auth	Categories
MusicBrainz	API	None	music, reference
TMDB	API	ENGINE_TMDB_API_KEY	movies, entertainment

Geography / GIS

Engine	Type	Auth	Categories
Nominatim (OSM)	API	None	geography, reference

Legal

Engine	Type	Auth	Categories
Oyez (SCOTUS)	API	None	reference, legal

Jobs / ATS

Engine	Type	Auth	Categories
Ashby	API	None	jobs
Greenhouse	API	None	jobs
Lever	API	None	jobs

Adding a new engine: See docs/ENGINE_ADAPTERS.md for the full adapter reference — contract rules, data types, lifecycle hooks, and the category system.

Capability metadata: every adapter additionally declares an audited capability surface (supported_result_types, failure_classes, cost_class from the closed vocabularies in slopsearx/adapter.py). The MCP capability catalog exposes these for enabled and disabled engines; a declaration is a static capability hint and must not be confused with configured availability (enabled), authentication state (auth.class/auth.configured), or observed health (last_known_status). See docs/MCP_CONTRACT.md §7.4.

MCP Server (agents)

SlopSearX ships a Model Context Protocol server for AI agents. It exposes intent-level search (no URL strings), capability discovery, scope explanation, snapshot-based pagination, asynchronous research jobs, scheduled change detection, and attributed retrieval receipts — built on the same pipeline as the HTTP API. Search results carry a machine-readable retrieval handoff record so a downstream reader (e.g. GroktoCrawl) can capture pages and link them back to the originating result (see docs/RETRIEVAL_HANDOFF.md).

# Install (already included in the package dependencies)
pip install -e ".[dev]"

# Run over stdio (default transport)
slopsearx-mcp            # or: python -m slopsearx.mcp

# Run over HTTP for remote clients (e.g. Hermes Agent on another host)
MCP_TRANSPORT=http MCP_HOST=0.0.0.0 MCP_AUTH_TOKEN=change-me slopsearx-mcp

# Remote gateway: stdio MCP server that proxies to a remote SlopSearX server
slopsearx-mcp --remote http://<slopsearx-host>:8000/mcp   # token: MCP_REMOTE_TOKEN=…

# Gateway against an OAuth-mode remote: --oauth runs the standard MCP OAuth
# flow (loopback callback; tokens persisted for reuse)
slopsearx-mcp --remote http://<slopsearx-host>:8000/mcp --oauth

# OAuth 2.1 mode (required by OAuth-only clients, e.g. Claude Web connectors)
MCP_TRANSPORT=http MCP_OAUTH_ENABLED=1 MCP_OAUTH_ISSUER_URL=https://mcp.example.com slopsearx-mcp

• 38 tools: slopsearx_search, slopsearx_search_targeted, slopsearx_search_jobs, slopsearx_search_security, slopsearx_search_science, slopsearx_list_capabilities, slopsearx_explain_search_scope, slopsearx_get_service_status, slopsearx_read_results, slopsearx_read_result, slopsearx_read_entities, slopsearx_plan_research, slopsearx_plan_query_variants, slopsearx_plan_research_followup, slopsearx_start_research, slopsearx_get_job, slopsearx_cancel_job, slopsearx_retry_research, slopsearx_extend_research, slopsearx_update_research, slopsearx_create_saved_search, slopsearx_get_saved_search, slopsearx_update_saved_search, slopsearx_pause_saved_search, slopsearx_delete_saved_search, slopsearx_read_saved_search_reports, slopsearx_read_saved_search_events, slopsearx_ack_saved_search_events, slopsearx_submit_retrieval_receipt, slopsearx_read_retrieval_receipts, slopsearx_export_research_manifest, slopsearx_preview_staged_search, slopsearx_search_staged, slopsearx_get_staged_search, slopsearx_retry_staged_search, slopsearx_start_dependency_dossier, slopsearx_get_dependency_dossier, slopsearx_get_artifact_lineage
• Resources: slopsearx://capabilities, slopsearx://capabilities/{engine}, slopsearx://routing-profiles, slopsearx://health/summary
• Artifact references and bounded workflow lineage are documented in docs/ARTIFACT_LINEAGE.md.
• Supported artifact-to-workflow transitions, source lifecycle errors, and conflict rules are documented in docs/WORKFLOW_COMPOSITION.md.
• Specialist tools (jobs, security, science, research, saved searches and events, retrieval receipts, staged search, dependency dossiers) are disabled until the operator grants them (MCP_GRANT_JOBS=1, MCP_GRANT_SECURITY=1, MCP_GRANT_SCIENCE=1, MCP_GRANT_RESEARCH=1, MCP_GRANT_SAVED_SEARCHES=1, MCP_GRANT_RETRIEVAL_RECEIPTS=1, MCP_GRANT_SAVED_SEARCH_EVENTS=1, MCP_GRANT_STAGED_SEARCH=1, MCP_GRANT_DEPENDENCY_DOSSIER=1).
• Sensitive engines (hibp, dehashed) are unreachable from generic routing, categories, and intent profiles, and are rejected by every explicit-engine search path (generic explicit engines, targeted, jobs, security, science) unless the operator sets MCP_TARGETED_SENSITIVE_ALLOWED=1 (deliberate, uniform policy boundary enforced by one shared gate).
• There is no separate "advanced search" tool. Richer include/detail (card vs. full record) semantics on the existing search/read tools and the per-engine capability matrix (slopsearx_list_capabilities) cover requires_* needs — see docs/MCP_SERVER.md §6.14.

Full installation, configuration, client setup (Claude Desktop, Cursor, Hermes Agent — see "Hermes Agent (Nous Research)" in the guide — and generic MCP clients), and agent usage guidance: docs/MCP_SERVER.md.

Quick Start

Build from source

The Dockerfile requires a unique security-refresh value so cached builds cannot silently reuse an old Debian package layer:

DEBIAN_SECURITY_REFRESH="$(date +%s)" docker compose build

VPN and proxy deployments

Google and DuckDuckGo are best-effort HTML-scrape adapters. VPN, proxy, and datacenter IPs can receive consent, challenge, or block pages; configure a Brave API key (ENGINE_BRAVE_API_KEY) for a reliable API-backed web-search source. A Brave key supplements the other active Tier-1 engines; it does not disable them.

Optional Jev routing and result reranking

Set TYPESAFE_API_KEY to enable TypeSafe Jev specialist routing and result reranking automatically. SlopSearX keeps its ordinary general-engine base and asks Jev which eligible specialist engines can add distinctive evidence for the query. Every specialist meeting the 0.65 threshold is added; there is no arbitrary engine-count cap. Without the key, or if Jev is unavailable, the existing deterministic routing path is used unchanged. Explicit engine, category, and media scopes bypass Jev routing.

After retrieval, Jev orders a bounded result-card shortlist using ordinary Score, across general and specialist source tiers. A configured reranker disables specialist promotion, including when advice fails. Without the key, or on invalid/unavailable reranking advice, results retain the configured deterministic presence/RRF order. Providing the key opts into sending query, bounded titles, sanitized URLs and snippets to TypeSafe; sensitive-engine scopes are excluded. See Jev result reranking for payload bounds, timeouts, caching, privacy and the replaceable provider contract.

See TypeSafe Jev specialist routing for configuration, failure behavior, policy boundaries, and routing-card maintenance.

To surface successful scrape responses that parse to zero results, enable the opt-in diagnostic flag. These entries appear in meta.empty_engines; they are warnings rather than failures because a search can legitimately have no matches.

FEATURE_EMPTY_SCRAPE_DIAGNOSTICS=true

Pre-built Docker images are available from GitHub Container Registry for linux/amd64 and linux/arm64. Docker selects the native architecture automatically, including on Apple Silicon. Builds run on every push to main (latest and a short commit SHA) and on version tags (X.Y and X.Y.Z, plus a short commit SHA). Each architecture is built on a native runner, Trivy-scanned, and smoke-tested before the combined manifest is published. Pull requests validate both architectures without publishing. Existing deployment digest pins retain their original platforms until promoted to a new multi-platform image digest.

The repository's own deployment surfaces (docker-compose.yml, k8s/deployment.yaml) pin the image by digest, so they always run exactly the artifact that CI built, Trivy-scanned, and smoke-tested. To deploy the same verified artifact as a given commit, resolve its short-SHA tag to the current digest:

docker buildx imagetools inspect ghcr.io/magnus919/slopsearx:<short-sha> --format '{{json .Manifest.Digest}}'

# Pull and run with Valkey for caching and rate limiting
docker run -d --name valkey valkey/valkey:8-alpine
docker run -d --name slopsearx -p 8080:8080 \
  -e VALKEY_URL=redis://valkey:6379/0 \
  --link valkey \
  ghcr.io/magnus919/slopsearx:latest

# Try it
curl 'http://localhost:8080/search?q=hello+world&format=json'

Standard HTML, CSV, JSON, and RSS formats are controlled by search.formats in config.yaml (or SEARCH_FORMATS=html,json). YAML is an additive SlopSearX format and is always available. A requested standard format that is not enabled returns HTTP 403; malformed search parameters return HTTP 400.

Browser portal

Open / for the human-facing search portal. A query submitted there uses the same search service, routing, cache, ranking, and engine policy as the API; use /search directly for a result page or for compatibility clients. The portal defaults to Dark mode and provides a Darker mode toggle. Operators can set SLOPSEARX_PORTAL_DEFAULT_THEME=darker to change the initial theme. An explicit user choice is retained in browser storage when available. The portal does not grant capabilities that the server-side policy would reject. Explicit selection of a sensitive engine (currently hibp and dehashed by default) also requires MCP_TARGETED_SENSITIVE_ALLOWED=true; the same operator policy is applied before either the browser or MCP surface dispatches a search.

Public search remains account-free. Operators can separately enable the OIDC and Valkey-backed /workflows supervisor console for tenant-scoped workflow inspection and explicitly granted actions. The workflow routes are disabled by default and do not change /, /search, or MCP contracts. See docs/WORKFLOW_PORTAL.md for its direct HTTPS deployment, identity, two-gate authorization, and rollback requirements.

See docs/PORTAL_DEPLOYMENT.md for the browser URL map, proxy/access modes, digest-pinned deployment, smoke check, rollback, and troubleshooting runbook.

License

MIT — see LICENSE.

Scholarly results are grouped by identified work before reranking, with bounded internal source provenance and optional SearXNG Paper metadata.

Caller-directed query planning previews decomposition, terminology variants and evidence-linked follow-ups without dispatch. See query planning.

The v5 PR review canary uses non-binding COMMENT mode and a pinned JavaScript publisher action. Its first publication and same-review replay were confirmed, and containment/re-enable checks succeeded. This configuration enables COMMENT, APPROVE, and REQUEST_CHANGES and disables only Droid Auto Review. It is deployed only when this change is merged after checks and review. Droid Tag and Wiki remain unchanged. Source coverage has been partial and semantic review accuracy is unknown. For run receipts, bounded history, and rollback, see PR review operations. No workflow automatically merges a pull request.