ScrapeVerse API

A production-ready scraping API that crawls a public website and normalizes its content into deterministic JSON. Built with Spring Boot 3, Java 21, jsoup, Caffeine, and the official Bright Data Web Scraper API v3.

Integrating the API? See DOCUMENTATION-AND-GUIDE.md for the full API contract, status/error codes, rate limits, frontend and middleware guides. Live instance: https://scrapeverse-api.onrender.com (Swagger UI at /swagger-ui.html).

Terminal client: an interactive Textual dashboard + headless CLI lives in tui-client/ (pip install -r tui-client/requirements.txt, then python tui-client/scrapeverse_tui.py).

Features

• Crawl engine: deterministic sequential BFS with maxPages/maxDepth caps, same-domain crawling, URL normalization/deduplication, canonical URL detection and rel=next / URL-pattern pagination.
• Content extraction: semantic HTML extraction (headings, text, links, images, tables, lists, forms, content blocks) plus meta / Open Graph / Twitter Card / JSON-LD structured data.
• Providers: brightdata (datasets v3 API — Web Scraper / Crawl API / Live Crawler; trigger → progress → snapshot, retries, circuit breaker), brightdata-unlocker (Web Unlocker API — real-browser JS rendering for SPAs) or direct (plain compliant GET for static sites and local dev). SCRAPER_PROVIDER=auto picks Bright Data when configured, otherwise the Web Unlocker, otherwise direct. Failover chains providers in the same order.
• Security:
  • SSRF protection on the start URL and every discovered URL: scheme allow-list, blocked hostname namespaces, DNS resolution + double-resolution (DNS rebinding) checks, and IPv4/IPv6/CIDR block lists.
  • robots.txt compliance (standard * rules) with a short-lived cache.
  • In-memory fixed-window rate limiting per client IP (429 + Retry-After).
  • Request-body size filter, security headers, per-request MDC request IDs, and URL query sanitization in logs. Secrets are never logged.
• API contract: stable success / request / site / pages / statistics / warnings / errors shape with every key always present. Errors use a consistent {"success":false,"error":{"code","message","details"}} shape with no stack traces.

Requirements

• JDK 21
• Maven 3.9+
• Docker (only for ContainerizedScrapeIT, skipped automatically when absent)

Build & test

mvn clean package        # build the jar
mvn test                 # unit + integration tests (164 tests)
mvn verify               # also runs failsafe ITs (incl. Docker-based test)
java -jar target/scrapeverse-api.jar

Run locally

export SCRAPER_PROVIDER=direct   # no Bright Data credentials needed
mvn spring-boot:run
# or
java -jar target/scrapeverse-api.jar

Then:

curl http://localhost:8080/api/v1/health
curl http://localhost:8080/api/v1/info
curl -X POST http://localhost:8080/api/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'
# GET variant (url must be URL-encoded)
curl -G http://localhost:8080/api/v1/scrape \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "maxPages=10" \
  --data-urlencode "includeImages=false"

Interactive OpenAPI docs: http://localhost:8080/swagger-ui.html

Configuration

Everything is configurable through environment variables (defaults in src/main/resources/application.yml). See .env.example for the full list.

Variable	Default	Purpose
PORT	8080	HTTP port
SCRAPER_PROVIDER	auto	auto / brightdata / brightdata-unlocker / direct
SCRAPER_MAX_PAGES	25	max pages per scrape
SCRAPER_MAX_DEPTH	2	max crawl depth
SCRAPER_TIMEOUT_MS	60000	hard deadline per scrape
SCRAPER_RESPECT_ROBOTS	true	obey robots.txt
SCRAPER_SAME_DOMAIN_ONLY	true	stay on the starting domain
SCRAPER_RATE_LIMIT	30	requests per window per client IP
SCRAPER_RATE_LIMIT_WINDOW_MS	60000	rate limit window
SCRAPER_CACHE_ENABLED	true	in-memory page cache
BRIGHTDATA_API_KEY	empty	Bright Data API token
BRIGHTDATA_DATASET_ID	empty	Bright Data dataset id
BRIGHTDATA_FORMAT	json	output format: json/html/markdown/text
BRIGHTDATA_API_BASE_URL	https://api.brightdata.com	API base URL
BRIGHTDATA_UNLOCKER_ZONE	empty	Web Unlocker zone name (JS rendering)
BRIGHTDATA_UNLOCKER_API_KEY	empty	Web Unlocker token (defaults to BRIGHTDATA_API_KEY)
BRIGHTDATA_UNLOCKER_FORMAT	raw	Web Unlocker output format: raw/markdown
BRIGHTDATA_UNLOCKER_RENDER	true	force browser JavaScript rendering
CORS_ALLOWED_ORIGINS	*	allowed CORS origins

API reference

POST /api/v1/scrape

Same endpoint is also available via GET /api/v1/scrape with the request fields as query parameters (url required and URL-encoded; maxPages, maxDepth, includeLinks, includeImages, includeMetadata, includeTables, includeStructuredData optional).

Request body:

{
  "url": "https://example.com",
  "maxPages": 10,
  "maxDepth": 2,
  "includeLinks": true,
  "includeImages": true,
  "includeMetadata": true,
  "includeTables": true,
  "includeStructuredData": true
}

maxPages and maxDepth are capped by server configuration. The include* flags empty the corresponding response fields when false.

Example response (abbreviated):

{
  "success": true,
  "request": {
    "url": "https://example.com",
    "timestamp": "2026-08-15T05:36:42Z",
    "pagesRequested": 25,
    "pagesProcessed": 1
  },
  "site": { "title": "Example Domain", "description": null, "language": "en",
            "canonicalUrl": null, "robots": null, "favicon": null },
  "pages": [
    {
      "url": "https://example.com",
      "depth": 0,
      "title": "Example Domain",
      "headings": { "h1": ["Example Domain"], "h2": [], "h3": [], "h4": [], "h5": [], "h6": [] },
      "links": [{ "text": "Learn more", "url": "https://iana.org/domains/example", "rel": [] }],
      "images": [], "tables": [], "lists": [], "forms": [],
      "metadata": {}, "structuredData": [], "openGraph": {}, "twitterCard": {},
      "schemaOrg": [], "contentBlocks": []
    }
  ],
  "statistics": { "pagesDiscovered": 1, "pagesProcessed": 1, "linksFound": 1,
                  "imagesFound": 0, "tablesFound": 0, "formsFound": 0 },
  "warnings": [],
  "errors": []
}

Error codes: INVALID_URL, UNSUPPORTED_PROTOCOL, SSRF_BLOCKED, ROBOTS_BLOCKED, INVALID_REQUEST, PAYLOAD_TOO_LARGE, RATE_LIMITED, PROVIDER_ERROR, PROVIDER_TIMEOUT, PROVIDER_UNAVAILABLE, PROVIDER_AUTH_ERROR, INVALID_PROVIDER_RESPONSE, EXTRACTION_FAILED, SCRAPE_TIMEOUT, CRAWL_LIMIT_REACHED, INTERNAL_ERROR.

Example error:

{
  "success": false,
  "error": { "code": "SSRF_BLOCKED", "message": "hostname '127.0.0.1' is a blocked address", "details": {} }
}

GET /api/v1/health

{ "status": "UP", "service": "ScrapeVerse API", "version": "1.0.0",
  "provider": { "name": "direct", "configured": true, "supportsRendering": false,
                "circuitState": "CLOSED", "consecutiveFailures": 0 } }

GET /api/v1/info

Provider capabilities and configured limits.

Docker

docker build -t scrapeverse-api .
docker run -p 8080:8080 -e SCRAPER_PROVIDER=direct scrapeverse-api

Deploy to Render

1. Push the repository to GitHub.
2. In Render, choose New → Blueprint, select the repo and the render.yaml blueprint, or create a new Web Service pointing at the repo (build command mvn clean package -DskipTests, start command java -jar target/scrapeverse-api.jar, health check path /api/v1/health).
3. Set environment variables in the Render dashboard (see .env.example). For production set BRIGHTDATA_API_KEY, BRIGHTDATA_DATASET_ID and SCRAPER_PROVIDER=brightdata; for JS-heavy targets also set BRIGHTDATA_UNLOCKER_ZONE and use SCRAPER_PROVIDER=brightdata-unlocker.

Scraping any website with Bright Data

The brightdata provider targets the datasets v3 API shared by the Web Scraper, Crawl API and Live Crawler products. To scrape arbitrary websites (JS-rendered, bot-protected):

1. In the Bright Data Control Panel open the Crawl API page and create a dataset (no-code: enter a root URL; or via API).
2. Copy the Crawl API dataset id (format gd_xxxxxxxx) into BRIGHTDATA_DATASET_ID, and the product API token into BRIGHTDATA_API_KEY.
3. Optionally set BRIGHTDATA_FORMAT=json (default, structured), html (raw HTML), markdown, or text. Markdown/Text records are wrapped into HTML so the extraction layer can still produce titles, text and metadata.
4. Set SCRAPER_PROVIDER=brightdata (or auto).

JS-rendered sites (SPAs): client-side rendered apps (React/Vue, <div id="root">) serve an empty shell over the wire. Two ways to get the rendered DOM:

• Preferred — Web Unlocker (brightdata-unlocker provider): create a zone in the Control Panel (Web Access → Web Unlocker → Create zone), set BRIGHTDATA_UNLOCKER_ZONE and either BRIGHTDATA_UNLOCKER_API_KEY or the shared BRIGHTDATA_API_KEY, then set SCRAPER_PROVIDER=brightdata-unlocker (or auto). The provider calls POST /request with render:true and returns the fully rendered page.
• Crawl API dataset: the dataset must have JavaScript rendering / unlocking enabled in the Control Panel to return the rendered DOM (not all datasets expose this toggle).

Until rendering is available:

• A page that returned only the un-rendered shell is reported with a JS_RENDER_REQUIRED warning.
• Client-side routes the server does not serve (origin 404, e.g. /projects on a history-router SPA) are reported as HTTP 404 NOT_FOUND instead of a generic extraction failure. The trigger always requests include_errors=true so Bright Data includes per-record error detail.

Do not use a site-specific Web Scraper dataset (e.g. an Amazon scraper) unless you only target that site — those reject URLs for other domains.

Security model

• SSRF checks run on the initial URL and on every URL discovered during the crawl, before any outbound request.
• robots.txt is honored per origin and cached briefly.
• Responses never include stack traces or internal class names.
• Log lines carry a requestId/correlationId; sensitive URL query parameters and API credentials are redacted.

Auto-healing

The service recovers from provider outages without manual intervention:

• Per-provider circuit breakers trip OPEN after N consecutive failures (SCRAPER_CIRCUIT_BREAKER_FAILURE_THRESHOLD) and re-enter service with a probe request after SCRAPER_CIRCUIT_BREAKER_OPEN_DURATION_MS. Each provider has its own breaker, so one provider's outage never blocks the other.
• Transient retry retries flaky provider calls (timeouts, 5xx, rate limits) with exponential backoff before counting a failure (SCRAPER_RETRY_MAX_ATTEMPTS, SCRAPER_RETRY_BACKOFF_*).
• Failover routes page fetches to the other configured provider when the primary is unavailable (SCRAPER_FAILOVER_ENABLED, default on).
• Active recovery probe periodically issues a lightweight fetch through a degraded provider so recovery is detected even with zero user traffic (SCRAPER_PROBE_ENABLED, SCRAPER_PROBE_INTERVAL_MS, SCRAPER_PROBE_URL).
• /api/v1/health reports each provider's circuit state under provider.circuits and returns DEGRADED only when every provider circuit is open. Docker HEALTHCHECK and the Render health check restart the container when the process itself is unhealthy.

Limitations

• The in-memory rate limiter and caches are single-instance; scale horizontally with a shared store if you run many replicas.
• The direct provider cannot execute JavaScript and will report structured failures for sites that block standard HTTP clients. Use brightdata for rendered/JS-heavy targets.

Tests

• Unit: URL/SSRF validation, robots parsing, normalization, rate limiting, circuit breaker, HTML content parsing, all providers (MockWebServer), crawl orchestration.
• Integration: full Spring context via MockMvc (contract, validation, SSRF, rate limit, provider failure mapping), and an end-to-end crawl against a local MockWebServer site graph.
• ContainerizedScrapeIT runs the full pipeline against a real nginx container when Docker is available and is skipped otherwise.