Add deepcrawl max_pages, path rules and JSON format, release 0.6.0

CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## 0.6.0 - 2026-11-05
+
+Adds keyword arguments to `start_deepcrawl` and `deepcrawl` on `Search1API` and `AsyncSearch1API`: `max_pages` (1-1000, default 1000, applied after the path rules in discovery order), `include_paths` and `exclude_paths` (up to 20 URL path rules each; a rule without `*` is a path prefix on segment boundaries, a rule with `*` matches the whole path; exclusion wins) and `format` (`"zip"`, the default, or `"json"`). Unset arguments are not sent. `DeepcrawlStatusResponse` adds `format`, `stats` (`discovered`, `matched`, `skipped`, `crawled`, `failed`), `failed` (`url`, `error`) and, for `format="json"`, `resultUrl` and `pages` (`url`, `title`, Markdown `content`; omitted above 5 MB, use `resultUrl`). Still 20 credits per task. The bundled OpenAPI snapshot is re-synced with the live contract.
+
 ## 0.5.0 - 2026-11-03
 
 Adds `maps(query, ...)` on `Search1API` and `AsyncSearch1API` for `POST /maps`: one Google Maps page of up to 20 places with name, place ID and Maps link, plus address, phone, website, rating, review count, coordinates and opening hours when available. Keyword arguments: `country`, `language`, `location` (`latitude`, `longitude`, optional `zoom`), `page` (1-6, pages after the first need `location`) and `max_results`. 3 credits per completed page; API key or OAuth only. The bundled OpenAPI snapshot also records the 50-item batch limit for search, news and crawl.

README.md

@@ -101,6 +101,32 @@ result = client.deepcrawl("https://example.com", type="all")
 print(result["zipUrl"])
 ```
 
+`max_pages` (1-1000, default 1000) caps the number of pages. `include_paths`
+and `exclude_paths` (up to 20 rules each) are matched against the URL path: a
+rule without `*` is a path prefix on segment boundaries (`/docs` matches
+`/docs/api` but not `/docsearch`); a rule with `*` must match the whole path,
+and `*` also matches `/`. Exclusion wins, and `max_pages` applies after the path
+rules, in discovery order. `format="json"` returns the pages in the result
+instead of a ZIP:
+
+```python
+result = client.deepcrawl(
+    "https://example.com",
+    max_pages=50,
+    include_paths=["/docs"],
+    format="json",
+)
+
+for page in result.get("pages", []):
+    print(page["url"], page["title"], len(page["content"]))
+```
+
+Each page has `url`, `title` (may be `None`) and `content` as Markdown.
+`pages` is omitted when the result is larger than 5 MB; download `resultUrl`
+instead, a JSON file with the same `pages` and `failed` lists. Finished tasks
+also report `stats` (`discovered`, `matched`, `skipped`, `crawled`, `failed`)
+and `failed` (`url`, `error`). A task costs 20 credits regardless of page count.
+
 Use `start_deepcrawl`, `get_deepcrawl_status`, and `wait_for_deepcrawl` when
 the application needs to control persistence or polling itself.