Skip to content

Add deepcrawl max_pages, path rules and JSON format, release 0.6.0 - #6

Merged
fatwang2 merged 1 commit into
mainfrom
deepcrawl-scope-json
Nov 5, 2026
Merged

fatwang2 merged 1 commit into
mainfrom
deepcrawl-scope-json

Conversation

@fatwang2

@fatwang2 fatwang2 commented Nov 5, 2026

Copy link
Copy Markdown
Member

What changed

  • Syncs openapi/search1api.openapi.json byte-for-byte with the current public contract (semantically identical to https://api.search1api.com/openapi.json as of 2026-11-05). For deepcrawl this adds max_pages, include_paths, exclude_paths and format to POST /deepcrawl and format, stats, failed, resultUrl, pages to the status response. The sync also brings in contract changes made since 0.5.0 that this PR does not model in the SDK types: X (search_service: "x") result metadata fields, failureCategory/upstreamStatus on /crawl and /extract errors, and 404/410 responses on those endpoints.
  • start_deepcrawl and deepcrawl on Search1API and AsyncSearch1API take new keyword arguments: max_pages: Optional[int], include_paths: Optional[List[str]], exclude_paths: Optional[List[str]], format: Optional[DeepcrawlFormat] (Literal["zip", "json"]). Only arguments that are set go into the request body. Existing calls send the same body as before.
  • DeepcrawlStatusResponse adds format, stats (DeepcrawlStats: discovered, matched, skipped, crawled, failed), failed (List[DeepcrawlFailedPage]: url, error), resultUrl (Optional[str]) and pages (List[DeepcrawlPage]: url, title: Optional[str], content).
  • README deepcrawl section: path rule syntax, max_pages, and a format="json" example reading result["pages"]. CHANGELOG 0.6.0 (2026-11-05), version bump in pyproject.toml and __version__.

Billing is unchanged: 20 credits per task.

How tested

The CI steps, run locally on Python 3.9.24 and 3.12.3:

python -m pip install -e ".[dev]"
ruff format --check .
ruff check .
mypy --python-version 3.9 src/search1api    # and 3.12
pytest                                       # 23 passed
python -m build                              # search1api-0.6.0 sdist + wheel

New tests use httpx.MockTransport (no live API calls): sync deepcrawl with all new arguments and a format="json" completion, sync start_deepcrawl sending only set fields, and async deepcrawl with the same arguments.

After release

  • Publishing to PyPI happens when a v0.6.0 GitHub Release is published after merge; the release workflow checks the tag against pyproject.toml.
  • pip install search1api==0.6.0, then one real client.deepcrawl(url, max_pages=5, include_paths=["/docs"], format="json") call: confirm the request is accepted, result["pages"] holds at most 5 pages under /docs, and result["stats"] is present.
  • Check the PyPI project page renders the updated README.

🤖 Generated with Claude Code

Sync the OpenAPI snapshot with the live contract (adds max_pages,
include_paths, exclude_paths and format to POST /deepcrawl and the
stats/failed/resultUrl/pages fields to the status response) and accept the
new fields as keyword arguments on start_deepcrawl and deepcrawl in the sync
and async clients. Unset arguments are not sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fatwang2
fatwang2 merged commit 0608309 into main Nov 5, 2026
2 checks passed
@fatwang2
fatwang2 deleted the deepcrawl-scope-json branch November 5, 2026 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant