Skip to content

docs(scrape): document the PDF/document size limit and whether it can be raised #1362

Description

@RichardBray

Parse documents a 50MB per-request cap. Scrape documents no size limit at all, and the two behave very differently when you cross one.

What I observed

Same 63MB PDF (a 60MB journal issue), both endpoints:

endpoint result time
/v2/parse Uploaded file exceeds maximum size of 50MB 9s
/v2/scrape SCRAPE_TIMEOUT 4m40s (at timeout: 280000)

A 26MB PDF through /v2/scrape parsed in 1.9s, so this is not "big PDFs are slow". I also ruled out a slow origin: curl pulled the full 63MB in 4.5s.

So scrape appears to have a size ceiling in the same neighborhood, but instead of a size error you get a timeout after several minutes. In a pipeline that reads as a flaky source, so the natural response is to retry, which costs more time and (per the billing docs, credits are charged whenever infrastructure processes a request) possibly more credits.

Requests

  1. Document scrape's document size limit. Even "documents above ~50MB may not complete" would let people plan.
  2. Say whether it can be raised. My understanding is that this ceiling is account-level rather than a request parameter; there is no setting for it in the dashboard (Overview / Activity Logs / Usage / API Keys / Settings). If raising it is possible, the docs should say who to ask; if not, it should say that too.
  3. Consider a clearer failure than SCRAPE_TIMEOUT. An explicit "document exceeds size limit" would be far more actionable. Filing here since it is a docs repo, happy to move that third point to the main repo if preferred.

Why it matters

The Parse page tells you the limit up front and fails fast and cheap. Scrape is the endpoint most people point at PDFs, and it currently fails slow and silent.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions