Skip to content

feat: support structured fetch extraction #90

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cjumel merged 2 commits into from
Aug 27, 2026
Merged

feat: support structured fetch extraction #90

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4d97742
chore: remove cross field valition from SDK
cjumel Aug 27, 2026
7170ea2
feat: support structured fetch extraction
cjumel Aug 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,12 @@ Keep the SDK aligned with the current public, stable Linkup API while:

- implementing a Pythonic public interface,
- documenting the features through docstrings,
- and adding type safety through type hints or input data validation with `pydantic` models when
relevant.
- and adding type safety through type hints and `pydantic` models when relevant.

However, **do not**:

- duplicate API request-semantic validation in the SDK: SDK models may validate local data shape,
but the API owns cross-field, conditional, and product constraints.

## Working Rules

Expand Down
17 changes: 12 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ markdown format, together with the website's favicon URL.
You can use the `render_js` flag to execute the JavaScript code of the page before returning the
content, set `include_raw_content` to include the raw page content and its content type in the
output, or set `mode` to `"pro"` for significantly higher success rates on hard-to-retrieve pages.
You can also pass an object JSON `schema`, with optional `instructions`, to extract structured data.

```python
import linkup
Expand All @@ -149,18 +150,24 @@ fetch_response: linkup.FetchResponse = client.fetch(
render_js=True,
include_raw_content=True,
mode="pro",
schema={
"type": "object",
"properties": {"title": {"type": "string"}},
},
instructions="Extract the page title.",
)
print(fetch_response.model_dump())
```

Which prints:

```bash
```python
{
markdown="Get started for free, no credit card required...",
favicon="https://favicons.linkup.so?domain=docs.linkup.so",
raw_content="<!DOCTYPE html><html lang=\"en\"><head>...</head><body>...</body></html>",
content_type="html"
"markdown": "The production-grade web search API for AI.",
"favicon": "https://favicons.linkup.so?domain=docs.linkup.so",
"raw_content": "<!DOCTYPE html><html lang=\"en\"><head>...</head><body>...</body></html>",
"content_type": "html",
"data": {"title": "The production-grade web search API for AI."},
}
```

Expand Down
75 changes: 47 additions & 28 deletions src/linkup/_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -261,10 +261,10 @@ def search(
True

Raises:
TypeError: If structured_output_schema is not provided or is not a string, dictionary,
or pydantic.BaseModel when output_type is "structured".
LinkupInvalidRequestError: If structured_output_schema doesn't represent a valid object
JSON schema when output_type is "structured".
TypeError: If structured_output_schema is not a string, dictionary, or
pydantic.BaseModel when provided.
LinkupInvalidRequestError: If the request parameters are invalid, including an invalid
or missing structured_output_schema when output_type is "structured".
LinkupAuthenticationError: If the Linkup API key is invalid.
LinkupInsufficientCreditError: If you have run out of credit.
LinkupBudgetLimitExceededError: If the API key has reached its configured budget limit.
Expand Down Expand Up @@ -458,10 +458,10 @@ async def async_search(
True

Raises:
TypeError: If structured_output_schema is not provided or is not a string, dictionary,
or pydantic.BaseModel when output_type is "structured".
LinkupInvalidRequestError: If structured_output_schema doesn't represent a valid object
JSON schema when output_type is "structured".
TypeError: If structured_output_schema is not a string, dictionary, or
pydantic.BaseModel when provided.
LinkupInvalidRequestError: If the request parameters are invalid, including an invalid
or missing structured_output_schema when output_type is "structured".
LinkupAuthenticationError: If the Linkup API key is invalid.
LinkupInsufficientCreditError: If you have run out of credit.
LinkupBudgetLimitExceededError: If the API key has reached its configured budget limit.
Expand Down Expand Up @@ -539,9 +539,8 @@ def research(
The newly created research task, with "pending" status and no output.

Raises:
TypeError: If structured_output_schema is not provided when output_type is
"structured", or if it is not a string, dictionary, or pydantic.BaseModel when
provided.
TypeError: If structured_output_schema is not a string, dictionary, or
pydantic.BaseModel when provided.
LinkupInvalidRequestError: If the request parameters are invalid.
LinkupAuthenticationError: If the Linkup API key is invalid.
LinkupInsufficientCreditError: If you have run out of credit.
Expand Down Expand Up @@ -612,9 +611,8 @@ async def async_research(
The newly created research task, with "pending" status and no output.

Raises:
TypeError: If structured_output_schema is not provided when output_type is
"structured", or if it is not a string, dictionary, or pydantic.BaseModel when
provided.
TypeError: If structured_output_schema is not a string, dictionary, or
pydantic.BaseModel when provided.
LinkupInvalidRequestError: If the request parameters are invalid.
LinkupAuthenticationError: If the Linkup API key is invalid.
LinkupInsufficientCreditError: If you have run out of credit.
Expand Down Expand Up @@ -1005,6 +1003,8 @@ def fetch(
timeout: float | None = None,
include_raw_content: bool | None = None,
mode: Literal["standard", "pro"] | None = None,
schema: type[pydantic.BaseModel] | dict[str, Any] | str | None = None,
instructions: str | None = None,
) -> LinkupFetchResponse:
"""Fetch the content of a web page using the Linkup API /fetch endpoint.

Expand All @@ -1025,6 +1025,10 @@ def fetch(
response.
mode: The fetch strategy to use. "pro" delivers significantly higher success rates on
hard-to-retrieve pages.
schema: An object JSON schema describing an optional structured data to extract.
Supported formats are a pydantic.BaseModel, a Python dictionary, or a JSON string.
instructions: Optional instructions guiding structured data extraction if schema is
passed.

Returns:
The response of the web page fetch, containing the web page content.
Expand All @@ -1039,13 +1043,15 @@ def fetch(
type.
LinkupTimeoutError: If the request times out.
"""
params: dict[str, str | bool] = self._get_fetch_params(
params: dict[str, Any] = self._get_fetch_params(
url=url,
include_raw_html=include_raw_html,
include_raw_content=include_raw_content,
render_js=render_js,
extract_images=extract_images,
mode=mode,
schema=schema,
instructions=instructions,
)

response: httpx.Response = self._request(
Expand All @@ -1066,6 +1072,8 @@ async def async_fetch(
timeout: float | None = None,
include_raw_content: bool | None = None,
mode: Literal["standard", "pro"] | None = None,
schema: type[pydantic.BaseModel] | dict[str, Any] | str | None = None,
instructions: str | None = None,
) -> LinkupFetchResponse:
"""Asynchronously fetch the content of a web page using the Linkup API /fetch endpoint.

Expand All @@ -1086,6 +1094,10 @@ async def async_fetch(
response.
mode: The fetch strategy to use. "pro" delivers significantly higher success rates on
hard-to-retrieve pages.
schema: An object JSON schema describing an optional structured data to extract.
Supported formats are a pydantic.BaseModel, a Python dictionary, or a JSON string.
instructions: Optional instructions guiding structured data extraction if schema is
passed.

Returns:
The response of the web page fetch, containing the web page content.
Expand All @@ -1100,13 +1112,15 @@ async def async_fetch(
type.
LinkupTimeoutError: If the request times out.
"""
params: dict[str, str | bool] = self._get_fetch_params(
params: dict[str, Any] = self._get_fetch_params(
url=url,
include_raw_html=include_raw_html,
include_raw_content=include_raw_content,
render_js=render_js,
extract_images=extract_images,
mode=mode,
schema=schema,
instructions=instructions,
)

response: httpx.Response = await self._async_request(
Expand Down Expand Up @@ -1458,11 +1472,6 @@ def _get_search_params(
include_inline_citations: bool | None,
include_sources: bool | None,
) -> dict[str, str | bool | int | list[str]]:
if output_type == "structured" and structured_output_schema is None:
raise TypeError(
"structured_output_schema must be provided when output_type is 'structured'"
)

params: dict[str, str | bool | int | list[str]] = {
"q": query,
"depth": depth,
Expand Down Expand Up @@ -1512,11 +1521,6 @@ def _get_research_params(
exclude_domains: list[str] | None,
include_domains: list[str] | None,
) -> dict[str, str | bool | list[str]]:
if output_type == "structured" and structured_output_schema is None:
raise TypeError(
"structured_output_schema must be provided when output_type is 'structured'"
)

params: dict[str, str | bool | list[str]] = {
"q": query,
"outputType": output_type,
Expand Down Expand Up @@ -1630,6 +1634,8 @@ def _get_tasks_payload(self, tasks: list[LinkupTaskInput]) -> list[dict[str, Any
render_js=task.render_js,
extract_images=task.extract_images,
mode=task.mode,
schema=task.schema_,
instructions=task.instructions,
),
}
)
Expand Down Expand Up @@ -1666,8 +1672,10 @@ def _get_fetch_params(
render_js: bool | None,
extract_images: bool | None,
mode: Literal["standard", "pro"] | None,
) -> dict[str, str | bool]:
params: dict[str, str | bool] = {
schema: type[pydantic.BaseModel] | str | dict[str, Any] | None,
instructions: str | None,
) -> dict[str, Any]:
params: dict[str, Any] = {
"url": url,
}
if include_raw_html is not None:
Expand All @@ -1680,6 +1688,17 @@ def _get_fetch_params(
params["extractImages"] = extract_images
if mode is not None:
params["mode"] = mode
if schema is not None:
if isinstance(schema, str):
params["schema"] = schema
elif isinstance(schema, dict):
params["schema"] = json.dumps(schema)
elif issubclass(schema, pydantic.BaseModel):
params["schema"] = json.dumps(schema.model_json_schema())
else:
raise TypeError(f"Unexpected schema type: '{type(schema)}'")
if instructions is not None:
params["instructions"] = instructions
return params

def _parse_search_response(
Expand Down
24 changes: 8 additions & 16 deletions src/linkup/_types.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,7 @@ class LinkupFetchResponse(_LinkupBaseModel):
content_type: The type of the raw page content, if returned.
raw_html: The optional raw HTML content. Deprecated; use raw_content instead.
images: The optional list of extracted images.
data: Structured data extracted from the webpage, if a schema was provided.
"""

markdown: str
Expand All @@ -127,6 +128,7 @@ class LinkupFetchResponse(_LinkupBaseModel):
content_type: str | None = pydantic.Field(default=None, validation_alias="contentType")
raw_html: str | None = pydantic.Field(default=None, validation_alias="rawHtml")
images: list[LinkupFetchImageExtraction] | None = pydantic.Field(default=None)
data: JSONObject | None = None


class LinkupSearchTaskInput(_LinkupBaseModel):
Expand Down Expand Up @@ -170,14 +172,6 @@ class LinkupSearchTaskInput(_LinkupBaseModel):
pydantic.Field(default=None, validation_alias="structuredOutputSchema")
)

@pydantic.model_validator(mode="after")
def _validate_structured_output_schema(self) -> LinkupSearchTaskInput:
if self.output_type == "structured" and self.structured_output_schema is None:
raise ValueError(
"structured_output_schema must be provided when output_type is 'structured'"
)
return self


class LinkupResearchTaskInput(_LinkupBaseModel):
"""Input for creating or retrieving a research task.
Expand Down Expand Up @@ -214,14 +208,6 @@ class LinkupResearchTaskInput(_LinkupBaseModel):
pydantic.Field(default=None, validation_alias="structuredOutputSchema")
)

@pydantic.model_validator(mode="after")
def _validate_structured_output_schema(self) -> LinkupResearchTaskInput:
if self.output_type == "structured" and self.structured_output_schema is None:
raise ValueError(
"structured_output_schema must be provided when output_type is 'structured'"
)
return self


class LinkupFetchTaskInput(_LinkupBaseModel):
"""Input for creating or retrieving a fetch task.
Expand All @@ -234,6 +220,8 @@ class LinkupFetchTaskInput(_LinkupBaseModel):
render_js: Whether JavaScript rendering should be enabled.
extract_images: Whether image extraction should be enabled.
mode: The fetch strategy to use.
schema_: The object JSON schema describing the data to extract, if any.
instructions: Instructions guiding structured data extraction, if any.
"""

url: str
Expand All @@ -244,6 +232,10 @@ class LinkupFetchTaskInput(_LinkupBaseModel):
render_js: bool | None = pydantic.Field(default=None, validation_alias="renderJs")
extract_images: bool | None = pydantic.Field(default=None, validation_alias="extractImages")
mode: Literal["standard", "pro"] | None = None
schema_: type[pydantic.BaseModel] | str | dict[str, Any] | None = pydantic.Field(
default=None, validation_alias="schema"
)
instructions: str | None = None


LinkupTaskInput = LinkupSearchTaskInput | LinkupFetchTaskInput | LinkupResearchTaskInput
Expand Down
Loading
Loading