Skip to content

fix(call): honor catalog host: on marketplace upstream URLs - #472

Draft
stonexer wants to merge 2 commits into
mainfrom
cursor/fix-diffbot-extract-host-2c95
Draft

stonexer wants to merge 2 commits into
mainfrom
cursor/fix-diffbot-extract-host-2c95

Conversation

@stonexer

@stonexer stonexer commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

What this does

Catalog host: was a dead field. Diffbot extract endpoints (feedback #178) declared host: api.diffbot.com and documented https://api.diffbot.com/v3/event, but live /call/diffbot.x.extract-event went to https://kg.diffbot.com/kg/v3/v3/event (wrong host + doubled /v3), got nginx 404, and charged $0.

Root cause, verified in code:

  1. Diffbot base_url is https://kg.diffbot.com/kg/v3.
  2. _marketplace_upstream always built provider.base_url + path and never read ep['host'].
  3. _normalize dropped host entirely, so even a later reader of the loaded catalog could not see it.

This PR makes host: real:

  • _normalize keeps a hostname-only host (scheme/path/port rejected).
  • When host is set, _marketplace_upstream builds https://{host} + path and does not prepend provider.base_url's path (/kg/v3).
  • Endpoints without host are unchanged (provider.base_url + path).

host: is not extra_tools. host: rewrites one catalog-id call's netloc while the same credential is injected. extra_tools (GA's google-analytics-admin) still creates a second Tool row for URL-passthrough / named-tool resolution. They coexist; this PR does not move Diffbot onto extra_tools.

Please review before merge. Do not deploy from this PR.

Blast radius

Any catalog endpoint that already declares host: will now resolve to that host instead of provider.base_url. That is the intended fix. 117 loaded rows currently carry host::

  • src/treg/catalog/diffbot.extended.yaml - Extract / Crawl / Bulk / account on api.diffbot.com; web-search on llm.diffbot.com; NL on nl.diffbot.com.
  • src/treg/catalog/google-analytics.extended.yaml - Admin API rows on analyticsadmin.googleapis.com (Data API rows stay on analyticsdata.googleapis.com).
  • src/treg/catalog/google-business-profile.extended.yaml - sibling My Business hosts (mybusinessbusinessinformation, qanda, placeactions, verifications, notifications, businessprofileperformance, legacy mybusiness.googleapis.com).

A catalog-id call to those rows previously hit the provider base_url host (wrong). After this change they hit the declared host. GA extra_tools named-tool / URL-passthrough paths are unchanged.

Out of scope: feedback #179 / connect --provider, settlement/billing/refund, Diffbot extra_tools refactor.

Fragments updated: architecture/catalog.md, architecture/proxy-model.md, architecture/auth-secrets.md.

How it was tested

New unit tests in tests/test_marketplace_call.py:

  • diffbot.x.extract-event / extract-article resolve to https://api.diffbot.com/v3/... and never /kg/v3/v3/.
  • diffbot.companies.enrich (no host) still uses https://kg.diffbot.com/kg/v3/enhance.
  • A GA Admin catalog-id row resolves to analyticsadmin.googleapis.com, not the Data API host.
  • _normalize keeps a hostname and rejects scheme/path/port.

Also tests/test_catalog_validate.py for the hostname-only host: contract.

Ran locally:

  • uv run --frozen python -m pytest -q tests/test_marketplace_call.py tests/test_catalog_validate.py tests/test_catalog_api.py - 413 passed
  • uv run --frozen python scripts/catalog_validate.py diffbot google-analytics google-business-profile - OK
  • Walked all 117 hosted catalog rows through _marketplace_upstream; each origin is https://{host} and none contain /kg/v3/v3/
  • uv run lint-imports - 14 contracts kept

Checklist

  • uv run --with pytest-xdist pytest -n auto -q passes locally (full suite not run here; targeted files above are green)
  • Added or updated tests for the change (if it affects behavior)
  • Updated the relevant docs/context/ fragment (if a subsystem changed)
  • No secrets in the diff (keys, tokens, .env values)
Open in Web Open in Cursor 

Diffbot extract endpoints declared host: api.diffbot.com but _normalize
dropped it and _marketplace_upstream always joined provider.base_url
(https://kg.diffbot.com/kg/v3) onto /v3/event, producing
kg.diffbot.com/kg/v3/v3/event (nginx 404, $0). Keep hostname-only host
on the catalog row and, when set, build https://{host} + path.

Co-authored-by: SToneX <shitianxin@gmail.com>
@github-actions github-actions Bot added the area:docs Documentation & design fragments label Sep 12, 2026
Co-authored-by: SToneX <shitianxin@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:docs Documentation & design fragments

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants