Skip to content

Add AnyAPI: 36 core endpoints, a generated 150-endpoint coverage tier, and measured per-call prices - #398

Open
kev1n wants to merge 24 commits into
superdesigndev:mainfrom
kev1n:anyapi-listing
Open

kev1n wants to merge 24 commits into
superdesigndev:mainfrom
kev1n:anyapi-listing

Conversation

@kev1n

@kev1n kev1n commented Sep 9, 2026

Copy link
Copy Markdown

Contact: support@getanyapi.com

AnyAPI (https://getanyapi.com) is a per-request USD gateway over 362 normalized endpoints across
social, search, maps, commerce and web scraping. One key, one prepaid wallet, no subscription. Each
SKU routes across several upstream sources behind one schema: the cheapest serves, a failed attempt
is retried on the next, and a request no source could serve is not charged.

Two files. anyapi.yaml is the curated core tier and stands alone. anyapi.extended.yaml is
generated from our OpenAPI document by scripts/catalog_ingest.py, and is easy to drop if you
would rather run the ingest yourselves.

Eligibility

Requirement AnyAPI
Self-serve keys, no sales call POST https://api.getanyapi.com/agent/signup returns a working key with no account at all (starter credit, per-key spend cap, expires in 7 days).
Key in a header X-API-Key, or Authorization: Bearer. Never in the path.
Free probe that rejects a bad key GET /v1/balance — reads the wallet, runs nothing, charges nothing.
Published pricing https://getanyapi.com/catalog, one public page per SKU, plus the rate card below.
OpenAPI at a stable URL https://api.getanyapi.com/openapi.json, with a per-operation example request body.

Probe bad-key behavior, observed live 2026-09-08: a bogus key returns
401 {"error":"invalid credential","code":"unauthorized"}, a missing key returns
401 {"error":"missing credential","code":"unauthorized"}, a valid key returns
200 {"usd": <balance>}. There is no 200-on-bad-key case, so the registry entry needs no
token_verify_field / token_ok_field / token_reject_field.

Pricing

Pay per request from a prepaid USD wallet, no subscription. Rate card:
GET https://api.getanyapi.com/v1/apis (and /v1/apis/{sku}) returns pricing.from,
pricing.failoverMaxUsd and the per-source price list for every SKU, free.

cost.value is a measured charge, not a list price. On 129 of the 186 rows it is the p90 of
every USD AnyAPI actually billed for that SKU over the trailing 60 days, exported to
scripts/data/anyapi_measured_charges.json, which is why cost.source is observed. The rate
card's cheapest-source price reads far below what buyers pay wherever the cheap source rarely wins
(ebay.search advertises $0.0005 against a $0.03795 median real charge) and far above it on a
per-result SKU nobody calls at the input maximum. 57 rows with fewer than 5 charged calls in the
window keep rate_card_api, because there is nothing to measure.

Three consequences worth knowing before you read a price:

  • A p90 is not a ceiling. About one call in ten settles above it, which
    domain/money/settlement.py:133-134 already designs for. Measured over the same 60 days under
    our older prices, that overrun was $18.66 on $695.05 settled, 2.7%.
  • Measured prices are clamped at both ends, to today's cheapest advertised price and to
    pricing.failoverMaxUsd, so no row quotes a price no source can charge. 19 rows are clamped and
    each says so in its own note.
  • Five core rows list their cheapest source instead (ANYAPI_CORE_ROWS_PRICED_AT_CHEAPEST_SOURCE).
    On those the p90 measures which source won rather than how much work the call did: maps.search
    runs $0.00175, $0.00297 and $0.06005 across three sources that all return 11-12 items, so its
    $0.07734 p90 would misprice the shelf 44x.

cost.reported_charge: {path: costUsd, unit: usd} is set on all 186 rows: every 2xx body carries
the exact USD charged, so _observed_cost_micro settles the real number rather than any estimate
above. ?max_cost_usd= on any run refuses a source dearer than a figure you name.

Self-verification ledger — run live 2026-09-10 on our own account

Every test_request in anyapi.yaml exactly as committed. 36/36 HTTP 200, $0.08993 for the
run.
Claimed is the cost.value in this PR; metered is the costUsd that call returned.

capability AnyAPI SKU HTTP claimed metered ms
x.search.posts twitter.search 200 $0.00075 $0.00065 * 3444
x.user.profile twitter.profile 200 $0.00022 $0.00022 1408
instagram.search.reels instagram.reels_search 200 $0.0015 $0.0012 * 6029
google.serp.organic google.search 200 $0.0009 $0.0009 2588
linkedin.search.posts linkedin.search_posts 200 $0.002 $0.0012 * 5433
x.user.posts twitter.user_tweets 200 $0.0005 $0.0005 3611
linkedin.search.jobs linkedin.jobs_thin 200 $0.0005 $0.0005 3228
tiktok.user.profile tiktok.profile 200 $0.0012 $0.0005 * 10252
linkedin.user.profile linkedin.profile_thin 200 $0.0012 $0.0008 * 9078
google.serp.maps maps.search 200 $0.00175 $0.00175 2481
linkedin.search.posts linkedin.search_posts_full 200 $0.0143 $0.005 * 4084
instagram.post.detail instagram.post 200 $0.0015 $0.0012 * 6276
instagram.user.reels instagram.user_reels 200 $0.002 $0.0012 * 4438
instagram.user.posts instagram.user_posts 200 $0.0024 $0.0012 * 4865
linkedin.user.profile linkedin.profile 200 $0.0042 $0.004 * 3523
x.user.posts twitter.user_posts 200 $0.00075 $0.0005 * 3531
instagram.hashtag.posts instagram.search_hashtag 200 $0.002 $0.0012 * 5689
tiktok.user.videos tiktok.profile_videos 200 $0.0012 $0.0012 3280
instagram.post.comments instagram.post_comments 200 $0.00144 $0.0012 * 2968
tiktok.search.general tiktok.search_top 200 $0.002 $0.0012 * 5025
tiktok.video.comments tiktok.video_comments 200 $0.002 $0.0008 * 2395
facebook.group.posts facebook.group_posts 200 $0.002 $0.0012 * 3988
facebook.user.posts facebook.profile_posts 200 $0.002 $0.0012 * 5360
facebook.post.detail facebook.post 200 $0.002 $0.0012 * 2897
tiktok.search.videos tiktok.search_keyword 200 $0.0012 $0.0012 2164
tiktok.video.captions tiktok.video_transcript 200 $0.002 $0.0012 * 2592
google.serp.news google.news 200 $0.00099 $0.00099 1394
x.post.comments twitter.replies 200 $0.0005 $0.0005 3567
linkedin.company.people linkedin.company_employees 200 $0.105 $0.033 * 6397
linkedin.email.work.find linkedin.email 200 $0.011 $0.011 9475
google.serp.ai_overview google.ai_overview 200 $0.0018 $0.0018 93797
linkedin.search.jobs linkedin.jobs 200 $0.0273 $0.0044 * 7128
tiktok.search.users tiktok.search_users 200 $0.0012 $0.0012 2298
google.maps.reviews maps.reviews 200 $0.0035 $0.00138 * 7913
google.serp.maps maps.place 200 $0.00175 $0.00175 2911
google.keywords.ideas google.autocomplete 200 $0.00099 $0.00099 1228

A starred row — 22 of 36 — metered below its claim, which is the expected result rather than a
discrepancy: the claim is a p90 across real traffic, a test_request is a deliberately minimal
input. linkedin.jobs claims $0.0273 and meters $0.0044 at limit=3. On the 14 flat rows, where
input size cannot move the price, claimed and metered agree exactly. Nothing metered above its
claim. items counts result rows, so a single-object answer reports items: 0 and is still a hit.

Two things to expect when you run it yourselves:

  • google.ai_overview is slow and returned 502 once. 111 seconds to a 502 on the first call,
    then 94 seconds to a 200 at exactly its claimed $0.0018, which is the committed example. Set a
    client timeout above two minutes; a 502 here is a retry, not a dead route.
  • Two datePosted values on instagram.reels_search will 422. The row documents
    last-hour | last-day | last-week | last-month | last-year; our input schema currently accepts
    only the last three. The committed test_request uses last-month, so the pass will not trip
    over it, but a reviewer trying the short windows by hand will. Say the word and we trim the enum.

Full surface map

The OpenAPI document publishes 369 paths: 362 POST /v1/run/{sku} operations and 7 support routes.

Catalogued (186)

  • 36 core, on six platforms — TikTok, LinkedIn, Facebook, Google (search and maps), X,
    Instagram — chosen by our own 60-day production request volume. Those six carry 74% of our
    traffic. linkedin.profile_thin and linkedin.jobs_thin are the cheap tiers of their operation
    and the full-record rows are listed separately; linkedin.search.posts carries two of our rows
    for the same reason, one returning text and link, the other author, engagement and attachments.
  • 150 generated, on those six plus Reddit, YouTube, web scraping and TikTok Shop. An earlier
    revision generated all 271 non-core SKUs across 54 platforms; the tail below TikTok Shop carried
    6,275 requests between 106 rows over 60 days (alibaba 13, douyin 97, several zero), so it is
    cut rather than asking you to host rows nobody calls.

Excluded, and why

  • 8 apollo.* — Apollo is listed here directly already.
  • 45 enrichment SKUs (email*, person*, people_search*, company_*, mobile_phone*,
    job_search*, technographics*), all served by one upstream aggregator. Listed in
    ANYAPI_EXCLUDE so a re-ingest cannot pull them back.
  • 20 SEO-tooling SKUs (seo.*, ahrefs.*, semrush.*) — resold keyword, ranking and backlink
    data. Every shelf they would land on is already held by the primary vendor of that data
    (google.keywords.ideas has six providers, web.backlinks.* is semrush's own), so a reseller
    row there is a dearer copy of a price a buyer can already compare. ANYAPI_EXCLUDE_PREFIX.
  • 2 dead rows, instagram.followers and instagram.following: both surviving sources fail on
    the canonical example, so listing them would sell something that does not work. ANYAPI_EXCLUDE_DEAD,
    and they return when the data does.
  • 1 busy row held back, instagram.profile (7,256 requests/60d): its cheapest source answers
    found: true with username null. Ours to fix, not yours to list; it sits in the coverage tier.
  • 7 support routes — the rate card, the probe, own-account bookkeeping, and public discovery.
    None is a data job.

There are no free count or preview routes to include. /v1/apis/{sku} is the closest thing: free,
and it returns the price and input schema so an agent can size a call before paying.

Capability mapping

Every core row was placed by searching capabilities.yaml and the other provider files for the job
it does and reusing the id where one existed. 34 of 36 sit on a shelf another provider is already
on.
Three ids are proposed in proposed_capabilities:, two of them naming an endpoint you
already have so both can be attached in one review:

  • linkedin.email.work.findcontactout.yaml already carries this endpoint, uncapped.
  • google.serp.ai_overviewserpapi.extended.yaml already carries "Google AI Overview panel
    by page token", uncapped. AI Overview is not AI Mode: one is the panel above an ordinary
    search result, the other is Google's separate conversational surface, and cloro, dataforseo and
    serpapi are all on google.serp.ai_mode selling AI Mode. We have no AI Mode endpoint, so this
    proposal costs us a populated shelf rather than winning one.
  • google.maps.reviews — a shelf of one, and the only one we could not avoid: no other
    provider file here sells Google Maps reviews. Happy to move it to the coverage tier instead.

One row moved off the wrong existing shelf. twitter.replies was on x.user.replies ("a
user's replies to others"), but it takes a post URL and returns that post's reply thread, which is
x.post.comments — where tikhub and justoneapi already have three rows keyed by tweet id.

This PR does not touch capabilities.yaml. An earlier revision added 18 platform slugs that
existed only to host rows in the tail we have since cut;
git diff origin/main -- src/treg/catalog/capabilities.yaml is empty.

The 150 generated rows carry no capability, per your extended-tier schema. The ones with an
obvious existing shelf, if you want to attach them in review:

generated row shelf
youtube.video_comments youtube.video.comments
reddit.post_comments reddit.post.comments
tiktok.user_videos tiktok.user.videos
twitter.followers x.user.followers
linkedin.company_employees linkedin.company.people

Checks

uv run --frozen python scripts/catalog_validate.py
OK — 98 provider file(s), 3434 endpoint(s), 0 error(s), 0 warning(s)

TZ=UTC uv run --frozen pytest -q
3797 passed, 6 skipped

One maintainer test was edited, and it deserves a look. In f936f5a,
tests/test_catalog_api.py::test_search_finds_the_job_across_providers_best_first had one ranking
assertion widened: AnyAPI is the fourth seller of tiktok.video.comments and ranks on price like
any other row, so that search's "top 4" became a top 5. The verified-stamp half of the assertion
excludes anyapi, and should go back to a plain all(...) once your own run adds the stamps. It
is the only maintainer test this branch touches.

tests/test_plugin.py::test_the_plugin_skill_is_not_stale needed the five plugin SKILL.md copies
regenerated with scripts/build_plugin.py: their frontmatter embeds a rounded catalog size, and
listing AnyAPI moves it past a rounding boundary.

No verified: stamps are committed — that assertion is yours to make from your own run. The 36
example responses are committed, as the bodies that run returned, so your pass has something to
diff against. They are AnyAPI's own envelope rather than an upstream body, and provider is pinned
to the literal "AnyAPI", so nothing identifying a source rides in them. No credential value
appears anywhere in the diff.

scripts/catalog_ingest.py anyapi run twice produces a byte-identical file: checked is a
hand-bumped constant, not date.today(), and measured prices come from a committed export rather
than a live read. Merged origin/main four times, most recently 8c700dc3; the catalog YAML has
never conflicted.

For the live verification run

Reply to support@getanyapi.com and we will send a funded key privately. Every test_request above
costs well under a cent — the whole 36-row ledger cost $0.08993.

AnyAPI added 2 commits September 8, 2026 19:28
AnyAPI (getanyapi.com) is a per-request USD gateway over 362 normalized
endpoints across social, search, maps, commerce and web scraping. One key, one
wallet, no subscription; every response reports the exact USD it charged.

Two things are worth a reviewer's attention:

- `cost.reported_charge: {path: costUsd}`. AnyAPI routes each SKU across several
  upstream sources and retries a failed attempt on the next one, so the charge
  is not always the catalogued figure. `cost.value` is the cheapest source's
  price, read from AnyAPI's own live rate card; `costUsd` on every 2xx body is
  the exact charge, and that is what settles.
- `proposed_capabilities: web.contents.get` mirrors the id exa.yaml already
  proposes for exa.web.contents.get. Same job, same wording, so both endpoints
  can be attached in one review.

Contact: support@getanyapi.com
…e card

Optional second commit - the first stands alone if you would rather generate
the coverage tier yourselves.

`ingest_anyapi` reads two of the vendor's own sources: the OpenAPI document at
api.getanyapi.com/openapi.json for request shapes (public, no key), and
GET /v1/apis for prices. The rate card returns a structured per-SKU price in
USD, so every generated row carries `source: rate_card_api` with a concrete
number rather than a parsed pricing page, and `reported_charge` settles each
call on the real charge.

362 SKUs minus the 15 curated in core and the 53 AnyAPI excludes from this
listing (ANYAPI_EXCLUDE) leaves 294. Re-run with any AnyAPI key in
ANYAPI_API_KEY; `POST https://api.getanyapi.com/agent/signup` mints a free one
with no account.

capabilities.yaml gains 18 platform slugs the taxonomy had no entry for
(hackernews, substack, nextdoor, indeed, glassdoor, upwork, fiverr, alibaba,
zillow, redfin, realtor, airbnb, booking, g2, capterra, sec, congress,
polymarket). Everything else maps onto platforms that already exist.
@github-actions github-actions Bot added the area:dashboard The web dashboard / landing / tutorials label Sep 9, 2026
Five fixes, all found by re-checking the branch against live behaviour rather
than against the snapshot it was built from:

- Reserve. `_anyapi_cost` recorded `per_result: perUnitUsd` for the 59 SKUs
  AnyAPI prices as a base plus a per-result rider, dropping the base: a default
  `polymarket.markets` call would have reserved $0.0132 against a $0.116 floor.
  Every row is now the row's own `maxUsd` as a flat per-call price, which is the
  conservative reserve in both pricing shapes. `reported_charge` still settles
  the truth from `costUsd`.
- Disclosure. All 294 generated rows now carry the same `note` the 15 core rows
  do, naming that SKU's source count and its failover ceiling. Previously only
  core disclosed it, which is the wrong way round: the generated rows are the
  ones nobody reads one at a time.
- Re-runnability. `checked` was `date.today()` on 294 cost blocks, so a re-run
  the next day rewrote 294 lines with no price change. It is now a constant
  bumped by hand when the rate card is re-read, matching ingest_openrouter.
  Verified: two consecutive runs produce a byte-identical file.
- Platform. The `maps` prefix mapped to `google-business`, whose shelf is the
  caller's OWN Business Profile via OAuth. Google Maps place and review data is
  `google`, where serpapi's equivalent already sits.
- `cursor` is no longer stripped from generated inputs. It selects the page, so
  unlike the routing controls it does change the answer.

`instagram.profile` is out of the core tier and back in the coverage tier: its
cheapest source currently answers `found: true` with zeroed follower and post
counts for a live account, which is an AnyAPI defect to fix rather than a row to
put on a comparison shelf. `facebook.user.profile` takes the slot.
@kev1n kev1n changed the title Add AnyAPI: 15 core endpoints, a generated 294-endpoint extended tier, and per-call reported charges Add AnyAPI: 15 core endpoints, a generated 294-endpoint coverage tier, and per-call reported charges Sep 9, 2026
AnyAPI added 3 commits September 8, 2026 20:23
The first cut of this file was chosen by looking at treg's catalog: which
capabilities already had several providers, and where AnyAPI's price beat the
cheapest listed one. That is a reasonable way to pick a fight and a bad way to
pick a listing - it produced 15 endpoints, only 8 of which are in AnyAPI's own
top 20 by request volume.

Re-picked against 60 days of production traffic, keeping only endpoints where
AnyAPI is also the cheapest provider on the shelf. Seven rows change:

  in   reddit.post              21,193 requests   reddit.post.detail
  in   instagram.reels_search   10,611            instagram.search.reels
  in   reddit.search             6,887 (263 teams - our widest audience)
  in   linkedin.search_posts     5,692            linkedin.search.posts
  in   linkedin.jobs_thin        3,774            linkedin.search.jobs
  in   youtube.search            3,370            youtube.search.videos
  out  google.news / facebook.profile / instagram.user_posts /
       tiktok.search_keyword / linkedin.company_thin / youtube.video

The six that leave are all in the coverage tier. Four of them (instagram user
posts, tiktok keyword search, linkedin company profile, youtube video detail)
were also the four rows where AnyAPI was 1.2x dearer than the cheapest provider
already on that shelf, so nothing is lost by not putting them side by side.

Ledger re-run live against all 15 committed test_requests: 15/15 HTTP 200,
claimed == metered on every row, $0.01092 for the run.
`linkedin.search.posts` now carries two AnyAPI rows, because the capability
covers two different products and the price gap only makes sense once you see
both output schemas:

  linkedin.search_posts       $0.0012   text, link, publish date
  linkedin.search_posts_full  $0.005    author record, engagement counts,
                                        attachments, poll options

The other provider on this capability searches Google's index of public
LinkedIn posts and returns what Google exposes. The $0.005 row does not, which
is what the 4x buys. Fifty provider+capability pairs in this catalog already
carry more than one row, so this is the existing shape for a tiered operation
rather than a new one.

`linkedin.search_posts_full` is AnyAPI's largest revenue endpoint over the
trailing 60 days; leaving it in the coverage tier meant the shelf showed our
cheap tier and hid the product.

Ledger, live, all 16 committed test_requests: 16/16 HTTP 200, claimed ==
metered on every row, $0.01592 for the run.
VENDORS.md says 8-15 core endpoints and I took that as a ceiling. It is not one:
nothing in catalog_validate.py counts endpoints, and ten core files in this repo
already exceed 15 (companyenrich 41, leadsforge 32, predictleads 27). Curating
to 15 left most of AnyAPI's real traffic in the coverage tier, on no shelf.

The core file is now every AnyAPI endpoint down to the 90th percentile of
trailing-60-day production requests, minus the SKUs AnyAPI excludes and minus
one that is broken. 51 rows. Six of them share a capability with another AnyAPI
row, always as a cheap-tier / full-tier pair with the difference stated in
`note` - the existing shape here, where fifty provider+capability pairs already
carry more than one row.

Seven capabilities are proposed rather than reused, each because the taxonomy
has no id for the job: three mirror ids another provider file already proposes
and has not had merged (web.contents.get from exa, linkedin.email.work.find
from contactout, both verbatim), two are Google Maps place and reviews, which
google.serp.maps does not cover (it is the result list), one is Google
autocomplete, and two are Hacker News, a platform this PR adds.

tests/test_catalog_api.py: one ranking assertion had to widen. AnyAPI is the
fourth seller of tiktok.video.comments and ranks on price like the rest, so the
"top 4" for that search is now a top 5. The verified-stamp half of the
assertion excludes anyapi, because a vendor does not stamp its own rows - that
line goes back to a plain all(...) once the maintainers' own verification run
adds the stamps. Flagged here because a vendor editing a maintainer's ranking
test deserves a look.

Ledger, live, all 51 committed test_requests: 51/51 HTTP 200, $0.10418 for the
run. Three rows metered BELOW their claim (linkedin.company.people,
linkedin.search.jobs.full, google.maps.reviews) because a per-result source was
cheaper for the small test input; each says so in its own note, and cost.value
stays the price at the input maximum so the reserve is never short.
@kev1n
kev1n marked this pull request as draft September 9, 2026 05:00
AnyAPI and others added 8 commits September 8, 2026 22:04
twitter.followers, twitter.following, instagram.followers and instagram.following
each pair a flat single-page source with a BULK source priced per row, whose
input maximum is around 97,000 rows on twitter and 1,000 on instagram. Both
prices are real: $0.00075 for a page, $16.50 for the full pull. Neither
describes the row.

Listing the cheap one reserves $0.00075 against a settlement that can reach
$16.50. Listing the dear one puts $16.50 on a shelf beside $0.001 rivals.
cost.table with `times` would model it, but it is mutually exclusive with
cost.reported_charge, and exact settlement is worth more here than a modelled
reserve.

These four sit an order of magnitude clear of everything else on both measures:
after removing them the largest gap between a catalogued price and its ceiling
is $0.225, and the largest multiple is 440x rather than 22,000x. That is a break
in the data, not a chosen cutoff.

They can come back the day treg can pin a per-call ceiling. AnyAPI already
accepts `?max_cost_usd=`, which refuses any source dearer than a named price
before the request runs; platform_request cannot set a query parameter today.

254 generated endpoints, down from 258.
The exclusion these two sat under was a price argument: each paired a flat
single-page source with a bulk source priced per row, so one endpoint could
legitimately settle anywhere from $0.00075 to $16.50 and no single scalar
could describe it. AnyAPI deleted those bulk sources on 2026-09-09. Both now
publish one source whose price and ceiling are the same $0.00075, so the
reserve can never fall short, and the extended tier goes 254 -> 256.

The instagram pair stays out for a different and current reason, now written
into the comment: both of their surviving sources are dead today, so listing
them would shelf two endpoints that return nothing. Their $0.018 ceiling is
ordinary and is not the reason. The constant is renamed to say so.

Re-reading the rate card also corrects instagram.profile, committed at
$0.0005 against a live $0.0012.
`anyapi.tiktok.user.profile` was catalogued at $0.0005, the price of a
source AnyAPI quarantined on 2026-09-09. With that source out, the
cheapest one that can serve the row costs $0.0012, so the entry reserved
2.4x short against its own settlement. Unlike an extended row this one
carries a `test_request` and is platform-callable, so the shortfall was
live.

The note's source count follows the same change, 3 to 2.

`reported_charge` already settled the true number from `costUsd`, so no
call was ever mischarged - only the reserve was wrong.

Every AnyAPI row in both tiers now matches the live rate card: 307
endpoints, 0 missing, 0 price drift, 0 ceiling drift.
# Conflicts:
#	src/treg/oauth_providers.py
#	tests/test_key_providers.py
#	tests/test_oauth_providers_m3.py
…ate card

`cost.value` is both the shelf price a buyer compares and the amount treg
reserves, and reading it off `pricing.from.maxUsd` - the CHEAPEST source's
price at the input maximum - got both jobs wrong. Checked against 60 days of
AnyAPI's own request ledger, 121 of the 307 listed rows sat BELOW the p90
charge really billed (ebay.search listed $0.0005 against a $0.03795 median:
the cheap source almost never wins), and 40 sat far above what anyone pays
(instagram.hashtag_analytics listed $0.0385 against a $0.00297 p90, because
a per-result SKU was priced at an input maximum nobody sends). It also
drifted every time a source was quarantined and `pricing.from` recomputed.

So `cost.value` is now the p90 of what AnyAPI actually charged for that SKU
over the trailing 60 days, exported to scripts/data/anyapi_measured_charges.json
the same way justoneapi_prices.json carries that vendor's price sheet. 217 of
the 307 rows have at least 5 charged calls to measure and take the measured
price; the other 90 keep the rate card's cheapest-source figure and say so in
their own note. `reported_charge` stays on every row either way - `costUsd` is
the only number that knows which source served and how many rows came back.

This deliberately REVERSES the rationale of the earlier commits on this branch,
which held `cost.value` at the price at the input maximum so the reserve could
never fall short. The listing now optimises for the shelf price being what a
buyer actually pays, and accepts that roughly one call in ten settles above the
reserve - which is what domain/money/settlement.py:133-134 already designs for:
the charge may exceed the reserve, the ledger takes the difference from the
balance, and the next reserve is the gate. Measured over the same 60 days under
the OLD, much worse prices, that overrun came to $18.66 on $695.05 settled, 2.7%.

`cost.source` is `observed` on a measured row, this catalog's existing word for
a price seen being billed rather than read off a page, and stays `rate_card_api`
on the fallback. `source_url` still points at the per-SKU rate card either way.
A maintainer live-tests every core row by hand against a credential they have
to arrange, so the core tier is the reviewer's entire workload and the extended
tier costs them nothing. Core is now the six platforms that carry 74% of
AnyAPI's production traffic - tiktok, linkedin, facebook, google, x, instagram
- which is 36 rows instead of 51 and about 30% less hand verification.

The 15 rows that leave do not leave the listing. `ingest_anyapi` skips whatever
`core_routes("anyapi")` already curates, so re-running it picks them all up:
reddit (5), youtube (4), tiktok-shop (2), hackernews (2), web.scrape and
facebook.company_ads. Extended goes 256 -> 271 and the total stays 307. They
arrive as ordinary generated rows, which means they lose the capability mapping
and the hand-run verification the core file carried; `catalog_verify_extended.py`
is what re-earns a stamp there. The three `proposed_capabilities` no remaining
row uses (web.contents.get, the two hackernews ones) go with them.

Every surviving core row is repriced by the same measured rule as the generated
tier, all 36 of them from at least 100 charged calls, and each note now says
whether its price is a 60-day measured p90 or the rate card's cheapest source.
Six notes that argued the old rationale are rewritten: three said `cost.value`
was the price at the input maximum or the flat source's price to keep the
reserve whole, which is no longer what the number is.

The largest move up is maps.search, $0.00175 -> $0.07734: its two rescue sources
are far dearer than the primary and serve often enough to drag the p90 up near
the $0.116 ceiling, even though half of all calls settle at or under $0.00297.
The note says exactly that.
… price

`cost.value` is the p90 of what AnyAPI really billed, and for almost every row
that is the right number - the median p90/p50 across the 217 measured rows is
exactly 1.00. On five CORE rows it is not, because their sources charge very
different amounts for the SAME number of results, so the p90 measures which
source won rather than how much work the call did.

maps.search is the clearest case. Measured per source over the same 60 days:
scrapertech $0.00175 flat (312 calls), serper $0.00297 flat (743 calls), apify
$0.06005 median (485 calls) - and all three average 11-12 items per response.
The blend's p90 is $0.07734, a price only the dearest source ever charges,
while the cheapest source serves the same query for $0.00175. On the
price-sorted google.serp.maps shelf that overstates the endpoint 44x.

So these five list the rate card's cheapest advertised price, with
`cost.source: rate_card_api` because that is what the number now is:

    maps.search      0.07734 -> 0.00175
    maps.place       0.00840 -> 0.00175
    maps.reviews     0.00445 -> 0.0035
    twitter.profile  0.00075 -> 0.00022
    twitter.replies  0.00290 -> 0.0005

They UNDER-reserve by design: a rescue on a dearer source settles above the
listing. That is the overrun domain/money/settlement.py:133-134 already accepts
and reports in reconcile, and `reported_charge` (`costUsd`) settles the real
number either way. Each row's note now says so instead of claiming a p90 it no
longer carries, and keeps its source count and failover ceiling.

The rule is deliberately narrow. It does not extend to the extended tier, which
is not shelf-ranked, so a lower number there would only be a smaller reserve
with no upside. It does not apply to a single-source per-result row either:
linkedin.company_employees and linkedin.jobs advertise $0.11 and $0.0286 at the
input maximum, ABOVE the $0.105 and $0.0273 p90 they already carry, so
switching them would raise the shelf price for nothing. Both stay measured.

`ANYAPI_CORE_ROWS_PRICED_AT_CHEAPEST_SOURCE` in scripts/catalog_ingest.py names
the exception with that evidence, so a re-ingest cannot silently revert it.
Core is 36 rows and extended 271, unchanged; the observed / rate_card_api split
moves 217/90 -> 212/95. A double ingest run is byte-identical.
A measured p90 is a statistic over the trailing 60 days, so it can sit BELOW
the price the catalog charges today: when a source is withdrawn or quarantined,
`pricing.from` recomputes upward and every historical charge was cheaper than
anything now on offer.

A live ledger run over all 36 core rows caught it on one: `tiktok.profile`
claimed $0.001 and metered $0.0012. The $0.001 source is gone. Measured against
the live rate card it is 13 rows, all at exactly 1.20x and all the same
withdrawn lane - `tiktok.profile` plus the weibo, zhihu and douyin families.

`_anyapi_flat_floor` raises a measured price to the rate card's
`pricing.from.maxUsd` when, and only when, the SKU is priced `model: flat` and
that figure is therefore a true per-call floor. On a per-result SKU the same
field is the price at the INPUT MAXIMUM rather than a floor -
`instagram.hashtag_analytics` advertises $0.0385 against a $0.00297 p90 - so
clamping there would reinstate exactly the overstatement the measured prices
exist to remove.

After this, zero flat rows sit below their floor, and a re-run of
`tiktok.profile` claims and meters $0.0012.
@kev1n kev1n changed the title Add AnyAPI: 15 core endpoints, a generated 294-endpoint coverage tier, and per-call reported charges Add AnyAPI: 36 core endpoints, a generated 271-endpoint coverage tier, and measured per-call prices Sep 10, 2026
The generated tier listed 271 rows across 54 platforms, and most of that tail is
traffic nobody sends. Measured over AnyAPI's own 60 days of production requests,
there is a clean break: tiktok-shop, the last platform kept, took 5,121 requests
and the next one down, hackernews, took 1,567. The 106 rows below that break
carry 6,275 requests BETWEEN THEM - alibaba 13, douyin 97, zhihu 38, and several
none at all. So the cut line is where the data already put it, not a number
picked to hit a row count.

Extended keeps the six platforms anyapi.yaml curates in core - tiktok, linkedin,
facebook, google, x, instagram - plus the four next busiest: reddit, youtube,
web and tiktok-shop. That is 165 generated rows, and the listing totals 201
instead of 307. Core is untouched at 36.

The 18 platform slugs this branch proposed for capabilities.yaml go with the
rows that needed them. Every one of hackernews, substack, nextdoor, indeed,
glassdoor, upwork, fiverr, alibaba, zillow, redfin, realtor, airbnb, booking,
g2, capterra, sec, congress and polymarket now has zero endpoints in the whole
catalog, from any provider, so this PR no longer touches the shared taxonomy at
all. The brightdata Glassdoor rows are unaffected: they sit on the `x` platform
and never used the slug.
@kev1n kev1n changed the title Add AnyAPI: 36 core endpoints, a generated 271-endpoint coverage tier, and measured per-call prices Add AnyAPI: 36 core endpoints, a generated 165-endpoint coverage tier, and measured per-call prices Sep 10, 2026
Upstream added a SUMBLE provider in the same three lists this branch appends
ANYAPI to, so all three conflicts resolved by keeping both entries: the
`REGISTRY` tuple in src/treg/oauth_providers.py and the service-name lists in
tests/test_key_providers.py and tests/test_oauth_providers_m3.py. Nothing from
either side was dropped, and the SUMBLE provider block is committed verbatim.

Also regenerates the five plugin SKILL.md copies via scripts/build_plugin.py.
Their frontmatter embeds a rounded catalog size, and listing AnyAPI moves it
from "3,200+ endpoints across 70 providers" to "3,400+ across 71" -
tests/test_plugin.py::test_the_plugin_skill_is_not_stale fails without this,
and it passes on a clean origin/main, so the staleness is this branch's to fix.
@github-actions github-actions Bot added the area:docs Documentation & design fragments label Sep 10, 2026
Re-checked every core capability against capabilities.yaml and the other
provider files, per the "Capability first" rules in web/vendor-listing.md.

One row was on the wrong EXISTING shelf. twitter.replies sat on
x.user.replies ("List a user's replies to others"), but it takes a post URL
and returns that post's reply thread. That is x.post.comments, where tikhub
and justoneapi already have three rows keyed by tweet id; tikhub's own
x.user.replies row takes a handle. An agent shopping the old shelf would
have bought a different product.

Two of the four proposed_capabilities duplicated a populated shelf, which
rule 1 forbids, so both proposals are dropped:
  - google.autocomplete -> google.keywords.ideas, where serpapi already
    lists "Google Autocomplete keyword suggestions for a seed term"
    alongside dataforseo, semrush, seranking, serpstat and google-ads.
  - maps.place -> google.serp.maps, where serpapi's row is "Google Maps
    place results for a query". It now sits beside our own maps.search and
    its note says which returns one place and which returns up to twenty.

AI Overview is NOT AI Mode, so google.ai_overview moves off
google.serp.ai_mode onto a proposed google.serp.ai_overview. AI Overview is
the panel above an ordinary search result; AI Mode is Google's separate
conversational surface, and cloro, dataforseo and serpapi are all on
google.serp.ai_mode selling AI Mode. serpapi.extended.yaml already carries
an uncapped "Google AI Overview panel by page token" row, named in the PR so
both can be attached in one review. AnyAPI has no AI Mode endpoint, so this
costs us a populated shelf rather than winning one.

google.maps.reviews stays proposed: no other provider file in this repo
sells Google Maps reviews, so it is a shelf of one by necessity.

Four generated rows also changed platform. treg splits SEO tooling between
google (keyword and domain-ranking data) and web (backlinks and authority),
and semrush.yaml splits its own endpoints exactly that way. A vendor prefix
can span both, which ANYAPI_PLATFORM cannot express, so ahrefs.keyword_ideas,
ahrefs.keywords, semrush.keywords and semrush.overview are named in the new
ANYAPI_PLATFORM_SKU; ahrefs.backlinks and ahrefs.overview stay on web, where
semrush.yaml files the same two jobs.

Re-ingested from the same cached snapshot: the only extended-tier change is
those four platform values, no price moved.

Validator OK - 98 provider file(s), 3449 endpoint(s), 0 error(s), 0
warning(s). Suite 3613 passed, 6 skipped under TZ=UTC.
The `seo.*`, `ahrefs.*` and `semrush.*` families are AnyAPI reselling keyword
volume, ranked keywords, domain overviews and backlink data. Every shelf those
20 rows would land on here is already held by the primary vendor of that data:
google.keywords.ideas has six providers, google.keywords.volume and
google.domain.ranked_keywords five each, google.domain.competitors four, and
web.backlinks.* is semrush's own. A reseller row on those shelves is a dearer
copy of a price a buyer can already compare, so AnyAPI does not list them.

Excluded by prefix in ANYAPI_EXCLUDE_PREFIX rather than by row id, so a
re-ingest cannot pull them back. ANYAPI_PLATFORM_SKU, added in the previous
commit purely to place four of these rows on the right platform, is deleted
with them, as are the three now-unreachable prefix entries in ANYAPI_PLATFORM.

Extended 165 -> 145, total 201 -> 181. Core is untouched: it never carried an
SEO row. Diffed every surviving row against the previous file - exactly those
20 are gone, nothing else moved.

Validator OK - 98 provider file(s), 3429 endpoint(s), 0 error(s), 0 warning(s).
Suite 3613 passed, 6 skipped under TZ=UTC.
@kev1n kev1n changed the title Add AnyAPI: 36 core endpoints, a generated 165-endpoint coverage tier, and measured per-call prices Add AnyAPI: 36 core endpoints, a generated 145-endpoint coverage tier, and measured per-call prices Sep 11, 2026
…ling

Three defects, all found by re-checking the committed files against the live
rate card and OpenAPI document rather than against the snapshot they were
built from.

CURSOR. carry_verification re-attaches `input` from the extended file it is
replacing, unconditionally. For Meta that is correct and the comment says
why: Meta publishes no machine-readable request schema, so a hand-reviewed
`input` must survive a deterministic re-ingest. For a provider that ships an
OpenAPI document it does the opposite - it freezes a generated row's
parameters at whichever revision first created it, and no later re-ingest can
refresh them. 36 AnyAPI rows lost `cursor` that way, including
youtube.channel_videos, facebook.post_comments and every YouTube channel
listing, while sibling rows generated later carried it: the catalog could
paginate some AnyAPI endpoints and not others, with nothing in the file
explaining the difference. Fixed with a carry_input keyword defaulting to
True, so every other ingester is unchanged, and ingest_anyapi passes False.
35 rows regain `cursor`; the 36th was an SEO row now excluded.

CEILING. A measured p90 was already floored at today's cheapest price. It
needs the same clamp at the top: pricing.failoverMaxUsd is the dearest price
any source can charge for the request, so a p90 above it quotes a source that
has been withdrawn. 21 rows shipped that way in an earlier revision, the worst
at 3x, and youtube.search_hashtag read "$0.0036 ... up to a $0.0012 ceiling"
in one sentence. 18 rows move - 14 generated, 4 core - and a clamped row now
says so in its own note instead of leaving a reader to reconcile two numbers.

DATES. ANYAPI_CHECKED was doing two jobs: the rate-card read date and the
measured window's end. Re-reading prices therefore restated every measured
price as covering days the ledger never saw. The window now comes from the
export's own as_of, so cost.checked is 2026-09-10 while the notes still say
60 days to 2026-09-09.

Also from the refresh: five Instagram SKUs AnyAPI shipped since the first pass
(location_posts, post_likers, search_locations, similar_profiles,
user_reposts), and two stale core source counts (tiktok.profile 2 -> 3,
instagram.post_comments 4 -> 3).

Extended 145 -> 150, total 186. No listed row is above its ceiling, missing a
spec field, or absent from the rate card.

Validator OK - 98 provider file(s), 3434 endpoint(s), 0 error(s), 0
warning(s). Suite 3613 passed, 6 skipped under TZ=UTC.
@kev1n kev1n changed the title Add AnyAPI: 36 core endpoints, a generated 145-endpoint coverage tier, and measured per-call prices Add AnyAPI: 36 core endpoints, a generated 150-endpoint coverage tier, and measured per-call prices Sep 11, 2026
Re-ran every core test_request live against AnyAPI's own account and kept the
body: 36/36 HTTP 200, $0.08993 for the run, none metering above its claim.
Each body is committed as src/treg/catalog/examples/anyapi.<id>.json with
arrays trimmed to 2 items, per the core-tier guidance in
docs/context/architecture/catalog.md (172 KB for the set).

These are AnyAPI's own envelope rather than an upstream body: `output` in
AnyAPI's normalized schema plus costUsd, items, resultId and `provider`, which
its OpenAPI pins to the literal "AnyAPI". Nothing identifying the source that
served rides in the response, so an example stays valid when a source is
swapped underneath it.

`verified` is deliberately still absent. A vendor stamping its own rows as
verified is not evidence; catalog_verify.py writing the stamp over its own run
is. The bodies are here so that pass has something to diff against.

Two things the run surfaced, both in the PR body:
  - google.ai_overview returned HTTP 502 after 111 seconds on the first
    attempt and succeeded on a retry in 94 seconds at exactly its claimed
    $0.0018. It sits at the edge of a two-minute budget, which its note
    already warns about; the committed example is the successful body.
  - tiktok.profile metered $0.0005 against a $0.0012 claim, because a cheaper
    source has appeared since the measured window closed. The previous run
    caught the opposite case, which is what the flat floor exists for.

Also merged origin/main (7c18043); no conflicts.

Validator OK - 98 provider file(s), 3434 endpoint(s), 0 error(s), 0
warning(s). Suite 3613 passed, 6 skipped under TZ=UTC.
AnyAPI's live spec accepts sortBy 0 (relevance), 1 (most liked) and 2 (newest
first); the catalogued row was generated before 2 shipped and offered only the
first two, hiding a sort a caller would want. Enum and field note now match the
spec.

Validator OK - 98 provider file(s), 3434 endpoint(s), 0 error(s), 0
warning(s). Suite 3613 passed, 6 skipped under TZ=UTC.
AnyAPI's source set moved between the last read and now, so the listing was
describing a catalog that no longer existed:

  - instagram.post priced $0.002 against a ceiling that fell to $0.0015 when a
    source was withdrawn - the same defect the ceiling clamp exists to catch,
    reappearing because the clamp runs at ingest and the rate card moved after.
  - Stale source counts in five notes: tiktok.profile 3 -> 2, instagram.post
    3 -> 2, instagram.post_comments 3 -> 4, tiktok.video_comments 3 -> 2, and
    reddit.post 5 -> 4 with its ceiling $0.0036 -> $0.00225.

Extended regenerated with --refresh so the committed file is reproducible
rather than hand-patched; core, which is hand-curated, edited in place.
ANYAPI_CHECKED and every cost.checked move to 2026-09-11. No row was added,
removed or repriced beyond instagram.post, and no input field changed.

This is the drift your 90-day staleness flag is for: five rows in one day,
from a catalog nobody touched on purpose.

Validator OK - 98 provider file(s), 3434 endpoint(s), 0 error(s), 0
warning(s). Suite 3797 passed, 6 skipped under TZ=UTC.
@kev1n
kev1n marked this pull request as ready for review September 12, 2026 00:07
@kev1n

kev1n commented Sep 12, 2026

Copy link
Copy Markdown
Author

Two bugs we hit building this listing. Both are in your code rather than ours, and neither blocks the PR — raising them separately so they are not buried in a vendor description.

1. relay.py:176 drops a relayed path's own query string

infra/upstream/relay.py:176 passes params= to build_request unconditionally, and httpx replaces a URL's existing query whenever params is given — even an empty list:

build_request('POST', '.../run/maps.search?max_cost_usd=0.003', params=[])          -> .../run/maps.search
build_request('POST', '.../run/maps.search?max_cost_usd=0.003', params=[('q','1')]) -> .../run/maps.search?q=1

This affects your own first-party rows, not just ours: facebook.extended.yaml:74 is /{comment_id}?is_hidden=true and linkedin.yaml:84,102,120 carry ?action=initializeUpload. Every relayed call to those four loses its query string.

You already have the fix and a comment describing this exact trap, in health.py:70-82 — "Merge onto the URL rather than passing params= … would silently strip a probe path's own query" — which builds httpx.URL(url) and applies copy_add_param per pair.

For us it would unlock pinning a per-request price ceiling in path (/v1/run/maps.search?max_cost_usd=0.003), since platform_request only pins body. fields and our ceiling is a query param. That is a nice-to-have, not a request.

2. carry_verification freezes a generated row's input forever

scripts/catalog_ingest.py:carry_verification re-attaches reviewed fields from the extended file it is replacing, and input is on that list unconditionally. For Meta that is correct, and the comment says why: Meta publishes no machine-readable request schema, so a hand-reviewed input must survive a deterministic re-ingest.

For a provider that ships an OpenAPI document it does the opposite — it pins a generated row's parameters to whichever revision first created it, and no later re-ingest can refresh them. 36 of our rows lost cursor that way (youtube.channel_videos, facebook.post_comments, every YouTube channel listing), while sibling rows generated later carried it. The catalog could paginate some of our endpoints and not others with nothing in the file explaining the difference.

The fix is in this PR: a carry_input keyword defaulting to True, so every other ingester behaves exactly as before, and ingest_anyapi passes carry_input=False. Happy to split it into its own PR if you would rather take it separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dashboard The web dashboard / landing / tutorials area:docs Documentation & design fragments

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant