Skip to content

fix: batch of verified billing, routing and catalog fixes from feedback #11-#45 - #400

Open
stonexer wants to merge 14 commits into
mainfrom
fix/feedback-10-45-batch
Open

fix: batch of verified billing, routing and catalog fixes from feedback #11-#45#400
stonexer wants to merge 14 commits into
mainfrom
fix/feedback-10-45-batch

Conversation

@stonexer

@stonexer stonexer commented Sep 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Batch of the verified billing, routing and catalog fixes from the 2026-09-08/09 feedback triage (reports #11 to #45). Every root cause was traced in code, then confirmed on 2026-09-09 against the production ledger and the providers' own credit meters with controlled test calls. One commit per fix, plus one reconciliation commit and one review fix. The overflow disclosure and aggregator-strike change is split out into its own PR because it changes capacity policy for every overflowed call.

Billing (settlement and reservation)

  • icypeas submission acknowledgements (people.email.find, bulk.search and the sibling bulk routes) settle at 0 instead of the 20-row estimate; bulk jobs reserve one row per submitted row. Live: a NOT_FOUND cost 0 credits upstream while treg charged $0.38; every platform success since 08-20 was charged exactly $0.38.
  • companyenrich people.search and .scroll settle on returned items (2 credits each) with the catalog-declared 2-credit minimum on an empty page. Live: an empty pageSize 10 page cost 2 credits upstream while treg charged 20.
  • serpstat: the estimator reads params.size inside the JSON-RPC envelope; row-priced routes use unit: row; a top-level error object settles at 0 and a success counts the returned rows. Live: error envelopes cost 0 lines while treg charged 20 credits; a 12-row SERP cost 12 lines.
  • seranking keywords.ideas reserves by limit and settles per returned keyword. Live: 5 keywords cost 50 credits while treg charged 1 unit; an empty result cost 0.
  • apify linkedin.search.jobs: maxItems/resultsLimit size the reserve, the actor's bare dataset array is counted at settle, and the input notes carry the actor's real enum values instead of README labels (3,019 platform calls settled at a flat $0.02).

Routing

  • X-Treg-Route-Max-Cost is applied per candidate; the 402 names the truly cheapest candidate only when every candidate is over the cap.
  • Adapter verification fails any adapter that cannot fill a required input; the findymail domain-search adapter now supplies roles (it deterministically 4xx'd and ended treg.people.search as 422 with earlier charges kept).
  • Catalog calls resolve by provider identity before host, so meta-ads next to instagram-page-tools no longer 409s. Two connected accounts of ONE provider still refuse, now with a structured 409 that names each account's /call/<tool>/<path> form: nothing in the request says which account is meant, and a silent default would route one account's request through the other's credential.

Catalog

  • lusha.x.decision-makers retired (Lusha removed the path on 2026-08-12) and superseded by lusha.x.buying-group, the path that accepts contactsLimit.
  • The 26 DataForSEO task_post rows are platform_blocked on the shared key because their consumers are blocked or unlisted; hints stop claiming the key is injected for blocked rows; "website audit" aliases to the one-shot page audit.

Review notes (manual, after the agent review pass was interrupted)

Read every hunk of the twelve commits against AGENTS.md and money.md. One change made as a result: the multi-account default above. Findings left as they are, for the reviewer to weigh:

  • icypeas submissions settle at 0, so treg absorbs the credits Icypeas takes on hits until terminal settlement exists. Bounded at 1 credit per single search, but icypeas.bulk.search accepts up to 5,000 rows per job (up to about $95 absorbed per job). Before this PR big jobs were already under-billed at a flat $0.38, so this is not a new exposure class, but blocking bulk.search on the shared key until async settlement lands is a reasonable alternative.
  • serpstat treats every top-level error object as free. Verified live for -32000 only; the 32017 not-found envelope is assumed free too (safe for customers, at most 1 credit of treg revenue per call).
  • seranking keywords.ideas now reserves limit rows with the API's default of 100, so a call that names no limit reserves $0.179 instead of $0.0018. That is the vendor's real price.
  • The stricter adapter verifier required two catalog adjustments to keep every adapter verified (moz url_metrics distributions const, lusha people search pagination); all 298 adapters verify on this branch as on main.
  • lusha.x.decision-makers answers 410 for the 13 orgs that used it in the last 30 days, and 21 orgs' DataForSEO task_post calls answer credential_missing on the shared key. Both intended; both deserve a note to those teams.

Verification

Not included here: the overflow disclosure (separate PR), the Hunter companies.emails repricing, the routed-row pricing notes, and the catalog docs hygiene item were interrupted before commit and will follow separately. The findymail/companyenrich changes do not issue refunds; historical corrections are a separate grant.

@github-actions github-actions Bot added area:cli The treg CLI area:dashboard The web dashboard / landing / tutorials area:docs Documentation & design fragments labels Sep 9, 2026
@stonexer
stonexer force-pushed the fix/feedback-10-45-batch branch from df7da0b to 1111db8 Compare September 9, 2026 14:25
@stonexer stonexer mentioned this pull request Sep 9, 2026
Merged
@github-actions github-actions Bot removed the area:dashboard The web dashboard / landing / tutorials label Sep 9, 2026
@stonexer
fix(catalog): retire lusha.x.decision-makers and add lusha.x.buying-g…
899dc95 
…roup

Lusha removed POST /v3/contacts/decision-makers on 2026-08-12 (API changelog 2.9.0)
in favour of POST /v3/contacts/buying-group, the only operation that accepts
contactsLimit and personas. Commit 5c52ae2 documented those parameters on the
legacy path, but its surviving handler rejects them with 400 'property should not
exist', so the documented per-company spend cap never applied: companies-only
bodies still answered, uncapped at Lusha's 60-contacts-per-company default.

- mark lusha.x.decision-makers `status: retired` with the removal story and
  `superseded_by: lusha.x.buying-group`; a direct call now answers 410 naming the
  successor before any credential loads or a hold is placed
- add lusha.x.buying-group per Lusha's V3BuyingGroupRequest (companies 1-25 of
  {domain|id, clientReferenceId}, personas, contactsLimit default 60 minimum 1,
  pagination), same per_result 1-credit price and the people.decision_makers
  capability; no live probe was run, so it ships `skipped` with no example fixture
  instead of an invented one
- tests: the tombstone leaves discovery and the successor stays platform-eligible;
  a POST at the retired id is 410, names the successor and moves no ledger entry
- docs: money.md's contactsLimit paragraph now says the cap only reserves what the
  provider honours, and names the tombstone and its successor
@stonexer
fix(catalog): block DataForSEO task_post rows on the shared key
4030fa5 
A task_post only enqueues work; the answer comes back through task_get,
which ingest drops, or through id-keyed readers that are already
platform_blocked. The 26 task_post rows were still platform-eligible, so
a shared-key caller paid to start crawls and SERP tasks whose results
treg could never serve (213 platform calls across 21 orgs since 09-04).

- stamp platform_blocked on every DataForSEO task_post row with a reason
  that says results come back through task_get, to connect an own key,
  and names the one-shot sibling treg does serve (dataforseo.web.page.audit
  for on-page, brightdata.x.trustpilot-reviews for Trustpilot reviews)
- test: a task_post may be offered on treg's key only while a consumer of
  its family is; on-page consumer notes may only name ids that exist
- fix the on-page consumer notes that pointed at the non-existent
  dataforseo.x.on-page-instant-pages and declare the {id} pathParam on
  dataforseo.x.on-page-summary-id
- catalog hints (search, endpoint detail, MCP catalog_get via the same
  route) and `treg catalog get` no longer say "key injected server-side"
  for a platform_blocked row; they say the team's own key is needed and
  carry the reason
- aliases: `website` and `site` bridge to `on-page`, so "website audit"
  and "site audit" find the one-shot instant_pages row first instead of
  only the async task_post; drop the duplicated `ig` alias key
@stonexer
fix(billing): settle Icypeas async submissions at zero and reserve bu…
2309bb4 
…lk jobs per row

Icypeas' /email-search, /domain-search and /bulk-search answer 2xx with an
acknowledgement and no result rows; the credit for a hit is only visible later
on the free poll route. Settling at the estimate billed every submission the
20-row page default (380000 micro), hit or miss. An acknowledgement now settles
at 0 and closes the hold; synchronous bodies with data rows and the per_call
verify route keep the estimate. icypeas.bulk.search reserves one credit per row
of the body's top-level data array, capped at the platform row maximum, without
changing how other providers' bodies are read.

Catalog notes and the money fragment state that submissions settle at 0 today
and that treg absorbs found-email credits until terminal settlement exists.
@stonexer
fix(billing): settle CompanyEnrich search pages on returned rows
a8a1f21 
CompanyEnrich bills its search pages per row returned, with a documented
floor on an empty page (2 credits for people, 1 for companies, 5 for
lookalikes). No settlement rule counted the rows, so every 2xx settled at
the reserve, the whole requested page: live 2026-09-09 an empty pageSize
10 people search cost 2 credits upstream and treg charged 20.

- settle: a companyenrich rule charges len(items) rows at the catalog
  per-row price, never below the cost block's `minimum_units` floor; a
  body without an `items` list keeps the estimate.
- catalog: `minimum_units: 1` on people/companies search, their scroll
  routes and companies.similar, validated by catalog_validate.check_cost.
- tests: direct platform pages (empty, partial, full, unreadable) and a
  routed treg.people.search whose companyenrich child misses.
- docs: money.md provider rules and sources, MAP.md regenerated.
@stonexer
fix(billing): settle Serpstat calls on the JSON-RPC envelope, reserve…
dbe20aa 
… on params.size

Serpstat meters one API credit per returned row and answers a rejected
request as HTTP 200 with an `error` object. Verified live 2026-09-09
against the account's own limits meter: an error envelope cost 0 lines
while treg settled the 20-row estimate (10,000 micro), and a getKeywordTop
that returned 12 rows cost exactly 12 lines while treg settled 20 credits.

Two estimator defects fed that. `_body_limit` never looked inside a
JSON-RPC `params` object, so `size` was ignored and every row-priced
route reserved the page default. The routes priced per RETURNED row
(ranked_keywords, keywords.ideas, linking_domains.list) carried
`unit: keyword`/`domain`, which the per-input counter read as exactly
one credit regardless of `size`. With no settle rule the estimate was
the charge either way.

- resolve: a JSON-RPC envelope (`method` + `params` object) is read at
  the top level and then inside `params`, generically, for the same
  limit keys and entity lists as any body.
- catalog: those three routes are `unit: row`; keywords.volume and
  domain.overview stay per input. Notes say what is verified and that an
  error envelope is free; the unsourced "since 2025-06-15" minimum is
  restated as the documented, unverified floor.
- settle: `application/call/serpstat.py` counts rows in `result.data[]`,
  `result.data.top[]` or the keyed entries, settles an `error` envelope
  at 0 and a served empty result at the 1-credit floor; an unrecognised
  shape keeps the estimate.
@stonexer
fix(billing): charge SE Ranking keyword ideas per returned keyword
d06458c 
seranking.google.keywords.ideas bills 10 credits per keyword RETURNED, but
the catalog priced it per input keyword, so every call reserved and settled
one unit (1790 micro-USD) whatever limit asked or the answer carried.
Verified live 2026-09-09 on treg's own meter: 5 keywords returned cost 50
credits while treg charged 10; an empty answer cost 0 while treg charged 10.

- catalog: the route is now unit: row with page_default: 100 (the API's own
  default page), source observed, confidence verified; keywords.volume stays
  per input keyword.
- estimate: a row-priced cost may carry page_default, the rows the provider
  answers when the caller names no limit; the reserve is limit x unit, capped
  at the platform max, and the validator checks the key.
- settle: the returned keywords list is the bill (the influencersclub rule);
  an empty list settles at 0 and a non-JSON body settles at the estimate.
- tests: limit 5 reserves 5 units, 5 rows charge 8950, empty charges 0, an
  unparseable body settles at the estimate, siblings are untouched.
@stonexer
fix(billing): size Apify job searches by maxItems and settle on the d…
96c8a8e 
…ataset array

apify.linkedin.search.jobs documented the actor README's display labels
(Past month, On-site, Full-time) where the actor's input schema takes
month, office, full-time; the five enum fields now carry the schema's
values as enum lists and the test_request sends postedLimit so a
re-verify exercises one. The maxItems note called the query-string cap
the only hard spend limit, which is true for pay-per-result actors but
not for this pay-per-event one: maxTotalChargeUsd is the platform's
abort and the body maxItems is the actor's own cap.

Every platform call on the route settled at a flat 20,000 micro (3,019
calls since 2026-08-20) because maxItems was not a reserve signal and
no settle rule counted the response. resolve reads maxItems and
resultsLimit as limit params (positive integers only; 0 keeps the page
default) and settle counts a bare top-level JSON array for apify
per_result rows, falling back to the estimate on gzip, truncated or
unexpected shapes, mirroring the Bright Data rule.
@stonexer
fix(routing): verify adapters can fill every required input; findymai…
4b17157 
…l domain search needs a role

The adapter verifier compared only the keys an adapter maps against the
endpoint's test_request, so an adapter that could never produce a
required input still verified. findymail.search.domain accepted
{company_domain} alone while its body.roles is required: every routed
call to it was a vendor 4xx by construction, which the router read as
the caller's fault and ended treg.people.search with 422
route_caller_fault, keeping the earlier children's charges.

- contracts.py: missing_required_inputs runs each accepted variant of an
  adapter through to_upstream on a placeholder identity (choosing the
  variant the router would send) and verify() fails any adapter that
  cannot fill an input the endpoint marks required: true. Bare-array
  bodies are read through their first element; an input of type array
  that labels the array itself is satisfied by a non-empty array.
- adapters.yaml: findymail.search.domain accepts only
  {company_domain, title} and sends roles: [title];
  moz.web.backlinks.summary now sends distributions: true, which the
  endpoint is defined and priced by.
- lusha.yaml: the two search inputs name the pagination field the live
  API takes (pagination, size >= 10) instead of the docs' pages, as
  their notes, test_requests and adapters already did.
- tests: the verifier rejects an adapter missing a required body field;
  a domain-only routed people.search never reaches findymail and misses
  honestly instead of 422.
- docs: catalog.md adapter verification section.
@stonexer
fix(route): apply X-Treg-Route-Max-Cost per candidate, not to the top…
6c37f28 
…-ranked one

The routed pre-check compared the ceiling with plan.candidates[0] only and
refused the call, calling that candidate "the cheapest" even though the
plan is ranked by specificity before price. people.phone.find for
{email, full_name} leads with leadsforge (245000 micro) ahead of tomba
(44500 micro), so a $0.05 cap was refused while the embedded plan showed
an affordable row and the attempt loop would have skipped past it.

The loop already skips an over-cap candidate; the pre-check is gone and
the 402 route_max_cost is raised only after the loop, when every
candidate was skipped for cost and nothing was asked or reserved. It
names the true minimum (cheapest_micro, cheapest_endpoint_id). The
unused next() over a miss in tried, which raised StopIteration when
every attempt was skipped, is removed.

Docs: the catalog fragment now states the actual specificity rule (a
derived key covers the caller's key it was derived from) and the
per-candidate ceiling.
@stonexer
fix(call): resolve catalog calls by provider identity before host
46baf88 
Connect provisions every account of a provider on the same host with the
same provider-tagged secret, so a second connected account or two providers
sharing a host (meta-ads beside instagram-page-tools on graph.facebook.com)
tied in the URL-passthrough matcher and every catalog call for that provider
failed with 409 target_ambiguous.

- _resolve_marketplace_call now asks _provider_tool_grant for a tool bound to
  a connection of the endpoint's provider on the upstream host before matching
  by host; the method filter is bypassed for endpoints that declare none, the
  bare service name wins among several accounts, then the newest connection.
- _resolve_call's tiebreak treats a tool as provider-owned only when its
  secret's provider owns the upstream host (_providers_owning) and raises
  AmbiguousTarget carrying the tie.
- A catalog call that still ties restates the 409 in catalog terms: the
  endpoint id, the colliding tools and the /call/<tool>/<path> form of each.
- Own-account tools remain unmetered and unrouted on every branch.

Fragments updated: docs/context/architecture/proxy-model.md,
docs/context/architecture/instagram-oauth.md, docs/context/interface/api.md
@stonexer
fix(billing): reconcile the batched feedback fixes after cherry-pick
56a4bae 
Restore the icypeas platform fixture body, drop the duplicated contactsLimit
entry in _LIMIT_PARAMS, merge the routers/catalog hints expression and the
money.md row-count paragraph that both-sided cherry-picks had concatenated,
and regenerate MAP.md.
@stonexer
fix(call): refuse a catalog call that ties between two connected acco…
e0619a6 
…unts

Provider identity still resolves meta-ads beside instagram-page-tools, but
two accounts of one provider are a genuine tie: a silent default to the
bare-named tool would route one account's request through the other's
credential. Answer the structured 409 with each account's named form instead.
@stonexer
docs(context): regenerate MAP.md after rebase
482a8c8
@stonexer
test(routing): keep the max-cost story stable beside the new quickenr…
caf758d 
…ich phone candidate

quickenrich.people.phone.find (main, 2026-09-09) also covers {first_name,
last_name, domain} and undercuts tomba, so the two max-cost tests exclude it
and compare tomba against leadsforge by the plan's usd field.
@stonexer
stonexer force-pushed the fix/feedback-10-45-batch branch from 1111db8 to caf758d Compare September 9, 2026 14:39
@AdamBerri AdamBerri mentioned this pull request Sep 16, 2026
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area:cli The treg CLI area:docs Documentation & design fragments

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@stonexer