Skip to content

feat(catalog): add Scrubby email verification - #533

Merged
shehjad-dev merged 1 commit into
mainfrom
codex/provider-scrubby
Sep 16, 2026
Merged

shehjad-dev merged 1 commit into
mainfrom
codex/provider-scrubby

Conversation

@shehjad-dev

@shehjad-dev shehjad-dev commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Adds Scrubby (scrubby) as a pasted-key email-verification provider: all five documented endpoints, with single-email verification on BYOK and treg's platform key and the four asynchronous bulk/deep tools on BYOK only.

Why now: Scrubby adds both immediate verification and a 72-hour deep-verification workflow. Its responses report exact credits_used, including zero for cached retries, so treg can settle the shared single-email tool from provider evidence instead of assuming every successful response costs one credit.

Listing owner / contact: treg team (platform key held in private deployment configuration); vendor behavior was clarified directly with Scrubby support. No credential is included in this PR.

What's in the diff

  • catalog/scrubby.yaml — the complete five-POST OpenAPI surface with captured, sanitized examples: single verification; quick-bulk submit/fetch; deep submit/fetch.
  • oauth_providers.py / config.py — pasted x-api-key connection, exact free miss probe, and TREG_PLATFORM_KEY_SCRUBBY platform-key slot. A fixed descriptive User-Agent intentionally overwrites the caller value on probes and BYOK/platform calls because Scrubby's edge rejected Python's default urllib profile before it reached the API.
  • application/call/settle.py — provider-specific settlement from top-level credits_used, following the established Sumble/QuickEnrich pattern. Only non-negative integers are accepted; zero remains a free cached response.
  • catalog/adapters.yaml — maps scrubby.people.email.verify into the existing people.email.verify route. That one tool participates in automated routing and Enrich Arena; all Scrubby verdicts are answers rather than routing misses.
  • domain/capacity/policy.py / domain/capacity/collectors.py / catalog/fx.yaml — documented 25 requests/second pacing, manual credit capacity, explicit NO_BALANCE_API classification, and the supplied $50 / 6,250-credit acquisition rate ($0.008/credit).
  • The four bulk/deep lifecycle tools are explicitly BYOK-only. Scrubby publishes no batch maximum, so treg cannot bound a shared-key reservation; fetch identifiers are carried in JSON bodies, while the existing shared-key ownership guard covers declared request parameters.
  • logos/scrubby.svg — the exact supplied Scrubby artwork (SHA-256 8e7874baf3cd70ad485528f1df7ce04d7c30274d729a61d9e2efb201dfa1e9e2).
  • tests/test_scrubby.py plus shared registry/capacity assertions; Scrubby context fragment and generated context/plugin mirrors.
  • No generic relay, credential, router, Arena, ownership, or capacity contract was extended for this provider.

Verification evidence ledger (2026-09-16)

Meter for charged rows: top-level credits_used in Scrubby's response, multiplied by the frozen $0.008/credit acquisition rate. Fetch tools are priced free because Scrubby charges at submission and their responses expose no charge field.

tool / upstream path http live target credits observed catalog price access / integration matches?
scrubby.people.email.verify · /validate_email 200 one fresh reserved-domain address 1 $0.008/success BYOK + platform; routed people.email.verify; Enrich Arena
same tool, exact retry 200 same address within 24h 0 $0 cached same
scrubby.people.email.verify.bulk · /validate_bulk_emails 202 two addresses 2 $0.008/fresh record BYOK only
scrubby.people.email.verify.bulk.results · /fetch_bulk_results 200 processing, then completed batch 0 free poll BYOK only
scrubby.people.email.verify.deep · /validate_bulk_emails/deep/ 202 one address 3 $0.024/record BYOK only
scrubby.people.email.verify.deep.results · /fetch_bulk_results/deep 200 pending 24/48/72-hour windows 0 free poll BYOK only

Quick bulk returned an identifier and a 30-second retry interval. Deep polling returned a live 259,200-second retry interval. No pagination is documented.

Bogus-key probe (live, local treg on this branch): no credential and a bogus credential returned structured upstream 401 responses. POST /connections/token with the bogus key returned treg HTTP 422. A real key returned HTTP 200. Both upstream-supported auth forms (x-api-key and Authorization: Api-Key …) worked; treg uses the documented x-api-key form.

The free connection probe posts a fixed nonexistent identifier to /fetch_bulk_results. A valid key returns HTTP 404 with exact detail No results found for this identifier.; 401/403 remain rejection statuses. The probe consumed no credit.

Platform settlement and BYOK precedence: the call-path test reserves one credit and settles credits_used: 1 to x-treg-cost-micro: 8000; malformed, negative, Boolean, fractional, and missing evidence retain the reserved estimate for reconciliation. A live cached platform call returned 200 with credits_used: 0 and settled at zero. A live BYOK call returned 200 with no treg cost header or balance change, confirming that a team's own key wins and is unmetered.

Timeout and retry evidence

  • A direct fresh single verification completed after 108.5 seconds and reported one credit, so the tool warns that a fresh call can take about two minutes.
  • One local treg request reached Cloudflare 524 near 125 seconds. Retrying the exact request returned HTTP 200 with credits_used: 1, indicating that particular timed-out attempt had not consumed a Scrubby credit; a later retry was cached with credits_used: 0.
  • Scrubby support separately confirmed that an upstream 408 can consume a credit and that retrying the same email within 24 hours is then free and returns the same result. Single-email responses expose no reconciliation identifier. treg therefore relays provider behavior and settles only from valid response evidence; it does not invent a refund.
  • Structured transient Cloudflare 502 responses were observed during asynchronous operations and remained upstream failures.

Capacity and routing decisions

  • Published rate limit: 25 requests/second (1,500/minute) per API key.
  • Capacity is manual because Scrubby exposes no free standalone account-balance or usage endpoint; NO_BALANCE_API reports that deliberate limitation instead of no fetcher written yet. remaining_credits appears only on verification responses, and treg does not spend a verification merely to collect capacity. There is no collector or overflow route.
  • Only scrubby.people.email.verify is platform-enabled, routed, and included in Enrich Arena.
  • The other four tools remain direct BYOK tools for the reservation and ownership reasons above.
  • The out-of-credit response shape was not forced; it is explicitly recorded as an unobserved exhaustion signature rather than guessed.

Validation after rebasing current main

  • Exact CI command: uv run --with pytest-xdist pytest -n auto -q4,268 passed, 8 skipped.
  • Focused Scrubby and capacity-collector suite after the review fixes → 63 passed.
  • catalog_validate.py107 provider files, 3,399 endpoints, 0 errors, 0 warnings.
  • Import lint → 14 contracts kept, 0 broken.
  • build_plugin.py --check → all five generated mirrors match.
  • git diff --check origin/main...HEAD passes.
  • Context drift maps every changed Scrubby source with no missing fragment.
  • Rebased onto main at 90edbf9e.

Ops

  • This PR does not install a production key, modify private production configuration, or enable shared capacity.
  • After merge, set TREG_PLATFORM_KEY_SCRUBBY in the private deployment and add scrubby to TREG_PLATFORM_PROVIDERS, then run a production smoke check before treating platform capacity as available.
  • The assigned key and canonical local environment file were never printed or changed. Live verification stayed below the authorized 25-credit ceiling.

Sources

@github-actions github-actions Bot added area:dashboard The web dashboard / landing / tutorials area:docs Documentation & design fragments labels Sep 16, 2026
@shehjad-dev
shehjad-dev force-pushed the codex/provider-scrubby branch from 59eaa0a to 9aac53b Compare September 16, 2026 20:13
@shehjad-dev
shehjad-dev force-pushed the codex/provider-scrubby branch from 9aac53b to 418be58 Compare September 16, 2026 20:53
@shehjad-dev
shehjad-dev marked this pull request as ready for review September 16, 2026 21:12
@shehjad-dev
shehjad-dev merged commit 346ddaa into main Sep 16, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dashboard The web dashboard / landing / tutorials area:docs Documentation & design fragments

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant