Social scrape: optional `posts=N` depth so X + YouTube Shorts scrapes return per-post metrics (docs + api)

[sweetmantech]

Tracking issue for making the Apify social-scrape endpoints return recent per-post engagement instead of a single latest item. Perspective (out of scope here): internal weekly social measurement needs per-post X and YouTube Shorts numbers, and the X connector analytics path is a dead end (TWITTER_GET_POST_ANALYTICS → 403 client-not-enrolled; post lookup → 401) — the scrape pipeline is the only viable source, verified 2026-07-02.

Goal

POST /api/socials/{id}/scrape and POST /api/artist/socials/scrape accept an optional posts=N depth parameter (default 1, today's behavior) that flows into the platform Apify actor inputs, so a scrape of an X or YouTube profile returns the last N posts with engagement metrics in the GET /api/apify/runs/{runId} dataset:

• X — apidojo/twitter-scraper-lite already returns likeCount, retweetCount, replyCount, quoteCount, bookmarkCount, and viewCount (impressions) per tweet; we just cap it at 1 item (api/lib/apify/twitter/startTwitterProfileScraping.ts L14: maxItems: 1).
• YouTube — streamers/youtube-scraper is capped at 1 long-form video and explicitly excludes Shorts (api/lib/apify/youtube/startYoutubeProfileScraping.ts L18-L19: maxResults: 1, maxResultsShorts: 0).

Instagram already works (its profile actor bundles latestPosts — 12 posts with likes/comments/plays in one profile scrape); X and YouTube reach parity via the new param. Existing callers see zero behavior change.

PRs (updated 2026-07-02)

PR	Item	Base	State
docs#258	docs: posts param on both scrape endpoints + per-platform dataset shapes (OpenAPI social.json + releases.json)	main	✅ merged 2026-07-02 — see Done
api#741	api: plumb posts → Twitter maxItems, YouTube maxResults/maxResultsShorts (default 1/0 unchanged)	test	✅ merged 2026-07-02 — see Done
docs#259	docs: bill social scrape on credits.mdx (5 + posts, per profile) + 402s + auth on bulk endpoint	main	✅ merged 2026-07-02 (f70b9ff)
api#744	api: credit-gate both scrape endpoints (5 + posts per profile) + auth on bulk	main (retargeted from test)	✅ merged 2026-07-02 — see Done

Merge sequencing: docs → api (documentation-driven; the OpenAPI change is the contract the api PR fulfills). api targets test, then test→main.

Context — verified 2026-07-02

All three platforms tested live via both POST /api/artist/socials/scrape (bulk) and POST /api/socials/{id}/scrape (direct); every run returned exactly 1 dataset item:

• X (runs rMCM4jzjd9V9Y05mU, lbm3U2eqNzLpfCv8q): 1 item — the latest timeline entry (a retweet), but carrying full public metrics incl. viewCount: 699267. The account's own 4 posts from 2026-06-24 → 07-01 were not reachable.
• YouTube (runs ZDGoi2IlDQj0KZL6k, 3OTYCnusYtUeUlrGU): 1 item — a long-form video from 2025-11-25; the channel's 4 Shorts published 2026-06-24 → 07-01 were all absent (maxResultsShorts: 0).
• Instagram (run ewcxyNIY97dfQrdQb): profile item with latestPosts = 12 posts, each with likesCount/commentsCount/videoViewCount — the parity target.

Interim X workaround in use: the public syndication endpoint (cdn.syndication.twimg.com/tweet-result) — likes + conversation count only, no impressions.

Done

• docs#258 — posts param on both scrape endpoints + per-platform dataset shapes.
  ✅ Shipped 2026-07-02 (merged to main, merge commit 615cf5d).
  Adds the optional posts param (integer, 1–100) to POST /api/socials/{id}/scrape (query) and POST /api/artist/socials/scrape (body), expands the Scraper Results example with X tweet + YouTube video items, and moves the artist socials-scrape page into the Social Media nav group. Two rounds of review: descriptions reconciled with live preview behavior (YouTube depth is per content type — up to N videos + N Shorts; X returns latest N timeline items incl. RTs), then KISS'd per human review and default: 1 removed (omitted ≠ posts=1 on YouTube: omitted excludes Shorts, posts=1 includes one — a schema default would mislead SDK generators).
  Verified: both OpenAPI files parse after every edit; contract semantics match the api#741 implementation exactly, which was itself verified live (see the preview verification).

• api#741 — thread posts from the scrape routes into the platform actor inputs.
  ✅ Shipped 2026-07-02 (merged to test 4290c72, promoted to main via api#743 6ae6e79, test re-synced d5f649f — live on prod: posts=0 probe on api.recoupable.dev flipped 401→400 post-deploy, confirming the new validation is serving).
  Optional posts (1–100) on both endpoints, validated before auth, threaded through scrapeProfileUrl/scrapeProfileUrlBatch into Twitter maxItems: posts ?? 1 and YouTube maxResults: posts ?? 1 / maxResultsShorts: posts ?? 0 (a requested depth includes Shorts; omitted keeps the byte-identical legacy input — unit-tested). TDD red→green (20 failing tests first); full suite 3745/3745.
  Verified twice on preview (second time on the final head c868d45 via the authed path with a preview-env Privy token): X posts=20 → 20 tweets incl. pinned id with viewCount=188 (impressions); YouTube posts=20 → 40 items (20 videos + 20 Shorts) incl. all 4 pinned Shorts; all 400/401 paths. Final results table. Note: the older pinned tweet 2069618388926701915-era ids fall outside a 20-item timeline window (X depth counts RTs/replies) — documented in docs#258 rather than special-cased.

Open — contract first (shipped — see Done)

Follow-ups (shipped — see closure note)

• Credit-gate the scrape endpoints (docs#259 + api#744) — 5 credits + 1 per requested post, per profile.
  ✅ Shipped 2026-07-02 (docs merged f70b9ff; api merged direct-to-main f45cd0f, test re-synced d72c826 — live on prod: unauthenticated bulk probe flipped 200→401 post-deploy).
  Pricing from measured Apify cost (cost table: worst case YouTube ~$0.003 × 2×posts items; 5+posts keeps ≥1.75× margin at posts=100, 3–20× at default). Implementation mirrors the research family: ensureSocialScrapeCredits gates (402 + checkoutUrl, auto-recharge path) before any Apify run; deductSocialScrapeCredits deducts only for runs that started, never throws. POST /api/artist/socials/scrape gained auth + artist-access (was fully open). Docs: price lives on credits.mdx's What's-billed table (per review, not per-endpoint); OpenAPI documents the new 402s + bulk auth. TDD red→green (10 failures first); 324/324 affected-domain tests.
  Verified on preview with balance reads around every call (results): default −5 exactly, posts=20 −25 exactly, bulk ×4 profiles −20 exactly; bulk 401 without auth; no-socials → [] with zero charge; 400s before auth. 402/true-403 unit-covered (test account is org-admin with ~1B credits).

Out of scope

• LinkedIn per-post scrape. The LinkedIn path uses a person-profile actor (no post fields at all; verified against runs LfeuRTBHFITdh51SP, 9R4lnMCOCKigxFH46) and isn't in scrapeProfileUrl.ts's PLATFORM_SCRAPERS — supporting posts means selecting a different actor. Follow-up tracker if wanted; the LINKEDIN_LIST_REACTIONS connector action covers per-post reactions today.
• Enrolling the X dev app for connector analytics (fixes the 403 at the OAuth-app level) — independent of this change, and the scrape path is strictly better (impressions without API-tier gating).
• Deciding whether scrapes should charge credits — promoted to an Open item (2026-07-02, sweetman): credit-gating + Apify cost research now tracked above.
• recoup-internal-social-ship-posts skill update to consume posts=N — lands in recoupable/skills after the api PR ships.

Architecture decisions

• Default depth stays 1. Apify actors bill per dataset item; the existing scrape callers are profile-snapshot flows (onboarding, follower refresh) that don't need posts. Depth is strictly opt-in, so cost and behavior are unchanged unless a caller asks.
• Scrape over connector for X metrics. The connector requires the linked X dev app to be enrolled in a paid API project for analytics (403 client-not-enrolled) and 401s on post lookup; the Apify actor returns richer public metrics (incl. impressions) with no per-account OAuth scope wall.

Source references

• Actor configs: lib/apify/twitter/startTwitterProfileScraping.ts, lib/apify/youtube/startYoutubeProfileScraping.ts, routing in lib/apify/scrapeProfileUrl.ts.
• Endpoint docs: Social Scrape, Artist Socials scrape, Scraper Results.
• Discovery session: 2026-07-02 weekly engagement pull (strategy repo, sweetman/posts-log.md → "Measurement paths").

[sweetmantech]

Credits audit (2026-07-02, from source while implementing api#741): neither scrape endpoint charges credits today, and they differ on auth:

• POST /api/socials/{id}/scrape — authenticated (validateAuthContext) + per-artist access check (checkAccountArtistAccess over owning artists), but no credit deduction.
• POST /api/artist/socials/scrape — no auth and no credits: postArtistSocialsScrapeHandler validates only the body and calls Apify directly (source: lib/artist/postArtistSocialsScrapeHandler.ts). Anyone with an artist_account_id can trigger Apify runs billed to Recoup. The root middleware.ts only applies x402 payment to GET /api/x402/image/generate, so nothing upstream gates this route either.
• The credit infrastructure exists and has a per-route pattern to copy: the research family charges 5 credits/call via ensureResearchCredits → ensureCreditsOrShortCircuit (source: lib/research/ensureResearchCredits.ts).

api#741 deliberately does not change auth or billing (out of scope; posts defaults keep Apify cost identical for existing callers). But since posts=N raises the per-call Apify cost ceiling ~100×, two follow-up decisions are worth their own item/issue:

1. Add validateAuthContext + artist-access to POST /api/artist/socials/scrape (straight security/cost gap, independent of this feature).
2. Decide whether scrapes should charge credits (e.g. flat per scrape, or scaled by posts), mirroring ensureResearchCredits.

[sweetmantech]

Apify cost research — scrape endpoints (2026-07-02)

Sources: Apify actor store pricing (GET /v2/acts/{actor} → pricingInfos, PAY_PER_EVENT for all 7) + observed usageTotalUsd from 13 real runs today (run ids in this issue/PR). Credit anchor: 1 credit = $0.01 (UNIT_AMOUNT_CENTS_PER_CREDIT = 1); research precedent: 5 credits/call.

Platform	Actor	Fixed / query cost	Per-item cost	How posts scales items	Observed @ default	Observed @ posts=20	Projected @ posts=100 (max)
X/Twitter	apidojo/twitter-scraper-lite	$0.016 per profile query (includes first ~40 results)	$0.0004–0.002/item (tiered, ≤5 queries → $0.0004)	1 query regardless of depth; items ≤40 covered by query fee	$0.016 (2 runs)	$0.016 (2 runs — flat!)	~$0.016–0.06 (>40 items unobserved; needs 1 confirming run)
YouTube	streamers/youtube-scraper	—	$0.003/video observed ($0.004 list; +$0.0013 if date-filter)	2×posts (N videos + N Shorts)	$0.003 (1 item, 2 runs)	$0.12 (40 items, 2 runs)	~$0.60 (200 items) — the cost ceiling
Instagram	apify/instagram-profile-scraper	—	$0.0023/profile observed ($0.0026 list; +$0.007 w/ about-account)	none — latestPosts (~12) bundled free	$0.0023 (2 runs)	$0.0023 (ignores posts)	$0.0023
TikTok	clockworks/tiktok-scraper	$0.001 actor-start	$0.0037/result (+$0.0013 per filter; $0.00125/comment; $0.048/transcription-min)	~1×posts (resultsPerPage)	~$0.005 est. (no runs — no account attached)	~$0.075 est.	~$0.37 est.
Threads	apify/threads-profile-api-scraper	—	$0.005/profile	none — profile only	$0.005 est.	$0.005	$0.005
Facebook	apify/facebook-pages-scraper	—	$0.012/page	none — page only	$0.012 est.	$0.012	$0.012
LinkedIn	harvestapi/linkedin-profile-scraper	—	$0.004/profile ($0.01 w/ email search — not used)	none — profile only	$0.004 (3 runs)	$0.004	$0.004

Shape of the cost: default-depth scrapes cost $0.002–0.016 everywhere. posts only moves the needle on YouTube (2×posts items — worst case $0.60 at posts=100), TikTok ($0.37), and X (flat $0.016 up to ~40 items, then tiered). Instagram/Threads/Facebook/LinkedIn are profile scrapers and ignore posts entirely.

Recommendation: 5 + posts credits (5 credits when posts omitted; 5 + N when posts=N; bulk endpoint = sum over profiles scraped).

• Omitted → 5 credits ($0.05): parity with the research family, 3–20× margin on every platform.
• posts=20 → 25 credits ($0.25): ≥2× margin vs the YouTube worst case ($0.12).
• posts=100 → 105 credits ($1.05): ~1.75× margin vs YouTube's ~$0.60 ceiling, covers TikTok's ~$0.37.
• One rule for all platforms → one sentence in the docs; no per-platform matrix to maintain. Per-platform pricing would be more cost-accurate (X/IG barely scale) but adds contract complexity for <$1 of spread per call.

Caveats: X beyond 40 items is projected, not observed (one posts=100 X run would close it); TikTok/Threads/Facebook are store-price estimates (no connected account to run); YouTube observed rate ($0.003) already reflects Recoup's Apify plan tier.