Scrape pipeline: account keys get Forbidden polling their own Apify runs + TikTok/X results never persist

[sweetmantech]

Tracking issue for two scrape-pipeline bugs that block sandbox agents from using their own scrape results, found 2026-07-02 while production-testing an artist's daily-briefing task. Sibling of chat#1839 (sandbox delivery fixes) — same failure shape (the agent does everything right; the platform drops the result), different subsystem.

Goal

1. An agent authenticated with an account-scoped ephemeral key that starts a scrape via POST /api/socials/{social_id}/scrape can poll that run's status and results via GET /api/apify/runs/{runId} — today it gets Forbidden because the endpoint is admin-only (api/lib/apify/validateGetScraperResultsRequest.ts calls validateAdminAuth).
2. Successful TikTok and X/Twitter Apify runs persist their results (follower counts to socials, post rows to posts/social_posts) the way YouTube runs already do, so GET /api/artists/{id}/posts and embedded account_socials counts reflect reality.

PRs (updated 2026-07-03)

PR	Item	Base	State
api#752	GET /api/apify/runs/{runId}: validateAdminAuth → validateAuthContext — any valid API key / Bearer can poll any run	main	✅ merged 2026-07-03, prod-verified
api#753	TikTok/X post persistence: shared persistPostsForSocial + handlers write posts/social_posts rows (socials counts + webhook registration already fixed by api#738/#740 under chat#1833)	main	✅ merged 2026-07-03, e2e-verified

The two api PRs are independent — merge in either order. docs#262 closed unmerged: after the capability-model decision the existing docs page already matches the new behavior (it never documented the admin gate), so there is no contract drift to fix. database#39 (runId→account ownership map) was closed unmerged when the design moved to the capability model — see the architecture decision below. Preview note (2026-07-03): api preview deployments do not share the prod key store (a prod-minted key 200s on prod, 401s on previews), so key-based Done-when checks run against the test deployment post-merge — round-1 verification tables are on both api PRs.

Evidence (2026-07-02 production run, chat eac0b159-7473-4d72-a12b-f8bde3f084fc)

• Forbidden polls, 7/7 across two rounds. The agent kicked off 4 scrapes via POST /api/socials/{social_id}/scrape with its ephemeral $RECOUP_API_KEY — all accepted, runIds returned (RMpBmxlGtptIyBnb3, 9TZRbi9gz2TgBMeMH, OeGhuGV3obde3CDGc, XBsWtbWYPlSnpWj5O; retry round 6fzZN5NjsVRIWS1Yi, HoJaej0Ue3PYafw5M, FulNVd3JxH2SeyQ9n). Every GET /api/apify/runs/{runId} with the same key returned {"status":"error","message":"Forbidden"}. The same polls with an admin token succeed. Root cause is by-design: validateGetScraperResultsRequest is admin-only ("Apify run identifiers are not account-scoped, and the poller is a backend-only caller (tasks)") — but the public docs (docs.recoupable.dev Scraper Results page), the recoup-platform-api-access skill, and existing customer task prompts (e.g. the OneRPM roster-wide YouTube report) all instruct agents to poll this endpoint. The asymmetry (can start, can't read) makes every scrape-dependent agent run fail its data step.
• TikTok/X results succeed on Apify but never persist. Runs 9AYX8xyaHWyHtnGtC (TikTok) and bx3asRqfbNnkKgogG (X) reached SUCCEEDED with real dataset items (the artist's TikTok posts and tweets, verified via admin poll 2026-07-02 ~23:10Z). The corresponding socials rows (d497a17b, 291753e9) kept followerCount: 0 and stale updated_at (2025-07-20); zero rows landed in posts/social_posts for the artist. By contrast the YouTube run from the same account persisted within minutes (socials.followerCount = 507, updated_at 2026-07-02 22:57), so the webhook/persist path (api/lib/apify/apifyWebhookHandler.ts + lib/apify/tiktok, lib/apify/twitter handlers) works for YouTube but not for TikTok/X runs started from POST /api/socials/{id}/scrape. Unknown whether the webhook isn't registered for those actors on this path or the handlers fail silently — first investigation step.
• Downstream symptom: GET /api/artists/{id}/posts returned a 500 on first call, then {"posts": [], "total_count": 0} — the "What You've Been Posting" section of the briefing had nothing to read because nothing is ever written.

Done

• Data fix: relinked the affected artist's socials to live handles. ✅ 2026-07-02 (data-layer change, no PR). The artist's linked Instagram was a dead handle (brauxelion.gd, Apify not_found — account renamed) and TikTok was the non-canonical tiktok.com/brauxelion. Linked the live instagram.com/@brauxelion and tiktok.com/@brauxelion socials to the artist and unlinked the dead IG. Verified: account_socials now carries both live rows (+1 each) and the dead link is removed. Note: there is no API endpoint for linking/unlinking socials to an artist — tracked below as a gap.

Open — fixes (independent; pick either first)

• Open the scrape-results endpoint to any authenticated key. ✅ 2026-07-03 — api#752 merged to main (first PR under the retired-test-branch flow, api#750) and verified on prod: a freshly minted non-admin agent key polled run 9AYX8xyaHWyHtnGtC → 200 SUCCEEDED with data (this exact call was 403 the same morning). Preview verification tables on the PR.

  • Why: validateGetScraperResultsRequest admin-gates all polls — while docs/skills/prompts direct account-key agents to poll. Agents can commission scrapes they can never read.
  • Fix: swap validateAdminAuth → validateAuthContext (any valid API key / Bearer passes; 401 stays for anonymous callers); no docs change needed — the existing page already matches (docs#262 closed unmerged). Unit tests: auth-error propagation, 400 bad runId, authenticated passthrough.
  • Done when: an account key starts a scrape and successfully polls the same runId to completion (no admin token involved).

• Make TikTok/X scrape results persist like YouTube's. ✅ 2026-07-03 — socials counts shipped via api#738/Added tool result UI for web search #740 (chat#1833); posts rows via api#753, merged to main (prod deploy from 177c0410 success). Verified live end-to-end against the production DB (previews share it): TikTok + X scrapes → webhook → 10+10 posts/social_posts rows → GET /api/artists/{id}/posts went 49 (all Instagram) → 69 (49 ig + 10 x + 10 tiktok). Full results table on the PR. Review hardening: toIsoDate extracted to its own lib file; TikTok dates normalized so one malformed createTimeISO can't fail an upsert batch.

  • Why: succeeded Apify runs for TikTok/X leave socials/posts untouched (evidence above); YouTube's path persists. Suspect webhook registration or the per-platform result handlers (lib/apify/tiktok, lib/apify/twitter) on the POST /api/socials/{id}/scrape path.
  • Fix: (1) trace one TikTok and one X run end-to-end: was a webhook registered at actor-start, did POST /api/apify receive it, which handler ran, what did it write; (2) fix the broken link (registration or handler mapping or payload-shape mismatch); (3) add a test per platform handler asserting a SUCCEEDED payload writes followerCount + post rows.
  • Done when: POST /api/socials/{id}/scrape on a TikTok social and an X social → within one webhook cycle, socials.followerCount/updated_at are fresh and GET /api/artists/{id}/posts returns the scraped posts.

Architecture decisions

• Open the endpoint to any authenticated key (decided 2026-07-03, supersedes the earlier owner-or-admin plan). Scrape datasets are public social content and Apify run ids are long random strings, so possession of a runId plus a valid credential is sufficient; authentication stays so anonymous callers can't use the endpoint as a free Apify proxy. This made the runId→account ownership map unnecessary — database#39 closed unmerged. Tightening later would break agents that depend on open reads, so this is the deliberate end state, not a stopgap.

Inherited gaps / Source references

• Sibling tracker: chat#1839 (web_fetch empty-body + ephemeral key refresh). This issue completes the "agent-blocking platform bugs" family surfaced by the 2026-07-02/03 task-email work.
• Known gaps noted, out of scope here: no API endpoint to link/unlink a social to an artist (Done item above needed direct data access); POST /api/socials/{id}/scrape returns "Unsupported social media platform" for Spotify; POST /api/emails restricts accounts without a payment method to their own address (documented behavior discovered 2026-07-02 — correct, but surprising to test flows).
• Key files: api/app/api/apify/runs/[runId]/route.ts, api/lib/apify/validateGetScraperResultsRequest.ts, api/lib/apify/apifyWebhookHandler.ts, api/lib/apify/getScraperResultsHandler.ts, api/lib/apify/tiktok/, api/lib/apify/twitter/, api/app/api/socials/[id]/scrape path.

[sweetmantech]

Reproduced 2026-07-02 ~23:50Z against prod with a freshly minted non-admin ephemeral key (same minting path as mintEphemeralAccountKey, scoped to the affected artist's owner account; key deleted after the test):

1. POST /api/socials/{social_id}/scrape with the key → 200 {"runId":"Q97vBzJtcY2V6jkV0","datasetId":"Lw3hpb9yoe1TRfQTM"}
2. GET /api/apify/runs/Q97vBzJtcY2V6jkV0 with the same key → 403 {"status":"error","message":"Forbidden"}
3. Control: GET /api/artists/{artist}/posts with the same key → 200 (key is valid; owner-gating works on endpoints that have an ownership chain)

Exact rejection chain: app/api/apify/runs/[runId]/route.ts → lib/apify/getScraperResultsHandler.ts → lib/apify/validateGetScraperResultsRequest.ts L24 (validateAdminAuth) → lib/admins/validateAdminAuth.ts L21-27 emits the 403 when checkIsAdmin (= membership in RECOUP_ORG_ID) fails. The endpoint never looks at the runId's provenance — there is nothing to look at, since no table maps runId→account. That's the structural difference from owner-gated endpoints like the scrape POST (lib/socials/validatePostSocialScrapeRequest.ts: resource → account_socials links → checkAccountArtistAccess).

Bug-2 findings from the same dig: only the Instagram starters attach the persistence webhook (getApifyWebhooks is imported solely by startInstagramProfileScraping/startInstagramCommentsScraping), and apifyWebhookHandler's dispatch handles only the two Instagram actorIds (everything else → "Unhandled actorId" warn). TikTok/X/YouTube starters attach no webhook in code. Corroboration: after relinking the artist's live Instagram handle, GET /api/artists/{id}/posts went from [] to returning real post rows — the IG pipeline persists end-to-end; TikTok/X have no code path to. Open sub-question for the PR: something did persist the YouTube followerCount on 2026-07-02 22:57 and it isn't visible in api or tasks code (tasks' pollScraperResults.ts is unreferenced) — identify the writer before assuming YouTube works by design.

[sweetmantech]

Closing — both fixes shipped to main and verified 2026-07-03:

1. api#752 — any valid API key/Bearer can poll GET /api/apify/runs/{runId} (capability-model decision recorded in Architecture decisions; the owner-map design and its database PR were superseded and closed unmerged). Prod-verified: a freshly minted non-admin key polled run 9AYX8xyaHWyHtnGtC → 200 SUCCEEDED (7/7 Forbidden the same morning).
2. api#753 — TikTok/X posts persist like Instagram's. Verified end-to-end (preview deployment, shared production DB): live scrapes → webhook → 20 post rows → the affected artist's GET /api/artists/{id}/posts went from 49 posts (all Instagram) to 69 (+10 tiktok, +10 x). Next natural confirmation: the artist's weekly briefing scrape (Mon 10:30Z).

Data fix along the way: the artist's TikTok link pointed (twice) at the stale legacy tiktok.com/brauxelion row while handlers persist to the canonical tiktok.com/@brauxelion — relinked at the data layer, one more case for the no link/unlink-social API gap noted above.

New env findings for whoever hits them next: preview deployments share the prod Supabase but hash API keys with a preview-specific secret and use a separate Privy app that is at max_accounts_reached (so POST /api/agents/signup 500s on previews); a few throwaway agent+ test accounts were left in prod from verification.

Follow-up (separate scope, feeds the original briefing goal): posts persist URL + timestamp only — a "best performing content" section needs per-post engagement metrics (columns + handler mapping, or in-run reads of the scrape dataset now that #752 allows agents to poll).