feat(apify): scrape alerts fire only on genuinely new posts (diff against stored posts)#759
Conversation
… posts first The Instagram scrape alert fired whenever a scrape returned posts, with no comparison against posts already stored — every scrape re-announced the profile's recent feed as new (observed: 6 of 7 alerts in one day announced posts up to 10 days old). New reusable filterNewPostUrls diffs candidate URLs against posts BEFORE upsert; the alert is gated on a non-empty result. Persistence unchanged (recoupable/chat#1855, PR #3 of 6). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
📝 WalkthroughWalkthroughA new ChangesNew Post Filtering for Email Notifications
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related issues
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
No issues found across 4 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Architecture diagram
sequenceDiagram
participant ApifyCloud as Apify Cloud Webhook
participant Handler as handleInstagramProfileScraperResults
participant Filter as filterNewPostUrls (NEW)
participant DB as Supabase (posts table)
participant Upsert as upsertPosts / upsertSocials / upsertSocialPosts
participant Email as sendApifyWebhookEmail
participant FollowUp as handleInstagramProfileFollowUpRuns
Note over ApifyCloud,FollowUp: Instragram scrape alert — NEW: only notify on genuinely new posts
ApifyCloud->>Handler: POST /apify/instagram (scrape result)
Handler->>Handler: Extract post URLs from latestPosts
Note over Handler,Filter: Must run BEFORE upsert (chat#1855)
Handler->>Filter: filterNewPostUrls(postUrls)
Filter->>DB: getPosts({ postUrls })
DB-->>Filter: existing posts (post_urls)
Filter-->>Handler: newPostUrls (URLs not in DB)
Handler->>Upsert: upsertPosts(postRows) (all scraped URLs)
Handler->>Handler: getPosts() to fetch upserted records
alt newPostUrls.length > 0
Handler->>Email: sendApifyWebhookEmail(firstResult, emails)
Email-->>Handler: sentEmails
else newPostUrls.length === 0
Note over Handler: NEW: skip email — all posts already stored
end
Handler->>FollowUp: handleInstagramProfileFollowUpRuns(...)
FollowUp-->>Handler: follow-up runs
Handler-->>ApifyCloud: response (social, posts, etc.)
Auto-approved: Adds filterNewPostUrls to diff scraped URLs against stored posts, gating alert emails on genuinely new posts only. Tests pass for both the helper and the handler's no-email path.
Re-trigger cubic
Preview verification — 2026-07-09Preview: Method: same isolated-fixture pattern as api#758's verification — a
This is exactly the issue's Done-when: "re-running a full scrape twice back-to-back yields exactly one digest; a scrape finding zero unseen post URLs yields none" — observed live as send → suppress on identical back-to-back deliveries. Notes:
Verdict: the newness gate does exactly what it claims on the preview — send on new, silence on seen, persistence unaffected. Ready in merge order (after database#41 per chat#1855 sequencing; no hard dependency on it, but the issue's ordering keeps the digest stack clean). |
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
lib/apify/instagram/handleInstagramProfileScraperResults.ts (1)
33-122: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoffConsider extracting sub-steps to reduce function length.
handleInstagramProfileScraperResultsspans ~90 lines, well beyond the 20-line guideline. While this is pre-existing, the PR adds new logic to an already long function. Consider extracting cohesive blocks (e.g., social upsert + lookup, email + follow-up side effects) into named helpers to improve readability and testability.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/apify/instagram/handleInstagramProfileScraperResults.ts` around lines 33 - 122, `handleInstagramProfileScraperResults` has grown too long, and the new logic makes it harder to read and test. Extract cohesive blocks from this function into named helpers in the same module or nearby, especially the social upsert/lookup flow and the email/follow-up side effects, while keeping the main `handleInstagramProfileScraperResults` orchestration concise.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/apify/instagram/handleInstagramProfileScraperResults.ts`:
- Around line 44-47: The new pre-upsert read in
handleInstagramProfileScraperResults can prevent persistence if
filterNewPostUrls (and its underlying getPosts read) throws, so update this flow
to fail open. Wrap the filterNewPostUrls call in a try-catch inside
handleInstagramProfileScraperResults, and on any read/error path default
newPostUrls to all post_urls from postRows so upsertPosts still runs
unconditionally. Keep the existing postRows handling intact and ensure the
fallback preserves the prior behavior where write success is not blocked by read
failures.
---
Nitpick comments:
In `@lib/apify/instagram/handleInstagramProfileScraperResults.ts`:
- Around line 33-122: `handleInstagramProfileScraperResults` has grown too long,
and the new logic makes it harder to read and test. Extract cohesive blocks from
this function into named helpers in the same module or nearby, especially the
social upsert/lookup flow and the email/follow-up side effects, while keeping
the main `handleInstagramProfileScraperResults` orchestration concise.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: a92e147a-2b02-40ba-b1d4-8f8dfce2f1f1
⛔ Files ignored due to path filters (2)
lib/apify/instagram/__tests__/handleInstagramProfileScraperResults.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**lib/socials/__tests__/filterNewPostUrls.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**
📒 Files selected for processing (2)
lib/apify/instagram/handleInstagramProfileScraperResults.tslib/socials/filterNewPostUrls.ts
| // Diff BEFORE upserting — afterwards every scraped post exists and nothing | ||
| // is distinguishable as new (chat#1855). | ||
| const newPostUrls = await filterNewPostUrls(postRows.map(p => p.post_url)); | ||
| await upsertPosts(postRows); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Unguarded filterNewPostUrls call can block post persistence.
filterNewPostUrls runs before upsertPosts. If the DB read inside getPosts throws (transient network/DB error), upsertPosts never executes and scraped posts are lost. Previously, persistence was unconditional — the email logic came after. This introduces a new failure mode where a read error blocks the write.
Wrap the diff in a try-catch and default to all URLs being "new" on failure, so persistence proceeds and the worst case is a duplicate email (the pre-PR behavior).
🛡️ Proposed fix: degrade gracefully on diff failure
// Diff BEFORE upserting — afterwards every scraped post exists and nothing
// is distinguishable as new (chat#1855).
- const newPostUrls = await filterNewPostUrls(postRows.map(p => p.post_url));
+ let newPostUrls: string[];
+ try {
+ newPostUrls = await filterNewPostUrls(postRows.map(p => p.post_url));
+ } catch (error) {
+ console.error("[WARN] filterNewPostUrls failed, assuming all posts are new:", error);
+ newPostUrls = postRows.map(p => p.post_url);
+ }
await upsertPosts(postRows);📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // Diff BEFORE upserting — afterwards every scraped post exists and nothing | |
| // is distinguishable as new (chat#1855). | |
| const newPostUrls = await filterNewPostUrls(postRows.map(p => p.post_url)); | |
| await upsertPosts(postRows); | |
| // Diff BEFORE upserting — afterwards every scraped post exists and nothing | |
| // is distinguishable as new (chat#1855). | |
| let newPostUrls: string[]; | |
| try { | |
| newPostUrls = await filterNewPostUrls(postRows.map(p => p.post_url)); | |
| } catch (error) { | |
| console.error("[WARN] filterNewPostUrls failed, assuming all posts are new:", error); | |
| newPostUrls = postRows.map(p => p.post_url); | |
| } | |
| await upsertPosts(postRows); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/apify/instagram/handleInstagramProfileScraperResults.ts` around lines 44
- 47, The new pre-upsert read in handleInstagramProfileScraperResults can
prevent persistence if filterNewPostUrls (and its underlying getPosts read)
throws, so update this flow to fail open. Wrap the filterNewPostUrls call in a
try-catch inside handleInstagramProfileScraperResults, and on any read/error
path default newPostUrls to all post_urls from postRows so upsertPosts still
runs unconditionally. Keep the existing postRows handling intact and ensure the
fallback preserves the prior behavior where write success is not blocked by read
failures.
Preview re-verification after branch update — 2026-07-09Branch updated with New preview:
One honest caveat: the live Resend header read for delivery 1 wasn't possible this round — the test-session Privy token expired and preview deployments salt API keys differently from prod, so the admin-emails endpoint was unreachable. The BCC behavior of this exact code path was live-verified on #758's preview (Resend record: Fixture fully cleaned after the run (artist, social, 12 posts + social_posts + post_comments incl. rows re-created by follow-up comment-scrape webhooks; final sweep: 0 remaining). Verdict: combined #758+#759 behavior confirmed on the updated branch — ready to merge (next in the chat#1855 sequence; database#47 is merged and applied). |
1 participant
Summary
Kills the alert false positives: the notifier sent "new posts" whenever a scrape returned any posts — no diff against what's stored — so every scrape re-announced the profile's recent feed. Evidence on recoupable/chat#1855: 6 of 7 alerts to one customer in one day announced posts that were 2–10 days old and already in the database.
Tracking issue: recoupable/chat#1855 (PR #3 of 6 — independent; biggest notification-volume win).
What changed
lib/socials/filterNewPostUrls.ts(new, reusable across platforms): given candidate post URLs, returns those with no existingpostsrow. Documented constraint: must run before the upsert (afterwards nothing is distinguishable as new).lib/apify/instagram/handleInstagramProfileScraperResults.ts: computes the diff pre-upsert and only sends the alert when ≥1 URL is genuinely new. Persistence is unaffected — posts/socials/social_posts/follow-up scrapes run exactly as before; only the notification is gated.Tests (RED→GREEN)
filterNewPostUrls: known-vs-new partition, all-new passthrough, empty-input short-circuit (no query).vitest run lib/socials lib/apify/instagram→ 13 files, 63 tests passed; eslint clean.Relationship to the digest PRs
The digest aggregation (PR #4) consumes this same helper per platform. This PR is deliberately shippable alone for immediate relief on the current per-platform email.
🤖 Generated with Claude Code
Summary by cubic
Stop Instagram scrape alerts from re-announcing old posts by diffing scraped URLs against stored
postsbefore upsert. This gates emails on genuinely new posts and fixes false positives (recoupable/chat#1855).filterNewPostUrlsand used it in the Instagram handler pre-upsert to detect genuinely new URLs.Written for commit b13343a. Summary will update on new commits.
Summary by CodeRabbit