feat: implement safeFetch utility for enhanced HTTP requests handling

[alhaymex]

• Added safeFetch utility in packages/backend/convex/utils/safeFetch.ts to handle HTTP requests with robust error handling, including support for timeouts, redirects, and content type validation.
• Introduced fetchMetadataHtml and fetchSmallJson functions for specific use cases.
• Implemented IP address blocking logic to prevent requests to private or blocked IPs.
• Created comprehensive tests for safeFetch functionality in packages/backend/tests/safeFetch.test.ts, covering various scenarios including blocked IPs, unsupported content types, and redirect handling.
• Added utility functions to detect blocked pages in packages/common/utils/blockedPage.ts to identify common patterns in blocked responses.

Summary by CodeRabbit

Release Notes

• New Features

  • Safeguarded network requests with DNS filtering, timeout enforcement, and content validation
  • Enhanced anti-bot detection for improved extraction reliability

• Improvements

  • Stealth capabilities for more reliable website interaction
  • Better metadata fallback handling when primary retrieval fails

[coderabbitai]

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Free

Run ID: 105aec32-2ed0-431f-b633-173388f919c5

📥 Commits

Reviewing files that changed from the base of the PR and between 6373035 and faf7792.

⛔ Files ignored due to path filters (2)

• packages/backend/convex/_generated/api.d.ts is excluded by !**/_generated/**
• pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml

📒 Files selected for processing (16)

• TODO.md
• apps/extractor/package.json
• apps/extractor/src/lib/browser.ts
• apps/extractor/src/lib/extractArticle.ts
• apps/extractor/src/routes/extractArticle.ts
• apps/extractor/tests/extractArticle.test.ts
• packages/backend/convex/links/actions.ts
• packages/backend/convex/utils/extractorClient.ts
• packages/backend/convex/utils/safeFetch.ts
• packages/backend/convex/utils/youtube.ts
• packages/backend/package.json
• packages/backend/tests/getOpenGraph.test.ts
• packages/backend/tests/safeFetch.test.ts
• packages/common/index.ts
• packages/common/schema/linkSchema.ts
• packages/common/utils/blockedPage.ts

---

📝 Walkthrough

Walkthrough

This PR introduces a comprehensive safe-fetching and anti-bot detection system for article extraction. It adds DNS-guarded HTTP fetching with IP blocking, timeout enforcement, and response validation; deploys shared anti-bot page detection across extraction and metadata fetching; hardens the browser launcher with stealth plugins; adds structured logging throughout; and refactors OpenGraph and extractor handling to use the new safe-fetch infrastructure and block-detection helpers.

Changes

Safe Fetching and Anti-Bot Infrastructure

Layer / File(s)	Summary
Shared Anti-Bot Detection packages/common/utils/blockedPage.ts, packages/common/index.ts	New looksLikeBlockedPage function evaluates URL, title, and content against blocked-page regex patterns (strong and weak) to identify captcha/challenge pages. Re-exported from common for use across services.
Hostname Validation packages/common/schema/linkSchema.ts	Renamed isPublicHostname to isPublicUrlHostname for clarity; added isPublicIpHostname to check if a hostname resolves to a public IP address. Both support IPv4/IPv6 validation and integrate with URL schema parsing.
Safe Fetch Module packages/backend/convex/utils/safeFetch.ts, packages/backend/package.json	New safeFetch.ts implements guarded HTTP fetching with URL validation, DNS-based private IP blocking, timeout/abort control, response content-type and size enforcement, manual redirect handling, and structured error reasons. Exports safeFetchText, fetchMetadataHtml, fetchSmallJson, and helpers for checking public IPs. Added undici dependency for guarded DNS lookup.
Extractor Client Timeouts packages/backend/convex/utils/extractorClient.ts	Updated callExtractor to parse/canonicalize URL via urlSchema, set default timeoutMs of 30 seconds, enforce hard abort-based timeout of 35 seconds, validate JSON content-type, and conditionally throw on status === "error". Exports EXTRACTOR_REQUEST_TIMEOUT_MS constant.
YouTube oEmbed Safe Fetch packages/backend/convex/utils/youtube.ts	Changed fetchYouTubeOEmbed to use fetchSmallJson instead of raw fetch, enforcing timeout, max redirects, max size, and allowed content types. Removed debug logging and switched to JSON.parse on validated response text.

Extractor Service Hardening

Layer / File(s)	Summary
Browser Stealth Plugin apps/extractor/src/lib/browser.ts, apps/extractor/package.json	Replaced puppeteer with puppeteer-extra and registered StealthPlugin to hide automation signals. Expanded Chromium launch flags to include --disable-blink-features=AutomationControlled and origin isolation disabling. Added puppeteer-extra-plugin-stealth and tsx dev dependency.
Extraction with Logging & Anti-Bot apps/extractor/src/lib/extractArticle.ts	Integrated shared looksLikeBlockedPage for detection; removed local blocked-page patterns. Added granular console logging for startup, navigation, settling, HTML retrieval, readability results, unreadable content, and errors. Pre-readability diagnostics test expanded block-pattern set and warn on matches. Post-readability blocked check uses article-extracted content. Conditionally close browser only if created (via let/finally guard).
Route Handler Logging apps/extractor/src/routes/extractArticle.ts	Added try/catch wrapping request handling. On invalid body, log Zod errors and return HTTP 400 with details. On success, log receipt and completion with content metrics (title/htmlFragment/textContent lengths, finalUrl, errorCode, errorMessage). Catch and log unexpected extraction errors before re-throwing.
Extraction Anti-Bot Tests apps/extractor/tests/extractArticle.test.ts	Added two test cases: blocking asserted when anti-bot page title is detected even with article-like HTML; blocking asserted when readability returns challenge text from benign-looking shell. Both verify extractReadableArticle is not called and browser is closed.

Metadata and Link Handling Integration

Layer / File(s)	Summary
OpenGraph Fetching Refactor packages/backend/convex/links/actions.ts	Replaced direct fetch and inline extractor fallback with fetchMetadataHtml (safe fetch). Introduced tryExtractorFallback helper that calls callExtractor, validates eligibility via looksLikeBlockedPage and hasUsableArticleContent, and updates OpenGraph fields on success. Direct fetch failure now updates embed links with hostname-derived title and fallback favicon, then attempts fallback. Successful parsing uses fetchMetadataHtml's resolved URL as base for sanitization.
OpenGraph Test Updates packages/backend/tests/getOpenGraph.test.ts	Mocked safeFetch functions (fetchMetadataHtml, fetchSmallJson) via vi.mock and safeFetchMocks. Updated assertions to check safe-fetch calls instead of globalThis.fetch. Added tests for: embed metadata fallback when direct fetch fails, parameterized extractor fallback for various safe-fetch failure reasons, invalid canonical URLs marked as error without fetching, extractor fallback content that still looks blocked stored with no HTML/text, and both fetch paths failing.
Safe Fetch Test Suite packages/backend/tests/safeFetch.test.ts	Added comprehensive Vitest suite covering: blocking private IPv4/IPv6 and IPv4-mapped DNS/hostnames, rejecting localhost/private hostnames, enforcing allowed content-type, response size limits, abort/timeout handling, relative redirect following with final URL resolution, blocking redirects to private IPs, redirect count limits, unsupported URL schemes (file://, ftp://, javascript:), HTTP error and malformed redirect responses, and empty response bodies. Also tests isPublicResolvedAddress for public/private IP classification.

Sequence Diagram

sequenceDiagram
    participant Client
    participant Handler as OpenGraph Handler
    participant SafeFetch
    participant DNS as DNS Lookup
    participant Extractor as Extractor Service
    participant Readability as Readability Parser

    Client->>Handler: handleOpenGraph(linkId)
    Handler->>SafeFetch: fetchMetadataHtml(url)
    SafeFetch->>DNS: resolve hostname
    DNS-->>SafeFetch: public IP or blocked error
    SafeFetch->>SafeFetch: validate response content-type & size
    SafeFetch-->>Handler: {ok, text} or {ok: false, reason}
    
    alt Direct fetch succeeded
        Handler->>Handler: parse HTML, sanitize, detect blocked page
        alt Not blocked & content usable
            Handler-->>Client: OpenGraph fields updated
        else Still looks blocked or no content
            Handler->>Extractor: callExtractor(url)
            Extractor->>Readability: extractReadableArticle
            Readability-->>Extractor: article or null
            Extractor-->>Handler: extraction result
            Handler->>Handler: validate extracted content not blocked
            Handler-->>Client: OpenGraph with fallback content or error
        end
    else Direct fetch failed
        Handler->>Handler: update embed with fallback favicon/title
        Handler->>Extractor: tryExtractorFallback
        Extractor->>Readability: extractReadableArticle
        Readability-->>Extractor: article or null
        Extractor-->>Handler: extraction result
        alt Extractor succeeded & content usable
            Handler-->>Client: OpenGraph with extractor fields
        else Extractor failed or looks blocked
            Handler-->>Client: OpenGraph error or minimal data
        end
    end

Loading

Estimated Code Review Effort

🎯 4 (Complex) | ⏱️ ~60 minutes

---

Poem

A rabbit built walls 'round the web so bright, 🐰
With DNS guards that block what's not right,
Anti-bots fled when stealth came to play,
Safe fetches and logging lit up the day,
Now articles flow through defenses so tight! ✨

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

• Generate code and open pull requests
• Plan features and break down work
• Investigate incidents and troubleshoot customer tickets together
• Automate recurring tasks and respond to alerts with triggers
• Summarize progress and report instantly

Built for teams:

• Shared memory across your entire org—no repeating context
• Per-thread sandboxes to safely plan and execute work
• Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get your free trial and get 200 agent minutes per Slack user (a $50 value).

---

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Review rate limit: 2/3 reviews remaining, refill in 20 minutes.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}