Skip to content

LinkedIn scrape: run harvestapi/linkedin-profile-posts when posts is requested #1838

Description

@sweetmantech

Tracking issue for making the LinkedIn scrape return per-post engagement when a posts depth is requested — the last platform gap in the posts param shipped via #1836 (where this was parked as out-of-scope). Perspective (out of scope here): closes the one remaining connector dependency in the weekly social measurement loop.

Goal

When posts is passed to POST /api/socials/{id}/scrape or POST /api/artist/socials/scrape, a LinkedIn profile scrapes via harvestapi/linkedin-profile-posts (targetUrls + maxPosts: posts) instead of the profile actor, returning post items with engagement (likes, comments, shares, reaction breakdown). posts omitted → the existing harvestapi/linkedin-profile-scraper profile snapshot, unchanged. This mirrors X/YouTube semantics exactly: depth requested → post items in the dataset; no depth → legacy snapshot.

PRs (updated 2026-07-02)

PR Item Base State
docs#260 docs: posts param descriptions gain LinkedIn (post items w/ engagement) main ✅ merged 2026-07-02 (4977fdb) — see Done
api#745 api: startLinkedinProfileScraping(handle, posts) → posts actor when depth requested main ✅ merged 2026-07-02 (21ee50b) — see Done

Merge sequencing: docs → api (contract-first). api targets main directly per current flow; sync test after.

Context — actor verified live 2026-07-02

  • Actor: harvestapi/linkedin-profile-posts (21.6k users). Input: {targetUrls: ["https://www.linkedin.com/in/<handle>"], maxPosts: N}.
  • Pricing (store, PAY_PER_EVENT): $0.002/post + $0.00005 actor-start (+$0.002/reaction and /comment only if those sub-scrapes are enabled — we don't).
  • Verified run aKsHPucJ3thxuosSd (maxPosts=5 on a real profile): 5 items, cost $0.01005 (exactly list), each with engagement: {likes, comments, shares, reactions: [{type, count}]}, postedAt, content, linkedinUrl, entityId. Notably richer than the connector: LINKEDIN_LIST_REACTIONS cannot see comments/shares on personal posts, and there is no list-my-posts connector endpoint at all.
  • Credits: no change needed. Under the shipped 5 + posts rule (per profile), LinkedIn at posts=100 costs ~$0.20 vs 105 credits ($1.05) charged → 5.2× margin. YouTube ($0.60 at posts=100) remains the binding worst case the pricing was set against.

Done

  • docs#260 — LinkedIn joins the posts-aware platforms.
    ✅ Shipped 2026-07-02 (merged to main, 4977fdb). Both scrape endpoints' posts descriptions now state LinkedIn returns the profile's latest N posts with like/comment/share counts and a reaction-type breakdown; LinkedIn removed from the "other platforms ignore this parameter" bucket. Verified: both OpenAPI files re-parse; wording matches the live dataset shape from the api#745 preview verification.

  • api#745 — LinkedIn scrape runs the posts actor when posts is requested.
    ✅ Shipped 2026-07-02 (merged direct-to-main 21ee50b; production deployment confirmed success; test re-synced 60a4164).
    startLinkedinProfileScraping(handle, posts?): posts defined → harvestapi/linkedin-profile-posts with {targetUrls, maxPosts}; omitted → the profile actor with byte-identical input (unit-tested). Same guards/webhooks/error paths. TDD red→green; full suite 3762/3762. Billing unchanged (5 + posts covers LinkedIn at ~5× margin, verified $0.01005 for 5 posts).
    Verified on preview (results): default → profile item, −5 exactly; posts=5 → 5 post items with engagement incl. the pinned comments: 1 check, −10 exactly; surfaced a 12-like/10-comment post the connector could never see.

Out of scope

  • Scraping reactions/comments bodies (scrapeReactions/scrapeComments add $0.002 per item — the counts in engagement are enough for measurement).
  • Company-page posts (harvestapi/linkedin-company-posts) — personal profiles only for now.
  • Credit pricing changes — verified unnecessary (see Context).

Source references

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions