fix: Youtube Miner will interact with website to reveal more data bef… - #42
Conversation
There was a problem hiding this comment.
Pull request overview
This PR aims to address Issue #41 (“Youtube reports are sometimes missing date”) by ensuring the YouTube page is interacted with (expanding the description area) before scraping, so the date becomes available in the scraped content.
Changes:
- Add a pre-scrape interaction step for YouTube pages to click the “expand” control before extracting sections.
- Upgrade
@danilidonbeltran/webscrapperto2.3.0to support the new interaction capability. - Update the lockfile to reflect the dependency upgrade.
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| src/lib/data-mining/YoutubeMiner.ts | Adds a DOM interaction step (click expand) before scraping YouTube content to improve date extraction reliability. |
| package.json | Bumps @danilidonbeltran/webscrapper to 2.3.0 to enable interaction steps. |
| package-lock.json | Updates resolved version/integrity/metadata for the upgraded webscrapper dependency. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| browser: "chromium", | ||
| headless: true, | ||
| timeout: 30_000, | ||
| interactionSteps: [{ "event": "click", "target": "#expand", "wait": 1000 }], |
| browser: "chromium", | ||
| headless: true, | ||
| timeout: 30_000, | ||
| interactionSteps: [{ "event": "click", "target": "#expand", "wait": 1000 }], |
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #42 +/- ##
=======================================
Coverage 80.58% 80.58%
=======================================
Files 41 41
Lines 855 855
Branches 170 167 -3
=======================================
Hits 689 689
Misses 166 166 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…ore scraping
Fixes: #41