Repository navigation
Add automated RSS feed importer for AI4EOSC news - #7
Merged
Merged
Conversation
Contributor
|
@copilot this PR creates a new commit, but I would prefer that it opens a pull-request so that we can review it. |
Contributor
Author
|
@alvarolopez I've opened a new pull request, #8, to work on those changes. Once the pull request is ready, I'll request review from you. |
* Initial plan * chore: open PR instead of direct commit in RSS importer workflow --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
alvarolopez
approved these changes
Jun 22, 2026
alvarolopez
marked this pull request as ready for review
June 22, 2026 07:55
There was a problem hiding this comment.
Pull request overview
Adds automated ingestion of RSS feed items from AI4EOSC subdomains into src/content/stories/ and automates running it daily via GitHub Actions.
Changes:
- Introduces a Node.js RSS importer script that fetches feeds, parses items, deduplicates by
externalUrl, and writes story Markdown files. - Adds a scheduled GitHub Actions workflow to run the importer daily and open a PR when new stories are generated.
- Documents the importer’s behavior and manual usage under
scripts/.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
scripts/README.md |
Documents the RSS importer script, feed sources, and manual/automated execution. |
scripts/import-rss-feeds.mjs |
Implements fetching/parsing RSS and generating story Markdown frontmatter files. |
.github/workflows/import-rss-feeds.yml |
Runs the importer on a daily schedule and opens a PR with new story files. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+133
to
+148
| function createStoryContent(item) { | ||
| const dateStr = item.pubDate.toISOString().split('T')[0]; | ||
|
|
||
| // Truncate description to a reasonable length for excerpt | ||
| let excerpt = item.description; | ||
| if (excerpt.length > 200) { | ||
| excerpt = excerpt.substring(0, 197) + '...'; | ||
| } | ||
|
|
||
| const frontmatter = [ | ||
| '---', | ||
| `title: "${item.title.replace(/"/g, '\\"')}"`, | ||
| `type: News`, | ||
| `date: ${dateStr}`, | ||
| `excerpt: ${excerpt.replace(/"/g, '\\"')}`, | ||
| ]; |
Comment on lines
+74
to
+83
| if (title && link && pubDate) { | ||
| items.push({ | ||
| title: cleanText(title), | ||
| link: link.trim(), | ||
| pubDate: new Date(pubDate), | ||
| description: cleanText(description), | ||
| image: imageUrl, | ||
| feedUrl | ||
| }); | ||
| } |
Comment on lines
+119
to
+128
| function generateFilename(title, date) { | ||
| const dateStr = date.toISOString().split('T')[0]; // YYYY-MM-DD | ||
| const slug = title | ||
| .toLowerCase() | ||
| .replace(/[^a-z0-9]+/g, '-') | ||
| .replace(/^-+|-+$/g, '') | ||
| .substring(0, 50); | ||
|
|
||
| return `${dateStr}-${slug}.md`; | ||
| } |
Comment on lines
+213
to
+225
| for (const item of newItems) { | ||
| const filename = generateFilename(item.title, item.pubDate); | ||
| const filepath = join(STORIES_DIR, filename); | ||
| const content = createStoryContent(item); | ||
|
|
||
| try { | ||
| await writeFile(filepath, content, 'utf-8'); | ||
| console.log(`Created: ${filename}`); | ||
| createdCount++; | ||
| } catch (error) { | ||
| console.error(`Error creating ${filename}:`, error.message); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements automated daily ingestion of news from AI4EOSC domains (origin, fluid, arena) into the stories collection.
Implementation
scripts/import-rss-feeds.mjs: Fetches RSS feeds, parses XML, and generates story markdown files with proper frontmatter. Deduplicates by comparingexternalUrlfields against existing stories..github/workflows/import-rss-feeds.yml: Runs daily at 6 AM UTC. Commits new stories only when changes detected. Manual trigger available viaworkflow_dispatch.Generated Story Format
Filenames follow
YYYY-MM-DD-slug.mdpattern. Images extracted from feed content when available.