feat(news): ingest query-based RSS items and article-count snapshots - #40
Merged
Conversation
Implement the v0.3.0 news_rss fetcher against the documented Google News RSS search provider. Successful runs now normalize accepted feed entries into news_article content items and persist one news.article_count metric snapshot per binding. A small storage batch helper keeps the metric and content writes atomic for news ingestion.
Add deterministic news_rss tests with httptest fixtures. The suite covers successful parsing, max_items handling, rerun-safe content dedupe, malformed RSS failures, and the runtime path with the default registry updated for news_rss support.
Document the v0.3.0 provider/runtime behavior for query-based news bindings. The configuration docs now make the Google News RSS /rss/search execution path, news_article normalization, and news.article_count snapshot semantics explicit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implement query-based news RSS ingestion for v0.3.0 using the documented Google News RSS search provider. Successful runs now persist normalized
news_articlecontent items and onenews.article_countmetric snapshot per binding, with deterministic rerun dedupe.Linked issue(s)
Milestone fit
This PR delivers the
news_rssportion of the v0.3.0 ingestion milestone. It adds the first broad content stream for reviewer demos while keeping scheduler, API, dashboard, multi-provider selection, and enrichment work explicitly deferred.What changed
news_rssfetcher for query-scoped bindings using the documented Google News RSS search endpointcontent_itemsasnews_articlenews.article_countsnapshot per successful binding executiondocs/configuration.mdOut of scope
Validation
Mark what you actually ran:
python3 ./scripts/trim-whitespace.py --checkgo vet ./...go test ./...make ciDocumentation impact
README.mdSPEC.mddocs/AGENTS.md,CONTRIBUTING.md)CHANGELOG.mdReviewer focus
internal/fetch/news.gofor provider request/parse/normalize behaviorinternal/storage/ingestion.gofor atomic metric/content persistenceinternal/fetch/news_test.gofor rerun dedupe and malformed RSS handlingAuditability checklist
Notes for merge
Preferred merge method: