RSS Feed Intelligence Pipeline

An automated information pipeline that collects news and research from a configurable list of RSS / Atom / API sources, stores normalized items in a Notion database, enriches them with AI metadata (importance, urgency, signals, summaries) and sends topic-specific HTML digest emails.

---

What it does

• Aggregates RSS / Atom / API feeds across three topical streams: finance, biomed and technology. Sources are declared in sources.yml with per-source parser, lookback window, credibility weight, etc.
• Stores normalized items in Notion using a single source of truth schema (de-duplication via SHA-256 of source||link||title).
• Filters items locally with a small Ollama model (1-10 score), so only items worth the spend are sent to the Cloud LLM for fine-grained enrichment. This is the main cost-saving mechanism.
• Enriches items with importance, urgency, score reason, tickers, signal direction/horizon/strength, plus an optional concise summary and key points for high-importance items.
• Generates per-topic narrative digests (finance / biomed / technology) and sends HTML emails over SMTP, with optional per-topic recipient overrides.

---

Architecture

sources.yml                 fetchers/          parsers/         curators.py
 ┌─────────┐  Phase 1     ┌──────────┐      ┌──────────┐     ┌──────────┐
 │ sources │──────────────│ asyncio  │──────│ per-parser│────│  field   │
 │ config  │  concurrent  │ aiohttp  │      │  routing │     │ assembly │
 └─────────┘  fetch       └──────────┘      └──────────┘     └────┬─────┘
                                                                   │
         ┌───────────────────────────────────────────────────────────┘
         │ Phase 2
         ▼
 ┌──────────────┐  ┌─────────────────────┐  ┌────────────────────┐  ┌──────────────────┐
 │ Notion dedup │──│ Local coarse filter │──│  Cloud enrichment  │──│ Notion DB write  │
 │ (batch hash) │  │  (Ollama, 1-10)     │  │  (importance,      │  │ (single source   │
 └──────────────┘  └─────────────────────┘  │   urgency, summary)│  │  of truth)       │
                                            └─────────┬──────────┘  └──────────────────┘
                                                      │
                                                      │ Phase 3
                                                      ▼
                                            ┌────────────────────┐  ┌──────────────────┐
                                            │ Per-topic digest   │──│ HTML email       │
                                            │ generation (LLM)   │  │ via SMTP         │
                                            └────────────────────┘  └──────────────────┘

Phase	What happens
1	Concurrent fetch of all configured feeds with HTTP caching and 304 handling.
2	Parse → curate → deduplicate against Notion → score locally → send a small fraction to the Cloud model for fine enrichment → write to Notion.
3	For each topic with new items, build a structured narrative digest and send an HTML email to the configured recipients.

---

Repository layout

rss_feed/
├── main.py                # Pipeline entry: fetch → enrich → digest
├── config.py              # All configuration + env loading
├── sources.yml            # Source registry
├── fetchers/              # HTTP / API fetch layer
├── parsers/               # Feed entry parsers
├── storage/notion.py      # Notion read/write
├── pipeline/
│   ├── enrich.py          # Local coarse filter + Cloud enrichment
│   ├── digest.py          # Per-topic digest generation + email
│   ├── notifier.py        # SMTP delivery
│   ├── model_manager.py   # Local Ollama + Cloud client wrappers
│   └── prompt_loader.py   # Optional file-based prompt overrides
├── prompts_local/         # (optional, gitignored) your private prompts
└── requirements.txt

---

Setup

1. Create a virtual environment and install dependencies

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

If you use any fetchers backed by Playwright, also run:

playwright install

2. Configure environment variables

Copy the template and edit:

cp .env.example .env
$EDITOR .env

Minimum required values:

• NOTION_TOKEN — your Notion integration token
• NOTION_DATABASE_ID — the destination Notion database ID

To enable AI enrichment and digests you also need at least one Cloud LLM provider:

• BAILIAN_API_KEY (+ BAILIAN_BASE_URL, BAILIAN_MODEL) — Alibaba Cloud Bailian / DashScope, OpenAI-compatible
• or OPENROUTER_API_KEY
• or NIM_API_KEY (NVIDIA NIM)

To enable email digests, also fill in the SMTP block (see .env.example).

3. Configure your sources

Edit sources.yml and add or remove entries. Each entry declares a parser, category (finance / biomed / technology), an optional theme, lookback_hours, credibility, etc.

4. Smoke test

python main.py --topic biomed

Useful flags / env switches while iterating:

• DRY_RUN=1 — fetch and parse only, no Notion writes, no emails
• SKIP_ENRICH=1 — skip Cloud enrichment
• SKIP_EMAIL=1 — skip email sending
• MAX_CURATED_PER_SOURCE=10 — cap items per source for faster runs
• LOOKBACK_HOURS_OVERRIDE=24 — override the lookback window for all sources
• RUN_ONE_PER_PARSER=1 — keep only one source per parser type, useful for end-to-end tests

---

Customizing prompts (private overrides)

The repository ships with generic, public-safe prompts for both the per-topic digest narrative and the per-topic enrichment scoring guide. If you want a personalized assistant — for example reflecting your portfolio focus or your research domain — drop plain-text override files into a directory the pipeline can find. Defaults shipped in code stay untouched; the loader simply replaces them at runtime if a matching override file is present.

Default lookup order:

1. PROMPTS_DIR environment variable (absolute or relative path)
2. prompts_local/ next to the package (already in .gitignore)

Recognized filenames inside that directory:

File	Replaces
digest.finance.system.txt	FINANCE_SYSTEM in pipeline/digest.py
digest.biomed.system.txt	BIOMED_SYSTEM in pipeline/digest.py
digest.technology.system.txt	TECHNOLOGY_SYSTEM in pipeline/digest.py
enrich.finance.role.txt	_CATEGORY_ROLES["finance"] in pipeline/enrich.py
enrich.biomed.role.txt	_CATEGORY_ROLES["biomed"] in pipeline/enrich.py
enrich.technology.role.txt	_CATEGORY_ROLES["technology"] in pipeline/enrich.py
enrich.finance.focus.txt	_CATEGORY_FOCUS["finance"] in pipeline/enrich.py
enrich.biomed.focus.txt	_CATEGORY_FOCUS["biomed"] in pipeline/enrich.py
enrich.technology.focus.txt	_CATEGORY_FOCUS["technology"] in pipeline/enrich.py

See pipeline/prompt_loader.py for the implementation.

---

Cost-saving design

A two-stage scoring path keeps Cloud token usage low:

1. Local coarse filter — a small Ollama model (default qwen3.5:9b) scores each candidate item 1-10. Items below the threshold are dropped before any Cloud call. Configured through:
   • LOCAL_COARSE_MODEL (default qwen3.5:9b)
   • ENRICH_COARSE_ENABLED (default on)
   • ENRICH_COARSE_THRESHOLD (default 6)
2. Cloud fine-grained enrichment — only items that pass the local filter are sent to the Cloud model in batched requests. Per-batch input size is bounded by MAX_INPUT_TOKENS and INPUT_TOKEN_HEADROOM. Output length is also bounded by an explicit max_tokens.

Two more knobs reduce noise downstream:

• DIGEST_MIN_IMPORTANCE (default 75) — minimum importance for an item to be included in the digest email.
• MAX_CURATED_PER_SOURCE — drops whole items beyond a per-source cap (does not truncate abstracts).

The pipeline also caches Cloud enrichment results so re-runs of the same items don't pay twice.

---

Email digests

For each topic with at least one item passing DIGEST_MIN_IMPORTANCE, the pipeline:

1. Sends the surviving items to the Cloud LLM with the topic-specific system prompt.
2. Renders the LLM output to HTML, replacing inline [N] references with clickable links to the source items.
3. Builds a "signal table" (tickers, direction, horizon, strength) below the narrative, when applicable.
4. Adds a "sources cited" table at the bottom, ordered by importance.
5. Sends the email to the configured recipients.

Recipient configuration:

• SMTP_TO — default recipient list (comma-separated)
• SMTP_TO_FINANCE / SMTP_TO_BIOMED / SMTP_TO_TECHNOLOGY — optional per-topic overrides; fall back to SMTP_TO when empty.

The digest module also keeps a local state file (state/digest_sent.json) keyed on date + topic + recipients + content signature, so re-running the same day will not send duplicate emails. Set FORCE_SEND=1 to bypass this.

---

Scheduling (cron)

The repository ships with helper scripts:

• run_pipeline.sh — used by cron / systemd, writes logs under logs/
• setup_cron.sh / setup_schedule.sh — install / remove cron entries

A simple daily run at 05:00 looks like:

0 5 * * * cd /path/to/rss_feed && ./run_pipeline.sh

run_pipeline.sh activates the local venv and runs main.py. Adjust the arguments inside if you need topic filtering or alternative env switches.

---

Notion schema

The pipeline writes the following fields when present (see storage/notion.py):

Notion field	Type	Notes
Title	title	Original title (or translated if available)
URL	url	Normalized URL
Source	select	Source name
Hash	rich_text	SHA-256 of `source
Category	select	biomed / finance / technology
Subcategory	select	e.g. paper, preprint, news, policy
Theme	multi_select	Source-declared tags
Asset	multi_select	e.g. Equities, Commodities
Published	date	Publication time (UTC)
Abstract	rich_text	Source abstract or extracted body
Summary	rich_text	AI-generated short summary (high-importance items)
KeyPoints	rich_text	AI-generated bullet points
Importance	number	0-100
Urgency	number	0-100
ScoreReason	rich_text	Why the item scored as it did
Tickers	rich_text	Mentioned tickers (e.g. NVDA, GC=F)
SignalDirection	select	Bullish / Bearish / Neutral / Mixed
SignalHorizon	select	Time horizon in days
SignalStrength	number	0-100
Credibility	number	0-100
PMID / DOI / Journal / Authors	rich_text / select	Academic metadata
SEC Form / CIK / Company / AccNo	rich_text	SEC EDGAR fields

---

Tech stack

Layer	Tools
Language	Python 3.10+
Feed parsing	feedparser, beautifulsoup4, lxml, readability-lxml
HTTP	aiohttp + asyncio (concurrent), requests (sync fallback), Playwright (rendered pages)
Config	pyyaml, python-dotenv, dataclasses
Storage	notion-client (Notion SDK + REST fallback)
Cloud LLM	openai-compatible client (Bailian / DashScope, OpenRouter, NVIDIA NIM)
Local LLM	ollama (small coarse-filter model + embeddings)
Embeddings / vector	scikit-learn, numpy
Email	smtplib (SMTP/TLS)

---

Project status

Personal prompt customizations are intentionally not committed: prompts_local/ is git-ignored and only loaded at runtime if present.

---

License

This project is shared under the MIT License.