RSS News Engine

🔥 Live Demo: rss-news-engine.vercel.app

The ultimate goal of this project is simple: find the most interesting news with the lowest possible time investment.

Currently, it operates as an automated, curated news feed. It's backed by a custom ML pipeline that scores articles for sentiment and "hotness" (information density + objectivity), allowing the most factual and interesting stuff to bubble to the top.

Features

The project is broken down into four main pillars:

1. Ingestion: Asynchronously pulls down articles from various RSS feeds.
2. Sentiment Analysis: Runs headlines and summaries through a local Transformer model (via HuggingFace) to tag them as positive, negative, or neutral.
3. Hotness Scoring: Uses spaCy to calculate an "Information Density" score (measuring the density of entities, places, and numbers in the text). This is balanced with recency and objectivity metrics to rank articles that are actually worth reading.
4. MLOps & Observability: Because the news cycle changes rapidly, the pipeline periodically compares this week's news against last month's using Evidently and MLflow to detect data drift.

Architecture

%%{init: {
  'themeVariables': {
    'clusterBkg': '#ffffff',
    'clusterBorder': '#cccccc',
    'fontFamily': 'arial'
  }
}}%%
flowchart TD
    %% External Entities
    User((End User))
    Sources[\RSS Feeds/]

    %% Central Storage
    DB[(Central Database)]

    %% Backend / ML Subsystem
    subgraph Backend [Data & ML Pipeline]
        direction TB
        Ingest[Ingestion Engine]
        NLP[Sentiment Analysis<br>Model]
        Scorer[Hotness & Ranking<br>Engine]
    end

    %% Orchestration (Linux + systemd)
    subgraph Orchestration [Orchestration on Linux]
        direction TB
        Systemd[Systemd User Service and Timer<br>]
    end

    %% Observability Subsystem
    subgraph MLOps [Observability]
        direction TB
        Monitor[Data Drift Monitoring]
        Tracker[Metrics & Artifact<br>Tracking]
    end

    %% Frontend Subsystem
    subgraph Frontend [Client Presentation]
        direction TB
        WebUI[News Feed Web UI]
    end

    ModelReg[\Model Registry/]

    User -->|Views curated feed| WebUI
    WebUI -->|Queries enriched news| DB

    Sources -->|Polls updates| Ingest
    Ingest -->|Saves raw articles| DB

    DB <-->|Fetches raw data / <br>Persists sentiment| NLP
    DB <-->|Fetches data / <br> Persists scores| Scorer

    ModelReg -.->|Provides models| NLP

    DB -.->|Extracts data| Monitor
    Monitor -->|Logs reports & alerts| Tracker

    Orchestration -.->|Runs daily| Ingest
    Orchestration -.->|Runs daily| NLP

    classDef frontend fill:#e8f0fe,stroke:#1a73e8,stroke-width:2px,color:#1967d2,rx:6px,ry:6px
    classDef backend fill:#e6f4ea,stroke:#1e8e3e,stroke-width:2px,color:#0d652d,rx:6px,ry:6px
    classDef pipeline fill:#fef7e0,stroke:#f29900,stroke-width:2px,color:#b06000,rx:6px,ry:6px
    classDef storage fill:#fce8e6,stroke:#d93025,stroke-width:2px,color:#a50e0e,rx:6px,ry:6px
    classDef external fill:#f1f3f4,stroke:#80868b,stroke-width:1px,color:#3c4043,rx:6px,ry:6px,stroke-dasharray: 4 4
    classDef user fill:#e3f2fd,stroke:#1565c0,stroke-width:3px,color:#0d47a1,rx:20px,ry:20px

    class ModelReg,Sources external
    class DB storage
    class WebUI frontend
    class Ingest,NLP,Scorer backend
    class Systemd pipeline
    class Monitor,Tracker mlops
    class Orchestration,Backend,MLOps,Frontend cluster

Loading

Running it Locally

1. Database Setup

You'll need a Postgres database. The easiest route is to spin up a Supabase project and grab your keys.

I've included ready-to-run SQL scripts to generate the articles, sentiment_results, and hotness_scores tables.

• For Supabase: run backend/sql/init_supabase.sql
• For local Postgres: run backend/sql/init_postgres.sql

2. Backend Pipeline (ML & Data)

Navigate to the backend directory, set up your Python environment using uv, and configure your .env file with your Database connection strings.

cd backend
uv sync

uv run python -m spacy download en_core_web_sm

uv run python -m main

3. Frontend Web App

Navigate to the frontend directory, install dependencies, and create a .env.local file containing NEXT_PUBLIC_SUPABASE_URL and NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY.

cd frontend
npm install
npm run dev

Visit http://localhost:3000 to see your local news feed.

Orchestration & Observability

Pipeline tasks are orchestrated using Prefect. To spin up the local orchestration server and view the UI:

prefect server start

The dashboard will be available at http://127.0.0.1:4200.

(Note: Evidently data drift reports generated by the pipeline are saved locally within the backend/ directory).

Automation (systemd)

To keep the news feed fresh, the heavy machine learning pipeline is scheduled to run on a local Linux machine using systemd timers.

If you want to replicate this automated deployment, be sure to replace /absolute/path/to/rss-sentiment-engine with the actual path to the repository on your machine.

1. Create the service (~/.config/systemd/user/rss-sentiment-engine.service):

   [Unit]
   Description=Prefect Daemon: News Ingestion and Sentiment Analysis
   After=network-online.target
   
   [Service]
   Type=oneshot
   Restart=no
   
   # Update these paths to match your local setup
   WorkingDirectory=/absolute/path/to/rss-sentiment-engine/backend
   ExecStart=/absolute/path/to/rss-sentiment-engine/backend/.venv/bin/python main.py
   
   StandardOutput=append:/absolute/path/to/rss-sentiment-engine/backend/logs/prefect.out
   StandardError=append:/absolute/path/to/rss-sentiment-engine/backend/logs/prefect.err
   
   [Install]
   WantedBy=default.target

2. Create the timer (~/.config/systemd/user/rss-sentiment-engine.timer):

   [Unit]
   Description=Daily News and Sentiment Pipeline Timer
   
   [Timer]
   OnCalendar=*-*-* 23:00:00
   Persistent=true
   RandomizedDelaySec=5m
   
   [Install]
   WantedBy=timers.target

3. Enable and start the daemon:

   systemctl --user daemon-reload
   systemctl --user enable rss-sentiment-engine.service
   systemctl --user enable --now rss-sentiment-engine.timer

4. Verify it's running:

   systemctl --user list-timers --all