README.md

Kworb Scraper

Scrapes Spotify data from kworb.net and stores it in PostgreSQL.

Note: This fork is almost a complete rework of the original service.

Kworb presents all data on a static webpage which makes it incredibly easy to scrape and puts practically 0 stress on their servers!

Data Sources

• https://kworb.net/spotify/artists.html - Top streamed artists
• https://kworb.net/spotify/listeners.html - Monthly listeners
• https://kworb.net/spotify/songs.html - Top song streams

Usage

docker compose up --build -d

This starts:

• PostgreSQL database for storage
• Scheduler (ofelia) runs the scraper daily at midnight UTC

Querying the Data

docker compose exec db-scraper psql -U root -d kworb_data

SELECT * FROM spotify_top_streamed_artists LIMIT 10;
SELECT * FROM spotify_artist_monthly_listeners LIMIT 10;
SELECT * FROM spotify_top_song_streams LIMIT 10;

Manual Run

docker compose --profile manual run --rm scraper