42 CDN (pdf) scraper

This is a hobby project to scrape all subject.pdf files from the 42 intra cdn as they did not secure this. This has been made just to explore and continue working on the C projects after the move to more python projects for myself.

Installation

Clone this repository and build rust project

git clone https://github.com/ShadowThijs/42-cdn-scraper.git
cd 42-cdn-scraper
cargo build --release

Running

To run the binary and view help menu

./target/release/pdf-scraper --help

Check and download PDFs from 42's CDN

Usage: pdf-scraper <COMMAND>

Commands:
  fetch     Fetch valid PDF URLs and save them to a file
  download  Download PDFs from a URL list and rename them based on their title
  help      Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help

To fetch all urls for every en.subject.pdf

./target/release/pdf-scraper fetch --help

Fetch valid PDF URLs and save them to a file

Usage: pdf-scraper fetch [OPTIONS]

Options:
  -s, --start <START>              Starting number for the PDF IDs [default: 1]
  -e, --end <END>                  Ending number for the PDF IDs [default: 200000]
  -c, --concurrency <CONCURRENCY>  Number of concurrent requests [default: 500]
  -o, --output <OUTPUT>            Output file for valid URLs [default: valid_urls.txt]
  -t, --timeout <TIMEOUT>          Request timeout in seconds [default: 10]
  -h, --help                       Print help

To download all pdfs from a valid_urls.txt file (or specified otherwise)

./target/release/pdf-scraper download --help

Download PDFs from a URL list and rename them based on their title

Usage: pdf-scraper download [OPTIONS]

Options:
  -i, --input <INPUT>              Input file containing URLs (one per line) [default: valid_urls.txt]
  -o, --output-dir <OUTPUT_DIR>    Output directory for downloaded PDFs [default: pdfs]
  -c, --concurrency <CONCURRENCY>  Number of concurrent downloads [default: 10]
  -t, --timeout <TIMEOUT>          Request timeout in seconds [default: 60]
  -h, --help                       Print help

Do take note that all pdfs take a combined 23Gb (using the default values) as of right now (30-11-2025) so make sure you have enough storage and a fast enough internet connection. Otherwise this could take a very long time.