LazyWGET is a Bash-based utility that mimics wget -r for recursive website downloading, but in a controlled, step-by-step way. It’s designed to download websites while managing internal links and directory structures without relying on wget’s built-in recursion.
-
lwget.sh– Initializes the download:- Downloads the first page of the site.
- Creates temporary files to track links (
.promises.txt,.parcurgeri.txt). - Prepares a list of internal links for recursive downloading.
-
lget.sh– Handles recursive downloading:- Reads the links generated by
lwget.sh. - Creates the corresponding directory structure.
- Downloads pages and related assets (HTML, CSS, JS, images, etc.).
- Filters external links and avoids duplicates.
- Reads the links generated by
- Recursive website downloading similar to
wget -r. - Filters external links and avoids re-downloading the same pages.
- Preserves full website directory structure.
- Supports parallel downloading of verified links using
curl. - Pure Bash implementation with minimal dependencies.
- Download the first page:
./lwget.sh [wget options] URL- Continue recursive downloading:
./lget.sh- Do not use the -r option with lwget.sh; recursion is handled by lget.sh.
- Temporary files .promises.txt and .parcurgeri.txt are generated automatically to track download progress.
- Once you have given lwget for a site, the rest of the lgets will be for that site until you give another lwget.