Avoid creating excess traffic and storing excess files. These features would help:
Reducing crawling
- crawl a subset of pages
- limit link following
- interactive mode that reports on all the links it will follow and asks for approval
Reducing high bandwidth/storage
- don't download images (possibly past a threshold)
- don't download videos (don't actually know if it is now)
Avoid creating excess traffic and storing excess files. These features would help:
Reducing crawling
Reducing high bandwidth/storage